Connecting Twitter with SAP HANA

[Former Member]

Hello,

We are planning to connect SAP HANA with Twitter for doing the analytics.

Earlier, we thought of doing this with introducing Hadoop in between, and with the help of Plume, establish the real-time streaming of tweets.

But, now we have decided to directly connect SAP HANA with Twitter.

I would like to know how can be achieve this, using any intermediate tools, and whether real-time streaming tweets is possible.

Please help me in this regard.

Best Regards,

Ram

[Former Member · Accepted solution]

There are many ways to connect HANA to Twitter.

A great way to establish real-time connectivity is to leverage SAP Event Stream Processor in conjunction with SAP HANA.  SAP Event Stream Processor (ESP) is designed to capture and analyze multiple streams of high-speed, high-volume complex event data in real time, while loading real-time data into SAP HANA.

Twitter’s streaming APIs can be leveraged from within a custom SAP Event Stream Processor (ESP) input adapter, and then you can use SAP ESP’s HANA Output Adapter to load the HANA in-memory database at high volume and low latency. ESP adapters can be developed in, Java, .NET or C/C++, and there are also a variety of third party libraries available to ease interfacing with Twitter’s Streaming APIs.

To get started with development, Twitter offers free access to a random sampling of public Tweets in real-time up to a certain rate limit they define.  The Tweets supplied in this free level are those public Tweets that match filters that you define.  Higher levels of access (access on behalf of users, access to the full “firehose” of public Tweets) are available for a fee.

Mike

[Former Member · Accepted solution]

Just saw the ESP Roadmap:

This suggests ESP is moving to HANA,  to be know as 'Hana streaming'. 

This might be the logical tool to use when it's available????

In the meantime Python might be the easiest for a PoC, as Jose below mentioned.

Going forward I wonder how SAP's partnership with Hortonworks may turn out.

Hortonworks has an open source tool called 'Storm' which also could be used.

http://hortonworks.com/hadoop/storm/

My guess is that for small amounts of streaming data then 'Hana streaming' may be the perfect fit.

As the data grows then ESP standalone  or open source Hadoop Storm/Flume may need to be considered. 

[Former Member]

Hello there !

Do you know if there is a way to search for a tweet instead of track tweets?

More generally, how to use the search API instead of the Streaming API ?

thanks in advance,

[Former Member]

Hello Ram,

Can you please help me out how you have connected Twitter APi with SAP HANA. I am trying to replicate this process. Can you please elaborate in detail.

Thank you

Regards

[hari_ashvini]

Hi Ramkumar,

We are working on a project with HANA and Twitter and would like to get some insights from your work. If you are open for a discussion please feel free to send me a note at hari.ashvini@sap.com.

Best,

Hari

[Former Member]

There is a way you can connect Twitter to HANA using Python.

I think that the more friendly libraries in order to use Twitter are implemented with Python, and you can actually access a Python API that it's already included when you install the HANA Client in your system.

Former Member wrote a blog post where he describes how to use Python and HANA together. Then, all you need is to install a Python library for Twitter. Personally, I prefer Tweepy.

With Tweepy and HANA, we are doing some sentiment analysis and data mining. It's not that hard.