cancel
Showing results for 
Search instead for 
Did you mean: 

Connection between esp and hadoop

Former Member
0 Kudos

Hello,

I have some questions about the possibilities of the event stream processor.

I have a local installation of esp and a table in HANA.

Now I would like to get all datasets from Hana that are older than a month into hadoop.

Therefore I researched and found the following guide:

Hadoop - SAP Event Stream Processor - SCN Wiki

If I understand it right, then all default hadoop output adapter will write into an external file (csv/xml/jason).

But then I have to manually load the datasets from the file into my table in hadoop hive.

Now my questions:

Is there a possibiltity to directly push the datasets from esp into hive?

And if yes, what output adapter do I have to use?

Thanks in advance.

Best Regards

Stefan

Accepted Solutions (1)

Accepted Solutions (1)

JWootton
Advisor
Advisor
0 Kudos

First let me comment that ESP - or HANA smart data streaming - is not the right tool to use for moving data from HANA to Hadoop.  However,  you can certainly use ESP - or HANA smart data streaming (SDS) - to capture incoming data into Hadoop.  You can even have SDS apply rules to capture high value data in HANA, and push all the rest into Hadoop.

Note that while you asked about ESP,  in this situation you would actually be better off using HANA smart data streaming,  which is a version of ESP that runs as part of the HANA system.  More seamless.

The SDS File/Hadoop output adater writes the data to hdfs.  We don't currently support data loading directly to Hive - so if you need the data in Hive, you would need to write a simple job to load the hdfs data into Hive.  Others have asked about this, so I'm looking into whether we should add the ability to push data directly into Hive from ESP/SDS.  I'm  interested in the use cases.  Why do you want/need the data in Hive?

Former Member
0 Kudos

Hi Jeff,

thanks for the answer.

I need the data (or a part of it) in hadoop for my master thesis.

I got some sensordata and now I woud like to split the current data from historydata.

And because I'm no expert in hadoop, I thought I will send them directly into hive for easier handling.

But ok, then I have to look to write a job like you said.

Best Regards

Stefan

JWootton
Advisor
Advisor
0 Kudos

Just to add:  you don't need to load the data into Hive to be able to analyze it from HANA. While it needs to be in Hive to access it using HANA Smart Data Access,  the HANA Hadoop integration features let you access hdfs data directly:

http://help.sap.com/saphelp_hanaplatform/helpdata/en/fd/3b8d1cbc074b889c8e06b05ba7af54/content.htm?f...

Answers (0)