ESP vs. Hana (...) - Differences

[Former Member]

Hi guys,

i am working on my thesis at the moment and comparing streaming technologies with hana, hadoop and so on. ESP is a part of my work.

I want to ask you guys, are there some camparisons between hana and esp? What are the differences between them, for example, I/O behaviour? I want to ask: Do i need ESP if i have Hana, and why? Where are the limitations of hana, when do i need a CEP system like ESP? Are streaming technologies, like ESP or Apache Storm, really nessesary and when, when do we need them?

I hope you can help me with some useful information, links or something, especially about SAP ESP and HANA.

Thank you!

[JWootton]

ESP is a technology that complements SAP HANA.  As they are fundamentally different technologies, it really makes more sense to talk about how they complement each other rather than how they are different.  SAP HANA is a database and an application platform, while ESP is - well, an event stream processor.  So let's start with the fact that ESP is not a database and is not built on a database. It's a real-time event processing engine that operates on an entirely event-driven model.  Data streams into the ESP engine in real-time, flows through continuous queries that have been defined, and results are published - i.e. streamed out from the ESP engine - in real-time, in response to those  incoming events. ESP is does not provide permanent or even long-term data persistence and is not designed to support ad-hoc queries (yes, there is some very limited ability to query windows...but let's not let that complicate things.

Generally, SAP ESP is used with SAP HANA in either/both of the following ways:

1. To capture streaming data in SAP HANA, providing the ability to receive live data streams, apply any desired filters, transformations,  aggregations, correlations, etc and then capturing the desired data  in HANA (in the desired form)

2. Real-time response:  monitor data as it streams in to generate alerts, notifications, or initiate a response as fast as the information is received.

For an overview of what SAP ESP is, how it works, and how it can be used, I suggest this paper.  You might also find this video showing how ESP can be used with SAP HANA to be helpful. And there are a number of example use cases for ESP described in the Event Processing community on SCN.

[Former Member]

Hi Jeff,

thank you for your respond and links. If we are talking about streaming i have to think about the different I/O behaviour of both technologies. I know that hana isnt the best and insert new data, but, when we are talking about a stream of data and automatically insert new data sets into hana, analysis them if we need them and drop them after if not, isn't that - cause of the in memory database - nearly as fast as an event stream processor like sap ESP?

Just to be sure: i am working on my thesis, i am not very experienced in that topic. I have to ask critical questions and i am really happy about your answer.

By the way, do you know a comparison between stream processing and "batch" processing?

[JWootton]

Again, the processing model is very different between an event stream processor and a database. ESP is highly optimized to process individual events as fast as they arrive.  What's more, the language is optimized for this.  ESP uses what we call "continuous queries" - you define the query before having the data, and as the data arrives it flows through the continuous query to produce release.  This allows ESP to compute some things incrementally.  Take a moving average for example: in ESP you define a window - let's say a 3 hour window for example - and define an aggregation to compute the average value received from each sensor.  ESP can update the average incrementally with each new event received.  But ESP isn't saving the events - again, it's not a database - so if you later decide you want to know other statistics about the set of events - eg min, max, distribution, etc - you can't ask ESP, it wouldn't be able to tell you.  But if you had ESP collect all the events in a HANA table, then it would be very easy to do any analysis of that data set.

Because of this optimization around processing individual events arriving at high speeds through one or more continuous queries,  ESP can handle very high event rates -  100's of thousands or even millions of events per second.

Stream processing handles each event as it arrives, unlike batch processing that works on a scheduled basis processing sets of records.  Note, however, that when ESP captures streaming data into HANA,  it will perform what we call "micro batching" to optimize the efficiency of loading the data into HANA.

[Former Member]

Hello,

There are some good diagrams in the following link that try to illustrate the difference between the two:

   Event Stream Processor Compared to Databases

Thanks,

Neal

[Former Member]

Thank you guys,

you helped me alot 🙂

Two last questions, are there public benchmark or somethings for the limitations of ESP and/or Hana and where to find to real world use cases about ESP?

[Former Member]

Hi Jeff & Neal again,

do you have a idea, how to show the difference in processing data in hana and esp with a small example? I'm think about something like a WordCount or something.

Any ideas?

[former_member123979]

I agree with Jeff. Both are different processing models.

ESP will have data structures optimized to perform on streaming data, where as for HANA though data is in memory, but it will be more of a pull model of data to do analytics.

I will also suggest what Nathan Marz calls the Lambda Architecture(how to combine the batch analytics and real time analytics).

Lambda Architecture » λ lambda-architecture.net

So as an example of word count(say twitter streams). Real time aggregation can be done in ESP(say last 10 mins trends). and then for every hour a batch view is run on HANA to aggregate the tweets. Whenever a real time projection is needed, both can be combined.