The annual ACM SIGMOD/PODS conference* convenes this weekend in Melbourne and an SAP Research team will be presenting their paper: Quality-Driven Continuous Query Execution over Out-of-Order Data Streams.
This cutting edge research into event stream processing extended SAP Event Stream Processor (ESP) to study how to apply streaming analytics (aka continuous queries) to event streams that contain out-of-order events. This occurs naturally when individual messages get delayed over transmission networks, and can affect the accuracy of data when calculations are applied to sets of events. A simple - and very common - example would be if you are computing aggregate statistics over a slide window of the last n events, where a late arriving event means that the computed values prior to the late arrivals, contain erroneous data.
The traditional approach to handling this is to define a static "wait period" to wait for late arrivals before computing the statistics, but this introduces latency. The inherent trade-off is latency vs. accuracy.
The team here introduces an adaptive approach they call AQ-K-slack, which automatically balances acceptable latency with desired accuracy.
Congratulations to the SAP team behind this new research. Check it out.
Note: SAP ESP is now also available as an integrated capability of SAP HANA: SAP HANA smart data streaming.
*ACM SIGMOD/PODS is "a leading international forum for database researchers, practitioners, developers, and users to explore cutting-edge ideas and results, and to exchange techniques, tools, and experiences" (as taken from the conference web site).
