Problem with input/output to/from ESP

[Former Member]

Hi,

Our project has the following parameters:

A large number of continuous data producing clients (devices), which constantly provide data that needs to be processed and partly saved in HANA.

Several clients (users) that need the processed data for the application running on their machine.

Input:

Is there any way to connect a large number of clients delivering input constantly to an ESP project? Right now we use a server to accept the clients and consolidate all data into one queue which is then forwarded to ESP through a csv socket input adapter.

Output:

We need a snapshot of a window's content at a certain time (when queried). We tried the http output adapter, but it provides only a never ending stream of input, update or delete events. The same applies when connecting to a csv socket output adapter from a server and getting the window's content.

It would be great if anyone can help us with these queries.

Thanks,

Bhumi Patel.

[Former Member]

Hello,

Just curious with the CSV Socket Input Adapter, did you find some kind of limitation or bottleneck?  I don't know if you used the example from the SAP Wiki site but I think this example could be expanded to have multiple queues that each have a connection to the project.

I have seen several customers go with the Web Service Provider:

http://help.sap.com/saphelp_esp51sp08ag/helpdata/en/e7/7715d36f0f101488e3b595ee582190/content.htm?fr...

Since this adapter maintains a pool of connections to the project it reduces the load on the cluster manager particularly for applications that are heavy connect/disconnect types.

Thanks,

Neal

[Former Member]

Hi Neal,

I am not sure what kind of bottleneck you are thinking about, currently we are using an input consolidator which is basically taking all the requests from multiple clients and putting them in a queue, and we are piping that queue to the CSV Socket Input Adapter because I think each CSV Socket Input Adapter can receive input from just one client/server (CSV Socket Input Adapter can not handle data from multiple clients/sources). Is that correct or am I missing something here?

Right now CSV Socket Input Adapter is working fine for us in above mentioned setup.

But yes, we can look into Web Service provider you have mentioned.

Thanks for the information!

Regards,

Bhumi Patel.

[Former Member]

Hello,

Yes, each instance of the CSV Socket Input Adapter in a project reads from (or writes to) the server it is connected to.  This adapter acts as a client and connects to a server.

I had mistakenly thought you might be using the CSV Socket Server example that we have on our wiki site here:

  SocketServerExample - SAP Event Stream Processor - SCN Wiki

This is a server.  It reads from each client that connects to it and puts that data into a queue that is then published to the project.  In its current form it only maintains one connection to the project but since it is an example, it could easily be expanded to have multiple connections.  In a sense, it would be acting like a connection pool similar to the WSP adapter shielding the ESP cluster manager from the connect/disconnect activity.  This example is not as advanced as the WSP adapter so the WSP adapter may better suit your needs.

Thanks,

Neal

[former_member217348]

Hi Bhumi,

I will respond to the Output question, and I will let others chime in about the Input question. It is suggested to post separately for each question you have so that others can follow and find the information, please do start a new post for that one. Thanks!

To get a snapshot of a window's content at a certain time, you could use esp_subscribe, and use the -S option to just get a snapshot.

Here is the link to the ESP 5.1 SP08 document that describes esp_subscribe.

esp_subscribe - Utilities Guide - SAP Library

Thank you,

Alice

[JWootton]

Other options for the snapshot query include:

- HANA Smart data access:  map the ESP window to HANA and query HANA

- The "other" ESP http output adapter:  this adapter is poorly named, but accepts queries via http and delivers a snapshot result

- Us the ESP ODBC driver to run snapshot queries against ESP windows

As for the question on input:  Yes, you can have a large number of publishers that publish direct to ESP without having to implement a "consolidator".  There are several options:

1. Use the ESP Web Service Provider, REST interface,  and have all publishers publish directly to that

2. Use a message bus with either the ESP JMS or MQ input adapter, and have all publishers publish to the bus.  If you are interested in other messaging (eg ActiveMQ or RabbitMQ - let me know - it's an area we are gathering requirements on).

3. Use a messaging appliance, such as the one from Solace Systems, that provides very good scalability and connects to ESP using the standard ESP JMS adapter

4.  All publishers can connect directly to ESP using TCP Sockets and you can scale for large numbers of publishers by running a set of ESP "gateway projects" and then one or more ESP "consolidation" projects.  You spread the publishers across the gateways. So, for example, if  you assign 500 publishers to each gateway project, then running 10 gateway project instances will support 5,000 publishers.

[Former Member]

Hi Jeff,

Thanks for your responses.

We can try implementing one of the four options you have mentioned for input consolidator.

Regarding snapshot query, we had tried using http output adapter. We used sample HTTPAdapterClient.html file provided under Sybase/ESP-5_1/adapters/http/examples to query the server, but it returns a stream instead of a snapshot. Is there any specific setting which we are missing?

Please let me know.

Thanks,

Bhumi Patel.

[JWootton]

That's a different adapter.  There are two http output adapters.  They are unfortunately named, which make it confusing.  The http client output adapter publishes every output event to an http server, in real-time.  The "other" http output adapter listens for http requests for snapshot queries. Check the doc for both/each

[Former Member]

Oh ok, then we have to look again for both of those adapters.

Thanks for clarification!

Regards,

Bhumi Patel.