ESP Rest Provider, how to poll data from another web service?

[former_member214945]

Hello colleagues,

I have enabled a ESP project as a web service, with HTTP POST request, events are pushed into ESP (input_window_1). When specific event (for example, file download) occurs, I will need additional information from another web service (SRV2), data should be polled into input_window_2 and joined with input_window_1 on the same event.

How do I achieve this, polling data from another web service, only when specific event occurs?

I am still new on streaming, would be very grateful if a concrete example could be provided.

Thanks a lot for your expertise.

Regards,

Tao

[JWootton · Accepted solution]

Hi Tao,

I'm afraid right now there is not a way to do this - or at least not easily.  There is no way, using our existing web service adapters,  to do event-driven requests to the web service.

You can use the Web Service input adapter to poll a web service on a scheduled basis and load the returned data into a CCL input window, where it is then available within your data model (eg  you could join an incoming event to this window).  But this probably won't help you if you need to poll the Web Service for data related to the event, and if the data set held by the web service is too large to pre-load into a CCL window.

It is possible to do this if the data you needed was held in a database instead of a web service.  The CCL getdata() function lets you run event-driven queries against an external ODBC or JDBC database.  But we don't have the same thing for Web Services (or ODATA sources).  If we saw a need for this, it could be added to the roadmap.

Finally, to my earlier comment that there's no way to do it - at least not easily.  There is one way, if you're a programmer.  ESP does have an interface that lets you link external C++ or Java function libraries.  You could write a C++ or Java function to call the web service and return the result, and then link this library to your CCL project and call the function from a CCL Flex operator.  But clearly:  this is a lot of work involved.  Also note that the external function library interface in ESP is disabled by default, because it represents a security risk, but you can enable it and accept the risk, since you would be creating the external function.  Also note that support for external function libraries is only available in ESP and is not available in HANA SDS (because of that security risk).

[former_member214945 · Accepted solution]

Hello Jeff,

thank you, your reply is extremely helpful, as always.

We are still prototyping, so I take your advise, create a table in HANA with the additional informations, and add a flex window with the CCL getdata() function. It works.

I post my code here, could you please take a look if this is clean enough? In the case of assigning values to the outrec

outrec :=  [id = window1.id; ...]

is there a simple way to assign all values of window1 to outrec, then plus the 2 new column values?

Thanks and regards,

Tao

CREATE INPUT WINDOW window1

SCHEMA (

    id     long ,

    pdf   integer ,

    url    string ,

    wma integer

  )

PRIMARY KEY (id) ;

CREATE Flex pdf_flex

IN window1

OUT OUTPUT WINDOW window2 SCHEMA(

  id long ,

    pdf integer ,

    url string ,

    wma integer ,

  volume integer,

  price    float

) PRIMARY KEY (id)

BEGIN

   DECLARE

    typeof(window2) outrec;

    typedef[integer pdf;|integer volume; float price;] datarec;

    vector(datarec) pdfdata;

   END;

   ON window1 {

     if(isnull(pdfdata)) pdfdata :=new vector(datarec);

     if(window1.pdf=0 OR window1.pdf=2)

     {

       getData(pdfdata,'ana_odbc','select pdf, volumn, price from zhangta.pdfdata where pdf = ?',window1.pdf);

       for(rec in pdfdata)

         outrec := [id = window1.id; pdf = window1.pdf; url = window1.url; wma = window1.wma; volume = rec.volume; price = rec.price];

         output setOpcode(outrec,insert);

     }

     else

     {

       outrec := window1;

       output setOpcode(outrec,insert);

     }    

     resize(pdfdata,0);

   };

END;

[former_member217348 · Accepted solution]

Hi Tao,

Your question is great, but please do start a new thread so that others who are searching will be able to find it and follow it.

Thanks,

Alice

[JWootton · Accepted solution]

Your code looks fine (but I just gave it a quick look...).  As for your question: is there a simple way of assigning all values of window1 to outrec and then adding the additional fields?  sure.  it's just like you did in the else statement.  CCL will "coerce" the schema where it can. So just say:

outrec := window1;

outrec.volume := rec.volume;

outrec.price := rec.price;

output setOpcode(outrec, insert);

[JWootton · Accepted solution]

Actually,  after I posted my last reply, I realized that I was too focused on your specific question about how to set the values in outrec easily - and I missed a much bigger point (couldn't see the forest for the trees?).

A much simpler, and more performant approach to what you want to do is just join window1 with a CCL REFERENCE element that points to your HANA table.  This has a couple of benefits:

1. it's simpler to set up than a flex - it's just a slightly modified SQL statement

2. it will perform MUCH, MUCH better. Here's why:

- getdata() will execute a query on HANA every time an event with pdf = 1 or pdf = 2 arrives.

- the REFERENCE element has optional caching that will save the values retrieved from HANA and reuse them until they expire from the cache.

Here, it looks liek you are only pulling two values from HANA - and you are probably using them for multiple events.

If I get a chance later, I can show you a complete example, but it's basically just:

CREATE REFERENCE X (point it to zhangta.pdfdata table in HANA)...

<set your caching properties to on, and set an age as appropriate>

CREATE OUTPUT WINDOW Window2

...

Select...

From Window1 OUTER JOIN X where window1.pdf = x.pdf

[former_member214945 · Accepted solution]

Hi Jeff,

I tried this syntax already, but got compiler error: no stream named rec.

Regards,

Tao

[former_member214945 · Accepted solution]

Hi Alice,

thanks for the reminder. I will pay more attention to it in the future.

Regards,

Tao