This blog is part of a collection of blog entries that shows architectural concepts and configuration of the SAP PI REST Adapter. We also added some sample scenarios to make it easier for you to understand how your scenario can be implemented using the PI REST Adapter.
If you haven’t done so far, best is to start with the very first blog
PI Rest Adapter - Don't be afraid within the blog series covering the concepts of the REST adapter. A complete list of all blog entries can be accessed from here
PI REST Adapter - Blog Overview.
The current blog describes how to poll a REST API. The sender REST adapter in polling mode has been supported since release 7.31 SP16 / 7.4 SP11.
Scenario
We would like to frequently poll video information from Google's Youtube REST API, and store the same on a file system. Furthermore, the resultset should be split into individual messages per video ID, and already polled results should be discarded.
In the SAP Process Integration Designer perspective of the NetWeaver Developer Studio (NWDS), I have defined an Integration Flow with a REST sender channel and a file receiver adapter.
Let's focus on the configuration of the sender channel.
Configuring the REST sender channel
Double-click on the sender adapter to open the channel editor. Select the REST adapter type, and from the Message Protocol drop down menu the entry REST Polling.
Option 1: Incremental request based on timestamp of last call
In order to better understand the configuration below, let's take a look at a sample response in JSON format. The REST API returns an array of
items each having a unique ID stored in the
etag field.
First switch to sub-tab
Data Format below tab
Adapter-Specific.
The format is JSON, so select
JSON from the
Data type drop down menu. We would like to convert the JSON into XML, so select the
Convert JSON to XML check box, and add a wrapper element to ensure that the converted XML format contains one root element only.
As mentioned below, the message should be split into individual messages per video ID. Select
Split Result into Multiple Messages check box. As
Array Containing Messages maintain
items (See message format above).
Furthermore, duplicates should be removed. Select the
Filter out Duplicates check box, and maintain
etag as
Unique ID Element (See message format above).
To place incremental requests, we would like to use the timestamp of the latest call. The value is stored between the calls and can be used in the REST URL as a placeholder with name
incrementalToken. By the way, alternatively you can also use an XPath expression or a JSON element from the response of the last call such as
next Page indicator or similar (shown below). From the
Incremental Type drop down select entry
Timestamp of Last Call. The timestamp format of the API complies with ISO 8601, and needs to be specified as follows:
yyyy-MM-ddTHH:mm:ssZ with T and Z being constants. The REST adapter follows the Joda time format, see
DateTimeFormat (Joda time 2.2 API). So, we need to place the constants in quotes. Maintain the
Timestamp Format attribute as
yyyy-MM-dd'T'HH:mm:ss'Z', and define an initial value that complies with the format.
Switch to tab
HTTP Request, and maintain the
Target URL as follows:
https://www.googleapis.com/youtube/v3/search?key=<your API key>&part=id&q=SAP&maxResults=50&order=date&publishedAfter={incrementalToken}. Note, I added the placeholder
incrementalToken in curly brackets which holds the timestamp of the last call. As
HTTP Operation select
GET. Finally, define a polling interval, here 3600 seconds.
Option 2: Incremental request based on response content
As another option I would like to show you how to configure the incremental request in case that many pages need to be requested. As you can see below, in the response two additional elements are added holding the previous (
prevPageToken) and the next page token (
nextPageToken). Latter can be used in the URL to gather the next response page.
From the
Incremental Type drop down select the entry
Response Content, and maintain the incremental token element as
nextPageToken. This will store the value of the
nextPageToken element into the placeholder
incrementalToken.
From release 7.31 SP17 / 7.4 SP12 / 7.5 SP01 onwards, you can define an action in case that the very last page has been reached and hence the
nextPageToken element is missing. You have the following options:
- treat as error which is the default
- skip current poll and retry later
- use an empty token value
- use the initial token value
- use a custom token value
Here, we chose the option
Use Empty Value for Token.
On tab
HTTP Request, maintain the
Target URL as follows:
https://www.googleapis.com/youtube/v3/search?key=<your API key>&part=id&q=SAP&maxResults=5&pageToken={incrementalToken}. Note, I added the placeholder
incrementalToken in curly brackets which holds the token of the next page.
I hope this blog was helpful to understand how to configure the REST sender channel in polling mode. If you like to learn more, check out the other blogs in the series, accessible from the main blog
PI REST Adapter - Blog Overview.