With the SMP OData SDK, we can implement offline scenario with simple and easy API – if you’re not yet familiar with the SDK, there’re blogs and how-to-guides explaining practical details.
During the offline store creation, you might encounter the runtime error, simply because the OData collection you're trying to fetch is too big - you can copy & paste the OData endpoint URL in the web browser and see if it shows 500 timeout error. The issue in this case is a problem on the server-side. The HTTP requests for data were timing out because the backend producer was taking too long to query the data and encode it as OData.
In this case we should consider using server side paging on OData producer. There are two types of paging in OData – server side and client side.
Client side paging is driven by having a $top value and then incrementing the $skip values for each request (e.g. $top=100&$skip=0 for first request, then $top=100&$skip=100 for second request etc.) This type of paging will not help with populating the database in the offline store because the defining requests are fixed. (Of course, you can use $top or $skip to make requests for small amounts of data to the local store after the store has been populated.)
In server side paging, even if you request the entire big OData collection, the server will only return a portion of the results, but provide a “next link” info, which can be followed to get the next portion and so on. When the MobiLink is populating the local store it knows how to follow these next links. Server side paging via next links is OData Version 2.0 feature. The $skiptoken query is seen in OData URL if it is implemented.
This H2G explains how to build OData Query service which supports $skiptoken with SAP Gateway.
Prerequisites
You should have done the following steps:
#1 - OData CRUD Crash Course - Query
Implementing $skiptoken logic
Essentially the $skiptoken implementation is fairly simple. Once you have a table data, here’s the basic logic flow:
- Obtain the $skiptoken value if it exists in the URL.
- Measure the size of the table.
- If the size is beyond the predefined chunk size (ex. 1000 entities for each $skiptoken page), cut the whole table into pieces (= pages) with the chunk size.
- Set both the last index of the entity for the next $skiptoken call and one piece of the table data as the return table (= a page) of the get_entityset.
That's all. Here’s the code implementing the steps above:
Note: Appendix has the complete $skiptoken implementation code you can run by copy & paste in get_entityset method section.
01 * a) Obtain the $skiptoken value if it exists in the URL
02 lv_skiptoken = io_tech_request_context->get_skiptoken( ).
03
04 * b) Measure the size of the table
05 DESCRIBE TABLE lt_entityset LINES lv_table_size.
06 IF lv_table_size IS INITIAL.
07 CLEAR es_response_context-skiptoken.
08 EXIT.
09 ENDIF.
10
11 * c) If the size is beyond the predefined chunk size,
12 * cut the whole table into pieces with the chunk size
13 IF lv_table_size < lv_paging_max_chunk.
14 lv_index_beg = 1.
15 lv_index_end = lv_table_size + 1.
16 ELSE.
17
18 IF lv_skiptoken IS NOT INITIAL.
19 lv_token_len = STRLEN( lv_skiptoken ) - 1.
20 IF lv_token_len > 0.
21 lv_token_beg = lv_skiptoken(1).
22 lv_token_end = lv_skiptoken+lv_token_len(1).
23 IF lv_token_beg = '''' AND lv_token_end = ''''.
24 lv_token_len = lv_token_len - 1.
25 lv_skiptoken = lv_skiptoken+1(lv_token_len).
26 ENDIF.
27 ENDIF.
28 * Determine indices for BEG and END
29 lv_token_len = STRLEN( lv_skiptoken ).
30 IF lv_token_len > 0 AND
31 lv_token_len < 10 AND
32 lv_skiptoken CO '0123456789'.
33 lv_index_beg = lv_skiptoken.
34 ELSE.
35 lv_index_beg = 1.
36 ENDIF.
37 ELSE.
38 lv_index_beg = 1.
39 ENDIF.
40 lv_index_end = lv_index_beg + lv_paging_max_chunk.
41 ENDIF.
42 * d) Set both the last index of the entity for the next $skiptoken call
43 * and one piece of the table data as the return table of the get_entityset
44 IF lv_index_end <= lv_table_size.
45 es_response_context-skiptoken = lv_index_end.
46 ENDIF.
47 CONDENSE es_response_context-skiptoken.
48
49 LOOP AT lt_entityset INTO ls_entity.
50 IF sy-tabix < lv_index_beg. CONTINUE. ENDIF.
51 IF sy-tabix >= lv_index_end. EXIT. ENDIF.
52 INSERT ls_entity INTO TABLE et_entityset.
53 ENDLOOP.
In the line #02, “io_tech_request_context->get_skiptoken( )” method obtains the $skiptoken value if it exists in the URL.
#11 - #41 are the core trick algorithm to split one table into pieces, basically you can copy & paste for any single table - they are all about to calculate where is "begin" and "end" row of a table with a specific chunk size.
#44 - #46 set the last entity index number as a return parameter “es_response_context-skiptoken” so that the subsequent OData Query call can fetch the continuous collection data.
#49 - #53 extracts the entities from begin to end and set them as a returning OData entityset.
You should wonder how those variables are declared, so here's the one:
Naming convention:
l - local scope
t - table
s - structure
v - variable
01 DATA: lt_entityset TYPE TABLE OF stravelag,
02 * variables for $skiptoken
03 ls_entity TYPE stravelag,
04 lv_paging_max_chunk TYPE i VALUE 50,
05 lv_index_beg TYPE i,
06 lv_index_end TYPE i,
07 lv_skiptoken TYPE string,
08 lv_token_len TYPE i,
09 lv_token_beg TYPE c,
10 lv_token_end TYPE c,
11 lv_table_size TYPE i.
As you see this sample code has chunk size "50", which indicates each page should have 50 entities with this $skiptoken implementation.
Now you can activate it and test it.
Testing the $skiptoken
Try running the service from either the browser or REST Client. Have a look at the <link> data at the bottom of the OData payload – this is the $skiptoken we have implemented. You can try copy this generated $skiptoken URL as the OData URL and see how the next $skiptoken data looks like – for this example it continues to show the next chunk starting from the 51st entity.
Your $skiptoken implementation runs successfully!
Note: Once you implement it, in general, you should experiment a bit with the page sizes until you find a size that balances the backend performance and the response size of offline store. (For example, overall offline store initial load performance with a chunk size 1000 in one page might be slower than 5000, and so on.)
And - have you read this yet? - SMP3 OData SDK - Performance Tuning with Offline Store
Appendix – Query operation with $skiptoken
method TRAVELAGENCYSET_GET_ENTITYSET.
DATA: lt_entityset TYPE TABLE OF stravelag,
* variables for $skiptoken
ls_entity TYPE stravelag,
lv_paging_max_chunk TYPE i VALUE 50,
lv_index_beg TYPE i,
lv_index_end TYPE i,
lv_skiptoken TYPE string,
lv_token_len TYPE i,
lv_token_beg TYPE c,
lv_token_end TYPE c,
lv_table_size TYPE i.
SELECT * FROM stravelag INTO TABLE lt_entityset.
* a) Obtain the $skiptoken value if it exists in the URL
lv_skiptoken = io_tech_request_context->get_skiptoken( ).
* b) Measure the size of the table
DESCRIBE TABLE lt_entityset LINES lv_table_size.
IF lv_table_size IS INITIAL.
CLEAR es_response_context-skiptoken.
EXIT.
ENDIF.
* c) If the size is beyond the predefined chunk size,
* cut the whole table into pieces with the chunk size
IF lv_table_size < lv_paging_max_chunk.
lv_index_beg = 1.
lv_index_end = lv_table_size + 1.
ELSE.
IF lv_skiptoken IS NOT INITIAL.
lv_token_len = STRLEN( lv_skiptoken ) - 1.
IF lv_token_len > 0.
lv_token_beg = lv_skiptoken(1).
lv_token_end = lv_skiptoken+lv_token_len(1).
IF lv_token_beg = '''' AND lv_token_end = ''''.
lv_token_len = lv_token_len - 1.
lv_skiptoken = lv_skiptoken+1(lv_token_len).
ENDIF.
ENDIF.
* Determine indices for BEG and END
lv_token_len = STRLEN( lv_skiptoken ).
IF lv_token_len > 0 AND
lv_token_len < 10 AND
lv_skiptoken CO '0123456789'.
lv_index_beg = lv_skiptoken.
ELSE.
lv_index_beg = 1.
ENDIF.
ELSE.
lv_index_beg = 1.
ENDIF.
lv_index_end = lv_index_beg + lv_paging_max_chunk.
ENDIF.
* d) Set both the last index of the entity for the next $skiptoken call
* and one piece of the table data as the return table of the get_entityset
IF lv_index_end <= lv_table_size.
es_response_context-skiptoken = lv_index_end.
ENDIF.
CONDENSE es_response_context-skiptoken.
LOOP AT lt_entityset INTO ls_entity.
IF sy-tabix < lv_index_beg. CONTINUE. ENDIF.
IF sy-tabix >= lv_index_end. EXIT. ENDIF.
INSERT ls_entity INTO TABLE et_entityset.
ENDLOOP.
endmethod.
- SAP Managed Tags:
