Integration Gateway: Understanding REST data sourc...

CarlosRoggan · ‎02-17-2015

This blog is (once more) about Integration Gateway in SAP Mobile Platform 3.0.

It is meant for those of you who have already created OData services based on Integration Gateway in SAP Mobile Platform 3.0 (SMP).

If you want your OData service to support filtering, you have to implement it in the custom script.

This tutorial explains how it is done using the ExpressionVisitor interface, which is provided by Olingo.

By the way, this tutorial is based on SMP SP05.

We’re not going to start from scratch. So in order to follow this tutorial, you need to have the project in place that is created in my previous tutorial.

Prerequisites

My previous tutorial where we’ve created an OData service based on a REST data source:
http://scn.sap.com/community/developer-center/mobility-platform/blog/2015/02/12/integration-gateway-...
SAP Mobile Platform 3.0 SP 05
Eclipse Kepler with SAP Mobile Platform Tools installed

Preparation

The following steps were done in the previous tutorial:

Create implementation project
Define OData model
Generate custom code for Groovy
implement the processResponseData() method
Deploy it to SMP
Create/configure the destination
Test the query operation.

Filter Implementation

Overview:

Retrieve the Objects UriInfo and FilterExpression.
Create Visitor class that implements the ExpressionVisitor interface
Implement the 4 required methods
Invoke the visitor
Use the result to manipulate the relative URI

Step: Callback method processRequestData()

Implementing the callback method processRequestData() that is provided by the Integration Gateway Framework and which is generated into our custom script.

We have to understand:

This method is called, before the REST service (= the data source) is invoked.

The order is:

The user calls our OData service in SMP. E.g. https://<host>/gateway/odata/SAP/<SERVICE>/StreetSet?$filter=streetType eq ’PEDESTRIAN’

The OData call is received by Integration Gateway.

Integration Gateway requires our help, therefore it passes the OData-URI (which was called by the user) into our custom script, in the processRequestData() method

Here, we have to change the URI, because the filter statement in OData language is not understood by the REST service.

After changing the URI, we give it back to Integration Gateway.

Then Integration Gateway can safely call the REST service and will get the filtered data.

At this point we need to check the documentation of our REST service:

http://www.gisgraphy.com/documentation/user-guide.htm#streetwebservice

For this tutorial, I’ve chosen the streettype for filtering.

Example filter:

&streettype=PEDESTRIAN

The full URL:

http://services.gisgraphy.com/street/streetsearch?lat=52.52&lng=13.41&streettype=PEDESTRIAN

Note:

Please ignore that this URL actually contains 3 filters.

The background is that when we've created the Eclipse project, we’ve configured following URL

http://services.gisgraphy.com/street/streetsearch?lat=52.52&lng=13.41 as REST service-URL to provide data.

So, since the first 2 filters belong to the base URL (as we've defined it), we have to ignore these first 2 filters in our tutorial

So we have the base URL:

http://services.gisgraphy.com/street/streetsearch?lat=52.52&lng=13.41

and we add one filter to it:

&streettype=PEDESTRIAN

Step: Object UriInfo

The Olingo API provides the interface UriInfo

We need it, because this object contains information about the URI, which includes system query options.

The part of the URI, in which we’re interested, is the filter statement.

Step: Object FilterExpression

The Olingo API provides the interface FilterExpression

We need it in order to parse the OData filter statement.

We obtain the filter statement by asking the UriInfo for the FilterExpression:

FilterExpression filterExpression = uriInfo.getFilter();

The filterExpression object contains the actual filter that is specified in the URL of the invoked OData service on SMP.

This is the $filter parameter, as specified in OData.

Example:

If the full URL of the OData service invocation is as follows:

https://<SMPserver>/gateway/odata/SAP/<YOUR_SERVICE>/StreetSet?$filter=streetType eq ’PEDESTRIAN’

then the filterExpression object contains the information about

filter=streetType eq ’PEDESTRIAN’

If the full URL of the OData service invocation is as follows (just as an example):

https://<server>/gateway/odata/SAP/<YOUR_SERVICE>/StreetSet?$format=json

then the filterExpression object is null

So we have to explicitly handle the case that no filter is desired:

if (filterExpression == null){

return;

}

The filterExpression object is required to invoke the ExpressionVisitor.

Step: Interface ExpressionVisitor

The Olingo API provides the interface ExpressionVisitor

This interface has to be implemented by us.

Here it is where the filter implementation is realized.

The ExpressionVisitor is required, because a filter statement can be complex.

The Olingo framework traverses all the filter statements in the OData service call and passes the segments to us in the different methods of the interface.

In our script, we create an inner class that implement the ExpressionVisitor interface:

def class RestExpressionVisitor implements ExpressionVisitor{

We let Eclipse add the methods which have to be implemented.

Most of them can be ignored.

In following steps, we'll have a closer look at the relevant methods.

visitProperty()

visitLiteral()

visitBinary()

visitFilterExpression()

They are invoked in this order.

Example:

What happens, if our OData service is invoked with $filter=streetType eq ’PEDESTRIAN’

The methods are invoked in the following order:

visitProperty()	Here we have to deal with the property, e.g.	streetType
visitLiteral()	Here we have to deal with the literal, e.g.	‘PEDESTRIAN’
visitBinary()	Here we have to deal with the operator, e.g.	eq
visitFilterExpression()	Here we have to deal with our filterExpression, e.g.	streetType eq ‘PEDESTRIAN’

Note:

These sample values on the right column depend on the implementation.

Second example:

What happens, if our OData service is invoked with $filter=streetType eq ’PEDESTRIAN’ and name eq ‘East Lake Street’

visitProperty()	Deal with the property, e.g.	streetType
visitLiteral()	Deal with the literal, e.g.	‘PEDESTRIAN’
visitBinary()	Deal with the operator, e.g.	eq
visitProperty()	Deal with the second property, e.g.	name
visitLiteral()	Deal with the second literal, e.g.	‘East Lake Street’
visitBinary()	Deal with the operator, e.g.	eq
visitBinary()	Deal with the operator, e.g.	and
visitFilterExpression()	Deal with our filterExpression, e.g.	streetType eq ‘PEDESTRIAN’ and name eq ‘East Lake Street’

Note:

Again, these sample values depend on the implementation

We can see that the visitor traverses all segments and finally calls for the final expression

Note that In this tutorial we cover only simple filter statement.

Now let’s implement the methods for our example

Step: Method visitProperty()

The method signature:

Object visitProperty(PropertyExpression propertyExpression, String uriLiteral, EdmTyped edmTyped)

In this method, we get the possibility to analyze the property mentioned in the filter expression.

In our example:

$filter=streetType eq ’PEDESTRIAN’

Here, the text streetType is the name of a property (in the OData model).

When implementing the visitProperty() method, we can access the property as an object as follows:

EdmProperty edmProperty = (EdmProperty) edmTyped;

And the name of the property:

String propertyName = edmProperty.getName();

So what?

In our REST example service we really need this method.

Why?

Because the parameter, that is supported by the REST service is: streettype

(see docu)

but the name of the property (in both the OData model and the REST service) is: streetType

See it? They differ.

Without converting the OData property name to the REST parameter name, the filtering doesn't work.

But we can make use of the visitProperty() method in order to do a mapping from property name to parameter name

String restParameterName = null;

if(propertyName.equals("streetType")){ // we get the OData property name

restParameterName = "streettype"; // we convert it to REST parameter name

}else{

restParameterName = propertyName;

}

Our method has to return the name of the parameter that is used by the REST service.

See the sample implementation of the method here:

@Override

public Object visitProperty(PropertyExpression propertyExpression, String uriLiteral, EdmTyped edmTyped) {

if(! (edmTyped instanceof EdmProperty)){

throw new RuntimeException("Unexpected type");

}

EdmProperty edmProperty = (EdmProperty) edmTyped;

String propertyName = null;

try {

propertyName = edmProperty.getName();

} catch (EdmException e) {

((ILogger)log).logErrors(LogMessage.TechnicalError, "Failed to read property");

throw new RuntimeException("Unable to access name of property", e);

}

String restParameterName = null;

if(propertyName.equals("streetType")){ //in the REST service, the param is different

restParameterName = "streettype";

}else{

restParameterName = propertyName;

}

return restParameterName; // e.g. will lead to streettype=PEDESTRIAN

}

Step: Method visitLiteral()

The method signature:

Object visitLiteral(LiteralExpression literalExpression, EdmLiteral edmLiteral)

In this method, we get 2 objects, but we focus on the second only:

EdmLiteral can be used for obtaining the literal, which is the pure text of the filter.

What do we have to do?

As we’ve seen above, the syntax of our REST service for filtering is:

&<parametername>=<value>

Note that the value doesn’t require inverted comma.

:!: -> here we have the difference to OData

So in our example the OData segment ’PEDESTRIAN’

Has to be converted to the REST segment PEDESTRIAN

(without inverted commas)

This is easy to implement: just remove the inverted commas.

And it is even easier to implement, because the following statement returns the value without inverted comma

edmLiteral.getLiteral()

So our implementation looks as follows:

@Override

public Object visitLiteral(LiteralExpression literalExpression, EdmLiteral edmLiteral) {

return edmLiteral.getLiteral();

}

Step: Method visitBinary()

The method signature:

Object visitBinary(BinaryExpression binaryExpression, BinaryOperator operator, Object leftSide, Object rightSide)

In this method, we have to convert the operator.

As we’ve seen above, the syntax of our REST service for filtering is:



&<parametername>=<value>

The operator is: =

:!: -> this is the difference to OData

So in our example the OData operator: eq

Has to be converted to the REST operator: =

This has to be done for all supported operators, so we have to care about each operator in a switch statement.

Note that Olingo provides an enum for the OData operators:

org.apache.olingo.odata2.api.uri.expression.BinaryOperator

We get it as parameter in the method.

At the end, we have to return the whole expression.

In our tutorial, let’s support only 2 operators (it depends of course on the chosen REST service), to allow the above service invocation.

public Object visitBinary(BinaryExpression binaryExpression, BinaryOperator operator, Object leftSide, Object rightSide) {

String operatorForREST = "";

switch(operator){

case BinaryOperator.EQ:

operatorForREST = "=";

break;

case BinaryOperator.AND:

operatorForREST = "&";

break;

default:

throw new RuntimeException("Not supported: " + operator.toUriLiteral());

}

return leftSide + operatorForREST + rightSide;

}

Step: Method visitFilterExpression()

The method signature:

Object visitFilterExpression(FilterExpression filterExpression, String expressionString, Object expression)

In this method, we have the chance to check the final filter expression.

The parameter expression contains our REST filter expression, as we've defined it in the previous visitor-methods.

E.g. streettype=PEDESTRIAN

What are we going to do here?

In a simple example, we would simply return the instance of expression.

Our REST sample service is a bit special, because our so-called base-URI already contains some parameters. As explained above, we use it simply to get some data. The fact that the REST service needs 2 parameters for geolocation, doesn’t disturb us.

It only forces us to do a little work in the visitFilterExpresission() method.

The reason is:

Our expression looks fine: streettype=PEDESTRIAN

But it has to be appended to the base URL, which already contains parameters:

http://services.gisgraphy.com/street/streetsearch?lat=52.52&lng=13.41

So before appending our expression, we have to insert a &

That’s all.

Our sample implementation:

public Object visitFilterExpression(FilterExpression filterExpression, String expressionString, Object expression) {

if(expression instanceof String){

String expressionAsString = (String)expression;

if(! expressionAsString.startsWith("&")){

expressionAsString = "&" + expressionAsString;

return expressionAsString;

}

return expression;

}

Step: Use ExpressionVisitor

In the above steps, we've created the RestExpressionVisitor class, that implements the required interface and the necessary methods.

Now we can instantiate and invoke it:

RestExpressionVisitor visitor = new RestExpressionVisitor();

Object restFilterObject = null;

try {

restFilterObject = filterExpression.accept(visitor);

} catch (ExceptionVisitExpression e) {

log.logErrors(LogMessage.TechnicalError, "Failed calling accept()");

return;

} catch(ODataApplicationException oe){

log.logErrors(LogMessage.TechnicalError, "Failed calling accept()");

return;

}

The returned restFilterObject is the result of our effort: the REST-izised version of our OData-filter.

In our sample implementation, the resulting value is the following string:

&streettype=PEDESTRIAN&name=East Lake Street

What to do with it?

Remember that we’re implementing the callback method processRequestData(), where we’re supposed to manipulate the backend-URL, before it is called.

So we have to build the URI containing the filter string.

Usually, it is formed as follows:

<relativeURI> + ? + <filterString>

In our special case, the relativeURI already contains the ?

So we don’t need to add it.

We get the relativeURI string from the header, which is passed in the method signature in the message object.

In our example, the relative URI woud be:

/street/streetsearch?lat=41.89&lng=-87.64

The code looks as follows:

String relativeUri = getRelativeUri(message);

relativeUri = relativeUri + (String)restFilterObject;

and the code to obtain the relativeURI:

def String getRelativeUri(Message message){

Map headers = message.getHeaders();

Object object = headers.get("RelativeUri");

if(object instanceof String){

return (String)object;

}

return null;

}

The last step to do in the processRequestData() method is: to pass the new relativeUri to the Integration Gateway framework.

This is done by setting the header:

def void setRelativeUri(Message message, String newUri){

message.setHeader("RelativeUri", newUri);

}

Final: Run

That’s it.

At least, that’s it for the filter implementation in the method processRequestData().

Which is the focus of this blog.

We haven’t talked about the processResponseData(), as this was the focus of my previous blog

Note that you also need to add the required import statements and also to add required bundles.

I’m pasting the full script below, and I'm also attaching the relevant files of my Eclipse project.

Note that the attached files are based on SP05. So if you use them for a different version, you have to adapt them, e.g. update the dependencies in the manifest file.

Please refer to my previous blogs for detailed information (see Links section).

Testing the filtering

After generate and deploy the designtime artifact and after configuring the service on SMP, you can try calling several URLs with different filters:

First test without filter:

https://<host>:8083/gateway/odata/SAP/<YOUR_SERVICE>;v=1/StreetSet

Try different streetType filters:

https://<host>:8083/gateway/odata/SAP/<YOUR_SERVICE>;v=1/StreetSet?$filter=streetType eq ’PEDESTRIAN’

https://<host>:8083/gateway/odata/SAP/<YOUR_SERVICE>;v=1/StreetSet?$filter=streetType eq ‘FOOTWAY’

And also name filters:

https://<host>:8083/gateway/odata/SAP/<YOUR_SERVICE>;v=1/StreetSet?$filter=name eq ‘East Lake Street’

https://<host>:8083/gateway/odata/SAP/<YOUR_SERVICE>;v=1/StreetSet?$filter=name eq ‘North Canal Street’

Try combined filter

https://<host>:8083/gateway/odata/SAP/<YOUR_SERVICE>;v=1/StreetSet?$filter=streetType eq ‘PEDESTRIAN’ and name eq ‘East Lake Street’

There should be different and meaningful response to these calls.

Full source code



import org.apache.olingo.odata2.api.edm.EdmException

import org.apache.olingo.odata2.api.edm.EdmLiteral

import org.apache.olingo.odata2.api.edm.EdmProperty

import org.apache.olingo.odata2.api.edm.EdmTyped

import org.apache.olingo.odata2.api.exception.ODataApplicationException;

import org.apache.olingo.odata2.api.uri.UriInfo

import org.apache.olingo.odata2.api.uri.expression.BinaryExpression

import org.apache.olingo.odata2.api.uri.expression.BinaryOperator

import org.apache.olingo.odata2.api.uri.expression.ExceptionVisitExpression;

import org.apache.olingo.odata2.api.uri.expression.ExpressionVisitor

import org.apache.olingo.odata2.api.uri.expression.FilterExpression

import org.apache.olingo.odata2.api.uri.expression.LiteralExpression

import org.apache.olingo.odata2.api.uri.expression.MemberExpression

import org.apache.olingo.odata2.api.uri.expression.MethodExpression

import org.apache.olingo.odata2.api.uri.expression.MethodOperator

import org.apache.olingo.odata2.api.uri.expression.OrderByExpression

import org.apache.olingo.odata2.api.uri.expression.OrderExpression

import org.apache.olingo.odata2.api.uri.expression.PropertyExpression

import org.apache.olingo.odata2.api.uri.expression.SortOrder

import org.apache.olingo.odata2.api.uri.expression.UnaryExpression

import org.apache.olingo.odata2.api.uri.expression.UnaryOperator

import com.sap.gateway.core.ip.component.commons.ODataExchangeHeaderProperty

import com.sap.gateway.ip.core.customdev.logging.ILogger;

import com.sap.gateway.ip.core.customdev.logging.LogMessage;

import com.sap.gateway.ip.core.customdev.util.Message

/*

* ***************************

*  FRAMEWORK CALLBACK METHODS

* ***************************

*/

/**

  Function processRequestData will be called before the request data is

  handed over to the REST service and can be used for providing

  filter capabilities. User can manipulate the request data here.

*/

def Message processRequestData(message) {

    handleFilter(message);

    return message;

}

/**

  Function processResponseData will be called after the response data is received

  from the REST service and is used to convert the REST response to OData

  response using String APIs. User can manipulate the response data here.

*/

def Message processResponseData(message) {

    convertResponse(message);

    return message;

}

/*

* ***************************

*  RESPONSE HANDLING

* ***************************

*/

def void convertResponse(Message message){

    String restResponse = message.getBody().toString();

    /* CONVERT PAYLOAD */

    int index = restResponse.indexOf("<result>");

    if(index == -1){ // in case of empty payload

        restResponse = "<StreetSet><Street><gid></gid></Street></StreetSet>"; // currently the only possibility for returning nothing

    }else{

        restResponse = restResponse.substring(index);

        restResponse = restResponse.replaceAll("<result>", "<Street>");

        restResponse = restResponse.replaceAll("</result>", "</Street>");

        restResponse = restResponse.replaceAll("</results>", "</StreetSet>");

        restResponse = "<StreetSet>" + restResponse;

    }

    message.setBody(restResponse);

    message.setHeader("Content-Type", new String("xml"));

}

/*

* ***************************

*  REQUEST HANDLING

* ***************************

*/

/**

*     supporting $filter for one property only

* */

def void handleFilter(Message message){

    //retrieve the FilterExpression

    UriInfo uriInfo = getUriInfo(message);

    FilterExpression filterExpression = uriInfo.getFilter();

    if (filterExpression == null){

        // this is the case if the service is invoked without $filter

        return;

    }

    // Invoke the ExpressionVisitor

    RestExpressionVisitor visitor = new RestExpressionVisitor();

    Object restFilterObject = null;

    try {

        restFilterObject = filterExpression.accept(visitor);

    } catch (ExceptionVisitExpression e) {

        log.logErrors(LogMessage.TechnicalError, "Error while calling accept() on custom ExpressionVisitor");

        return;

    } catch(ODataApplicationException oe){

        log.logErrors(LogMessage.TechnicalError, "Error while calling accept() on custom ExpressionVisitor");

        return;

    }

    if(restFilterObject == null){

        log.logErrors(LogMessage.TechnicalError, "Couldn't retrieve the converted filter expression for the backend REST service.");

        return ;

    }

    // add the filter statement (converted to REST-format)

    String relativeUri = getRelativeUri(message); // the REST service URL without hostname. In our example it is: /street/streetsearch?lat=52.52&lng=13.41

    relativeUri = relativeUri + (String)restFilterObject; // we know that it is a String, because we've implemented it in the Visitor below

    // encode

    relativeUri = relativeUri.replaceAll(" ", "%20").replaceAll("'","%27");

    // Finally, set the newly composed URI

    setRelativeUri(message, relativeUri);

}

/**

* Implementation of ExpressionVisitor

*

* Supports combination of 2 filters

* e.g.

 * https://<host>/gateway/odata/SAP/<YOUR_SERVICE>/StreetSet?$filter=streetType eq ‘PEDESTRIAN’ and name eq ‘East Lake Street’

*

* */

def class RestExpressionVisitor implements ExpressionVisitor{

    /*

     * This implementation converts the OData-like operators (eq)

     * to operators in the format, that's understood by the REST service

     * */

    @Override

    public Object visitBinary(BinaryExpression binaryExpression, BinaryOperator operator, Object leftSide, Object rightSide) {

        String operatorForREST = "";

          

        switch(operator){

            case BinaryOperator.EQ:

                operatorForREST = "=";

                break;

            case BinaryOperator.AND:

                operatorForREST = "&";

                break;

                default:

                    throw new RuntimeException("This filter operator is not supported: " + operator.toUriLiteral());

            }

          

        return leftSide + operatorForREST + rightSide;

    }

  

    @Override

    public Object visitFilterExpression(FilterExpression filterExpression, String expressionString, Object expression) {

  

        // we need to add & because the filter is not the first URL parameter

        if(expression instanceof String){

            String expressionAsString = (String)expression;

            if(    ! expressionAsString.startsWith("&")){

                expressionAsString = "&" + expressionAsString;

                return expressionAsString;

            }

        }

    }

    @Override

    public Object visitLiteral(LiteralExpression literalExpression, EdmLiteral edmLiteral) {

        return edmLiteral.getLiteral();

    }

    @Override

    public Object visitMember(MemberExpression arg0, Object arg1, Object arg2) {

        return null;

    }

    @Override

    public Object visitMethod(MethodExpression arg0, MethodOperator arg1, List<Object> arg2) {

        return null;

    }

    @Override

    public Object visitOrder(OrderExpression arg0, Object arg1, SortOrder arg2) {

        return null;

    }

    @Override

    public Object visitOrderByExpression(OrderByExpression arg0, String arg1,List<Object> arg2) {

        return null;

    }

    /*

     * We need to do a mapping from OData-property-name to REST-parameter-name

     * */

    @Override

    public Object visitProperty(PropertyExpression propertyExpression, String uriLiteral, EdmTyped edmTyped) {

        if(! (edmTyped instanceof EdmProperty)){

            throw new RuntimeException("Unexpected type");

        }

  

        EdmProperty edmProperty = (EdmProperty) edmTyped;

        String propertyName = null;

        try {

            propertyName = edmProperty.getName();

        } catch (EdmException e) {

            ((ILogger)log).logErrors(LogMessage.TechnicalError, "EdmException occurred while accessing filter statment property");

            throw new RuntimeException("Unable to access name of property", e);

        }

  

        String restParameterName = null;

        if(propertyName.equals("streetType")){ // in the REST service, the filter parameter is spelled differently than the property name

            restParameterName = "streettype";

        }else{

            restParameterName = propertyName;

        }

  

        return restParameterName; // in our example, this will at the end lead to the expression streettype=PEDESTRIAN

    }

    @Override

    public Object visitUnary(UnaryExpression arg0, UnaryOperator arg1,Object arg2) {

        return null;

    }

}

/*

* ***************************

*  HELPER

* ***************************

*/

def String getRelativeUri(Message message){

    Map headers = message.getHeaders();

    Object object = headers.get("RelativeUri");

    if(object instanceof String){

        return (String)object;

    }

        return null;

}

def void setRelativeUri(Message message, String newUri){

    message.setHeader("RelativeUri", newUri);

}

def UriInfo getUriInfo(Message message){

    Map headers = message.getHeaders();

    Object object = headers.get(ODataExchangeHeaderProperty.UriInfo.toString()); // need dependency to bundle com.sap.gw.rt.ip.commons.camel-commons

    if(object instanceof UriInfo){

        return (UriInfo)object;

    }

    return null;

}