Using #SAPHCP (Trial) To See Where In The World Yo...

Robert_Russell · ‎08-19-2016

Background

A View of The Final Page

XS Twitter API

BITLY API

To BASE64

A Natural Earth

SHINE on HCP

How Did I Get HERE

Bringing It All Together With UI5

Background

I have personal projects based on maps and enjoy creating maps based on SAP technology. As the SAP Community Network (SCN) is worldwide it makes an interesting use case for mapping the world based on SCN data. The main Jive based SCN platform blog page is linked to an automated Twitter account called @SCNblogs which may (or may not) tweet about recently published blogs. I underlined may or may not due to the fact that the @SCNblogs account is currently not capable of tweeting all blogs and it does appear random in what it tweets about. When it does tweet about a new SCN blog it uses the BITLYhttps://bitly.com/ service to link to the actual blog on SCN, using these BITLY links it is possible to check where in the world the clicks occur. A few years ago @SCNblogs was a lot better at tweeting published blogs although even back then it was not perfect. Having said all that I still like the fact that using an HCP trial account I can utilise the twitter API and the linked BITLY api to map the SCN related world. So the following is based on an HCP Trail MDC HANA setup to find out where in the world people are clicking on your @SCNblogs tweet (if you're lucky to appear on the @SCNblogs twitter timeline ;))

A View of The Final Page

The below image shows a UI5 table of the latest tweets of @SCNblogs. By selecting a row in the table the bitly clicks per country will be shown on a map.

The page uses HANA XS services for the API calls, UI5 controls and Leaflet.js which is a Javascript map library to create maps from an HANA spatial GEOJSON service and the HERE map service. I’ll cover each of these areas in more detail in my blog.

XS Twitter API

My thanks to Wenjun Zhou for sharing his blog on an HANA XS based approach to search tweets. This was the inspiration to try the process on my HCP Multitenant Database Container (MDC) based trial account. I was aware of Wenjun’s blog for a while and it was published back in 2014 but with the original setup of the HCP Trial the HANA XS destinations always appeared restricted to HTTP only, which prevented any XS access to the Twitter API from my experience. Now that the trail HCP allows an MDC setup and so much more flexibility in the setup, I decided to try out the Twitter API approach. I followed Wenjun’s theory for the XS connection although the screens have been updated in the HANA SPS 10 on the HCP and some of the actual config was not required. So a few screenshots of the setup are below as I found differences to Wenjun’s original blog for the HANA XS destination configuration. I do find the layout of the XS admin screens frustratingly confusing at some points, and a few clicks on icons and buttons allowed the setup to work. I found if I created a xshttpdest file for the base connections as per Wenjun’s information I could modify them as required in the HCP trial XS admin screen.

https://{MDC_NAME}}{ID}trial.hanatrial.ondemand.com/sap/hana/xs/admin/

There was no need to configure the truststore and I left this section blank but kept SSL enabled.

Also the 3 lined icon offers more options in the XS Admin screen such as the Trust Manager but this was not required from my experience as the Twitter connection was trusted. I can’t find the list of trusted root certificates that the trial MDC supports by default but it must cover the twitter related root certificates :). I was aware of the previous HANA XS based trial account trusted root certificate list but I am sure it has been recently updated, maybe you can tell me where the official list of certificates are listed.

My final twitter.xshttpdest file is as follows

host = "api.twitter.com";

port = 443;

proxyType = http;

proxyHost = "proxy-trial";

proxyPort = 8080;

authType = none;

useSSL = true;

timeout = 30000;

sslHostCheck = true;

sslAuth = anonymous;

The code I used (with my required bearer ***token*** generated - as per Wenjun’s blog) is below and called twitter.xsjs

var destination_package = "hihanaxs.scnblogs";

var destination_name = "twitter";

var call = "/1.1/statuses/user_timeline.json?screen_name=scnblogs";

try {

var dest = $.net.http.readDestination(destination_package, destination_name);

var client = new $.net.http.Client();

var req = new $.web.WebRequest($.net.http.GET, call);

req.headers.set('Authorization', 'Bearer {*****TOKEN*******}');

client.request(req, dest);

var response = client.getResponse();

$.response.contentType = "application/json";

$.response.setBody(response.body.asString());

$.response.status = $.net.http.OK;

} catch (e) {

$.response.contentType = "text/plain";

$.response.setBody(e.message);

}

BITLY API

After following the external web site process to get a Twitter token I thought that HANA, by now, would support BASE64 encoding/decoding as standard. This way I could generate a BITLY token using XS/SQL calls to get an authorised BITLY API token. I wanted to base64 encode username and password following the documented at HTTP Basic Authentication Flow from BITLY. Searching SCN and I found a couple of options in either using an SQL approach https://scn.sap.com/thread/3764075 or trying a base64 server side library approach

http://scn.sap.com/community/developer-center/hana/blog/2013/12/23/db001-using-libraries-in-xs.

I went with the XS server side destination approach to get the BITLY token. My thanks to David Brookler for his xsjslib blog listed above and Aron MacDonald posting a BASE64 javascript solution on his github page https://github.com/AronMacDonald/HanaHbase/blob/master/base64.xsjslib

To BASE64

I had uploaded the wikipedia Javascript base64 code that Aron had used to a file called base64.xsjslib and created the following xsjs code to get the token. Once I had the token I didn’t need the XS code anymore but I had to try this approach :). It may have been quicker to get the token via a curl command, listed in the BITLY help page as I had done previously, but less fun :). It’s not cool to have a password in the code but it is a one off requirement and on the server side with XS so the approach may be useful to me in the future. Also due to the fact that there is reference to an inbuilt namespace encodeBase64(data) in the API reference pages I could have saved time again :wink: but at least I have a way to use external libraries in the future.

JSDoc: Index

JSDoc: Namespace: codec

$.import("hihanaxs.scnblogs", "base64");
var base64 = $.hihanaxs.scnblogs.base64;

//setup Destination
var destination_package = "hihanaxs.scnblogs";
var destination_name = "bitly";
var call = "/oauth/access_token";

//Setup auth for bitly
var u = "{CHANGE TO BITLY USER}";
var p = "{CHANGE TO BITLY PASSWORD}";
var t = u + ':' + p;
var up = base64.encode(t);

try {
       var dest = $.net.http.readDestination(destination_package, destination_name);
       var client = new $.net.http.Client();
       var req = new $.web.WebRequest($.net.http.POST, call);
       req.headers.set('Authorization', 'Basic ' + up);
         client.request(req, dest);
       var response = client.getResponse();

    $.response.contentType = "application/json";
       $.response.setBody(response.body.asString());
       $.response.status = $.net.http.OK;

} catch (e) {
       $.response.contentType = "text/plain";
       $.response.setBody(e.message);
}

Bitly.xshttpdest file

host = "api-ssl.bitly.com";

port = 443;

proxyType = http;

proxyHost = "proxy-trial";

proxyPort = 8080;

authType = none;

useSSL = true;

timeout = 30000;

sslHostCheck = false;

sslAuth = anonymous;

After creating the code in a file called bitlypost.xsjs and a destination file above I called bitlypost.xsjs service to generate my bitly token for the API. As shown below with a masked token.

Once I had the token I could use it in the actual code to query the country clicks is as follows

//setup Destination

var destination_package = "hihanaxs.scnblogs";

var destination_name = "bitly";

var tok = "**********BITLY_TOKEN*******";

var form = "json";

//var lnk = "http://bit.ly/1ifPTz8";

var lnk = $.request.parameters.get("l");

var call = "https://api-ssl.bitly.com/v3/link/countries?access_token=" + tok + "&format=" + form + "&link=" + lnk;

try {

var dest = $.net.http.readDestination(destination_package, destination_name);

var client = new $.net.http.Client();

var req = new $.web.WebRequest($.net.http.GET, call);

client.request(req, dest);

var response = client.getResponse();

$.response.contentType = "application/json";

$.response.setBody(response.body.asString());

$.response.status = $.net.http.OK;

} catch (e) {

$.response.contentType = "text/plain";

$.response.setBody(e.message);

}

To use the XSJS service calling the above code with a parameter l=http://bit.ly/1ifPTz8 would generate the results shown below (in my chrome browser) . And any public bitly link could be used as it does not have to be related to @SCNblogs

The above image shows all the country clicks on the bitly link. The bit.ly/1ifPTz8 link used in the image above relates to a map of old ABAP developers :). The country codes (e.g. AU in the above image) are linked to ISO country codes which can be used to map the clicks on a map. And there is a great free map of the world, which includes ISO codes, from Natural Earth.

A Natural Earth

At the Natural Earth link shared above you can download a world ESRI shapefile that can be uploaded to the HCP via Eclipse. The Natural Earth shapefile can be uploaded directly and used to generate GeoJSON files to map the BITLY clicks with the Javascript based map library Leaflet.js. I have covered choropleth maps on SCN previously and shared the approach below. By adapting the choropleth code in the shared google document, (linked in the blog) I used the ISO code in the Natural Earth shapefiles to match the bitly API results.

I have used the Natural Earth dataset a lot over the years and will get to talk about it during my SCN community teched session at Las Vegas in September - http://scn.sap.com/community/events/teched/blog/2016/07/27/my-session-at-las-vegas-teched-2016-getti...

Link to a my blog which explains how to upload and generate maps from uploaded shapefiles.

https://scn.sap.com/community/developer-center/cloud-platform/blog/2016/04/02/my-intent-to-create-ma...

A SHINE related diversion to get to HERE now follows

SHINE on HCP

With the new MDC based HCP trial accounts I was curious to see if the HANA SHINE (SAP HANA Interactive Education) package available on github would work on the HCP. I found version 11.1 did load on my trial HCP account from the following link.

https://github.com/SAP/hana-shine

The main reason to try SHINE from my point of view was to check out the spatial capabilities it covers. I was disappointed to find that the HERE based maps required a API key and had a limitation of 90 days. I find the 90 day limit a reason not to use the HERE service, however for a one off attempt I thought I would at least see if it would work on the HCP trial. The 90 day limit means I will not take the time to learn about HERE maps further. I found I could enable the base layer HERE maps as follows.

How Did I Get HERE

The first thing I noticed in the SHINE 11.1 version was that no map was showing in my Chrome browser. Chrome refuses to mix HTTP and HTTPS content as shown below in the console error messages.

It turns out that the HERE maps have a Javascript library and with a 90 day limit there does not seem much point for me to learn anything more than to get it at least up and running on the HCP.

I found that if I forced HTTPS in the main.controller.js code section below then the map would display.

I decided to leave the HERE javascript libraries alone and see if Leaflet.js could integrate with the base HERE maps. As leaflet is free from any limits but the base HERE map will expire after 90 days.

I found the following worked with Leaflet to use the HERE base maps on my @SCNblogs page.

var layer1 = L.tileLayer('https://{s}.{base}.maps.cit.api.here.com/maptile/2.1/{type}/{mapID}/normal.day/{z}/{x}/{y}/{size}/{format}?app_id={app_id}&app_code={app_code}&lg={language}', {

subdomains: '1234',

mapID: 'newest',

app_id: '{APP_ID}',

app_code: '{APP_CODE',

base: 'base',

maxZoom: 20,

type: 'maptile',

language: 'eng',

format: 'png8',

size: '256'

});

//A link to the full code here -

Bringing It All Together With UI5

I now had my API calls and maps ready I now needed to update my original UI5 page that controls the process. I retrieve the tweets of the @SCNblogs timeline into a table and allow a user to select a row to map the related Bitly links. The code can be found here but adapted to the HCP XS services. And once again as in my original @scnblogs post mentioned earlier, thanks to a Peter Muessig SCN post which allowed me to pick up the BITLY link from the nested Twitter JSON. A link to the original thank you here Data Geek Challenge: Analysing the @SCNblogs twitter timeline

Thanks for reading and I'll have to wait to see if my blog makes it onto the @SCNblogs timeline.

.