Technology Blogs by SAP
Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
cancel
Showing results for 
Search instead for 
Did you mean: 
JayThvV
Product and Topic Expert
Product and Topic Expert

I love maps. As a kid, one of the easiest ways to keep me quiet for a long time was to give me one of those big school atlases with different maps, from political and thematic maps to terrain, to weather or wind patterns and ocean current flows. It was therefore inevitable probably that at some point I would apply my D3 skills to this area.

There are other excellent articles on SCN with regards to maps and Lumira extensions, and you should read those. They definitely helped me in various aspects. We all build on top of the work of others:

These are great, and they touch on a number of important items, but I think there is space for yet another article on this topic. Specifically, what I wanted to do is do as much as I could myself - including the generation of the map from shape files - and how to create links between map elements, rather than just do something with individual map elements. I wanted to rely as little on external APIs (or more precisely, charting packages) as I could, so I could learn the most, but also had as much control over everything as I possibly could. Along the way I had to tackle some interesting problems.

Getting Started and generating the topoJSON map file

Like with everything D3.js, you go to the master, and everybody starts with Mike Bostock's Let's Make a Map. That has some good instructions on how this all works in general, as well as the installation of the required tool chain and where to get shape files. I am using the ne_10m_admin_0_countries from Natural Earth.

Basically, the workflow is like this. You use the ogr2ogr tool to manipulate the shape file and create a GeoJSON file. This then is run through the topojson tool to create a significantly smaller and more efficient topoJSON map file. The ogr2ogr tool allows you to put selection criteria, so you can select specific countries or exclude them with the -where option, which allows a SQL like syntax. This means you can specify whether to include or exclude specific countries as you wish. However, this doesn't always work too well. Some countries handle things a little differently than others for political and sovereignty reasons. France, for instance, considers Guadeloupe and Reunion as part of France, which means that in trying to create a map of Europe you get weird artifacts where small islands suddenly are included with France that are nowhere near Europe. It also messes with finding the center of the country (see later), as it affects the "bounding box" the country is in.

However, there is a good solution for this and that is to use the -clipdst option included with the ogr2ogr tool. It is well described in this article. What this allows you to do is create a "viewbox" of things that you're interested in. As described in that article, you can use Google Maps to find coordinates of bottom left and top right, and use those in the ogr2ogr command for the -clipdst option. That gave me the box what you see in the screenshot. That is, rather than have to rely on whatever charting package decided to give me, I have full control over what is included in the map and what is not. If you wanted to make a map of Scandinavia, for instance, you would change your selected values to create a box around those Nordic countries. Note that you could always run a second ogr2ogr command to filter the JSON file generated using -clipdst and further exclude countries, so you could get a free floating Norway-Sweden-Finland-Denmark map without bordering countries, if you so wished.

In any case, this clipped selection created my GeoJSON file. I then generated the topojson file using the following command:

topojson --id-property ADM0_A3 -p name=NAME -o <outputfile> <geojson input file>

The GeoJSON files have a lot of metadata included in it, which makes the file size bigger than it needs to be, and all of that would need to be downloaded to a client machine. This command (beyond reducing duplication from the country shapes themselves) reduces that to just an id and name property. The ADM0_A3 stands for an ISO standard of 3-letter country codes, which we will use both as identifier and as a label. We also specify to use "name" in lower case, from the uppercase NAME in the GeoJSON file.


D3.js and building the map with GeoJSON and TopoJSON


You are now ready to use the generated map in your D3.js development. I typically do the work in D3.js stand-alone first, but you can also use the approach of generating first an empty extension through VizPacker after naming your canvas and work from within the Lumira directory structure.


The first thing you'll probably want to decide is which projection to use. The world is round and our screens are flat, so we have to specify a way to deal with that. There are a good number of projections available. The ones that seem to work best for me are equirectangular or mercator. For this, since equirectangular distorts nearer to the poles, I used mercator. The next problem is to properly center and size the map according to your viewing screen, for which I used the answers from this Stack Overflow post.


Drawing the map and added the boundaries is easily done once all that is set up, and followed what you see in that Let's Make a Map article.


Drawing the arrows


The previous is all in the main render function. I wanted to make the map clickable, so that the import and export flows are shown by selected country. This is done in an updateMap() function, attached to an onclick event for each country shape. To achieve the visualization of these flows, we can get the centroid of each country element, as well as it's "bounding box". The image below should help clarify.

The labels like "DEU" and "ITA" are placed in the centroids of their shapes. Each country also has a rectangular "bounding box" in which the whole shape is contained. Drawing a line or arrow from each centroid is easy, but then we overlap the labels, and the more arrows we have, the less easy it would be to understand. So, we need something like this:

Basically, we want to start the arrow line a little later, and finish a little earlier. I ended up with doing this:

  1. Create a "guideline" from centroid to centroid for the link that we want to show - with width 0 and opacity 0. We're never going to show it, just use it to guide our line.
  2. The D3 library allows you to "follow" a path and get points along that path using path.getPointAtLength(). This takes a fraction of the total length of the path as a parameter, so  0 x path length is the beginning of the path, and 1 x path length is the end point. By "walking" along the guideline we can find the moment when we're passing out of the source country boundary box, and the moment when we are entering the target's one.
  3. Once we've figured out those points, we use those as the new starting and end points for the arrows we want to draw.

That solved one problem. It immediately raised another.If there are imports and exports both between the same two countries, these arrows are going to be on top of each other. So, I needed to somehow curve them, or offset them. Curves seemed more aesthetically pleasing. In my first attempt, I found out the center point of the guideline and then offset that. That worked, but was not ideal. I subsequently found this Stack Overflow answer, which gave a very elegant solution that could operate on the arrow line instead, and figured out the midpoint through some proper trigonometry. I am in fact using that function in my extension.

Finally, then, making the lines into actual arrows by appending an arrow head. You do this by specifying a marker element in the "defs" section of the SVG. I am simply appending these. There are three such markers: one white to serve as the background, one orange and one gold/yellow for the imports and exports.

You attach these markers to the line/path by appending a marker-end attribute and provide the URL to the marker:

.attr("marker-end", "url(#arrow_back)")

The really nice thing is that by default the arrowhead resizes with the stroke-width specified for the line itself. So, you can use a d3.scale.linear() scale to indicate volume and the arrow head grows with it. I attached this to the power metric so bigger flows result in bigger arrows. I then first draw the arrow in white with a 2 px extra stroke-width, followed by the main arrow for imports and exports (orange or gold), so that each arrow has a white border around it.

Porting to Lumira

Normally, porting your D3.js work to Lumira is not too complex, and mostly involves taking care of width and height, and mapping the data correctly. However, in this case I needed to include a couple of libraries that are not by default in Lumira, as well as the topoJSON map file. To accommodate this, in Lumira we use require.js. This retrieves the files that the client needs to download and makes sure that they are available when the code needs to run. You can use external URLs for that, but it is probably a better idea to bundle those with your extension so that there are no requirements for an internet connection and dependencies are fulfilled that you know work with your extension.

For this, you need to create an extension ZIP file through VizPacker, and unzip it within a folder in the main SAP Lumira/Desktop folder. Then, find your render.js file, and make sure you create a block like this:  (Note that mideurope.json is my topoJSON map file)

//Require JS block

try{

  require.config({

  'paths': {

  'd3': '../sap/bi/bundles/comsapsvcseuropowermap/libs/d3.v3',

  //'d3': '../../bundles/comsapsvcseuropowermap/libs/d3.v3',

  //'topojson': '../../bundles/comsapsvcseuropowermap/libs/topojson.v1.min',

  //'eur': '../../bundles/comsapsvcseuropowermap/libs/mideurope.json'

  'topojson': '../sap/bi/bundles/comsapsvcseuropowermap/libs/topojson.v1.min',

  'eur': '../sap/bi/bundles/comsapsvcseuropowermap/libs/mideurope.json'

  },

  shim: {

  d3: {

       exports: 'd3'

  },

  topojson: {

       deps: ['d3'],

       exports: 'topojson'

  },

  eur: {

       deps: ['topojson'],

       exports: 'eur'

  }

  }

  });

  // Define require.js module with all needed js libs

  define("runtime", function(require) {

  //var $ = require('jquery');

  var d3 = require('d3');

  var topojson = require('topojson');

  var eur = require('eur');

  // return the required objects - can be used when module is used inside a require function

  return {

       topojson: topojson,

       eur: eur,

       d3:d3

  }

  });

  // Exception handling for require.js In case of an error it alerts the message.

  require.onError = function (err) {

  if (err.requireType === 'timeout') {

       alert("error: "+err);

  } else {

       throw err;

  }  

  };

  require(["runtime"], function(runt){

  /* MAIN CODE BODY - INSIDE OF REQUIRE JS TO ENSURE TOPOJSON AND OUR MAP ARE THERE  */

  var topojson = runt.topojson;

  var d3 = runt.d3;

... <your main code for the map, apart from any update functions and helper functions>

  } catch (Exception) {

       console.log("Error: " + Exception)

  }


Apologies for that big chunk of code, but this is very important. I've seen other instructions on this, but it still took me a while to figure out, so I hope this helps others. Basically, what's happening here is that first we specify the links to the file locations. I created a "libs" folder to place these APIs in in the same place as the <extensionname>-bundle.js file and the <extensionname>-src folder. You see two sets of URLs there: the ones with ../sap/bi/ you need in the extension once ported to Lumira, the others for while you're testing still with the HTML file in the "examples" folder.


Next is to give the APIs and resources names, and specify any dependencies in the "shim" section, then tie those to variables you can call in the rest of your code. An error check is done, and if all is good, the require runtime is set up, followed by the main body of your D3.js code. It finally ends with the catch block to match the original try at the beginning.


My updateMap() function is then placed entirely outside the body that contains the render function, that is, outside of:


define("comsapsvcseuropowermap-src/js/render", ["comsapsvcseuropowermap-src/js/utils/util"], function(util){   .... }


This ensures that it is callable from the onclick event. This does mean that I had to declare a number of variables on global level, so that they are available to the updateMap() function.


Further extensibility, code reuse


This extension specifically aimed to display a particular section of Europe relevant to the power company I developed this for. But the code can be easily re-used and extended for other purposes. For instance, I built out this interactive map of migration flows (2005-2010) where you can see country by country where are people immigrating from, and where they are emigrating to, where virtually the only changes were the topoJSON map file generated and used, the colors of arrows, and an added linear color scale to show whether a country is a net immigration country (orange = high immigration) or a net emigration country (green = high emigration). Below you see a screenshot for Egypt. This map is using an equirectangular projection.

I hope this gave a good description of how this extension was built, and even more importantly, how you could create your own, without having to rely on a charting package that doesn't do exactly what you want. Using topoJSON and your own generated map files, you have complete control over what is included or not, and you decide what projection to use, what geographical elements and how to style it exactly to your needs. Rather than attaching the extension itself, therefore, I am attaching my render.js file. It's certainly not the best code possible as my JavaScript tends to be a little messy, but hopefully it is of some assistance to others.


Happy map-making for data visualization!


Code for this can now be found on GitHub.

4 Comments