(I say this is 'Part 1' because I'm not done with this but I promised myself I'd release what I had before end of 2014 and it's Dec 31st, so here we are...)
Update Feb 2015:
Part 2 is out now! Design Studio SDK (1.3/1.4) - Open Source Maps (Part 2) Choropleth Maps
Original Part 1 Post:
As I said at TechEd in October and in an ASUG Webinar in November on Design Studio SDK:
"Maps are HARD".
I still believe it to be true, for reasons already stated in those 2 sessions, but again here for SCN.
- Maps, unlike Charts, rely on a lot of data to tell their story. With a Chart, you define axes and plot points and you're done. With a Map, you need much more information for the data being visualized to make any sense. Information like geographic layers including coastal boundaries, administrative boundaries (countries, regions, subregions, cities, etc) along with even terrain information, etc. These types of Maps on the Web are usually served up as pre-rastered Slippy Map Tiles (OpenStreetMaps, MapQuest, Mapbox, ESRI) or scalable GeoJSON Vectors (TopoJSON D3). Each of these approaches have their merits which most graphic designers know (vector vs raster).
- Even after you've tackled drawing a Map, you need to supply latitude and longitude to a Map in order to even plot your date somewhere. You could 'Bring Your Own Latitude/Longitude' (BYOLL from hereafter) or supply Country/City/State/ZIP to a GIS provider to Geocode for you, or cache your own stockpile of locations and lat/lng entries, which is daunting and a maintenance issue.
- Licensing issue related to GIS Providers in general. Let's say you find a GIS Provider to handle the task for you, now call your lawyers or accountants to be allowed to use their GIS Services (Tiles or Geocoding) - Mapbox, ESRI and others each have different license structures and if you are a current ESRI customer hey that's great, knock yourself out but otherwise, get ready to pay someone for something.
- Relying on a GIS means poking holes through corporate firewalls since you will be relying on external GIS Tile/Geocoding assets.
- Oh yeah, and Maps are HARD, in case I've not said it enough!
Even though they are HARD for technical, practical, and licensing reasons, I wanted to see what I could make that poor man's generic map component that:
- Does not (but can) use external Map Tiles
- Does not use a Geocoding service because (while I had it working, it was annoying, and also remember licensing...)
- Does not HAVE to be fed lat/lng because many times you may not have that data in your dataset but you have probably 0CITY/0REGION/0COUNTRY fields available.
My Approach:
- I settled on using Leaflet as my main mapping API. This seems like the defacto JS Map library (aside from Google Maps and Bing). They are also Open-Source. There's also OpenLayers, but I didn't like it as much...
- Since I was using a Tile approach, I figure out that there's a TileJSON syntax that you can then easily use across any GIS Map Tile Provider (Example: kartena/leaflet-tilejson · GitHub)
- I also wanted a good Advanced Property Sheet experience, as the Default Property Sheet in Design Studio is pretty basic and I find it annoying to write JSON string by hand and I'd never expect a Design Studio designer to do this.
- And again, options to use all local resources (no reliance on Map Tile Providers or Geocoding services) - Also using only Public Domain data because I don't want to tie my stuff to a GIS license.
How Far Did I Get? Again, Maps are HARD. :smile:
- Map Tiles (External and Internal):
I used an application called TileMill (built by Mapbox on open source technology) to create my Map Tiles. This is a subject area all in it's own, but long story short, by using public domain data, I was able to generate license-free Map Tiles in two flavors (Standard and Terrain) - All data is public domain from NaturalEarth. Examples in Tile Mill below:
Standard:
Terrain:
In my opinion, these may not be perfect maps compared to Bing/Google/ESRI/Mapbox, but in my opinion, they are "Good Enough" and free.
So after generating these tiles (in the MPTiles format), I needed to extract just the PNG tiles because I was not interested in uploading these in the proprietary MapBox format. There was a nice tool called MB Liberator that handled the job nicely. I had my PNG tiles!
I'll pause here for a note about Map Tiles. They grow exponentially per Zoom Level (which goes from 1, the highest, to 22 or so, the lowest) - I believe they grow by a factor of 4 for each additional zoom level. This means stuff can get BIG, FAST. Tiles supporting Zoom Level 4 weigh around 5MB in all, while Zoom Level 7 grows to around 80MB!! Since I wanted to embed these tiles, I had to be conscientious of the size of the plugin so I settled for a default lightweight set of Zoom Level 4 tiles for both Standard and Terrain. I also have created separate optional feature installs using the heavier level 7 tiles that can be installed separately (more on this later)
So I was able to both achieve my goal of being GIS-independent, yet supporting GIS Tile Providers as well. Below is a screenshot in Design Studio Design time showing different map providers including my local tilesets in action, from simple, to exotic (Stamen Watercolor):
Not pictured, but also works just fine, are Mapbox and ESRI (I personally find ESRI tiles ugly though) - If you turn your attention to the Additional Properties panel, you can see I've added a few nice things like ability to set your map center with a visual map that you can click to set (or you can use the Default Location property in the default Properties pane) - Also you can control min and max zoom and default zoom levels. If you do not know the TileJSON URLs, that's ok I have added several defaults for you using the 'Pick...' button that will fill out most of the TileJSON stuff for you:
So, there's your tiles! - Geocoding/Geohierarchies
Wow, this one is hard, and I'm still struggling with how much of this is feasible or outgrows what is possible in Design Studio... As mentioned earlier, I was able to make this work with a MapBox Geocoder service but I was quickly annoyed by how you had to batch-pull only 25-50 locations per pull and it was an asynchronous spaghetti callback mess behind the scenes and this was only ONE implementation so what if you wanted to do similar with ESRI? Probably a whole different implementation so I backed this approach out and will revisit it another day or find someone equally as insane as me to help.
So I did something equally insane and found some public domain lat/lng data that went down to Postal/ZIP Code and using NodeJS voodoo, carved this flattened list of 500,000 entries into a Geohierarchy in separate JSON files that only get loaded if needed to help with performance. I was able to get this all into about 45MB size (again, loaded on-demand only, not all in your browser or DS client)
I'm leaving a lot of gory details out here, but basically what I ended up with was an ability to resolve location down to lat/lng (if in my 'JSON database').
In order for this to work, you first begin with building what I am calling a 'Geo Hierarchy' which includes at least Country, then Region, then City, then ZIP. It works best if you can supply Country and Region however it will 'guess' the best it can using reverse lookup JSON but I may pull this out as it's just insane, but for now it works:
I have built an example GeoHierarchy called 'Example':
And a quick glimpse of the Unsolved Locations:
So, we have a GeoHierachy and a poor man's Geocoding, what's next? - Map Layers
Whew. We can now draw a map and resolve lat/lng (mostly). Let's visualize some data already!!!!
Below I've added 2 Marker Layers to my Map, both of which use my 'Example' Geohierarchy to plot with. (Terrain Map in screenshot)
Per-layer, you see we can specify many other properties, including filtering! Example for 'AL Schools' layer:
This allows us to control properties such as Marker type (I used Marker in one layer and Circle in another, in the example above) - So plotting markers seems to work at a basic level.
There's still work to do here, as right now it just plots based on occurrence, so if you have multiple entries with the same location, it's just keeps dropping markers on top of the same spot (even if you cannot see it) so there's no 'aggregation' happening here, but it should be possible, given enough time.
However, we can at least gauge occurrence intensity/density using a Heat Layer, let's see:
As you can see, you can even control the Heat Layer's Blur, Radius, and color gradient and stops. I have also included color presets common in SAP tools and ColorBrewer colors:
And while incomplete, but will eventually get finished, a Cluster Layer:
What's Next/More Ideas
- I need a break.
- Many bug fixes
- Polygon Layers (if feasible)
- Chloropleth Layers (if feasible)
- Bubble Overlays by occurrence/measure (These are ugly to me, but seem to be popular)
- BIAL Methods
- Did I mentioned I need a break?
Can I play with what you have today?
Sure! I've updated the Preview Build of SCN Design Studio SDK Development Community distribution. Give it a try!
Source Code:
Install Links:
- Preview Release (Updating online within DS App): http://org-scn-design-studio-community.github.io/sdkinstall/releases/preview
- .ZIP Version: https://github.com/org-scn-design-studio-community/sdkinstall/blob/master/releases/preview/org.scn.c...
Optional Zoom Level 7 Feature Installs (90MB each):
etc
- Something isn't working.
Not surprising, a lot is buggy (hence preview release) - Feel free to leave a polite comment! - I have a crazy idea...
I'd love to hear it. - I think you are crazy for trying this.
Agreed.
Questions/Comments/Feedback welcomed.
Happy New Year!
- SAP Managed Tags:
10 Comments
