App to Data or Data to App: Rise of the Data Platform

[Former Member]

I went to a Hadoop Users Group meeting in the Valley recently and got into a long and interesting discussion with a Hadoop contributor about what it means to be an enterprise app developer (at least from a big picture perspective) with the rapid growth of Big Data, and widespread adoption of Cloud Computing and other technology trends. One of the things we spent quite a bit of time on is discussing the implications of "moving app to the data" vs "moving data to the app" as defined by the current enterprise software development paradigms.

Our discussion got me thinking about what a data application platform is and the current state. Since SAP announced the availability of its Flagship Business Suite on HANA couple of weeks ago, I've been doing some research and hope to share my learnings and thoughts via a series of blogs on the data platform topic in the coming days/weeks.

For starters:

• Rapid development and adoption of Hadoop-based solution ecosystem is enabling this "app-to-data" transition. See attached image below from The Register. Hadoop has come long way in 6 years from an Apache project to an enterprise-scale Big Data solution.
• This new development paradigm will finally break the data processing bottlenecks in a typical enterprise by freeing app developers to develop many "micro" apps that process/present data in various ways with direct access to data. Think iphone apps vs traditional, menu/navigation driven enterprise apps!
• Convergence of Cloud, Big Data, Mobility trends with rise of "hacking" development culture will lead to many new types of apps, frameworks, and a lot of rip-and-replacing of apps (esp. in the early stages when software doesn't work as expected and since licensing costs for these new pieces of software is low-to-zero). H/T to Krishnan Subramaniam for his excellent insights on this convergence of these trends. Image below are screengrabs from his slideshare presentation.
• Implementing and Developing Hadoop-based applications requires an understanding of Java, using Javascript, knowledge of MapReduce, etc. putting it beyond the capabilities of a typical data scientist/programmer. However, there are companies such as Continuuity and other major vendors that are developing integrated dev tools and packages that reduce the complexity. These are still in early stages but the general trend is very encouraging.
• Thanks to the sharing culture, there are many great examples of building enterprise-scale apps and data marts using Hadoop.

More in the next blog..