What were the top Big Data news stories of 2013? The usual pundits have been busy over the past week or so weighing in on this question, providing lists that include software releases, corporate acquisitions, new strategic directions, new offerings in the cloud, Hadoop without MapReduce -- all the things we would expect. But are these really the top stories? Perhaps by looking through the industry lens, we’re missing the real news.
Look at the top mainstream news stories of 2013 and you will see that Big Data is becoming a part of our social fabric. Worldwide, the top news stories included the devastating typhoon in the Philippines, the civil war in Syria, the birth of a prince, the election of a Pope, and the passing of a man who led his nation into a new era of justice and freedom. Top stories in the U.S. would include the roll-out of the Affordable Care Act, the Boston Marathon bombings, and the NSA scandal. Although none of these are Big Data stories, or even necessarily technology stories, each has a very real connection to the world of Big Data.
Increasingly, intensive analysis of social media and other communications content is driving our understanding of the major news stories of the day. Thanks to social and mass media channels, these stories become massively shared experiences. Collecting and digging into such content is a large-scale data mobilization effort. Such analysis provided context in understanding the impact of events as diverse as the birth of Prince George and the death of former South African president Nelson Mandela.
In the intelligence community, this kind of analytical effort is referred to as “chatter analysis,” and it is a critical component of modern intelligence work. As the tragic events in Syria unfolded, chatter analysis proved indispensable as it became clear that the conflict was driven in large part by an underlying information war. While each side in the conflict used traditional channels to promote its interpretation of what was taking place hundreds of thousands of Syrians used their mobile devices to report their individual experiences and to help shape a more complete understanding of events as they occurred. That sharing of information came at a high price to some, whose digital footprints made them trackable by the opposing side. Ongoing analysis of social media and other communications content continue to drive our understanding as events unfold both in Syria and the surrounding region.
In the US, intelligence-gathering of another sort took center stage as details from the 1.7 million National Security Agency (NSA) files leaked by former contractor Edward Snowden began to be made public. The NSA scandal at least bears a strong resemblance to a Big Data story, raising as it does major questions about privacy and data ownership. The NSA scandal is driving critically important debate within the public and the courts that will, in all likelihood, lead to a whole new approach to regulating and enforcing data privacy. As the idea that everyone leaves a digital footprint gains broader understanding, questions about who (if anyone) should have access to the various pieces of data that make up that footprint are becoming more urgent. Everyone in the business of collecting and analyzing data stands to be impacted by the answers that emerge to those questions.
Meanwhile, there can be little doubt that Big Data plays a rapidly growing role for both the intelligence and the law enforcement communities. In the investigation and manhunt that followed the Boston Marathon bombing, law enforcement officials were in many ways as reliant on computer technology as they were traditional methods. However, a subsequent review of the investigation showed that enhanced Big Data capabilities might have produced faster results. So demand for increased government access to and use of digital footprints is occurring in one context, while demand that such access be severely curtailed is occurring in another. Quick and easy resolutions to these conflicting sets of priorities seem unlikely.
But not all use of Big Data raise such seemingly intractable conflicts. In the case of Typhoon Haiyan, Big Data was pivotal in enabling relief efforts throughout the Philippines. An interactive map which synthesized geo-spatial, demographic, and social media data guided relief workers to the areas with the most urgent need for help, providing the most expeditious routes to these troubled zones. And a GPS-enabled asset-tracking system helped to ensure that resources were deployed to where they were needed the most. In a completely different vein, and demonstrating the diversity of applications for Big Data capabilities, odds-makers used advanced analytics in an attempt to predict the outcome of the Papal election in March.
Finally, in the U.S., the implementation of the Patient Protection and Affordable Care Act (ACA), commonly referred to as "Obamacare," proved to be less of a Big Data story than expected.
Obviously, launching a new national healthcare system for a country with population of more than 300 million has Big Data implications. The system had to accommodate the tens of millions who currently don’t have coverage, but would also impact the hundreds of millions who do. After all, many of those individuals would be expected to end up on the healthcare exchanges themselves one day and, in any case, it would be necessary to ensure that their existing coverage was compliant with all the new regulations. Such a system would require a whole new infrastructure for managing healthcare data. Each participant’s full history of medical conditions, treatments, and providers would have to be consolidated into one easily portable data profile, a profile made up of information that would now be accessible to more sets of eyes than ever before. The Big Data implications were enormous.
The expectation was that healthcare was about to become "the new financial services." In the financial services space, thousands of entities work together to provide an infrastructure that enables individualized credit ratings and simplified local, national, and global funds transfer. Now, in the healthcare space, thousands of entities would work together to provide an infrastructure to enable widespread analysis of treatments and outcomes and easy transfer of complete medical history from provider to provider.
But when Healthcare.gov was launched, a very different story emerged. That story centered on the basic operation and security of the site, and the tremendous difficulty encountered when attempting to get applicants through the registration process. Technical observers clustered around a consensus that the basic infrastructure of the system that was 10 years (or more) out of date -- a Web 1.0 solution for a Web 2.0 world. Clearly those issues would have to be addressed before Big Data could even enter the picture.
Not that long ago, the statement that all commerce is e-commerce would have been have been laughable. Today it is simply a straightforward description of how things have evolved. The notion that all companies are software companies would have seemed even more absurd. But now that preposterous idea is offered up as a commonplace. As technology becomes more deeply embedded in the fabric of society, we are approaching the day when it might well be said that all news is technology news, or even that all news is Big Data news.
In 2013, that was still not quite the case, although Big Data is increasingly becoming ingrained into the fabric of our societies, as evidenced by its role in many of the major stories of the year and in how we learned about and came to understand virtually all of them. We can expect these trends to continue in 2014 and beyond.
This blog has also been posted on saphana.com
