No matter how you feel about the implications of it, the government intelligence gathering program now known as Prism represents a tremendous revelation on many levels for both our friends and our adversaries. Lost behind the headlines, privacy, and geopolitical concerns about the US government’s massive intelligence gathering program is a profound validation of the concepts and technologies that underlie the Big Data movement.
Widespread acceptance and use of advanced technologies in the government intelligence arena is nothing new and many of these same agencies do a lot to foster advanced technology developments in the US (see Financial Times Silicon Valley rooted in backing from US military). In the 1970s a CIA program for gathering and processing intelligence grew beyond the confines of the agency and changed the world of technology as we know it. This program lead to the relational database and the product / company today known as Oracle.
No matter how you feel about Prism no one can argue that the relational database didn’t fundamentally change the IT landscape of the last thirty years. And yet it was some of these same agencies that spearheaded the creation of that technology that are at the heart of Prism. What we see today is that there are some very real world uses for Big Data applications – in the case of Prism both real and serious. This is also a validation of a technology platform that spans both the Cloud and Big Data. This program undoubtedly uses technologies that the rest of us would recognize as Cloud (massively distributed separation of hardware from software) and Big Data – namely Map-Reduce or Hadoop.
That’s not to say that Prism is using Apache Hadoop, but it’s probably using something fairly similar – a massively distributed file system that can hold unstructured data, tons of it, and process it in a fast and parallel way. Clearly there is value in both the information and the platform being used or the agencies involved would have changed course over time. Legal and privacy issues aside this is a major validation of the concepts and technologies of Big Data that have been dominating the press in recent times.
Perhaps the biggest change from previous intelligence / defense driven technology cycles is the speed at which these same technologies now find their ways into civilian hands. Today there are a variety of Hadoop enabled platforms available, some even in the Cloud, that give any organization access to a technology stack that the intelligence establishment clearly feels is working well.