Lately I could spend nearly every week attending conferences on big data. Two weeks ago I did just that. My week began at The Data Warehouse Institute’s annual World Conference and it ended at The Chicago Chapter of the American Statistical Association’s annual Statistical Advocate of the Year Award luncheon. As is the trend these days, both events featured big data themes.
While I attended many different sessions at the TDWI conference, the most compelling speaker of the week was Ken Rudin, Director of Analytics at Facebook. He opened the conference with an excellent and pragmatic discussion of big data and analytics. His talk even made headlines as he discussed how Facebook is now integrating traditional relational data warehouse technologies alongside their famous big data infrastructure. As the birthplace of tools like Cassandra and Hive, many were surprised by Mr. Rudin’s very pragmatic advice of “the right technology for the right purpose.”
While this didn’t seem particularly newsworthy, by the evening his talk had made headlines at Business Insider and other up-to-the-minute technology blogs.
What I found most useful from his talk was his focus on making an impact with data and analytics. Unlike most of the coverage of big data today, he did not focus on the amount of data they have at Facebook (it’s a lot!), but rather how his team focuses on business improvements above all else. His simple quote of “find a needle and move it” was a perfect summary of making analytics meaningful. He also reminded us that data without business insight is useless. You must work in partnership with your business users if analytics is to have a lasting and meaningful impact in your company.
After three days at TDWI, I attended an outstanding luncheon hosted by the Chicago Chapter of the American Statistical Association. (Full disclosure – I am a Board Member and Vice President of the Chapter.) Every so often, the Chapter presents the Statistical Advocate of the Year Award to recognize someone whose career is distinguished by their leadership in championing respect for data and the effective use of statistical reasoning and data analysis in business, public policy, healthcare, education and other sectors.
This year’s honoree was Dr. Roger W. Hoerl, Brate-Peschel Assistant Professor of Statistics at Union College in Schenectady, NY. Previously, he led the Applied Statistics Lab at GE Global Research where he led a team of statisticians, applied mathematicians, and computational financial analysts who worked on some of GE’s most challenging research problems.
In his presentation, Roger discussed the role that statisticians must play in this new world of big data. While of the coverage of big data implies that with enough data and software, meaningful insights and predictions are guaranteed, Roger, along with colleague Roger Snee, presented a more realistic approach to analytics success. Their goal is to create a new practice of “statistical engineering” that brings the best practices of statistical analysis such as sampling, test and control, and sequential learning and model validation to the data mining world of big data. He gave a very compelling argument for the statistics profession taking a leadership role in big data analytics.
It seems to me that both of these talks are part of a larger trend of bringing some practical considerations to the hype of big data. Just a quick look at my own blog history proves the point. With titles like “Little Data Makes Big Data More Powerful”, “Beware the Hidden Bias in Big Data”, “Is Big Data All It’s Cracked Up to Be”, and “The Value of Testing in Big Data”, just to name a few, it’s clear that practical advice is finally catching up with the hype. There has even been a backlash to the big data hype as Sir John Hagerty’s recent talk at Advertising Week Europe illustrated with provocative comments like “Supermarkets have an incredible amount of data coming in to them and they didn’t realize they were flogging horsemeat to people.”
This is all a good sign, even Mr. Hagerty’s comments. At West Monroe, our focus is always on the practical application of new technologies. While it’s true that our clients have more data than ever, it’s important that we focus on applications and analysis of these new data to, as Ken Rudin, said “move the needle.” We are not “data fundamentalists” who believe that having more data will solve your business problems. Instead, our goal is to combine deep industry knowledge and understanding of our clients’ business to design data and analytic solutions that have a real business impact.
While it’s exciting to finally see analytics become “all the rage,” it’s even more exciting to see data and analytics converge to truly transform business. It appears that we’re entering the next phase in the Big Data hype cycle as we seek practical ways to put our new data riches to work.