Big data has become a cross-industry term utilized by the capital markets industry, the pharmaceutical industry and the scientific community. Big data is used to describe extremely large datasets. Binary prefixes to describe the size of data have graduated from kilobytes and megabytes to data stores sized in terabytes, petabytes and exabytes (slightly more than 1 million terabytes) of storage.
Data management is an ongoing challenge for the capital markets industry. Due to a plethora of data sources, multiple asset classes spread across multiple geographic regions, and a premium on risk management, the amount of data continues to grow exponentially. Capital markets departments require access to massive amounts of data to uncover market opportunities and to satisfy compliance and regulatory reporting and risk management.
Furthermore, the regulatory climate post the 2008 financial crisis has forced firms to take a proactive view of risk. The Dodd-Frank Act will increase the amount of compliance and regulatory activity substantially. Specific details of DFA continue to be discussed, but the mandate is clear. It is time for financial institutions to lay the groundwork for compliance via sound data procedures and data management.
Wall Street banks are innovating and finding better ways to handle large datasets. Much effort is being spent managing petabytes of data for advanced analytics and regulatory compliance. IT vendors have responded accordingly. Hadoop, an open source product, is a favorite among capital market practitioners working with big data. Large database players have technology to assist with the big data issue Oracle, IBM, Sybase and Teradata. Smaller vendors and niche solutions are present and innovation in this area will continue over the foreseeable future.