More and more companies realize their proprietary data contains insights that drive tremendous competitive advantage. Enabling an organization to make data driven decisions is a long term process. Below is the current big data adoption process and where we are within it:
- Companies generate proprietary data whose volumes can’t be handled by existing tools. The main adopters of these technologies are financial services, healthcare, genomics and web companies.
- Companies build or buying the tools and expertise to store and process that data. Major vendors include Cloudera, MapR, HortonWorks, Splunk, GoodData, Vertica, Greenplum and many others.
- These new tools demand new skill sets within the organization: data storage expertise, data processing acumen, analytical ability, modeling skills, visualization expertise. As the infrastructure to support data analysis matures, data scientists become the bottleneck within the organization as they are beseiged by data questions.
- Tools emerge to enable analysis at edges of the organization so sales, marketing, product and everyone else can perform the analysis without having to submit a request to the data science team.
While the ecosystem is quite young and there will be innovation at every step in the process above, the next wave of innovation in the data science market will occur in the last stage of the process above: the democratization of data analysis.