A Moore's Law for Data

Since the first transistor, increasing speed has been at the core of much innovation in Silicon Valley . Over more than three decades, Moore’s Law has remained the engine of progress in chip technology. I’ve been wondering if a analogous productivity law will be written for data.

One level up the stack from the chip, software has benefitted tremendously from chip speed. I learned to code in Java, which I needed to compile before I could test my program. Each compilation required 3 or 4 minutes and this latency slowed me down. In comparison, the languages dominating today’s programming stacks provide instant feedback. I saw 90% time savings to write code when I shifted to Ruby.

Two levels up the stack, applications also profit from Moore’s law. Apps can process more data, larger images, bigger movie files - and all of it much faster than before. But no where is the benefit more apparent to me than in data processing.

At Google, I watched the company evolve its data processing technology from MapReduce to Dremel. Having left the Googleplex, I still keep tabs on innovations like F1. Today, companies outside of Google are replicating this transition from Hadoop to Impala to Spark and other databases.

While that change may seem intangible, much like an additional hundred million transistors squeezed onto a smaller chip, the benefit is real, concrete and massive. Today, with Redshift and Looker, a salesperson can analyze a few billion rows of data in a second. Not a data scientist or statistician or engineer. But a salesperson.

The perfomance improvement throughout the entire stack, from chip to database to application, has reduced the feedback cycles to sub-second responses. The magnitude of that change can’t be overstated, because it makes all the petabytes of data companies generate and collect useful to everyone in an organization.

I hope there will be a power law written one day that shows as the amount of data processed increases geometrically, the quality of decision-making within companies increases and so does their worth.

Published 2014-03-06 in trends 


I am partner at Redpoint. I write daily, data-driven blog posts about key questions facing startups. I co-authored the book, Winning with Data. Join more than 20,000 others receiving these blog posts by email.

Read this next: