In a triumph of statistics, Nate Silver predicted the outcome of every state in the Presidential election correctly. What makes this story so noteworthy isn’t that it proves data enables superior decision-making to human intuition. We know the math works.
Instead, Silver’s success highlights and challenges the prevailing culture, present in politics and in the workplace, that overvalues intuition and undervalues data.
Our analysis tools and our access to big data are forcing cultural change at the broadest scale. The latest field to face this upheaval is journalism. It won’t be the last.
Data is the most cogent argument. Because we compare Silver’s predictions to the outcome, we can understand his thought process and his logic. It’s black and white. Right or wrong. And if it’s wrong, we can fix the models.
Political parties, journalists, and startups that build cultures that use data to make dispassionate arguments and objectively measure performance will be the ones that will seemingly pull a rabbit out of a hat - just like Silver.
And these performances will become increasingly consistent. Who won’t pay attention to Silver’s data in four years' time?
We’re going to continue to hear about these rabbits with increasing frequency - whether it’s Nate Silver, Moneyball, the story of Oakland As, the high frequency traders on Wall Street, the search algorithms of Google, or the data science teams of LinkedIn.
These events will be more common because these companies aren’t using sleight of hand to pull these rabbits. They use linear regression.