In response to analyses published on this blog, a handful of readers have raised the idea of Simpson’s Paradox to me. Because I wasn’t familiar with it, I’ve been researching it and last week, I came across VUDLab’s beautiful website which uses a sexism-in-admission lawsuit against Berkeley to demonstrate the idea.

In 1973, the University of California-Berkeley was sued for sex discrimination. The numbers looked pretty incriminating: the graduate schools had just accepted 44% of male applicants but only 35% of female applicants. When researchers looked at the evidence, though, they uncovered something surprising:

If the data are properly pooled…there is a small but statistically significant bias in favor of women.

In other words, the researchers found that the aggregate metrics didn’t tell the real story. By examining the admission patterns of each department in closer detail, the researchers revealed a different truth. While on the whole, women were accepted at lower rates, it was because they were vying for spots against other women in humanities programs that were on the whole more competitive than sciences and engineering, which accepted more students from a predominantly male applicant pool.

I’ve come to think that at the heart of every disruptive technology startup is Simpson’s paradox. Most people in a market holds a particular worldview which ostensibly describes the way a market works. But by investigating further, a startup may reveal that a fundamental underlying driver has changed unexpectedly. Identifying and leveraging that change to win market share is the key to disruption.

The Berkeley example reminds me of the SpaceX’s formation story Elon Musk shared at the D conference this year. Musk implicitly knew launching satellites into space would be expensive. After all, NASA’s annual budget is about $19B. But when Musk and his team analyzed each cost component of a space launch, they found that less than 10% of the costs were the rocket and the fuel and the launch equipment. This meant Musk could conceivably reduce the costs of space shipping by 80%.

While it’s not a true statistical example of Simpson’s Paradox, the point is the same. The market held a worldview based on aggregate data. But Musk recognized the aggregate space costs didn’t tell the true story. By digging deeper, he and his team found a lurking explanatory variable and an opportunity to disrupt the industry.

These little secrets, understanding the admission patterns of a university or the cost-structure to launch a rocket, require detective work. But digging past the facade of the truth, there’s the possibility of a different explanation and a world of possibility.