At Google, we tested everything. User interfaces, advertising targeting models, even hiring practices. One product team tested 41 different shades of blue to ensure maximum click through rate; but the company is now testing black links. That’s one enormous advantage of A/B testing - all the sacred cows must prove their beatitude to maintain their divinity.

As Looker founder Lloyd Tabb explained to me, it doesn’t work for early stage SaaS companies. There’s a key ingredient to AB testing that most SaaS companies don’t have, something that Google profited tremendously from. Huge amounts of user traffic. The image above is a screenshot from A/B Testing marketshare leader Optimizely.

To detect a 20% change in a target metric with 90% confidence if it happens to 3% of visitors requires 12,000 visitors per test. In other words, the AB test require 24,000 visitors to test two variations.

Most early stage SaaS companies simply don’t have enough volume to achieve statistical significance in few days or even weeks. For many early-stage software companies achieving this kind of scale requires months or even quarters, an eternity in increasingly competitive markets.

The answer is not to reduce the confidence level or simply to run the test on a smaller scale and pick the one that seems to be performing better. Deciding that way is equally as arbitrary as deciding with no data at all.

What recourse do early-stage SaaS companies have to determine the best pricing strategy, product design, or landing page? It’s not numerical data. Rather, it’s qualitative data: interviewing users about the whys underpinning their points of view on price, reviewing the video of people exploring the product, and opinions about design. It’s the qualitative data, the acumen of an brilliant designer, the insight of a skilled product manager, the empathy of a master marketer.

At small-scale, intuition and research trumps statistical analysis. It’s not that statistical analysis doesn’t have its place. At scale, running large multivariate tests enables businesses like Google and others to consistently seek out a marginal 1 to 5% gain in product usage, performance, and revenue.

But the time frame required for statistical analysis at small-scale is not viable for early stage and even mid-stage SaaS startups.