3 minute read / Feb 3, 2020 /
Simpson's Paradox in Measuring Net Dollar Retention Rate
Net Dollar Retention is one of the most important metrics is a SaaS business. It measures the value of a cohort of customers over time including expansion, cross-sell, and churn (loss of revenue). But how do you measure NDR?
Imagine this is your company’s data. The first column is the cohort month for each cohort in a year. The second column is the revenue of this cohort in their first month. The second colum is a random number between -20 and +20 that is the NDR for that cohort. The last column is the ending revenue of that cohort.
| Month | Starting Revenue | Percent Change | Ending Revenue |
|---|---|---|---|
| 1 | 5812 | -20 | 4650 |
| 2 | 3342 | 0 | 3342 |
| 3 | 322 | 12 | 361 |
| 4 | 5438 | 0 | 5438 |
| 5 | 5549 | 14 | 6326 |
| 6 | 216 | 16 | 251 |
| 7 | 9217 | 16 | 10692 |
| 8 | 5383 | -6 | 5060 |
| 9 | 7660 | -8 | 7047 |
| 10 | 2159 | 5 | 2267 |
| 11 | 9147 | -11 | 8141 |
| 12 | 752 | -12 | 662 |
| Total | 54997 | 54235 |
You could calculate NDR as the average of the Percent Change column. You could calculate it as the percent growth in total ending revenue compared to the total starting revenue. You could aggregate the data by quarter and average that.
In the first case (average of each cohort), your NDR would be 0.5%. In the second case (annual total), your NDR would be -1.4%. In the third case (average of each quarter), your NDR would be -2.6%.
So which do you choose? If you’re fundraising, certainly the last one. And if you’re managing internal company goals, then certainly the first one. Just kidding.
Let’s take a step back. First, why is this happening? Why aren’t all the averages the same? Simpson’s paradox. The overall average of a data set may be different than the averages of different groups. We see this paradox all over the place: college admissions, sports performance data, and medicine.
Which is right? Well, there’s no industry standard today. These are the summaries from their public disclosures. Alex Clayton collected the list here. Zendesk reports the annual change in cohort monthly recurring revenue. Twilio calculates it as the annual change in cohort quarterly recurring revenue. Marketo uses the change in cohort revenue for each of the last month of each quarter; for example, June revenue/March revenue. Okta and Box measure 12 month bookings changes for each cohort.
So how do you pick the right one for your business? The first principle is to align it to your billing cycles. If your business has annual contracts, then measuring annual growth makes sense. If your business is utility based, monthly is a better measure or perhaps quarterly to smooth out revenue.
The second principle is to share how you calculate NDR and stick to it, so it’s consistent. And the third principle is to benchmark your business to others who measure themselves in the same way.