IMO, NPS is horseshit. And let’s face it, horseshit really isn’t for everybody.

I’ve lost count of the number of times an early-stage founder asked about measuring their Net Promoter Score (NPS).

Most times, I dismiss the suggestion out-of-hand. Because in 99 cases out of 100, it doesn’t make sense for them.

As the name would suggest, Net Promoter Score is a metric which requires calculation, rather than mere data collection.

It was created by Bain & Company in the early 2000s (shout-out to all the ‘management consulting’ OGs out there) in the Harvard Business Review article “The One Number You need to Grow” by Fred Reichheld.

In the article, Reichheld points out that “Companies spend lots of time and money on complex tools to assess customer satisfaction. But they’re measuring the wrong thing. The best predictor of top-line growth can usually be captured in a single survey question: Would you recommend this company to a friend?”

It’s a great question because it doesn’t ask directly what someone thinks of the company. Instead, it asks what action you would take because of your interaction with that company. This indirect question requires a more rational interpretation with an additional cognitive step, and the real answer can then be inferred from the response.

The question is most commonly posed as “how likely are you to recommend this company (or brand, product, service) to a friend?” The respondent must choose an option between zero and ten, with zero being “not at all” and ten being “very likely”

The data is then compiled ready for interpretation.

As the name suggests, NPS is not just a matter of adding up all the 1’s through 10’s. NPS is determined essentially by splitting these respondents into ranges.

Respondents who give a 0-6 answer are known as ‘detractors’.

Those who give a 9 or 10 response are ‘promoters’.

In considering the ‘NPS’ number, only these groups matter. Those in the ‘middle’ giving 7 and 8 responses are ‘passives’, and their responses are discarded.

The percentage of total responses which are ‘detractors’ is calculated, and subtracted from the percentage of promoters to achieve an NPS score.

For example, if there are 500 responses, with 250 being passive, 100 being detractors and 150 promoters, then the NPS is 10.

According to "Andrew Chen of Andressen Horowitz", NPS “correlates strongly with word-of-mouth virality.” That means that if the NPS is relatively high, people are more likely to talk about you.

Before we even get to why NPS is horsehit, there are two issues with such a description.

The first is knowing what is a ‘good’ vs a ‘bad’ score. That’s because the number itself is irrelevant, it only really matters how it compares to alternative products or services. In other words 10 is a bad score only if all your competitors are scoring higher. But while some public data exists on the NPS of a select group of global brands, knowing the NPS of all your competitors just isn’t possible.

Secondly, the way NPS is calculated is designed under the assumption that detractors are more likely to say negative things than promoters are likely to say positive things. Hence 1-6 is subtracted from 9-10. But this very calculation is what makes it horseshit for most businesses; the size of the audience required to accurately (and regularly) measure NPS is simply beyond the vast majority of early-stage businesses.

I worked for the eBay Classified business in London (mostly unsuccessfully) for about two years. We were obsessed with NPS, irrationally in my view. We measured NPS on a monthly basis and placed a lot of faith in a number that constantly fluctuated. In the classical MBA, business-school style in which that organization was run, it was one of our guiding metrics.

Look how that turned out.

In the Classifieds, unlike the P2P auction format for which eBay is most well known, we had P2P consumers AND businesses using our platforms. For this reason we needed to take measurements of not just our consumers but also our business customers. The consumer users numbered in the millions, which meant taking an NPS measurement wasn’t difficult. But on the B2B side - my area of responsibility - we could only get an accurate number every six months. We worked with a professional research agency, had thousands of business customers, our B2B revenue was more than half the total of the entire business AND YET… we couldn’t get an accurate and useful NPS any more frequently than every six months.

In "this excellent article"", Alex Bitca notes: “We have a simple rule for calculating an accurate NPS score: the more customers you have, the more responses you need.” and “A Net Promoter Score calculated from a sample of 100 customers is less accurate, and thus less useful, than one calculated from a sample of 2,000.

This has several implications for small startups with few customers. The first is that your initial NPS data will give you valuable insight into what your customers think about your startup, but it’s not the only metric you should pay attention to.”

This was the eBay problem described above; the issue here is statistical significance.

The issue here is also that data collected from a small sample size will fluctuate wildly. This means that you can’t panic when it is bad, just like you shouldn’t pop the champagne when it is pumping.

That’s why any startup founder reading this should know: if you’ve measured your NPS without expert help, it’s probably just a performative wank. It’s not all bad though. It also means that anyone who advised you to measure NPS probably doesn’t know what they’re talking about, and I’d encourage you to ignore their advice on almost any other topic.

This is where most mistakes are made and why I think it's mostly a waste of time at a certain stage in the life of a business.

Firstly, don’t ask the question on its own. It needs to be ‘masked’ within other questions, lest you risk an irrational answer; despite being an indirect question, you need to consider other questions around it, in order to ‘prime’ the respondent into thinking about your business. At the same time this allows you to gather other qualitative data that may be more useful to move the needle on any detractors that do exist.

This is the question to which there is no good answer, ironically.

Even Bain "concedes the point:" “Among Net Promoter practitioners, the quest for good benchmarks has proven an important and difficult issue. Specifically, the challenge has been how and where to get reliable, executable benchmarks.”

This means that NPS is just like all good consulting methodologies: it sounds good, can’t really be disputed and therefore must be important. Follow this blueprint to untold riches if you dare!

In their regular benchmark updates, they break down major US industries, none of which are going to apply well to your Martech SaaS. So, good luck with that.

Yes.

But not much.

As I’ve detailed here it is difficult to get an accurate score and do so on a regular basis. And doing so regularly is the only way to see if you’re improving, so if you can’t… then why measure in the first place?

The limited value comes from knowing that you have a very positive or very negative user base. Nothing else. Additional questions around an NPS that may unearth the reasons for promoters and detractors, including specific issues will be much more valuable.

Therefore, the issue here isn’t talking to your customers; quite the opposite. You should measure anything and everything you can, talk to them regularly and try and solve their real-world problems.

But to look for one number to rule them all is both useless and inaccurate.

You’d be better off shoveling a pile of horseshit into your meeting room.