Apple Computer, OS X®, iOS® and the Net Promoter® Score (NPS®)

[grahamperrin]

Scores – based on votes in MacRumors Forums

OS X Mavericks: 49
iOS 8.x: -13
OS X Yosemite: -17​

Is it normal to find so much detraction following the initial release of an operating system?



Related reading – MacRumors

… I treat some poll results as unscientific … still, somewhat entertaining. …

Up a little.

52.55% of voters here are updating now (or intend to do so within a short period).

Post-launch traffic measurements at https://www.gosquared.com/global/mac/yosemite/#launch suggest an adoption rate lower than the rate suggested in this poll. I assume that voters in the Yosemite sub-forums are relatively optimistic. …

As someone who deals in NPS professionally and frequently - asking the NPS question without asking anything else won't allow you to gain any insights as to why the NPS score is the way it is.

Put another way, NPS doesn't stand alone. I cringe when I see stand-alone surveys like this one, as they're doomed to yield uninteresting results.

… such votes are both statistically insignificant AND statistically unsound.

There are many threads in which it has been explained why a self-selecting sample of users who are over-represented from the "user-with-issues" category isn't a basis on which to extrapolate to any overall statistical assessment whatsoever.

Are You Going To Install Yosemite?

Are voters in MacRumors Forums more optimistic – about Apple products – than voters at large?

10.9 to be the last of the old school ?

Related reading – elsewhere

Study: Apple customers are by far the most loyal of any computer company – MacDailyNews - Welcome Home (2006-07-11, highlights)

The Ultimate Question | Computerworld (2006-07-24)

Net Promoter Score (NPS) Criticisms and Best Practices (2009-04-28, highlights)

… typical measures in the 30-40% range. …

Comparison of Three One-Question, Post-Task Usability Questionnaires (PDF, 2009)

What is Net Promoter – Aggreko (~2010)

… Net Promoter All Stars table (right - compiled by Fred Reichheld and Bain & Company in 2006) …

Another Myth Bites The Dust: How Apple Listens To Its Customers (2011-08-26, highlights)

How Tech Providers Improve Consumer Brand Image (2013-10-11)

… Despite Apple’s rank as the #1 brand WW by Interbrand, the NPS score for the company is weak (-3% NPS score). Nearly 40% of consumers would not recommend the Apple brand. …

Apple among the highest in customer loyalty in 2014 Satmetrix Net Promotor Benchmarks - Apple Daily Report (2014-03-05)

 

[grahamperrin]

Variance in NPS scores for OS X Yosemite

The score for Yosemite has improved
from -17
to -13 (details below).

Measured a week ago, with one hundred votes cast, the score was -15.

I have not measured a variance but I'm almost certain that whenever I glanced at the figures, probably once a day or more, the score was negative.

Seventy-seven detractors at present. I wonder how many of those people voted without trying the released build of OS X 10.10 …

Current percentages in the poll for Yosemite

Percentage of promoters
31.21 + 4.46
= 35.67

Percentage of detractors
4.46 + 5.73 + 1.27 + 4.46 + 1.91 + 7.64 + 23.57
= 49.04

NPS
35.67 - 49.04
= -13.37

 

[simonsi]

BTW you need to resist the urge to calculate to 2 decimal places, as my Math teacher would have said "just because you put 2 decimal places on it, it doesn't stop it being a guess!" (after he had read my homework ).

But my point is your sample size doesn't support % to 2dp so it is a bit misleading to imply that level of accuracy

 

[grahamperrin]

… sample size doesn't support % to 2dp so it is a bit misleading to imply that level of accuracy

There was no intention to mislead – all of the Net Promoter scores are given as integers.

The percentages were exactly as presented by the forum software.

 

[simonsi]

NPS
35.67 - 49.04
= -13.37[/SIZE]

2dp

I'm sure you don't intend to mislead but forum software-provided or not, that result is not accurate to 2dp as it suggests.

 

[grahamperrin]

My concern – the state of Apple software – and a plea for better measurement

Simon, those figures were copied from the Paper Tape window of Calculator.app in Mavericks. I guess I don't know how to make best use of the app; I'm neither a mathematician nor a statistician.

Whilst I'm not disinterested in 2dp accuracy, I am more interested in https://forums.macrumors.com/threads/1818418/ – because, I guess, there's a bias amongst MacRumors Forums voters that's not representative of Apple customers in general. But in that poll about optimism, there are too few votes.

Keywords from the opening post here: unscientific, entertaining

Joking aside: underlying all of the above, I'm deeply concerned about the recent state of Apple software – in particular, the design and UX of OS X Yosemite. If the plea for better measurements is acted upon, then maybe discussions within Apple will lead to software products that are less ridiculous.

 

[simonsi]

I think the point being missed here is that Apple know full well that UI changes delight and disgust users in equal measure irrespective of the change. So there is no point in trying to please everyone, it is an impossible aim to achieve.

However the changes do generate interest and discussion but don't think Apple will ever be aiming for some kind of "zero UI impact" state.

Functional issues are different but your describing the UI as "ridiculous" isn't an issue I can see Apple remotely being concerned about, for every individual who holds that impression, there will be another who is delighted by it - and maybe 3 potential users who look at it as a result of the publicity....1 of whom might become an Apple customer...

 

[grahamperrin]

The subject here: NPS – apparently dismal loyalties to recent Apple operating systems

… there is no point in trying to please everyone, it is an impossible aim to achieve. …

With respect: that's neither the point nor the aim. This topic is primarily about the NPS measurements; about the apparently dismal loyalties towards recent operating systems from Apple.

Consider the possibility that – without the constraints/bias of MacRumors Forums – the scores for iOS 8.x and for OS X Yosemite are worse than the -3% for Apple in 2013.

Apple suggests that with the design of Yosemite, the entire experience is enhanced. That public page addresses all Apple customers, without exception. A wide audience.

NPS measurements today, from the polls in MacRumors Forums:

OS X Mavericks: 45 (38 respondents)
iOS 8.x: -12 (147 respondents)
OS X Yosemite: -12 (164 respondents)​

A drop in loyalty, between Mavericks and Yosemite, can not be harmonious with an entirely enhanced experience for Apple's intended audience.​

From the Paper Tape window of Calculator.app for Mavericks, iOS 8.x and Yosemite:

47.37 + 21.05
= 68.42

2.63 + 0 + 2.63 + 0 + 0 + 0 + 18.42
= 23.68

68.42 - 23.68
= 44.74

----

33.33 + 4.08
= 37.41

2.72 + 6.12 + 5.44 + 3.4 + 5.44 + 1.36 + 25.17
= 49.65

37.41 - 49.65
= -12.24

----

31.71 + 4.88
= 36.59

4.27 + 5.49 + 1.22 + 4.27 + 1.83 + 7.93 + 23.78
= 48.79

36.59 - 48.79
= -12.2

 

[simonsi]

With respect: that's neither the point nor the aim. This topic is primarily about the NPS measurements; about the apparently dismal loyalties towards recent operating systems from Apple.

Consider the possibility that – without the constraints/bias of MacRumors Forums – the scores for iOS 8.x and for OS X Yosemite are worse than the -3% for Apple in 2013.

Apple suggests that with the design of Yosemite, the entire experience is enhanced. That public page addresses all Apple customers, without exception. A wide audience.

NPS measurements today, from the polls in MacRumors Forums:

OS X Mavericks: 45 (38 respondents)
iOS 8.x: -12 (147 respondents)
OS X Yosemite: -12 (164 respondents)​

A drop in loyalty, between Mavericks and Yosemite, can not be harmonious with an entirely enhanced experience for Apple's intended audience.​

From the Paper Tape window of Calculator.app for Mavericks, iOS 8.x and Yosemite:

47.37 + 21.05
= 68.42

2.63 + 0 + 2.63 + 0 + 0 + 0 + 18.42
= 23.68

68.42 - 23.68
= 44.74

----

33.33 + 4.08
= 37.41

2.72 + 6.12 + 5.44 + 3.4 + 5.44 + 1.36 + 25.17
= 49.65

37.41 - 49.65
= -12.24

----

31.71 + 4.88
= 36.59

4.27 + 5.49 + 1.22 + 4.27 + 1.83 + 7.93 + 23.78
= 48.79

36.59 - 48.79
= -12.2

LOL - these polls aren't statistically valid in themselves, you certainly can't draw any conclusions between them, there is simply no comparable baseline between them. This is anecdotal at best however remember the golden rule "anecdotes are not data".

I'll leave you to it.

 

[grahamperrin]

Again, a plea for better measurement

… This is anecdotal at best however remember the golden rule "anecdotes are not data". …

Understood. That's why I made the plea for better measurement, a week ago.

simonsi, instead of laughing and abandoning the subject, could you please do something to encourage better measurement? Thanks.

 

[thejadedmonkey]

XP was "new new new" when it came out. I remember it was all over the torrents, and very popular at my school.

Vista was not popular because it ran like **** on low end computers. But the chief complaint was that they couldn't use the OS, not that they didn't like using it when they could use it.

7 was perfect in every way imaginable.

8 was all sorts of negative because people don't like metro, but simple tweaks like Start8 made it usable, and while you'll find people who aren't updating to Windows 8 still, even Metro haters seem to be okay with 8 if they disable the Metro part.

And Windows 10 seems to be getting great reviews like 7 did.

So to answer your question, I don't think even Windows has as poor of a reception as Yosemite has right now.

My next laptop will probably be a 12" Macbook Air, but only because Apple makes cost efficient ultrabooks. The first thing I'll be doing is installing Windows on it. Yosemite is just unusable, from Safari not showing the URL to the damage it does to my eyes staring at florescent pinks and blues all day long.

 

[grahamperrin]

Better measurements of scores will be welcome

Yesterday, quoting a tweet from two weeks ago:

Better measurements of scores for OS X 10.9.x and 10.10 will be welcome.

Anyone who wishes to devalue the MacRumors polls can re-tweet that. The sooner we get better measurements, in public, the better.

None of the three polls have closed.

To avoid some of the the initial rush of negativity, I waited a considerable length of time before beginning the poll for OS X Yosemite. Glancing at star ratings in the Apple Store – worse for 10.10.1 than for 10.10 – I suspect that beginning the poll later would have resulted in an even worse score for Yosemite. …Post-release reactions to Yosemite were (in at least some areas) worse than pre-release reactions.

For 10.10.1, reactions appear to be worse than for the initial release. …

... wouldn't it make more sense to not retweet a link to the MacRumors poll?

Please tweet, post or e-mail whatever you think will be sensible … the plea is open to anyone:

• could you please do something to encourage better measurement?

All four topics (this one, plus the three polls) included a link to 'Net Promoter Score (NPS) Criticisms and Best Practices' – and more – in the opening posts. I never intended the results, from any of the three, to be conclusive.

It's not enough to simply criticise the results from these polls. I genuinely did, genuinely do want better measurement – and that was not a lazy afterthought. The plea for something better was tweeted less than five minutes after the graph was tweeted.

To anyone who would like to respond: please first read some of the items linked from the opening post. If your criticisms (of the polling method and so on) are significantly different from the criticisms already given – the 2009 blog post by Verint Systems might be a good starting point – then please, add relevant details to this discussion.

If you must criticise, then please also do something positive and constructive towards getting better measurements of Apple customer loyalties towards the three operating systems.​

(Personally, I'm less interested in iOS. But it seemed fair to poll about iOS at the same time as the other systems; I didn't want to be exclusive.)

Thanks

 

[grahamperrin]

Persuasion: trusted sources

Customer experience is more important than advertising (Infographic)

… 83% trust independent sources with whom they have a personal relationship over advertising. …

 

[grahamperrin]

Customer satisfaction/loyalty – Apple hardware, Apple software

From an Apple operating systems design topic:

… It's going to take more than a horrible skin and a few bugs to get people to stop using the iPhone. It will however, affect people's perception of Apple. Instead of being known for making the best products, they will slowly degrade to just another company. There was a report that came out a few weeks ago showing that customer satisfaction for iPad and iPhone had dropped. Hmm.. I wonder what's causing that. …

Did that report refer to the American Customer Satisfaction Index?

Personal computers

Without reference to software: satisfaction with Apple dropped three percent between 2013 and 2014. The press release Desktop Customer Satisfaction Surpasses Tablets and Laptops American Customer Satisfaction Index (2014-09-23) describes PC makers so I assume that this measure of satisfaction is hardware-oriented. (Not a measure of satisfaction with OS X.)

Cellular telephones

Again without reference to software: satisfaction with Apple dropped 2.5 percent. I assume that this measure is iPhone-oriented. (Not a measure of satisfaction with iOS.)

Mobile phones (NCSI-UK)

Without reference to software: up one percent between 2013 and 2014.​

Shareholder influence

… Licking shareholder fanny comes first now. At all costs. just saying.

… The shareholders have always been first, even when it didn't seem like it. Especially with the iphone etc

I don't know about first, but there's a sense that increasingly, as a result of excessive shareholder influence, too many Apple customers are dissatisfied with changes to products. I sort of wonder what goes on at shareholder meetings (but I'm not inclined to read relevant documents – shareholding is of no interest to me).

Influencing the influencers

If it's possible to capture the attention of a broad range of influencers, including shareholders – to make enough people across that range realise (in simple terms) that some types of change are detrimental to the brand – then there'll be a better chance of future redesigns pleasing a greater percentage of customers.

How to capture that range of attention? What do people love to tout? Numbers.

– I keep coming back to that because of the wish for better measurements of loyalty towards OS X Mavericks and Yosemite (and to whoever might make those measurements, it'll probably make sense to perform measurements for iOS around the same time). Any one of the following might be within the results:

• a simple negative correlation (for the increased version number, decreased loyalty)
• a simple positive correlation
• nothing remarkable …

I don't even know how much Tim Cook listens to the heads of the divisions. …

… if there's a negative correlation, and if that's significant enough to attract the attention of Apple and/or shareholders, it'll be for Apple to balance priorities in software design reviews. The positive aspect of this approach to negative attention could be that it's not singling out any individuals or groups as 'to blame'. As far as

----

If there's a positive correlation – if, say, there's greater loyalty towards Yosemite than towards Mavericks – then that result will be a good complement to the poll about optimism in these forums.