Siri Answers 83% of Questions Correctly in Test, Beating Alexa But Trailing Google Assistant

MacRumors

macrumors bot
Original poster
Apr 12, 2001
49,581
10,899



In an annual test comparing Google Assistant, Siri, and Alexa on smartphones, Loup Ventures' Gene Munster found that Siri was able to correctly answer 83 percent of questions, beating Alexa but trailing behind Google Assistant.

Munster asked each digital assistant 800 questions during the test to compare how each one responded. Alexa answered 79.8 percent of questions correctly, while Google Assistant answered 92.9 percent of questions correctly.


Compared to last year, Siri has seen improvement. During the July 2018 test, Siri answered 79 percent of questions correctly compared to the 83 percent of questions answered right this time around. Alexa last year was at 61 percent while Google Assistant was at 86, so there have been digital voice assistant improvements across the board.


This test covered smartphones specifically, comparing iPhones and Android devices. Munster says that smartphones were isolated from smart speakers because while underlying technology is similar, "use cases vary." Siri was tested on an iPhone running iOS 12.4, Google Assistant on a Pixel XL, and Alexa in the iOS app.

Questions were based on five categories and all assistants were asked the same 800 questions. Each question set was designed to "comprehensively test a digital assistant's ability and utility." Some of the sample questions across each of the categories:

[*]Local - Where is the nearest coffee shop?
[*]Commerce - Order me more paper towels.
[*]Navigation - How do I get to Uptown on the bus?
[*]Information - Who do the Twins play tonight?
[*]Command - Remind me to call Jerome at 2 pm today

Siri did best in the command, local, and navigation categories, faring less well in the information and commerce categories. Siri actually won out in the command category, but trailed behind Google Assistant in other categories.

Siri continues to prove more useful with phone-related functions like calling, texting, emailing, calendar, and music. Both Siri and Google Assistant, which are baked into the OS of the phone, far outperformed Alexa in the Command section.
Munster says that the continued rate of improvement "continues to surprise" based on the notable improvements that each voice assistant has demonstrated over the course of the last few years.

In the future, Loup Ventures expects to see further improvements from extending the feature sets of each voice assistant.

Article Link: Siri Answers 83% of Questions Correctly in Test, Beating Alexa But Trailing Google Assistant
 
  • Like
Reactions: Solomani

MallardDuck

macrumors 6502
Jul 21, 2014
426
776
[*]Local - Where is the nearest coffee shop?
[*]Commerce - Order me more paper towels.
[*]Navigation - How do I get to Uptown on the bus?
[*]Information - Who do the Twins play tonight?
[*]Command - Remind me to call Jerome at 2 pm today

And the responses:

Here's what I found on the web for coffee soup
I can't sort paper trowels in order
Playing Uptown girl by billy joel
I don't know who will win the game tonight
I've updated your reminder to call home


Most of Siri's brain-deadness is in the voice recognition portion of the system.
 

mrow

macrumors 6502
Aug 15, 2009
346
482
This is my experience as well. My problem is Siri on HomePod won’t just read you the damn answer to your question out loud 50% of the time. She’ll say “I’m sorry but I can’t get the answer to that on HomePod” and then you ask on an iOS device and the answer pops up on the screen in plain text. WHY?!?!? Why can’t you parse that text to read me the answer?! Google and Alexa can!
 

alpi123

macrumors 65816
Jun 18, 2014
1,374
1,426
What I found most interesting for me about Siri is that some voice settings have better voice recognition than others. Siri US understand me better than UK, although I'm from neither of those countries and my accent is kinda mixed between the two.
 
  • Like
Reactions: bestdressedguessed

spazzcat

macrumors 68030
Jun 29, 2007
2,745
1,585
I'd like to see a stat of the percentage of iPhone users who've given up on Siri entirely & switched her off forever. Probably mostly everyone.
92% of iPhone users have no idea they can turn Siri off... I would say less then 1% have turn it off.
 

MrGimper

macrumors 604
Sep 22, 2012
6,545
7,135
Andover, UK
My experience of Siri is that it’s truly garbage. I’ve resorted to just using it to set timers. I have several Echo’s in the house with a Sonos setup and Hue lights and Alexa is beyond amazing at understanding what we’re saying and controls all that. As a consequence I no longer look for HomeKit compatibility, rather Alexa compatibility.
 

LordVic

macrumors 603
Sep 7, 2011
5,559
12,089
Interesting... in my un-scientific testing, Alexa consistently beats Siri at recognizing what I want, and more importantly, the gibberish requests that my 4 year old spouts "Alexa play more Frozen!" (kill me)
oh come on. whats wrong with frozen... let it go!

Mixed household with a combination of different voice reps and your anecdote is similar. Alexa seems to understand my nieces very easily. Google home seems to have the widest range of understandin, even if the answers times make me scratch my head. Siri usually understands my questions, but doesn't give me answers in usable forms. More often than not, it's just Siri saying "here's what I found for you on the web"


This is my experience as well. My problem is Siri on HomePod won’t just read you the damn answer to your question out loud 50% of the time. She’ll say “I’m sorry but I can’t get the answer to that on HomePod” and then you ask on an iOS device and the answer pops up on the screen in plain text. WHY?!?!? Why can’t you parse that text to read me the answer?! Google and Alexa can!
Hi, Here's what I found on the web! <links to wiki>

JUST READ ME THE DAMN WIKI
 

falainber

macrumors 68020
Mar 16, 2016
2,215
2,730
Wild West
Obviously, the questions in these tests are too simple. Neither of these assistants are that good. If they asked more difficult questions the results would drop for all assistants but we would get a better measure of their quality (like 15%, 40% and 60% or something like this). As is, the test results are fairly meaningless. All assistants look great (and yet people keep ignoring them because the results in real life are not that great).
 

MrTemple

macrumors regular
Jun 11, 2013
200
374
Canadian Pacific North Wilderness
Siri on my XS is terrific at understanding me, but on my Apple Watch Series 4, it is DREADFUL.

Must be the microphone, because I can barely whisper into my iPhone to control my smart lights and it almost always comprehends. On the watch though, no matter how clearly I speak, it screws up half the time. It's gotten to the point I don't even bother.
 

ignatius345

macrumors 68030
Aug 20, 2015
2,857
3,829
I still have erratic results from Siri -- and sometimes get different results from saying the same or nearly the same exact thing.

I will say this: the quality of the voice is WAY better in the iOS 13 beta. So, that's something I guess.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.