Siri Answers 83% of Questions Correctly in Test, Beating Alexa But Trailing Google Assistant

[MacRumors]

In an annual test comparing Google Assistant, Siri, and Alexa on smartphones, Loup Ventures' Gene Munster found that Siri was able to correctly answer 83 percent of questions, beating Alexa but trailing behind Google Assistant.

Munster asked each digital assistant 800 questions during the test to compare how each one responded. Alexa answered 79.8 percent of questions correctly, while Google Assistant answered 92.9 percent of questions correctly.

Compared to last year, Siri has seen improvement. During the July 2018 test, Siri answered 79 percent of questions correctly compared to the 83 percent of questions answered right this time around. Alexa last year was at 61 percent while Google Assistant was at 86, so there have been digital voice assistant improvements across the board.

This test covered smartphones specifically, comparing iPhones and Android devices. Munster says that smartphones were isolated from smart speakers because while underlying technology is similar, "use cases vary." Siri was tested on an iPhone running iOS 12.4, Google Assistant on a Pixel XL, and Alexa in the iOS app.

Questions were based on five categories and all assistants were asked the same 800 questions. Each question set was designed to "comprehensively test a digital assistant's ability and utility." Some of the sample questions across each of the categories:

[*]Local - Where is the nearest coffee shop?
[*]Commerce - Order me more paper towels.
[*]Navigation - How do I get to Uptown on the bus?
[*]Information - Who do the Twins play tonight?
[*]Command - Remind me to call Jerome at 2 pm today

Siri did best in the command, local, and navigation categories, faring less well in the information and commerce categories. Siri actually won out in the command category, but trailed behind Google Assistant in other categories.

Siri continues to prove more useful with phone-related functions like calling, texting, emailing, calendar, and music. Both Siri and Google Assistant, which are baked into the OS of the phone, far outperformed Alexa in the Command section.

Munster says that the continued rate of improvement "continues to surprise" based on the notable improvements that each voice assistant has demonstrated over the course of the last few years.

In the future, Loup Ventures expects to see further improvements from extending the feature sets of each voice assistant.

Article Link: Siri Answers 83% of Questions Correctly in Test, Beating Alexa But Trailing Google Assistant

 

[coolfactor]

Why isn't Cortana on this list?

 

[JetLaw]

I'm not sure about the methodology they used to reach these results, but my anecdotal experience is that Alexa does a MUCH better job than Siri on almost everything I've tried to use them for.

 

[Blackstick]

Interesting... in my un-scientific testing, Alexa consistently beats Siri at recognizing what I want, and more importantly, the gibberish requests that my 4 year old spouts "Alexa play more Frozen!" (kill me)

 

[airwalk331]

Does anyone else feel like this is crap? 99.8% of the time Siri doesn’t work for me (though my friends’ Alexa does). Period. It’s even worse when I try to use it and there are 2 or more apple devices in the vicinity.

 

[diego.caraballo]

Siri better than Alexa? Fake news

 

[cashville2400]

I don't need a list to tell me how horrible Siri is! I can form my own opinion just from my user experience with Siri and other personal assistants. Siri might "understand", but she can not do much.

 

[mediotanque]

This is ********. Siri is the dumbest Ai at the moment. Even Cortana works better than Siri.

 

[MallardDuck]

[*]Local - Where is the nearest coffee shop?
[*]Commerce - Order me more paper towels.
[*]Navigation - How do I get to Uptown on the bus?
[*]Information - Who do the Twins play tonight?
[*]Command - Remind me to call Jerome at 2 pm today

And the responses:

Here's what I found on the web for coffee soup
I can't sort paper trowels in order
Playing Uptown girl by billy joel
I don't know who will win the game tonight
I've updated your reminder to call home


Most of Siri's brain-deadness is in the voice recognition portion of the system.

 

[mrow]

This is my experience as well. My problem is Siri on HomePod won’t just read you the damn answer to your question out loud 50% of the time. She’ll say “I’m sorry but I can’t get the answer to that on HomePod” and then you ask on an iOS device and the answer pops up on the screen in plain text. WHY?!?!? Why can’t you parse that text to read me the answer?! Google and Alexa can!

 

[now i see it]

I'd like to see a stat of the percentage of iPhone users who've given up on Siri entirely & switched her off forever. Probably mostly everyone.

 

[ChrisMoBro]

From my experience of using both, Siri doesn’t even come close to Alexa. Using it is just a painful experience in comparison.

 

[spazzcat]

Does anyone else feel like this is crap? 99.8% of the time Siri doesn’t work for me (though my friends’ Alexa does). Period. It’s even worse when I try to use it and there are 2 or more apple devices in the vicinity.

Nope, never got all the siri hate. I have very few issues with using Siri, I use it daily around my house.

 

[alpi123]

What I found most interesting for me about Siri is that some voice settings have better voice recognition than others. Siri US understand me better than UK, although I'm from neither of those countries and my accent is kinda mixed between the two.

 

[medialook]

Does anyone else feel like this is crap? 99.8% of the time Siri doesn’t work for me (though my friends’ Alexa does). Period. It’s even worse when I try to use it and there are 2 or more apple devices in the vicinity.

Siri doesn't work properly for me. Not sure how they've managed to get this results.

 

[psac]

... and yet, "Call Home" is a problem.

 

[spazzcat]

I'd like to see a stat of the percentage of iPhone users who've given up on Siri entirely & switched her off forever. Probably mostly everyone.

92% of iPhone users have no idea they can turn Siri off... I would say less then 1% have turn it off.

 

[MallardDuck]

... and yet, "Call Home" is a problem.

You mean pictures of falling gnomes aren't what you want it to show?

 

[MrGimper]

My experience of Siri is that it’s truly garbage. I’ve resorted to just using it to set timers. I have several Echo’s in the house with a Sonos setup and Hue lights and Alexa is beyond amazing at understanding what we’re saying and controls all that. As a consequence I no longer look for HomeKit compatibility, rather Alexa compatibility.

 

[LordVic]

Interesting... in my un-scientific testing, Alexa consistently beats Siri at recognizing what I want, and more importantly, the gibberish requests that my 4 year old spouts "Alexa play more Frozen!" (kill me)

oh come on. whats wrong with frozen... let it go!

Mixed household with a combination of different voice reps and your anecdote is similar. Alexa seems to understand my nieces very easily. Google home seems to have the widest range of understandin, even if the answers times make me scratch my head. Siri usually understands my questions, but doesn't give me answers in usable forms. More often than not, it's just Siri saying "here's what I found for you on the web"

This is my experience as well. My problem is Siri on HomePod won’t just read you the damn answer to your question out loud 50% of the time. She’ll say “I’m sorry but I can’t get the answer to that on HomePod” and then you ask on an iOS device and the answer pops up on the screen in plain text. WHY?!?!? Why can’t you parse that text to read me the answer?! Google and Alexa can!

Hi, Here's what I found on the web! <links to wiki>

JUST READ ME THE DAMN WIKI

 

[jerryk]

Doesn't matter what the stats are today. In the era of Machine Learning, data is king. Google has more training data to answer questions. Amazon has more data on the queries people ask, and on which they will improve their models.

 

[ReneR]

Not enough human low-wage transcriber slave workers anymore? :-/

 

[falainber]

Obviously, the questions in these tests are too simple. Neither of these assistants are that good. If they asked more difficult questions the results would drop for all assistants but we would get a better measure of their quality (like 15%, 40% and 60% or something like this). As is, the test results are fairly meaningless. All assistants look great (and yet people keep ignoring them because the results in real life are not that great).

 

[MrTemple]

Siri on my XS is terrific at understanding me, but on my Apple Watch Series 4, it is DREADFUL.

Must be the microphone, because I can barely whisper into my iPhone to control my smart lights and it almost always comprehends. On the watch though, no matter how clearly I speak, it screws up half the time. It's gotten to the point I don't even bother.

 

[ignatius345]

I still have erratic results from Siri -- and sometimes get different results from saying the same or nearly the same exact thing.

I will say this: the quality of the voice is WAY better in the iOS 13 beta. So, that's something I guess.