Siri Answers 83% of Questions Correctly in Test, Beating Alexa But Trailing Google Assistant

Discussion in 'MacRumors.com News Discussion' started by MacRumors, Aug 15, 2019 at 12:05 PM.

  1. MacRumors macrumors bot

    MacRumors

    Joined:
    Apr 12, 2001
    #1
    [​IMG]


    In an annual test comparing Google Assistant, Siri, and Alexa on smartphones, Loup Ventures' Gene Munster found that Siri was able to correctly answer 83 percent of questions, beating Alexa but trailing behind Google Assistant.

    Munster asked each digital assistant 800 questions during the test to compare how each one responded. Alexa answered 79.8 percent of questions correctly, while Google Assistant answered 92.9 percent of questions correctly.

    [​IMG]

    Compared to last year, Siri has seen improvement. During the July 2018 test, Siri answered 79 percent of questions correctly compared to the 83 percent of questions answered right this time around. Alexa last year was at 61 percent while Google Assistant was at 86, so there have been digital voice assistant improvements across the board.

    [​IMG]

    This test covered smartphones specifically, comparing iPhones and Android devices. Munster says that smartphones were isolated from smart speakers because while underlying technology is similar, "use cases vary." Siri was tested on an iPhone running iOS 12.4, Google Assistant on a Pixel XL, and Alexa in the iOS app.

    Questions were based on five categories and all assistants were asked the same 800 questions. Each question set was designed to "comprehensively test a digital assistant's ability and utility." Some of the sample questions across each of the categories:

    [*]Local - Where is the nearest coffee shop?
    [*]Commerce - Order me more paper towels.
    [*]Navigation - How do I get to Uptown on the bus?
    [*]Information - Who do the Twins play tonight?
    [*]Command - Remind me to call Jerome at 2 pm today

    Siri did best in the command, local, and navigation categories, faring less well in the information and commerce categories. Siri actually won out in the command category, but trailed behind Google Assistant in other categories.

    [​IMG]
    Munster says that the continued rate of improvement "continues to surprise" based on the notable improvements that each voice assistant has demonstrated over the course of the last few years.

    In the future, Loup Ventures expects to see further improvements from extending the feature sets of each voice assistant.

    Article Link: Siri Answers 83% of Questions Correctly in Test, Beating Alexa But Trailing Google Assistant
     
  2. coolfactor macrumors 601

    Joined:
    Jul 29, 2002
    Location:
    Vancouver, BC CANADA
  3. JetLaw macrumors regular

    JetLaw

    Joined:
    Jan 21, 2009
    #3
    I'm not sure about the methodology they used to reach these results, but my anecdotal experience is that Alexa does a MUCH better job than Siri on almost everything I've tried to use them for.
     
  4. Blackstick macrumors 6502a

    Blackstick

    Joined:
    Aug 11, 2014
    Location:
    Sunny South Florida
    #4
    Interesting... in my un-scientific testing, Alexa consistently beats Siri at recognizing what I want, and more importantly, the gibberish requests that my 4 year old spouts "Alexa play more Frozen!" (kill me)
     
  5. airwalk331 macrumors newbie

    Joined:
    Mar 20, 2012
    #5
    Does anyone else feel like this is crap? 99.8% of the time Siri doesn’t work for me (though my friends’ Alexa does). Period. It’s even worse when I try to use it and there are 2 or more apple devices in the vicinity.
     
  6. diego.caraballo macrumors member

    Joined:
    Oct 18, 2013
  7. cashville2400 macrumors regular

    cashville2400

    Joined:
    Nov 29, 2011
    #7
    I don't need a list to tell me how horrible Siri is! I can form my own opinion just from my user experience with Siri and other personal assistants. Siri might "understand", but she can not do much.
     
  8. mediotanque macrumors member

    Joined:
    Oct 31, 2013
    Location:
    Uruguay
    #8
    This is ********. Siri is the dumbest Ai at the moment. Even Cortana works better than Siri.
     
  9. MallardDuck macrumors 6502

    Joined:
    Jul 21, 2014
    #9
    [*]Local - Where is the nearest coffee shop?
    [*]Commerce - Order me more paper towels.
    [*]Navigation - How do I get to Uptown on the bus?
    [*]Information - Who do the Twins play tonight?
    [*]Command - Remind me to call Jerome at 2 pm today

    And the responses:

    Here's what I found on the web for coffee soup
    I can't sort paper trowels in order
    Playing Uptown girl by billy joel
    I don't know who will win the game tonight
    I've updated your reminder to call home


    Most of Siri's brain-deadness is in the voice recognition portion of the system.
     
  10. mrow macrumors 6502

    Joined:
    Aug 15, 2009
    #10
    This is my experience as well. My problem is Siri on HomePod won’t just read you the damn answer to your question out loud 50% of the time. She’ll say “I’m sorry but I can’t get the answer to that on HomePod” and then you ask on an iOS device and the answer pops up on the screen in plain text. WHY?!?!? Why can’t you parse that text to read me the answer?! Google and Alexa can!
     
  11. now i see it macrumors 68040

    Joined:
    Jan 2, 2002
    #11
    I'd like to see a stat of the percentage of iPhone users who've given up on Siri entirely & switched her off forever. Probably mostly everyone.
     
  12. ChrisMoBro macrumors 6502

    ChrisMoBro

    Joined:
    Oct 31, 2016
    #12
    From my experience of using both, Siri doesn’t even come close to Alexa. Using it is just a painful experience in comparison.
     
  13. spazzcat macrumors 68030

    spazzcat

    Joined:
    Jun 29, 2007
    #13
    Nope, never got all the siri hate. I have very few issues with using Siri, I use it daily around my house.
     
  14. alpi123 macrumors 6502a

    alpi123

    Joined:
    Jun 18, 2014
    #14
    What I found most interesting for me about Siri is that some voice settings have better voice recognition than others. Siri US understand me better than UK, although I'm from neither of those countries and my accent is kinda mixed between the two.
     
  15. medialook macrumors newbie

    medialook

    Joined:
    Nov 12, 2017
    Location:
    Bracknell, United Kingdom
    #15
    Siri doesn't work properly for me. Not sure how they've managed to get this results.
     
  16. psac macrumors 6502a

    psac

    Joined:
    Jul 6, 2009
  17. spazzcat macrumors 68030

    spazzcat

    Joined:
    Jun 29, 2007
    #17
    92% of iPhone users have no idea they can turn Siri off... I would say less then 1% have turn it off.
     
  18. MallardDuck macrumors 6502

    Joined:
    Jul 21, 2014
    #18
    You mean pictures of falling gnomes aren't what you want it to show?
     
  19. MrGimper macrumors 603

    MrGimper

    Joined:
    Sep 22, 2012
    Location:
    Andover, UK
    #19
    My experience of Siri is that it’s truly garbage. I’ve resorted to just using it to set timers. I have several Echo’s in the house with a Sonos setup and Hue lights and Alexa is beyond amazing at understanding what we’re saying and controls all that. As a consequence I no longer look for HomeKit compatibility, rather Alexa compatibility.
     
  20. LordVic macrumors 603

    Joined:
    Sep 7, 2011
    #20
    oh come on. whats wrong with frozen... let it go!

    Mixed household with a combination of different voice reps and your anecdote is similar. Alexa seems to understand my nieces very easily. Google home seems to have the widest range of understandin, even if the answers times make me scratch my head. Siri usually understands my questions, but doesn't give me answers in usable forms. More often than not, it's just Siri saying "here's what I found for you on the web"


    Hi, Here's what I found on the web! <links to wiki>

    JUST READ ME THE DAMN WIKI
     
  21. jerryk macrumors 601

    Joined:
    Nov 3, 2011
    Location:
    SF Bay Area
    #21
    Doesn't matter what the stats are today. In the era of Machine Learning, data is king. Google has more training data to answer questions. Amazon has more data on the queries people ask, and on which they will improve their models.
     
  22. ReneR macrumors 6502

    Joined:
    Jun 18, 2008
    Location:
    Berlin, Germany
    #22
    Not enough human low-wage transcriber slave workers anymore? :-/
     
  23. falainber macrumors 65816

    falainber

    Joined:
    Mar 16, 2016
    Location:
    Wild West
    #23
    Obviously, the questions in these tests are too simple. Neither of these assistants are that good. If they asked more difficult questions the results would drop for all assistants but we would get a better measure of their quality (like 15%, 40% and 60% or something like this). As is, the test results are fairly meaningless. All assistants look great (and yet people keep ignoring them because the results in real life are not that great).
     
  24. MrTemple macrumors regular

    Joined:
    Jun 11, 2013
    Location:
    Canadian Pacific North Wilderness
    #24
    Siri on my XS is terrific at understanding me, but on my Apple Watch Series 4, it is DREADFUL.

    Must be the microphone, because I can barely whisper into my iPhone to control my smart lights and it almost always comprehends. On the watch though, no matter how clearly I speak, it screws up half the time. It's gotten to the point I don't even bother.
     
  25. ignatius345 macrumors 68020

    Joined:
    Aug 20, 2015
    #25
    I still have erratic results from Siri -- and sometimes get different results from saying the same or nearly the same exact thing.

    I will say this: the quality of the voice is WAY better in the iOS 13 beta. So, that's something I guess.
     

Share This Page

137 August 15, 2019