Siri on iOS Answered 78.5% of Queries Correctly in Latest Test, Trailing Google Assistant at 85.5%

Discussion in 'MacRumors.com News Discussion' started by MacRumors, Jul 25, 2018.

  1. MacRumors macrumors bot

    MacRumors

    Joined:
    Apr 12, 2001
    #1
    [​IMG]


    Five months after performing a test that put the smart speakers of multiple companies in the spotlight to determine how well they performed in various categories, Loup Ventures is back today with an IQ test focused entirely on digital AI assistants. To get the necessary results, the researchers asked Siri, Google Assistant, Alexa, and Cortana 800 questions each on a smartphone, and compared their findings to a previous AI test held in April 2017.

    [​IMG]

    For Siri in the new test, Apple's AI helper understood 99 percent of the queries and answered 78.5 percent of them correctly. That's an improvement on a similar AI-focused test from April 2017 (66.1 percent of 800 questions answered correctly). While Loup Ventures looked at similar methodologies when testing smart speakers in February, the researchers explain that it's "not worthwhile to compare" the results across these tests since "the use cases differ greatly between digital assistants and smart speakers."

    This is particularly true for Siri on HomePod, which performs well in certain areas but is largely limited to the amount of actions it can perform on the speaker itself. This led Apple's HomePod to become relegated to the "bottom of the totem pole" in an AI assistant performance test during Loup Venture's smart speaker research in February, with Siri answering 52.3 percent of 782 total questions correctly, across the same five categories as the new test.

    Loup Ventures grades each digital assistant on two metrics: "Did it understand what was being asked?" and "Did it deliver a correct response?" Questions came from five categories, including Local (example: "Where is the nearest coffee shop?"), Commerce ("Can you order me more paper towels?"), Navigations ("How do I get to uptown on the bus?"), Information ("Who do the Twins play tonight?"), and Command ("Remind me to call Steve at 2pm today").

    [​IMG]
    Charts via Loup Ventures


    Questions were asked of Siri on an iPhone running iOS 11.4, Google Assistant on a Pixel XL, Alexa on the iOS app, and Cortana on the iOS app. Siri's best category was Command (90 percent of questions answered correctly), outperforming all rivals when asked to control aspects of the iPhone, smart home products, Apple Music, and more. Following Command, Siri performed well in Local (87 percent), Navigation (83 percent), and began dipping in Information (70 percent) and Commerce (60 percent).
    Google Assistant was the top digital assistant in all categories except Command, with Loup Ventures particularly liking Google's "featured snippets" feature that reads off search results of voice queries and is often "exactly what you're looking for." Both Alexa and Cortana were lesser performers in the test due to the iOS app for each limiting what the assistants can do on an iPhone, unlike Siri's ability to perform tasks all over iOS and not just in one app.

    In total, Google Assistant answered 85.5 percent of the 800 questions asked correctly and understood all of them, compared to Siri's 78.5 percent answered correctly and 11 misunderstood. Alexa correctly answered 61.4 percent and misunderstood 13, while Cortana was the "laggard" and correctly answered 52.4 percent and misunderstood 19.

    [​IMG]

    Over the 15 month period since April 2017, Siri improved by 13 percentage points, with Loup Ventures pointing out that it was "impressed with the speed at which the technology is advancing" for most of the assistants. The researchers went on to explain that many of the issues they had last year were erased by "improvements to natural language processing and inter-device connectivity."

    Loup Ventures also sees more improvements coming down the line, particularly for Siri. This fall, the digital assistant will gain new abilities with Apple's upcoming "Siri Shortcuts" feature in iOS 12, which will let users create automations between apps that can be easily triggered through voice commands. The researches say they are "eager to test that feature" when it launches within iOS 12, likely coming in September.

    Article Link: Siri on iOS Answered 78.5% of Queries Correctly in Latest Test, Trailing Google Assistant at 85.5%
     
  2. Heineken macrumors 6502a

    Heineken

    Joined:
    Jan 27, 2018
    #2
    Impressive number without any personal data mining.
     
  3. OC40 macrumors 6502

    Joined:
    Sep 20, 2013
    Location:
    Chicago, IL
    #4
    She (it’s a she on my devices, British to be exact) is 1/1 on my new mbp.

    Siri, go to amazon dot com.

    Boom! Saved my fingers some heavy lifting.
     
  4. gnipgnop macrumors 65816

    Joined:
    Feb 18, 2009
    #5
    Certainly puts to rest all the claims of "no improvement" in Siri, as well as a sizable dent in the claim that Alexa is more accurate.
     
  5. ikir macrumors 65816

    ikir

    Joined:
    Sep 26, 2007
    #6
    Absolutely not. Sure whiners will always say that.
    Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
    Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
    Cortana is quite bad for example, want to bash something? Cortana is there.
     
  6. Boatboy24 macrumors 6502a

    Joined:
    Nov 4, 2011
    Location:
    1 Infinite Loop
    #7
    This is good news. Though it isn't shocking, since the test was performed on a phone. I'd like to see the same thing done a Google Home, Echo, and HomePod. (Is there a Cortana enabled speaker? I don't know). I imagine Alexa would fare much better, though not certain.
     
  7. Mike MA macrumors 68000

    Mike MA

    Joined:
    Sep 21, 2012
  8. djcerla macrumors 68000

    djcerla

    Joined:
    Apr 23, 2015
    Location:
    Italy
    #9
    Things are going to improve quickly, as Apple poached Google’s chief of AI and search. Never understimate Apple’s will and resources.
     
  9. H3LL5P4WN macrumors 68000

    H3LL5P4WN

    Joined:
    Jun 19, 2010
    Location:
    Pittsburgh PA
    #10
    I'd like to see a bit more fair comparison; ask Alexa on a Kindle (or even Android) device, and Cortana on a Windows PC (or even Android device).

    That being said, I'm glad we have some quantifiable numbers that Siri is indeed improving. For my purposes, she's always been spot on.
     
  10. TheFluffyDuck macrumors 6502

    Joined:
    Jul 26, 2012
    #11
    I call BS. I have never got anywhere near 70% accuracy. Mind you I am not an American living in the Bay Area.
     
  11. DrJohnnyN Suspended

    DrJohnnyN

    Joined:
    Jan 27, 2010
    #12
    You use Siri for work? You must not do anything meaningful.
     
  12. Stella macrumors G3

    Stella

    Joined:
    Apr 21, 2003
    Location:
    Canada
    #13

    Personal Data mining is irrelevant.

    Siri still sucks.
     
  13. Solomani macrumors 68040

    Solomani

    Joined:
    Sep 25, 2012
    Location:
    Alberto, Canado
    #14
    Siri is getting smarter! :p

    …. and I don't understand why Alexa is so over-rated. Both Google and Siri perform better than Alexa… and yet Alexa gets the lion's share of the hype.
     
  14. adammusic macrumors 6502

    adammusic

    Joined:
    Jun 6, 2007
    #15
    Siri performs at like a 10% for me.
    I despise her.
     
  15. MrGimper macrumors 603

    MrGimper

    Joined:
    Sep 22, 2012
    Location:
    Andover, UK
    #16
    You know what's more annoying than "whiners?", people who label anyone who has an opinion or criticism that doesn't align with their own, a "whiner".

    Glad you use it that much for work, you must have a job where you have to set a lot of timers :cool:
     
  16. DNichter macrumors 604

    DNichter

    Joined:
    Apr 27, 2015
    Location:
    Philadelphia, PA
    #17
    I am glad to see some improvement. It's very impressive that Apple is able to keep pushing forward in this area without mining our personal data. Command is all I use Siri for so I am happy to see it being the best here. Looking forward to the Shortcuts app in the fall for even more control.
     
  17. err404 macrumors 68020

    Joined:
    Mar 4, 2007
    #18
    I regularly use both Google and Siri, and while there is little doubt that Google is much better, neither are anywhere near the 80% mark. In both cases I need to query only very topics that I already know will work and I need to phrase things very carefully.
    Basically in order to work, I had to be trained as much as the AI.
     
  18. PickUrPoison, Jul 25, 2018
    Last edited: Jul 25, 2018

    PickUrPoison macrumors 68020

    Joined:
    Sep 12, 2017
    Location:
    Sunnyvale, CA
    #19
    So Siri’s not nearly the train wreck MacRumors forum posters endlessly and loudly claim it to be.

    Never could have guessed...

    Most improved, usually beaten by google, usually better than Alexa, and always beats Cortana; a lot of folks here aren’t going to want to hear this. Time to shoot the messenger I suppose.
     
  19. BlandUsername macrumors regular

    BlandUsername

    Joined:
    Jan 18, 2016
    Location:
    In yer server room, fixing the tubes
    #20
    I've given up on all these AI's. Disable Cortana and Siri on everything. Refuse to use Amazon or google phones or speakers. Life is better when you dont repeat yourself 10 times for the wrong answer.
     
  20. jerryk macrumors 601

    Joined:
    Nov 3, 2011
    Location:
    SF Bay Area
    #21

    Alexa is the user interface to a much more powerful system. Question answering is one thing, but Alexa controls for all devices in your home, online shopping, etc. Also, the cost for entry is very low. $20 or so, depending on the current sale, for an Echo Dot. So Alexa is found in many more households, or as in my case, in many more rooms of the same house.
     
  21. wigby macrumors 68000

    Joined:
    Jun 7, 2007
    #22
    Compared to what? According to these results, Siri is a little or a lot better than both Alexa and Cortana. Siri is not quite as Good as Google Assistant but these tests are important because they invalidate all hyperbolic claims that Siri is worthless and/or Google Assistant is leaps and bounds ahead. We all have the feeling that Siri has fallen behind so these numbers seem to validate that. They do not validate opinions that claim Siri is either far ahead or way behind the competition.
     
  22. pika2000 macrumors 601

    Joined:
    Jun 22, 2007
    #23
    What surprises me is how bad Cortana is. I mean Microsoft looks like an incompetent company considering they are a primarily software company and how AI is part of their vision. And it shows how bad Amazon does as well, considering how many conversations their machines have eavesdropped. Compared to those, Siri looks amazing considering Apple's method of data collection.

    And I do expect Google to be better than this. I mean consider the amount of data they have mined so far, it’s no excuse that they are not hugely better than Siri. What an embarrassment.
     
  23. antonis macrumors 68020

    antonis

    Joined:
    Jun 10, 2011
    #24
    Sure you don't mean "Never underestimate Apple's will and Google's resources" ?
     
  24. H.E. Pennypacker macrumors 6502

    H.E. Pennypacker

    Joined:
    Oct 23, 2017
    Location:
    NYC
    #25
    And yet Siri still cannot effectively do voice to text without messing up 75% of the time.
     

Share This Page