Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test

Discussion in ' News Discussion' started by MacRumors, Feb 10, 2018.

  1. MacRumors macrumors bot


    Apr 12, 2001

    In a new test shared today by Loup Ventures, Apple's HomePod was put through its paces in categories including Siri, sound quality, and ease of use. For Siri, Loup Ventures' Gene Munster reported that while the AI assistant understood 99.4 percent of queries asked of it, it answered only 52.3 percent of them correctly. Loup Ventures tested three separate HomePods and asked 782 queries total.

    Compared to previous tests of rival speakers, HomePod is "at the bottom of the totem pole" in the AI assistant performance category. Google Home answered 81 percent correctly, Amazon's Alexa answered 64 percent correctly, and Microsoft's Cortana answered 57 percent correctly.


    Munster broke this information down further, stating that Siri is good for "local" and "commerce" queries, like asking about nearby coffee shops or assisting in buying new shoes. In this area, Siri beats Alexa and Cortana but still falls behind Google Assistant on Google Home.

    Despite the low percentage of correctly answering the 782 total queries asked of it, Munster said Siri's overall performance rose above expectations "given the limited scope of HomePod's music focus."

    Chart via Loup Ventures

    The researchers explained that over time HomePod and Siri should grow to match, or surpass, rival assistants by simply adding query domains like calendar, email, calling, and navigation.
    As discovered in the research, where HomePod excelled was its "superior" listening skills. The HomePod allows users to speak at a normal volume, even when music from the speaker is particularly loud, and Siri will pick up on the voice and hear the user. "This was HomePod's most stellar feature," according to Munster.

    Loup Ventures also favored HomePod's sound quality, saying that "it sounds incredible." Like other reviews and impressions of the device, Munster's enjoyment of the HomePod's audio quality was countered with the speaker's lackluster Siri performance, which he thinks will be changing fairly soon.
    Visit Loup Ventures to read more of the information gathered from the HomePod "smart speaker gauntlet," including the publication's prediction for demand and market share of HomePod through the rest of 2018.

    Article Link: Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test
  2. Kebabselector macrumors 68030


    May 25, 2007
    Birmingham, UK
    So long as it can play/stop/skip music that's the end of the smart technology I would want. I don't care that others answer questions better, I wouldn't use it for that.
  3. Mobster1983 macrumors 6502a

    Sep 8, 2011
  4. Relentless Power macrumors Penryn

    Relentless Power

    Jul 12, 2016
    I think Siri is more than adequate for what most need it for the HomePod. Is it better than Google Home? No. Does it need to be better than the competition? Not necessarily. I use Siri all the time and I really don’t have any general complaints, aside from not understanding some of the things I ask it. None of these voice assistants are perfect.
  5. unobtainium macrumors 68020

    Mar 27, 2011
    Siri sucks. I don’t think I’ve used it in over a year.
  6. jasvncnt macrumors 6502


    Jan 20, 2011
    New Jersey
    What good is "superior" listeng skills if it can't answer many of the questions?
  7. No. 44 macrumors member

    No. 44

    Oct 29, 2016
    Electric Larryland
    "Munster broke this information down further, stating that Siri is good for "local" and "commerce" queries, like asking about nearby coffee shops..."

    So an Apple product is good at helping one to locate a Starbucks?

    Go figure... ;)
  8. SeattleMoose macrumors 68000

    Jul 17, 2009
    Der Wald
    Siri is Apple's biggest failure. They have had years to make it at least 80% functional and have failed to do so. Just like the ATV, it seems to be just a "hobby" for Apple.
  9. nfl46 macrumors 604

    Oct 5, 2008
  10. Polaroid macrumors 6502a

    Oct 1, 2013
  11. Pelea macrumors 6502

    Oct 5, 2014
  12. chublet macrumors member

    Feb 10, 2018
    I occasionally try Siri to see if there has been any improvements but have found it totally useless across all my devices, a HomePod with a screen that could access my local library easily would be a must buy, a HomePod that could never play the song I want because it can’t find it is a waste of money
  13. KidAKidB macrumors 6502


    Oct 1, 2014
    You say that, but I guarantee that if Siri was better you would be using it for more than the basics.
  14. RudySnow macrumors 6502


    Aug 27, 2016
    Tyler, TX
    Me: “Hey Siri, why are you so terrible at answering questions?”

    Siri: “I found several articles on the web about crustaceans.”
  15. MrGuder macrumors 68030

    Nov 30, 2012
    If you ask Siri to turn the volume up does she say, "I didn't get that, would you like me to search the web for how to turn the volume up?" No...Ok. Argh!!!
  16. foobarbaz macrumors 6502a

    Nov 29, 2007
    Frankly, I'd be happy if my Echo understood "louder" or "pause" 100% of the time. And if it didn't randomly start reading train schedules or weather reports while I'm watching TV.

    Everybody's needs are different, but I just want basic (mostly music) controls to work reliably. I don't need smart audio (as I almost always find visual and hyperlinked information more appealing than linear voice output).
  17. D-a-a-n macrumors regular

    Mar 22, 2010
    So true, I have an ATV 4th gen and indeed it just seems a hobby. Already have it for 2 years and the Siri functionality still doesn't work in Belgium...
  18. palmerc2 macrumors 68000


    Feb 29, 2008
    Los Angeles
    I can only imagine the amount of gloating on these forums if Siri was #1 on the list instead of Google.
  19. buklauu macrumors newbie

    Sep 14, 2017
    That's the first step. If the device can't even hear you, it's never going to be able to answer you. Apple seems to have gotten the first part of the equation right, but is lagging significantly on the second.
  20. Seoras macrumors 6502


    Oct 25, 2007
    Scotsman in New Zealand
    I measure the success of any purchase by the price / years of use.
    There's nothing in the Home Pod that can't be improved upon over the next year or so that requires new hardware.
    In that respect the Home Pod is a clear success. The software, and AI, are fixable.
    Can you say the same about the other smart speakers?
  21. artfossil macrumors 65816


    Oct 5, 2015
    You’d lose that bet. I’ll be using my HomePod to escape the world, not seek it out.

    Or, as Simon & Garfunkel sang:

    “I get the news I need on the weather report.
    I can gather all the news I need on the weather report.”
  22. nibfiledotcom macrumors member


    May 3, 2015
    Columbus, OH, USA
    I'm honestly surprised that it did that well. It's fairly well known that Siri's third-party capabilities are limited to certain domains, and on HomePod, that's even further restricted to just three: Messaging, Lists, and Notes. And it can't even do those yet... it's a test-only feature in the new iOS Betas.

    However, in my experience, if you want to do something that it actually can do, Siri (on iOS, anyway) is better than its competitors. It's far more capable of interpreting my request (regardless of how I ask) and subsequently helping me complete the actual task, filling in any missing information. I hope Apple is willing and able to open that up someday soon, because if Siri had the skills to match the smarts, I think she'd easily top the bunch.

    Siri is also unmatched in the category of not mistakenly activating all the damned time— no, I didn't say anything even close to "Alexa," you stupid Echo! Perhaps, though, that's strategic on Amazon and Google's part... the more often it activates, the more data it can gather from you.
  23. The Game 161 macrumors P6

    The Game 161

    Dec 15, 2010
    It does whats required for basic tasks

    It's certainly no alexa but for me doesn't need to be
  24. dkemmerer macrumors newbie


    Feb 10, 2018
    Unfortunately, the article doesn't give many specifics about the questions it used. If they were all simple yes/no questions, 50-some percent would be abysmal. But if they're open-ended or obscure questions, it's probably not too bad.
  25. DNichter, Feb 10, 2018
    Last edited: Feb 11, 2018

    DNichter macrumors 604


    Apr 27, 2015
    Philadelphia, PA
    That’s what we get for Apple keeping our information private. I feel like it’s a very small price to pay and I am not surprised at all that Google leads the way here. Their sole focus as a company is mining your data for advertising. Ultimately, it doesn’t affect me as I don’t use voice assistants this way. Smart home stuff and music is really it. I’m sure the HomePod works well for these.

Share This Page