Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test

[Rogifan]

Siri on HomePod can’t do X therfore it’s at the bottom of the pack when it comes to doing X. Well duh. If the product you want is the smartest AI assistant then HomePod isn’t for you. Then again Apple isn’t marketing HomePod as that. If you want to argue that HomePod should be a different product than it is, fine, but that’s a different conversation. Reviewing/grading HomePod on things you know it can’t do and Apple never said it could do is dumb.

 

[PJS_SoCal]

In a new test shared today by Loup Ventures, Apple's HomePod was put through its paces in categories including Siri, sound quality, and ease of use. For Siri, Loup Ventures' Gene Munster reported that while the AI assistant understood 99.4 percent of queries asked of it, it answered only 52.3 percent of them correctly. Loup Ventures tested three separate HomePods and asked 782 queries total.

Compared to previous tests of rival speakers, HomePod is "at the bottom of the totem pole" in the AI assistant performance category. Google Home answered 81 percent correctly, Amazon's Alexa answered 64 percent correctly, and Microsoft's Cortana answered 57 percent correctly.

Munster broke this information down further, stating that Siri is good for "local" and "commerce" queries, like asking about nearby coffee shops or assisting in buying new shoes. In this area, Siri beats Alexa and Cortana but still falls behind Google Assistant on Google Home.

Despite the low percentage of correctly answering the 782 total queries asked of it, Munster said Siri's overall performance rose above expectations "given the limited scope of HomePod's music focus."

Chart via Loup Ventures​

The researchers explained that over time HomePod and Siri should grow to match, or surpass, rival assistants by simply adding query domains like calendar, email, calling, and navigation.
As discovered in the research, where HomePod excelled was its "superior" listening skills. The HomePod allows users to speak at a normal volume, even when music from the speaker is particularly loud, and Siri will pick up on the voice and hear the user. "This was HomePod's most stellar feature," according to Munster.

Loup Ventures also favored HomePod's sound quality, saying that "it sounds incredible." Like other reviews and impressions of the device, Munster's enjoyment of the HomePod's audio quality was countered with the speaker's lackluster Siri performance, which he thinks will be changing fairly soon.
Visit Loup Ventures to read more of the information gathered from the HomePod "smart speaker gauntlet," including the publication's prediction for demand and market share of HomePod through the rest of 2018.

Article Link: Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test

This test is bogus. The “correct answer” rate of 52% for HomePod includes questions that HomePod doesn’t yet support. This is wildly deceptive. The test should measure how accurate HomePod is at handling the queries it supports.

HomePod is a version 1.0 product. Of course it has fewer capabilities - for now!

Categorizing HomePod’s non-answers to questions it doesn’t yet support as wrong answers is crap.

 

[falainber]

That’s what we get for Apple keeping our information private. I feel like it’s a very small price to pay and I am not surprised at all that Google leads the way here. Their sole focus as a company is mining your data for advertising. Ultimately, it doesn’t affect me as I don’t use voice assistants this way. Smart home stuff and music is really hit. I’m sure the HomePod works well for these.

Are you sure? Nowhere in this report did they indicate that any queries had anything personal in nature. Is it possible that Siri is just dumb?

 

[Rogifan]

Oh my at that navigation. Lol.

I didn’t buy HomePod so it could give me driving directions. I think Apple is pretty upfront about what skills HomePod has outside of playing music. No one buying it should have any illusions about its skills. Now if it’s terrible at what Apple says it can do then that’s another story but we’re not hearing that because people are reviewing the product as they wished it to be rather than what it is.

 

[Heineken]

It does whats required for basic tasks

It's certainly no alexa but for me doesn't need to be

64% for Alexa isn't exactly groundbreaking either.

 

[jjjjjooooo]

Also, in any case that iPhone-based Siri would bring up Google search results, HomePod would reply, "I can't get the answer to that on HomePod," which forces you to use your phone or give up on the question altogether.

This is interesting. I wonder if Apple is building its own knowledge base, perhaps on its way to an Apple search engine.

They already have their Applebot
web crawler and are presumably learning how to deliver real-time results with Apple News.

 

[Wags]

Since the Home Pod is targeting the Apple ecosystem they will expect to get away with this performance. A big miss for 6 years of development and even with initial launch.

 

[dennyc69]

So Apple announced this as a Music device, they didn't announce it as a "Smart Home" device, yet reviewers and other websites focus on how bad Siri is doing and how it doesn't work with Spotify. Apple is just getting into this and nobody knows publicly what Apple has in store for the HomePod, so for a first gen product, running BETA software, it does great.

People seem to forget how not so great Alexa was when it first came out or how the Google speaker had its problems as a "Smart Home" device, but that's okay. We humans seem to want to downplay everything at first because it doesn't suit what the individual thinks it should be. Give the product at least a year, then attack it if you don't like it.






In a new test shared today by Loup Ventures, Apple's HomePod was put through its paces in categories including Siri, sound quality, and ease of use. For Siri, Loup Ventures' Gene Munster reported that while the AI assistant understood 99.4 percent of queries asked of it, it answered only 52.3 percent of them correctly. Loup Ventures tested three separate HomePods and asked 782 queries total.

Compared to previous tests of rival speakers, HomePod is "at the bottom of the totem pole" in the AI assistant performance category. Google Home answered 81 percent correctly, Amazon's Alexa answered 64 percent correctly, and Microsoft's Cortana answered 57 percent correctly.

 

[The Game 161]

I didn’t buy HomePod so it could give me driving directions. I think Apple is pretty upfront about what skills HomePod has outside of playing music. No one buying it should have any illusions about its skills. Now if it’s terrible at what Apple says it can do then that’s another story but we’re not hearing that because people are reviewing the product as they wished it to be rather than what it is.

this is the thing why would people want to use homepod for directions? not like you would use it to take you right there.

 

[PJS_SoCal]

Siri on HomePod can’t do X therfore it’s at the bottom of the pack when it comes to doing X. Well duh. If the product you want is the smartest AI assistant then HomePod isn’t for you. Then again Apple isn’t marketing HomePod as that. If you want to argue that HomePod should be a different product than it is, fine, but that’s a different conversation. Reviewing/grading HomePod on things you know it can’t do and Apple never said it could do is dumb.

Exactly. It should be judged on how well it does what it does, not on what it doesn’t yet do.

 

[DNichter]

Are you sure? Nowhere in this report did they indicate that any queries had anything personal in nature. Is it possible that Siri is just dumb?

Yes. Google feeds all of its data into its AI algorithms. They understand how people use machines better than anyone.

 

[Heineken]

I just love this time of the year when the uninformed think they know something.

 

[falainber]

Siri on HomePod can’t do X therfore it’s at the bottom of the pack when it comes to doing X. Well duh. If the product you want is the smartest AI assistant then HomePod isn’t for you. Then again Apple isn’t marketing HomePod as that. If you want to argue that HomePod should be a different product than it is, fine, but that’s a different conversation. Reviewing/grading HomePod on things you know it can’t do and Apple never said it could do is dumb.

Then, of course, Apple is marketing HP as a music speaker but it does not even have stereo sound (and when it gets it I suspect it will be inferior to regular stereo systems because HP was not really designed for stereo)

 

[SoApple]

Apple gets many things right.
However, siri isn’t of one the them.

The only non apple gadget I own is a google home and the assistant is best in class.

 

[Heineken]

Then, of course, Apple is marketing HP as a music speaker but it does not even have stereo sound (and when it gets it I suspect it will be inferior to regular stereo systems because HP was not really designed for stereo)

You have no idea what counts as stereo, do you?

 

[puma1552]

This thing just seems like one of Apple's most half-baked 'me too' products. All it is is a speaker (albeit a nice one) and the same crappy Siri that's on my phone. And I suppose some Homekit stuff, but that's it.

That's literally all Apple could muster being this late to the game? I still feel these things answer a question nobody asked. IMO these type of things complicate your life, not simplify it; it's another device that needs to be configured/updated/dealt with in your ecosystem and doesn't really make any appreciable difference in your life. If I want a speaker, I have a Bose SoundDock that has required me doing absolutely nothing but turning it on when I've wanted music over the last 10 years. No setting up with my phone, computer, wifi, etc. It's not part of some wifi ecosystem or whatever.

A smart home is a nice idea in theory, but it's really just adding complexity to your life to replace tried and true things like keyed door locks and light switches that have worked for decades without even needing to be thought about. The simpler my life is, the better, and I don't think this thing makes my life any simpler.

 

[funnyboy88]

Gene Munster! That’s the analyst who for years swore an Apple television set was right around the corner.

 

[Heineken]

Apple gets many things right.
However, siri isn’t of one the them.

The only non apple gadget I own is a google home and the assistant is best in class.

Listen, if people actually bothered to get informed and educated themselves on the matter we wouldn't be having this "discussion".

This is what Siri on HomePod can do: help.apple.com/homepod/#/apd81431732e

It should have been tested based on that, because that's what it is able to to at this moment. No more, no less.

 

[hazeydavey]

That’s what we get for Apple keeping our information private. I feel like it’s a very small price to pay and I am not surprised at all that Google leads the way here. Their sole focus as a company is mining your data for advertising. Ultimately, it doesn’t affect me as I don’t use voice assistants this way. Smart home stuff and music is really hit. I’m sure the HomePod works well for these.

I really wanted to like it.

Doesn’t touch my Sonos 5 soundwise. Hate the closed environment, as I love Sirius, Spotify, amazon, tune in etc.

And Siri just crapped out on nearly every album request in my library. Really disappointed. Packed up to go back.

So wanted to like it, just not quite there sound wise or usefulness wise. Look forward to this in a few years. But my sonos system just superor.

 

[The Game 161]

apple has always said sound speaker first which does tasks that takes advantage of the sound afterall...

 

[44267547]

Apple gets many things right.
However, siri isn’t of one the them.

The only non apple gadget I own is a google home and the assistant is best in class.

I Agree that Siri can be inconsistent. But a lot of users likely don’t you utilize Siri to its full potential besides asking basic questions or commands. Given that Google is faster at deciphering and seems more accurate, I don’t think Siri is as bad as Some make it appear to be. And this is coming from somebody who uses Siri every single day with my iPhone, AirPod commands, Apple Watch commands, etc.

 

[Rogifan]

Exactly. It should be judged on how well it does what it does, not on what it doesn’t yet do.

I just think those are two separate conversations. Even if the reviewers conclusion is don’t buy this the review should still be based on whether the product does what it advertises or not and and if it does it well or not.

 

[PJS_SoCal]

Are you sure? Nowhere in this report did they indicate that any queries had anything personal in nature. Is it possible that Siri is just dumb?

Actually, the test is downgrading the HomePod for not being able to answer queries in categories is explicitly does not support, such as giving direactions and dealing with calendars. Nearly half of the questions in this test are for tasks that the HomePod doesn’t support - by design.

This means that for the tasks HonePod does support, it is nearly 100% accurate! That’s hardly dumb, especially when Apple is free to add more capabilities via future device and cloud updates.

 

[Naraxus]

......and I am not surprised at all that Google leads the way here. Their sole focus as a company is mining your data for advertising...

Nice misinformation there. Google doesn't sell your personal data any more than Apple does.

 

[spiddyman007]

So long as it can play/stop/skip music that's the end of the smart technology I would want. I don't care that others answer questions better, I wouldn't use it for that.

What about my music. Not just apples music