Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test

[TheAustrianGuy]

And if you consider that Siri is even worse and basically useless in German (except for setting timers), you can imagine how frustrating it is to “use” it.

I’m fully invested in Apple gear.
And I can’t even articulate how disappointed I am in Siri.

 

[Donoban]

When you have too much time on your hands. Lolz. What an era we live in.

 

[Apple_Robert]

In my opinion, that percentage is way too low for the price of the pod.

 

[WatchFromAfar]

Siri on HomePod can’t do X therfore it’s at the bottom of the pack when it comes to doing X. Well duh. If the product you want is the smartest AI assistant then HomePod isn’t for you. Then again Apple isn’t marketing HomePod as that. If you want to argue that HomePod should be a different product than it is, fine, but that’s a different conversation. Reviewing/grading HomePod on things you know it can’t do and Apple never said it could do is dumb.

The problem is Apple/tech fans know Homepod isn't a digital assistant first but try telling the general public they should buy a Google Home if they want a voice assistant and an Apple Homepod for great music; the truth is they are only going to buy one and people can think things will get better via software updates (blind faith?) but Apple are playing a dangerous game here.

 

[Pulpdiction]

Pretty much like most product releases since the accountants took over, underwhelming performance and a beyond premium price. Apples reputation over the past 5 years has dropped through the floor for new product releases and each release is followed by the inevitable dot release 2 weeks later to fix basic issues that should have been fixed in testing. That or ‘beta’ versions on some devices, which is 5he new code word for we wanted to include this s but ran out of time to get the release out in time for the next gift buying season.

 

[BruceEBonus]

You’re riffing. The numbers Apple just put up in Q1 proves Apple has insane loyalty and pricing power. $800 iPhone ASP. $88.3BB in revenue in 90 days. $20B in profit, the most ever by Apple and the most by any company ever. 13% sales growth y/y. 99% customer satisfaction. I could go on.

Software and ecosystem still matter and Apple customers will pay for iOS.

The Honor 9 is not an iPhone. Apple customers won’t go Android, at least now. Not everyone wants a budget phone. If it works well for you, great. Some of us pay for software.

It’s just like cars. Any car will get you from point A to point B. But people by Mercedes and BMWs for a reason.

Every thing you say is true. Tho the difference with the Honor is that it’s a high end phone but at a budget price. I think it’s the beginning of a new era. Quality and quantity. Never seen before. When people realise you are getting exactly the same camera performance, processed speed, features and build for a quarter of the Samsung and Apple price will it be enough to be the new “smart”?

Like you say in Apple’s case it’s ecosystem (read Big Brother) keeps people cocooned and isolated quite happily and it’s a wrench and hassle to break free. In Android case despite the fact it’s universal and limitless you get Samsung aficionados who will have nothing but the best. The best in both cases being increasingly questionable as Chinese competition catches up and is overtaking. Yet. As you say no matter what the logic and sense pervading - blind loyalty and brand obsession will win regardless no matter what.

Fans will get the HomePod they have no say in th matter. They’d buy a cat if it answered to the name of Apple. The question is whether people ... especially new ones ... will keep coughing up double costs for stuff that’s been out for ages yet offer no more. I’m a useless predictor of such trends because I apply normal rules. But in Apple’s case rules don’t apply. Call back again in 3 months and Ill tell you then.

(“Cut it here!”- M.R. Ed)

 

[jbizzybeetle]

In a new test shared today by Loup Ventures, Apple's HomePod was put through its paces in categories including Siri, sound quality, and ease of use. For Siri, Loup Ventures' Gene Munster reported that while the AI assistant understood 99.4 percent of queries asked of it, it answered only 52.3 percent of them correctly. Loup Ventures tested three separate HomePods and asked 782 queries total.

Compared to previous tests of rival speakers, HomePod is "at the bottom of the totem pole" in the AI assistant performance category. Google Home answered 81 percent correctly, Amazon's Alexa answered 64 percent correctly, and Microsoft's Cortana answered 57 percent correctly.

Munster broke this information down further, stating that Siri is good for "local" and "commerce" queries, like asking about nearby coffee shops or assisting in buying new shoes. In this area, Siri beats Alexa and Cortana but still falls behind Google Assistant on Google Home.

Despite the low percentage of correctly answering the 782 total queries asked of it, Munster said Siri's overall performance rose above expectations "given the limited scope of HomePod's music focus."

Chart via Loup Ventures​

The researchers explained that over time HomePod and Siri should grow to match, or surpass, rival assistants by simply adding query domains like calendar, email, calling, and navigation.
As discovered in the research, where HomePod excelled was its "superior" listening skills. The HomePod allows users to speak at a normal volume, even when music from the speaker is particularly loud, and Siri will pick up on the voice and hear the user. "This was HomePod's most stellar feature," according to Munster.

Loup Ventures also favored HomePod's sound quality, saying that "it sounds incredible." Like other reviews and impressions of the device, Munster's enjoyment of the HomePod's audio quality was countered with the speaker's lackluster Siri performance, which he thinks will be changing fairly soon.
Visit Loup Ventures to read more of the information gathered from the HomePod "smart speaker gauntlet," including the publication's prediction for demand and market share of HomePod through the rest of 2018.

Article Link: Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test

Seems Siri response record about same percentage as my husband.

 

[interessiert]

The heaven is already between us - and we don't realize.
Intelligent scrap - improvement not in sight
[doublepost=1518346917][/doublepost]

Seems Siri response record about same percentage as my husband.

You hit the point - maybe It's a must to open a Siri jokes corner yours would be my favorite

 

[chfilm]

For me, living in Germany, the worst thing with Siri is that she doesn’t understand English pronounciation when I’m speaking German with her.
For example I wanna tell her in German words: “in things, create a new to do called bla bla”.
Then she wouldn’t understand the TH in things and gets “sings” and therefore doesn’t get what app I’m talking about.
Same disaster with music. It’s not possible to ask her for ANY music artist or song because obviously most of them have English names. I’m sure this is the number 1 holdup why it’s not available in non English speaking countries yet!

On the other hand- this gives me hopes that they’re working to fix this in the near future and hopefully not just for music commands...

Yes I could set her to English, but then I feel like a douche speaking to my Siri in English all the time.

For all of you out there who are frustrated: I really recommend you try out the shopping list app
“Grocery”
https://itunes.apple.com/app/id1195676848
If you have an Apple Watch, this just works AMAZING! I use it all the time and am really happy with Siri on that one.
Maybe someone could try if that works with the homepod as well yet?

 

[Heineken]

It's a smart speaker being compared to smart speakers. Are you saying the comparison should only include the things the HomePod can do and ignore what the other speakers can do? Apple chose to emphasize the speaker over the smart. Over time, that might end up being considered the better decision. But no comparison should be based on what 1 contender does or doesn't have. Run whatcha brung.

But to suggest it get special consideration because Apple gimped it's initial capability is... yeah, no. I mean, should the Harmon Invoke's Cortana get a pass on commerce since it did just as badly as Siri did with Navigation? No, it shouldn't and neither should the HomePod.

Most of us don't buy based on these comparisons. We buy based on our use case. If I don't need nav on my smart speaker, why would I care if it did badly? I wouldn't. Trying to stack the deck for Siri really doesn't help the perception of Apple's assistant. Siri's rep is a fairly cemented consensus in the general sense. Whether you agree with the majority or the minority... still doesn't matter. People buying the HomePod know Siri's limitations and buy anyway. My brother is an Apple freak. He purchased the HomePod to Apple Music from a dedicated iPad. No Siri. The speaker is going to sell to those who want it. No need for a handicap in comparisons.

It is clearly stated in the manual what it does at this point. How hard is it to grasp? But no one wants a fair comparison. If it failed at what it says it can do 50% of the time then it would be a problem.

 

[phatspider]

And thats why mine is going back on Wednesday

 

[Heineken]

And thats why mine is going back on Wednesday

Just to buy it again after they push a software update? Not very smart on your part either.

 

[gnipgnop]

LOL...using the term "accuracy" and then admitting that they asked questions from categories that they already knew weren't supported on HomePod.

 

[B60boy]

Every time Siri just ‘searches the web for that’ it makes me less and less likely to actually bother to interact with it.

Siri internet search after it scan your phone via Spotlight and Apple News is Google Search.
[doublepost=1518351140][/doublepost]

In Chicago we have this thing called traffic. Can turn a 45 run to the airport to double that. It's nice to kno before you leave.

Dilly Dilly from a fellow Chicagoian!

 

[Bacillus]

Siri and Apple Maps. Both huge colossal failures.

2 licenses could overcome that.
Admit Different (like OLED...) !!

 

[PeLaNo]

It's just easier to ask on your phone. Or are we playing a convenience card again so that its a chore to ask it on the phone?

What if you sut in a room with HomePod iPhone and  watch. And only HomePod picking up your voice (since it has the best microphone). It’s about convenience to just say Hey Siri and request whatever you want on device you want.
And it isn’t that hard for Apple to do this since they control all devices.
[doublepost=1518352910][/doublepost]

Sorry to say that I think you are living in a dream world if you don't think that Google collects more information about its users than does Apple.

Google collects just about everything, they even scan your emails if you use gmail. However, Apple may have the most restrictive data sharing arrangement in the industry.

Take, for example, Siri and Google's voice assistant. Request made through Siri are completely masked as to identity even to Apple, while Google's voice assistant tags everything you request with your identity.

In terms of sharing user information Google is probably the worst in the industry, while Apple may be the best at protecting customer information and identity.

You forgot Facebook I think.

 

[spriter]

Siri sucks. I don’t think I’ve used it in over a year.

Quite. Despite the number of Apple products I have owned and do own, that Siri hasn't really made a significant step forward since inception is pathetic. It's an expensive one-dimensional product which is a shame given what it could be.

Without the search backend, if you want information out of a smart speaker then it's obvious where you go. Personally, I wanted something that could do smart device control, good audio quality and be able to answer questions (great for children to explore and help with things like homework or fact-checking).

 

[AldoRaine]

I measure the success of any purchase by the price / years of use.
There's nothing in the Home Pod that can't be improved upon over the next year or so that requires new hardware.
In that respect the Home Pod is a clear success. The software, and AI, are fixable.
Can you say the same about the other smart speakers?

Oh yeah, that is why i am so happy with my 2011 iPhone 4s purchase. My promoted so very helpful assistant Siri still can be fixed. Just need to be a bit more patient. With iOS 12 my 4s will have a renaissance and finally, Siri will work as promised. That is true value! Good point.

Oh wait ...

(hardware wise, I admit, the 4s was a very refined product)

 

[Bacillus]

Don’t worry! Siri will be fixed by 2040!

The sad part is that Apple didnt even invent Siri. They bought it and have nothing done with it since.

As "6000 employees in an Indian office" were to save the Apple Maps dilemma, I expect 600k far-eastern operators to ultimately solve the Siri catastrophy.
Hire Different !

 

[robjulo]

It that is your criteria, you’d be elated with the Sonos One, since it doesn’t need fixing, sounds great and is far superior to the Home Pod in terms of services and price.

I measure the success of any purchase by the price / years of use.
There's nothing in the Home Pod that can't be improved upon over the next year or so that requires new hardware.
In that respect the Home Pod is a clear success. The software, and AI, are fixable.
Can you say the same about the other smart speakers?

 

[iChrila]

For me, living in Germany, the worst thing with Siri is that she doesn’t understand English pronounciation when I’m speaking German with her.

Yes I could set her to English, but then I feel like a douche speaking to my Siri in English all the time.

Same problem here. I set Siri to English, otherwise Siri just isn't useful for anything expect setting alarms or timers. Using a voice assistent in public in Germany is cringe anyways , so I only use her at home and don't mind talking in English.

I've never used another voice assistant so far and I don't know if other voice assistants handle this better but I really hope Apple will fix this.

 

[AppleScruff1]

It is clearly stated in the manual what it does at this point. How hard is it to grasp? But no one wants a fair comparison. If it failed at what it says it can do 50% of the time then it would be a problem.

When I saw demonstrated at the Apple Store, it was about. 50-50 shot of playing the correct song and usually took 2-3 Hey Siri’s to activate, it reminded me of a Microsoft presentation back in the old days when the computer blue screened.

The HomePod sounded good, but dealing with the frustration of Siri?

 

[EvilEvil]

2 licenses could overcome that.
Admit Different (like OLED...) !!

But you have to turn the brightness all the way up to see the screen LOL.

 

[laz232]

I measure the success of any purchase by the price / years of use.
There's nothing in the Home Pod that can't be improved upon over the next year or so that requires new hardware.
In that respect the Home Pod is a clear success. The software, and AI, are fixable.
Can you say the same about the other smart speakers?

Until Siri 2.0 becomes a HomePod II only feature (ref Siri non-support on iPhone 4(IIRC)) ... Siri has been crap for years with little signs of improvement. Recently used Google Assistant on an Android phone (at 1/3 price of my iPhone SE) whilst SE was in for battery replacement. Can confirm that Google Assistant is significantly better on way cheaper HW.

 

[DoctorTech]

I'm honestly surprised that it did that well. It's fairly well known that Siri's third-party capabilities are limited to certain domains, and on HomePod, that's even further restricted to just three: Messaging, Lists, and Notes. And it can't even do those yet... it's a test-only feature in the new iOS Betas.

However, in my experience, if you want to do something that it actually can do, Siri (on iOS, anyway) is better than its competitors. It's far more capable of interpreting my request (regardless of how I ask) and subsequently helping me complete the actual task, filling in any missing information. I hope Apple is willing and able to open that up someday soon, because if Siri had the skills to match the smarts, I think she'd easily top the bunch.

Siri is also unmatched in the category of not mistakenly activating all the damned time— no, I didn't say anything even close to "Alexa," you stupid Echo! Perhaps, though, that's strategic on Amazon and Google's part... the more often it activates, the more data it can gather from you.

I had noticed the same thing about unintentional activation on my Echos. I bought my first Echo about 2-3 months after they first hit the market in 2015. Things were fine and it worked well for a year or two but all of a sudden it seemed to start unintentionally activating a lot when I or a voice in a movie said anything remotely close "lex" (i.e. Lexington, Alex, etc.). However, more recently (the past 2 months roughly) that problem seems to have gotten a LOT better.

My guess is that Amazon pushed an update to all Echos about a year ago to make them more sensitive and then got complaints about them being too sensitive so they pushed a second update to back off the sensitivity a bit. My echo still accidentally activates every once in a while but no where near as often as it was doing just a couple months ago.