Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test

[citysnaps]

So? My point is Apple is marketing this as a great sounding speaker with limited voice capabilities. Review it for what it is. And then let people decide if that’s what they want or not. Apple isn’t out there overselling Siri on this thing. The company is mostly focusing on sound quality.

I'd go one step further and say Siri on HomePod provides a user interface letting users access their music on the device. And it does that very well. For me, to perfection, so far.

Apple is not promoting Siri on HomePod as a general purpose digital assistant. HomePod, and its Siri user interface, is about music. Full stop.

 

[sirozha]

If you're content (and apparently desire) accessing every aspect of the service from a web browser, you're pretty much Google's ideal customer. Most of us don't share your view on the matter, and increasingly people have given up keeping their own MP3 collections in favor of convenient (and legal) subscription services. Amazon even retired the option to upload your own music in December.

I don't think $25/year is unreasonable, but yes Google offers a similar service for free (until they decide to axe it like Amazon did).

Why they chose to omit home sharing from HomePod, I don't know - but no feature like that exists on any competing device either, you have to have your music uploaded to their services online and I don't see much outrage about that.

Home sharing is also proprietary. Sonos can stream from a file share, or if you want to use Metadata tags, Sonos can stream from a Plex server and probably from a DLNA server.

I've been comparing Apple's way to Plex for a while now. I think I will go all the way Plex and Sonos vs Apple's ever shifting directions and half-baked solutions (iTunes with airport express speakers, vs iTunes with Home Sharing, vs iTunes in the cloud, vs Apple Music) I've given Apple a decade to clean up their mess, but it's getting messier and messier.

HomePod married to an individual Apple Music account is not a solution. It's only good for single people living alone in a condo - like Tim Cook.
[doublepost=1518441692][/doublepost]

Siri is dumber than Google Assistant, that's I can understand. But dumber than Alexa, Cortana and Bixby? Siri is so much behind these new comers. Bixby can control every aspect of a phone and Siri says "I cannot __". Shame on Apple. It cannot make a smart AI even with all resources available.

Give them a break. They were building a new campus.

 

[spacemnspiff]

I am not surprised to see Siri under-performed.

But what I am really surprised in how poorly Google Home and Alexa performed in many instances, only place Google did really well is navigation.
From all the horn tooting about Google Home and Alexa on here and elsewhere, my expectations were sky high, for the most part to be above 90+% in all categories, even after Google cataloging most of the internet, the command category performance was horrid. Commerce and Information categories were also not great.

And going by the results, Siri for the most part is usable and be helpful in basic functions and not too far behind the competition. I was expecting Siri to be in the 10% range compared to the competition, pleasantly surprised that it it's better than that.

 

[ChrisCW11]

Of course, Apple did NOTHING to improve Siri, they just focus on form over function again. Well, I mean they made it a very functional speaker, but as a home assistance device it is a fail.

Also, because Apple doesn't actually adapt or improve based on customer input there is little chance HomePod will ever rival Amazon or Google's devices which are constantly changing and improving based on customer usage stats and feedback. You can claim that is a violation of privacy, but at the end of the day learning from its mistakes is now anything improves. Apple is too arrogant to learn from their mistakes which is why Siri is so slow to evolve or improve in the same amount of time that both Amazon and Google have risen to dominate this market.

So, if you want HomePod to use like an Echo or Google Home, forget about it, it won't work at all different then your iPhone does currently, but if you were in the market for a Sonos competitor , this seems to be a better choice purely for its sound alone

 

[^^BIGMac]

Well I'm pretty much tired of hearing about the HomePod.

There are like 20 threads dealing with this toy.

 

[ChrisCW11]

Unfortunately, the article doesn't give many specifics about the questions it used. If they were all simple yes/no questions, 50-some percent would be abysmal. But if they're open-ended or obscure questions, it's probably not too bad.

I am sure the test compered the same response from Siri as with google Home or Echo with the same questions. I mean if the test was to ask Siri "Hey, what is the meaning of life" and Siri didn't answer it without context in relation to the competition's response, then that was a stupid test.

 

[Stella]

So, this is my prediction: Siri is going to beat the other voice assistants with a clear margin in less than 24 months. When 2020 comes around, we are going to look at Amazon’s voice assistant devices and wonder why we bought them. We are also going to wonder if we want to keep them in our homes, because the convenience of their voice assistance has waned, while their security risks dominate the conversation around these products. Google’s voice assistants will fare better, but they will also be dwarfed by Apple and Siri.

Other voice assistants aren't going to stand still, they will be developed further. In order for Siri to beat Google Assistant, Apple are going to have to put in a hefty amount - something we have yet to see.
[doublepost=1518444510][/doublepost]

You can do this on HomePod as well, using the exact same question.

Except it uses Apple Maps, which, if you're outside the states, isn't so great.

I've given Apple maps several tries, but Google Maps and Navigon just are more reliable.

 

[DNichter]

If the Siri was any good, you can bet Apple would be marketing it as the smartest AI assistant around.
I guess one of the problems for Apple is that it is like a dinosaur when trying to open things up to make them useful.
[doublepost=1518432838][/doublepost]
you don't use any of googles services?

Nope.

 

[AppleScruff1]

So, this is my prediction: Siri is going to beat the other voice assistants with a clear margin in less than 24 months. When 2020 comes around, we are going to look at Amazon’s voice assistant devices and wonder why we bought them. We are also going to wonder if we want to keep them in our homes, because the convenience of their voice assistance has waned, while their security risks dominate the conversation around these products. Google’s voice assistants will fare better, but they will also be dwarfed by Apple and Siri.

Why will it go like this? Because Siri will mature into a true Voice Assistant - you know, an entity able to actually help. And not a function primarily there to help you buy stuff on Amazon or to collect information on you.

So two more years for Siri to be on top? It's been 7 years and still on the bottom.

 

[decafjava]

Why isn't privacy more important to people? I'd rather do without an AI assistant at all if it meant protecting my privacy.

 

[ONEGRUMPYOLDMAN]

In a new test shared today by Loup Ventures, Apple's HomePod was put through its paces in categories including Siri, sound quality, and ease of use. For Siri, Loup Ventures' Gene Munster reported that while the AI assistant understood 99.4 percent of queries asked of it, it answered only 52.3 percent of them correctly. Loup Ventures tested three separate HomePods and asked 782 queries total.

Compared to previous tests of rival speakers, HomePod is "at the bottom of the totem pole" in the AI assistant performance category. Google Home answered 81 percent correctly, Amazon's Alexa answered 64 percent correctly, and Microsoft's Cortana answered 57 percent correctly.

Munster broke this information down further, stating that Siri is good for "local" and "commerce" queries, like asking about nearby coffee shops or assisting in buying new shoes. In this area, Siri beats Alexa and Cortana but still falls behind Google Assistant on Google Home.

Despite the low percentage of correctly answering the 782 total queries asked of it, Munster said Siri's overall performance rose above expectations "given the limited scope of HomePod's music focus."

Chart via Loup Ventures​

The researchers explained that over time HomePod and Siri should grow to match, or surpass, rival assistants by simply adding query domains like calendar, email, calling, and navigation.
As discovered in the research, where HomePod excelled was its "superior" listening skills. The HomePod allows users to speak at a normal volume, even when music from the speaker is particularly loud, and Siri will pick up on the voice and hear the user. "This was HomePod's most stellar feature," according to Munster.

Loup Ventures also favored HomePod's sound quality, saying that "it sounds incredible." Like other reviews and impressions of the device, Munster's enjoyment of the HomePod's audio quality was countered with the speaker's lackluster Siri performance, which he thinks will be changing fairly soon.
Visit Loup Ventures to read more of the information gathered from the HomePod "smart speaker gauntlet," including the publication's prediction for demand and market share of HomePod through the rest of 2018.

Article Link: Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test

52.3% is an "F" in grade school, middle school, high school and college. This is Apple's best after 8 years of developing Siri???

 

[cfurlin]

Why did you highlight text in red without actually answering anything? Weird.

*woosh*

 

[mike0steff]

That navigation score is no surprise.......terrible

 

[samden]

That’s what we get for Apple keeping our information private. I feel like it’s a very small price to pay and I am not surprised at all that Google leads the way here. Their sole focus as a company is mining your data for advertising. Ultimately, it doesn’t affect me as I don’t use voice assistants this way. Smart home stuff and music is really it. I’m sure the HomePod works well for these.

This is a pretty big misconception that gets yelled around the Apple echo chamber a little too much. What really holds Siri back is the complete lack of API, developers can't do jack with Siri and it's why Siri in turn can't do anything. I've developed multiple skills for both Alexa and Google Assistant and none of them invade privacy whatsoever. In fact Alexa/GA developers aren't provided any user data other than a randomized user ID, so any info we might want has to be provided by the user. Apple wants everything to be within their eco and by their rules instead of providing options to users and developers, that's why Siri sucks.

 

[cyb3rdud3]

Siri needs to be able to do home automation - Home Kit.
Siri is bad when it comes to accents, especially the Scottish accent

It does do HomeKit, I have no issues with that at all.

It also understand Anglo Dutch

 

[sparky08]

It only costs 3.5 times as much as Google Home. What did you expect?

 

[Leguna24]

Well I'm pretty much tired of hearing about the HomePod.

There are like 20 threads dealing with this toy.

Maybe it's because others don't view it as a toy? Perhaps you should take a break from forums that challenge your ability to ignore content that you don't care for, for your own well-being?

 

[^^BIGMac]

Maybe it's because others don't view it as a toy? Perhaps you should take a break from forums that challenge your ability to ignore content that you don't care for, for your own well-being?

Oh thanks for that.

Very predictable response.

 

[Leguna24]

Oh thanks for that.

Very predictable response.

Oh, you're very welcome.

Of course it's predictable, most would agree, I have no doubt. What else would one expect when you complain about a very current product being heavily represented on a brand-specific message board/forum?

 

[DNichter]

This is a pretty big misconception that gets yelled around the Apple echo chamber a little too much. What really holds Siri back is the complete lack of API, developers can't do jack with Siri and it's why Siri in turn can't do anything. I've developed multiple skills for both Alexa and Google Assistant and none of them invade privacy whatsoever. In fact Alexa/GA developers aren't provided any user data other than a randomized user ID, so any info we might want has to be provided by the user. Apple wants everything to be within their eco and by their rules instead of providing options to users and developers, that's why Siri sucks.

Siri works fine for me, but can definitely be improved. My only point was that Google's should be the best because they have the most consumer data to be pulled into their AI algorithms. They understand how we interact with machines better than any of them because that is their business model. Their sole reason for existing is to take consumer data through their products and services and feed them ads.

 

[jsmith78]

I swear by all that’s holy, the only thing Siri is good at it training me not to use Siri. The first one’s from today. No clue why I decided to try it again. The second is one of my favorites from Siri’s Greatest Misses:

Where were you when you asked for Walgreen's?, as there doesn't appear to BE a Walgreen's on King Street in or around Santa Cruz Ca, the closes to that is 1718 Soquel Ave, Santa Cruz, CA 95062, and this is confirmed via Google Maps (as far as I can tell)

Now I do think that Siri has some serious issue with non-English place names (businesses, streets, etc) and some slight trouble with voice recognition (at least on my iphone)
[doublepost=1518464204][/doublepost]

I've read surveys asking women to describe the perfect man. One dominant want in most of them: "a great listener that doesn't try to actually solve any of their issues." Maybe this product is aimed at women?
[doublepost=1518312491][/doublepost]

That's right! Change the test so that Apple can win it. That's the only way any such tests should be given.

After all, if you test only the queries it supports, it will get ALL OF THEM right. Perfection... exactly as we all knew it would be.
[doublepost=1518312761][/doublepost]

I have to believe that sometimes some of us write down an Apple defense and don't read it before posting. That was a classic.

No one wants to "change the test", but it is disingenuous to imply low reliability for features that the product doesn't have. There is a different between features and reliability.

One wouldn't complain that the Civic has a malfunctioning transmission because you cant get it to shift into 8th gear! or that it's 8th gear only works 0% of the time, when it doesn't HAVE an 8th gear.
You can point out that it would be nice if it had more gears, but that's NOT a reliability issue, the gears it does have work well.

And the same concept can be applied here, the voice features the HomePod has work 67% of the time (not amazing, but middle of the pack) while also stating that the other assistants have more voice features.

 

[deadoctopi]

If you ask Siri to turn the volume up does she say, "I didn't get that, would you like me to search the web for how to turn the volume up?" No...Ok. Argh!!!

You have to assume it’ll hear you over any other sound or music when you’re standing right next to it.

 

[jsmith78]

52.3% from a product (Smart Speaker) being touted as the end all where voice commands are concerned? Pretty darn pathetic!
My ex-Girlfriend responds more!
Without sounding like a hater i'm fed up with Apple releasing products in their infancy (Beta and other stages) IMHO just to appease the Board Of Directors.
Looks like this is just another early release leaving the customer in the lurch waiting for the fix. Typical.

As I stated in another reply; 52.3% is disingenuous, as that includes as FAILURES, features that it doesn't have.
you cannot tally those, those should be counted as an N/A; Its "actual" score is 67%, also from the ad's I've seen the HomePod is being advertised primarily based on it's sound and music, not as a central home IA computer.

 

[Kabeyun]

Where were you when you asked for Walgreen's?, as there doesn't appear to BE a Walgreen's on King Street in or around Santa Cruz Ca, the closes to that is 1718 Soquel Ave, Santa Cruz, CA 95062, and this is confirmed via Google Maps (as far as I can tell)

That is entirely my point. I was in a New England town with a Walgreens on King Street, driving ON King Street, AND, best of all, that Walgreen’s is in my contacts. So naturally Siri thinks I’m talking about some nonexistent Walgreens 3000 miles away. Good. Bloody. Lord.

I even, out of inexplicable charity, then asked her my location, just in case location services was wonky. Here’s the answer I got:

Never have I been more tempted to throw my iPhone out the car window.

Siri is a blithering idiot, full stop. I have an entire photo album dedicated to her idiocy. I love most things Apple, but either Siri needs to be made smarter or Apple needs to give up, swallow hard, and hammer out a licensing deal with Google.

 

[AgentAnonymous]

I think Siri is more than adequate for what most need it for the HomePod. Is it better than Google Home? No. Does it need to be better than the competition? Not necessarily. I use Siri all the time and I really don’t have any general complaints, aside from not understanding some of the things I ask it. None of these voice assistants are perfect.

It kind of sounds like you're doing your best here to rationalize HomePod's horrible performance, but don't really have anything besides vague "well its okay not to be the best" sentiments.
[doublepost=1518473557][/doublepost]

I measure the success of any purchase by the price / years of use.
There's nothing in the Home Pod that can't be improved upon over the next year or so that requires new hardware.
In that respect the Home Pod is a clear success. The software, and AI, are fixable.

Overpriced, average audio, bad AI.
[doublepost=1518473670][/doublepost]

People seem to forget how not so great Alexa was when it first came out or how the Google speaker had its problems as a "Smart Home" device, but that's okay. We humans seem to want to downplay everything at first because it doesn't suit what the individual thinks it should be. Give the product at least a year, then attack it if you don't like it.

This is some incredible damage control. If you have to "give the product a year", then it's a horrible product that should've been released a year later.