Siri on HomePod Asked 800 Questions and Answered 74% Correctly vs. Just 52% Earlier This Year

[Just sayin...]

Hey [Siri/Alexa/Google/Whavever] - turn off the entry lights in 30 minutes. NOT ONE can accomplish this...

However, Bixby CAN “put the TV to sleep in 30 minutes” - where Apple TV, Alexa and Google home hub cannot.

 

[AustinIllini]

But...music sounds better 100% of the time. Sounds good to me. I just bought another.

I have a Sonos 5.1 setup so that's not a big problem for me

 

[digitalexplr]

I feel like the issue with these devices is we're taking a completely incorrect approach to designing the UX.

The current approach seems to be to make a ridiculously gimped command line that has no "help", "man", "find", "grep", pipes, or variables.

I think a way better approach would be to take an apps approach, like a smartphone.

Me: "What were we talking about?"
Device: "We weren't talking about anything. What app would you like to launch?"
Me: "Oh, right. What apps do I have installed?"
Device: "Currently installed are weather, stocks, messages, phone..."
Me *cutting off*: "Open messages. Who texted me last?"
... so on...
Me: "Close apps" or "Goodnight" or "Goodbye" or something along those lines.

These AIs suck for general purpose. They need context and we need context. They need to be able to contextually tell us what our options are. They need to prompt us about what they can do. Apps can provide that context.

I think you are on the right track.

I also think that the AI is already better than most humans. In a day, how many humans give you the correct answer the first time with out saying, "could you repeat that", or "I don't know what you are talking about", etc. Yet we expect AI to be 100% correct 100% of the time?

 

[AustinIllini]

"Miles behind Alexa." LOL. I guess you didn't actually read the article.

I was referring specifically to the whole package. Almost every smart device supports Alexa and skills make her a lot more powerful than Siri.

It's more than just voice recognition.
[doublepost=1545324404][/doublepost]

I think you are on the right track.

I also think that the AI is already better than most humans. In a day, how many humans give you the correct answer the first time with out saying, "could you repeat that", or "I don't know what you are talking about", etc. Yet we expect AI to be 100% correct 100% of the time?

I think I saw somewhere human recognition is around 95% for clear speakers. That is to say, if you speak clearly to me like you speak to alexa/siri/google assistant, I'll recognize you for 19/20 words. Google Assistant is the closest to that bar but not quite there yet.

 

[ignatius345]

Probably the same reason Apple hasn't opened up the Watch to 3rd Party watch faces, they don't want it to turn into an utter mess.

You could have made the same argument about the iPhone when it came out, and I'm sure a lot of people at Apple did make that argument. Apple had full control of everything the iPhone did because there were no third-party apps. And until the totally revolutionary idea of the App Store, iPhone was a niche product. Is iPhone a "mess" now?

HomePod is now is a lot like the first versions of the iPhone. Nice, minimal, and non-expandable.

 

[WilliamG]

I have a Sonos 5.1 setup so that's not a big problem for me

HomePod isn’t competing with Sonos 5.1.

 

[AustinIllini]

Asked siri to turn on Podcasts while I was driving. "You have to unlock your phone first" was Siri's answer, and when I unlocked it (while driving) it didn't open the app. So freaking stupid.

Yeah, Android has a smartlock feature where, when the device detects you are connected to a specific bluetooth device, wifi network, or the device is on your body, the phone simply stays unlocked.

That being said, I don't believe I ever had that issue while driving with Siri

 

[Vanilla35]

You cannot equate smart assistance with keyboards or music. They're not even remotely comparable. A more appropriate comparison would be using only autocomplete to write a sentence, or getting a piece of software to recognise a tune from you simply humming it.


EDIT: Thinking about it, what was the failure rate of early typewriters, and what percent quality of the original recording was audible on original gramophones? Hmmm.


The trouble with smart assistants is they have to understand the question, understand the context - and then get it right first time - which is the hard part.

How often do you use search engines, and how often is the result you want the first link? In my case at least, rarely.

The point is it's not perfect, but it is getting there. Give it some time.
[doublepost=1545319574][/doublepost]

Try looking up 'Skyrim Very Special Edition'

The simple matter of the comparison is that you either type in what you want manually, or you use voice assistant. Understandably the voice assistant technology is much more complicated, however until people can use voice assistant very reliably, people will default back to manual entries. These differences show the varying thresholds between products.

Specifically when speaking about a smart speaker, which practically interacts on voice alone, how useful is one going to be if it doesn't break the threshold of accuracy and recognition? That's a very expensive and depressing purchase if the only thing it is designed to do, it doesn't even do well.

I don't know if my experience is more common, but when I type something into google search, it pulls up exactly what I'm looking for, and has done so for 5-6 years now. And I type in some pretty elaborate sh**

 

[truthertech]

I was referring specifically to the whole package. Almost every smart device supports Alexa and skills make her a lot more powerful than Siri.

It's more than just voice recognition.

Actually, Alexa is much less powerful than Siri. She can speak many fewer languages and, as this test and others show, when it comes to integration with your key items like your phone and commands, Alexa is left in the dust. Don't confuse the hype you hear about Alexa "skills." Surveys show that most people don't even activate or use even one of the 4000 skills. Yes, she can "fart on command," but EVERY survey shows that Siri already does what the vast majority of smart phones/speakers want their digital assistant to do- control Homekit stuff, get you directions, read messages, check on traffic, play music/podcasts, set reminders/alarms, control features on your phone, answer basic questions, do math, etc. Sir works great in the car, with the TV, etc.

And there's even bigger, equally meaningless hype about putting Alexa into things like your refrigerator and toaster. This is all a gimmick that manufacturers are shoving into things but people don't care about, like 3D TV. They are now advertising a toilet with Alexa built in, but I don't recall the huge clamor from people wanting to have Amazon control their flushes. Also, people may later regret as Amazon has much less security built in, and voice controlled switches is another point of failure. Apple has no intention of playing in that silly arena.

https://www.usatoday.com/story/tech/2018/01/11/do-we-really-need-alexa-powered-toilet/1024646001/

 

[gnomeisland]

Why do we even talk about any of these percentages like they're even remotely acceptable?

If your keyboard only produced the correct character 80% of the time, or your touchscreen discarded 20% of touches, or your speaker only streamed 80% of a song, not a soul would be okay with that.

Siri on your iPhone or Mac is a funny novelty. The moment it becomes the only way you can interact with a device? Who on earth would find that acceptable? Maybe you'd be willing to buy such a device if it was priced like a whoopie cushion or silly putty (so, you know, ~$5), but for $300?

And this isn't even getting into the fact that this basic test is EASY! We're not expecting the devices to be productive contributors every conversation - these are tailored queries - softballs - meant to be easily within the realm of what the devices are supposed to be capable of doing!

I think Apple would agree with your idea if not the degree you're taking it. That is why they positioned HomePod as a speaker first (and compared to other mid-range, airplay speakers it is appropriately priced) unlike Alexa, or Google Home, which was always an assistant first. While Apple clearly sees more value Siri than just a "novelty", I don't see them pushing it as a primary interface until it reaches the accuracy & dependability of a keyboard, mouse, or multi-touch (something around 95+%—we all hit accidental key or click the wrong thing from time to time).

 

[AustinIllini]

HomePod isn’t competing with Sonos 5.1.

No, but its incompatibility with Siri is yet another reason not to get a Homepod.
[doublepost=1545325141][/doublepost]

user didn't know how to use quotes so I'm having trouble quoting his post

Tim? Is that you?

But seriously, you can do so much more with Alexa and she supports so many more products. I appreciate the language thing because that certainly matters but Apple just needs to catch up on the functionality side.

 

[macduke]

The biggest barrier to buying this for me is that it's only tied to my account, so what makes it any better than asking my Apple Watch, iPhone, iPad, or Mac which are usually within range? Especially on the latest devices like the XS and 3rd generation iPad Pro which can hear my voice from further away. I see the HomePod as a community device. Other people in my house should be able to access their things, depending on voice, and I don't want my kids using it to text my business contacts or who knows what could happen accidentally. I get that may be a hard thing to accomplish, but that's what I think it needs to be. Especially when you consider that everyone else in the house has an Apple device capable of using Siri, this thing becomes only as useful as a speaker. A good speaker, but still a speaker. And honestly I use Siri a lot more now that I have a Series 4 Watch and can just raise my wrist and it responds quickly. It works great with HomeKit. My wife has one too (her first Apple Watch) and now HomePod is practically useless for our family.

Maybe, but with skills Alexa becomes much more capable. That influences why I think Alexa is far ahead of Siri.

Either way, there is no question GA is the best in the business and it's not particularly close.

Aren't Alexa Skills similar to Siri Shortcuts?

 

[haganah]

I just don't understand why there aren't third-party apps for HomePod yet. I'd buy an interactive story game thing, or Duolingo... there are tons of possibilities given the quality of the hardware in the device already.

Because public data points indicate nobody uses these yet. I'm not saying they are not valuable in the long run, but right now third party skills are a failure that Amazon dresses up to be a success. A while back you were able to see how many skills had reviews and the number was very small. Of that number, a handful had five star reviews. Amazon realized this and started proactively pushing users to rate the skills but if you look on their site at the top skills it is things like "whale sounds".

I'm not sure what it takes to get there. Amazon is invested in Alexa but the third parties are not as invested in their skills is my humble opinion. When that relationship becomes more equitable I think we will see really great third party skills out there.

 

[Ma2k5]

It can only do so much with a wall around privacy, Apple will never overtake them probably because they are so far behind

This is one limiting factor however there are other problems. A lot of leadership at Apple actually want to rebuild Siri from scratch because it exists as a poor implementation with a very bad code base - further limiting its potential integration. This video released today goes into explaining the history of Siri:

 

[ignatius345]

I was referring specifically to the whole package. Almost every smart device supports Alexa and skills make her a lot more powerful than Siri.

It's more than just voice recognition.
[doublepost=1545324404][/doublepost]
I think I saw somewhere human recognition is around 95% for clear speakers. That is to say, if you speak clearly to me like you speak to alexa/siri/google assistant, I'll recognize you for 19/20 words. Google Assistant is the closest to that bar but not quite there yet.

And we're much, much better at context, so we can usually infer very well that 20th word, or catch the gist of a conversation even when our recognition rate gets lower (in a loud room or whatever). The AIs in these speakers aren't so great at inferring intent yet.

 

[Vanilla35]

You really need to look up why we tend to use the QWERTY keyboard my dude, then you need to consider that Voice input is still, basically, in that typewriter stage.

I know you and others on here think it should be perfect, but trust me, this is a really, really, hard thing to get 100% right.

I think the problem people have with it is that tech companies are releasing these new products (smart speakers) and features that make it seem like the voice assistant technology is fully baked, when it isn't at all. Then when people get it and have issues they're irritated.

 

[AustinIllini]

The biggest barrier to buying this for me is that it's only tied to my account, so what makes it any better than asking my Apple Watch, iPhone, iPad, or Mac which are usually within range? Especially on the latest devices like the XS and 3rd generation iPad Pro which can hear my voice from further away. I see the HomePod as a community device. Other people in my house should be able to access their things, depending on voice, and I don't want my kids using it to text my business contacts or who knows what could happen accidentally. I get that may be a hard thing to accomplish, but that's what I think it needs to be. Especially when you consider that everyone else in the house has an Apple device capable of using Siri, this thing becomes only as useful as a speaker. A good speaker, but still a speaker. And honestly I use Siri a lot more now that I have a Series 4 Watch and can just raise my wrist and it responds quickly. It works great with HomeKit. My wife has one too (her first Apple Watch) and now HomePod is practically useless for our family.

Yeah, voice recognition on Google Assistant is an amazing addition. The fact that my wife and I are differentiated is really nice. I'm not thrilled they store that information but at the same time it's worth not stumbling around on one account at a time.

Aren't Alexa Skills similar to Siri Shortcuts?

Yes, but Alexa skills are way ahead of the curve right now.
[doublepost=1545325478][/doublepost]

And we're much, much better at context, so we can usually infer very well that 20th word, or catch the gist of a conversation even when our recognition rate gets lower (in a loud room or whatever). The AIs in these speakers aren't so great at inferring intent yet.

Well said. Context is still difficult on all these devices. There's sort of a stream of conscience that needs to happen for these devices to keep context.

 

[kaneda]

My wife laughed at me when I used Siri. It is not because I am talking to the phone, because it never gave me the answer I am looking for.

 

[ignatius345]

Because public data points indicate nobody uses these yet. I'm not saying they are not valuable in the long run, but right now third party skills are a failure that Amazon dresses up to be a success. A while back you were able to see how many skills had reviews and the number was very small. Of that number, a handful had five star reviews. Amazon realized this and started proactively pushing users to rate the skills but if you look on their site at the top skills it is things like "whale sounds".

I'm not sure what it takes to get there. Amazon is invested in Alexa but the third parties are not as invested in their skills is my humble opinion. When that relationship becomes more equitable I think we will see really great third party skills out there.

Nobody uses these things yet because Amazon "skills" probably aren't so hot. Apple is a lot better at this kind of thing than Amazon. And doesn't take a lot of imagination to come up with sound-based apps that could be genuinely useful or entertaining, and if potential HomePod developers were provided with access to the great hardware in the HomePod I'm sure they could come up with something better than whale sounds. Hell, I'd pay for Duolingo right now, or voice training app, or an interactive storytelling game.

 

[DNichter]

It really depends how you use the product. The thing is, if you don't want to use Google or Alexa for music, they can coexist with HomePod. They are cheap and fit anywhere with a power plug.

I have the Google Home Mini and I just don't really have much use for it. The Amazon Dot, on the other hand, has about 3 skills I use regularly and that makes it worthwhile to even keep one in the same room as the HomePod. I mainly use the HomePod for music, controlling lights, and alarms. The combo works for me.

I have noticed that the HomeKit stuff on the HomePod have gotten faster recently. It is nice not to have a delay when waiting for the lights to come on after giving it a command. The Dots (at least the versions I own) are much slower with my Hue Bulbs than Siri.

If I had a child that was asking me "why is the sky blue" kind of questions all the time, I would definitely have more use for Google Home. Most of the time when I have questions about something, I have a phone in hand.

Yea I am with you, I just personally didn't like crossing streams and having multiple voice assistants like this. I originally had Echo's, but ended up getting two HomePod's and moved all of my smart home stuff to HomeKit and I have been really happy. Since I moved to HomePod's, all my smart home stuff has been perfect.

 

[truthertech]

No, but its incompatibility with Siri is yet another reason not to get a Homepod.
[doublepost=1545325141][/doublepost]


Tim? Is that you?

But seriously, you can do so much more with Alexa and she supports so many more products. I appreciate the language thing because that certainly matters but Apple just needs to catch up on the functionality side.

Can you give us a list of the things that Alexa can do that Siri can't that large numbers of people are clamoring for? Yes, we know that Siri is not built in to toilets, and I saw a $600 microwave on Amaxon.com today with Alexa built in that instead of pushing the popcorn button on the front, I can say "Alexa cook the popcorn," but what else are Apple users being deprived of? Maybe, more of us will give up our privacy so we can have Jeff flush the toilet for us if you can assemble a list like that. Also, there's a reason that Amazon keeps getting lower scores on the functionality comparisons. Just pointing that out.

 

[pika2000]

My problem with Siri has always been not correctly recognizing my voice. After all these years, Siri is just not there yet, and recognizing your voice for the correct command is literally the make or break of a virtual assistant. It cannot be 80% good enough. It has to be close to 100% for me to actually start using it regularly.

Google has a huge head start thanks to Google Voice. Imo Apple will always behind, although the gap can diminish over time.

 

[itguy06]

Good to see Siri improving. And Alexa, like all Amazon stuff is still ****.

 

[haganah]

This is one limiting factor however there are other problems. A lot of leadership at Apple actually want to rebuild Siri from scratch because it exists as a poor implementation with a very bad code base - further limiting its potential integration. This video released today goes into explaining the history of Siri:

I really want them to do just that but I also think this video is full of nonsense. Most particularly it cites Stone Temple so if you look up what that it, it's an SEO firm that makes a lot of money off of Google. And their test was filled with a lot of random fact questions that Google will always win because it's a search engine. Doing Search and doing Assistant work are very different - the two should live side by side. But Google will own search forever...or at least I say those things and end up totally surprised by how tech evolves.

 

[Shanghaichica]

W

Actually, Alexa is much less powerful than Siri. She can speak many fewer languages and, as this test and others show, when it comes to integration with your key items like your phone and commands, Alexa is left in the dust. Don't confuse the hype you hear about Alexa "skills." Surveys show that most people don't even activate or use even one of the 4000 skills. Yes, she can "fart on command," but EVERY survey shows that Siri already does what the vast majority of smart phones/speakers want their digital assistant to do- control Homekit stuff, get you directions, read messages, check on traffic, play music/podcasts, set reminders/alarms, control features on your phone, answer basic questions, do math, etc. Sir works great in the car, with the TV, etc.

And there's even bigger, equally meaningless hype about putting Alexa into things like your refrigerator and toaster. This is all a gimmick that manufacturers are shoving into things but people don't care about, like 3D TV. They are now advertising a toilet with Alexa built in, but I don't recall the huge clamor from people wanting to have Amazon control their flushes. Also, people may later regret as Amazon has much less security built in, and voice controlled switches is another point of failure. Apple has no intention of playing in that silly arena.

https://www.usatoday.com/story/tech/2018/01/11/do-we-really-need-alexa-powered-toilet/1024646001/

I think the difference is that Alexa is not connected to any mobile OS so it’s limited in what it can do that requires integration with a mobile phone e.g sending text messages, reading notifications etc. It’s slightly better than Siri for general knowledge. I mean for example at least it can do conversions between different time zones. The HomePod can’t.

I think it doesn’t help the conversation when there are different versions of Siri on different devices and they all do different things. However the HomePod Siri is the least capable.
[doublepost=1545326517][/doublepost]

The biggest barrier to buying this for me is that it's only tied to my account, so what makes it any better than asking my Apple Watch, iPhone, iPad, or Mac which are usually within range? Especially on the latest devices like the XS and 3rd generation iPad Pro which can hear my voice from further away. I see the HomePod as a community device. Other people in my house should be able to access their things, depending on voice, and I don't want my kids using it to text my business contacts or who knows what could happen accidentally. I get that may be a hard thing to accomplish, but that's what I think it needs to be. Especially when you consider that everyone else in the house has an Apple device capable of using Siri, this thing becomes only as useful as a speaker. A good speaker, but still a speaker. And honestly I use Siri a lot more now that I have a Series 4 Watch and can just raise my wrist and it responds quickly. It works great with HomeKit. My wife has one too (her first Apple Watch) and now HomePod is practically useless for our family.


Aren't Alexa Skills similar to Siri Shortcuts?

Not really. Siri shortcuts are generally confined to automation or tasks. Alexa skills can be anything, including games.
[doublepost=1545326657][/doublepost]

Nobody uses these things yet because Amazon "skills" probably aren't so hot. Apple is a lot better at this kind of thing than Amazon. And doesn't take a lot of imagination to come up with sound-based apps that could be genuinely useful or entertaining, and if potential HomePod developers were provided with access to the great hardware in the HomePod I'm sure they could come up with something better than whale sounds. Hell, I'd pay for Duolingo right now, or voice training app, or an interactive storytelling game.

Of course Alexa has useful sound based skills. White noise, ambient noise etc.

Also there are interactive story telling skills for Alexa already.