Some sure, but it doesn't explain why 'create reminder to x' results in 'i've changed your reminder'. That's just broken. Or why it correct recognizes 'american airlines reservations' but can't reliably find that in my contact list.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Siri Answers 83% of Questions Correctly in Test, Beating Alexa But Trailing Google Assistant
- Thread starter MacRumors
- Start date
- Sort by reaction score
Some sure, but it doesn't explain why 'create reminder to x' results in 'i've changed your reminder'. That's just broken. Or why it correct recognizes 'american airlines reservations' but can't reliably find that in my contact list.
That is actually good to hear. It would be interesting to know what's different from how YOU use it... versus how others do so. Is it the scope of questions? Individual or Regional language patterns?
It amuses me that Alexa still hasn’t caught up with Siri since the big improvements in 2018. It’s no surprise, since Apple poached Google’s lead AI guy in early 2018.
https://www.nytimes.com/2018/04/03/business/apple-hires-googles-ai-chief.html
https://www.nytimes.com/2018/04/03/business/apple-hires-googles-ai-chief.html
... and that is at least part of the problem. Consumers want to ask lots of questions. Alexa can answer a wide range (but not all) of those questions. Siri can answer a much smaller subset.
Yeah.... Saying "here's what I found on the web" every time doesn't count as answering a query in my book
Yes, that sounds like crap. Siri works well for me.
You forgot schedule/create an appointment in my calendar. That NEVER works correctly for me.
[doublepost=1565902539][/doublepost]
Lucky you. The only thing Siri works for me is to start a timer or asking it the weather. Otherwise, its easier to just type it myself.
FWIW I have Hue lights and Wemo plugs hooked up to both Siri and Alexa. That’s about the only thing I use the assistants for: lights and turning on the window ACs.
I had to rename the lights with distinctive weird words because Alexa never understood what I was saying. Even before I did that though, Siri would always understand me when I used my phone. It’s just that it’s a pain to use the phone to turn things on. And I’m too damn lazy to stand up and use the light switch
The only reason I use the Echo Dots is because at $25 a pop on Black Friday, I can cheaply sprinkle them around the house. If Apple ever comes out with an inexpensive HomePod, the Dots are all going on eBay.
I had to rename the lights with distinctive weird words because Alexa never understood what I was saying. Even before I did that though, Siri would always understand me when I used my phone. It’s just that it’s a pain to use the phone to turn things on. And I’m too damn lazy to stand up and use the light switch
The only reason I use the Echo Dots is because at $25 a pop on Black Friday, I can cheaply sprinkle them around the house. If Apple ever comes out with an inexpensive HomePod, the Dots are all going on eBay.
You can take the aggregate of all the Macrumors, AppleInsider, 9to5, and Apple Reddit posts and get a sense of at least Apple related forum results. I think it's erroneous to completely throw out anecdotal because there are some valid inferences made there.
Also do keep in mind though that Loup Ventures (and Gene Munster) is a heavy bull on Apple.
Siri: “I’m sorry, I can’t do that on Apple Watch.”
That’s my daily experience with Siri. And random activation.
If I have to use my phone to ask even a simple question, why is Siri on the watch? So I can tell her to set a timer?
[doublepost=1565906169][/doublepost]
That’s my daily experience with Siri. And random activation.
If I have to use my phone to ask even a simple question, why is Siri on the watch? So I can tell her to set a timer?
[doublepost=1565906169][/doublepost]
True. I prefer an answer even if I have no idea if it’s right or wrong or the source...Yeah.... Saying "here's what I found on the web" every time doesn't count as answering a query in my book
Why didn’t they test Alexa on an Amazon device, like a $50 Kindle, instead of on an iOS device?
In an annual test comparing Google Assistant, Siri, and Alexa on smartphones, Loup Ventures' Gene Munster found that Siri was able to correctly answer 83 percent of questions, beating Alexa but trailing behind Google Assistant.
Munster asked each digital assistant 800 questions during the test to compare how each one responded. Alexa answered 79.8 percent of questions correctly, while Google Assistant answered 92.9 percent of questions correctly.
Compared to last year, Siri has seen improvement. During the July 2018 test, Siri answered 79 percent of questions correctly compared to the 83 percent of questions answered right this time around. Alexa last year was at 61 percent while Google Assistant was at 86, so there have been digital voice assistant improvements across the board.
This test covered smartphones specifically, comparing iPhones and Android devices. Munster says that smartphones were isolated from smart speakers because while underlying technology is similar, "use cases vary." Siri was tested on an iPhone running iOS 12.4, Google Assistant on a Pixel XL, and Alexa in the iOS app.
Questions were based on five categories and all assistants were asked the same 800 questions. Each question set was designed to "comprehensively test a digital assistant's ability and utility." Some of the sample questions across each of the categories:
[*]Local - Where is the nearest coffee shop?
[*]Commerce - Order me more paper towels.
[*]Navigation - How do I get to Uptown on the bus?
[*]Information - Who do the Twins play tonight?
[*]Command - Remind me to call Jerome at 2 pm today
Siri did best in the command, local, and navigation categories, faring less well in the information and commerce categories. Siri actually won out in the command category, but trailed behind Google Assistant in other categories.
Munster says that the continued rate of improvement "continues to surprise" based on the notable improvements that each voice assistant has demonstrated over the course of the last few years.
In the future, Loup Ventures expects to see further improvements from extending the feature sets of each voice assistant.
Article Link: Siri Answers 83% of Questions Correctly in Test, Beating Alexa But Trailing Google Assistant
And they need to break it down into better categories and situations. Outside with and without using headphones. In the home with a TV and people speaking in the background.
Will these findings affect anyone’s purchase decisions though?
More often than not, I suspect people buy products for a myriad of other reasons. If I am using an iPhone and Apple Watch, I am using Siri for the lack of any other alternative. If I buy an amazon echo, it’s likely due to the price, and not due to Alexa being any better.
Is there anyone here switching from an iPhone to an android phone specifically for google assistant baked into the OS at a system level? I highly doubt it.
Ultimately, however superior google assistant may be, it doesn’t matter to the end user because of the control that Apple wields over their ecosystem.
More often than not, I suspect people buy products for a myriad of other reasons. If I am using an iPhone and Apple Watch, I am using Siri for the lack of any other alternative. If I buy an amazon echo, it’s likely due to the price, and not due to Alexa being any better.
Is there anyone here switching from an iPhone to an android phone specifically for google assistant baked into the OS at a system level? I highly doubt it.
Ultimately, however superior google assistant may be, it doesn’t matter to the end user because of the control that Apple wields over their ecosystem.
What is truly disturbing is that Apple sees these tests and thinks they are doing a good job. There is little chance SIRI will be improved under these circumstances. I broke my iwatch and didn't replace it because of how poorly SIRI performs.
Google is destroying everyone at their correct search results. 92.9% versus 83.1 Siri, 79.8% Alexa. That is a huge difference in accuracy from a Machine Learning perspective. And it is all about the mountains of data Google has to search and the volume of searches they get against that data.
In the understanding the query they all virtually tied at near perfect And that is not surprising. Speech recognition is a pretty well known problem with a lot of very good models and solutions. But processing the query and returning useful, contextual information is a much more difficult problem.
I don't agree. I dumped my Alexa devices because I got tired of terrible results when I asked it questions. The Google Assistant in the Google Home Hub is so much smarter.
Siri does everything I ask it to do........I only ask it to do about 8 different things(HomePod) and it does it well.
I guess when Siri tells me she will google it for me, it is considered a 'correct' answer... That's all I keep getting from this stupid thing... I put Alexa all over my house because it does what I want...
And when I whisper to Alexa to turn off the bedroom lights, she doesn't reply to me like Siri does, at full volume saying that she has sent the request, and waking up everyone in the house... especially the dogs.
And when I whisper to Alexa to turn off the bedroom lights, she doesn't reply to me like Siri does, at full volume saying that she has sent the request, and waking up everyone in the house... especially the dogs.
I only use Siri on my HomePod. Since the last HomePod update, it has been spectacularly awful. Even worse than before. A crash of a kitchen pan will evoke Siri to comment on some oddity or play some millennial hip-hop-pop tune - I'm still stuck in the 1960's. I now avoid Siri and just use my iTunes app to select music.
I’m happy to see Apple making improvements here. I only use Siri on my HomePods for commands so I’m glad it is considered the best in that area.
I honestly don't know why people still use Alexa.. gave up that garbage years ago for a Google Home and never looked back.. its good to see Apple making progress as well though.. Amazon just sucks when it comes to AI. Still, until it can do better then Google, wish they would give people a choice on what assistant to use
I wonder how Samsung Bixby would have done. No matter how much people hate Bixby Samsung seems to keep pushing it one way or another.