Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
I would say Apple have come a long way in a short time. Siri does close to 100% on my voice dictation. And, answering questions is quickly improving. This is still in it's early days. It wasn't but a few years ago this wasn't even available. And, I paid 10.00 a month to a third party service to have "accurate" voice dictation on my Blackberry 8700. Can't forget what edge gps was like on those phones.
 
I agree. This isn’t a test conducted by or expressly for Apple, it was done to compare functionality of all the major assistants.

I don’t know anything about the company testing, but frankly when I see any comparison article on test results I don’t really know how trustworthy or impartial the company really is. If this is horribly inaccurate then I’m sure another test will be done by someone refuting these results.

Personal results vary on even the best AI. Just because it doesn’t work for you, personally, doesn’t mean it’s not as good or better than other assistants.

I’m someone who has criticized Apple for wasting the lead that they had in voice assistants. If they are catching up to be one of the better ones then that is an improvement that I didn’t expect. I hope these results are verified by other independent tests.
Since AI training and testing is always biased, any of these tests will be biased as well, especially when they come directly from the company behind that product. While it is great to see a metric for Siri improvement, I don't think anyone will be even noticing the improvements. Unlike Google and Amazon which distribute press releases and marketing regularly regarding updates to Google Assistant and Echo, Siri is one of Apple's few services that gets no marketing attention when it comes to improvements so no one really knows how often or when Apple incrementally improves Siri.
 
I call BS. I have never got anywhere near 70% accuracy. Mind you I am not an American living in the Bay Area.

Sounds about right to me. Even as a non-native speaker using Siri on the HomePod, I'd say my accuracy is probably more along the lines of 90-95%. Really happy with Siri. Also, her voice is so much better than Google Assistant's imo.
 
  • Like
Reactions: ErikGrim
There is more to it than just a simple we asked a question. Can't Google answer two Q's one after the other as it retains context?
The only thing that matters to me is does the assistant meet or exceed expectations and how often? If no one is asking two contextual questions in a row, it really doesn't matter if it's a feature or not. If they ran a test only involving multiple questions, we already know the outcome of a Google vs. Siri so that data has limited value for me and most users. Maybe a lot people ask follow questions but I've never heard anyone try it unless they were told to by someone during a demo.
 
I'm sure the test were made in US. I wonder how would be the results (especially under navigation category) if it was made in EU or in any other country other than US. Apple's map app is complete garbage (it is freaking useless) here in EU and probably that would affect the Siri.
 
I call BS. I have never got anywhere near 70% accuracy. Mind you I am not an American living in the Bay Area.

I am a UKer with a Bristol accent and I reckon this is about right for me - 3 out of 4. Some things it never seems to fail on like Facetiming my contacts, playing music or setting a timer but sending text messages with the right text can be tricky sometimes...
 
How can people actually defend this?

-6ieLCh5-LyQMHi2SawpiIC0V-QkhSnMX4RjOR9WhMQ.png
 
  • Like
Reactions: ErikGrim and FFR
Who is Loup Ventures? If they're anything like JD Power it's a marketing company that accepts money in return for skewed data disguised as research to favor the client (in this case Apple).

Here's the real progress indicator...

https://www.reddit.com/r/sirifail

w96xubvj50c11.jpg


6dmwvipwtqb11.jpg


6n572gpw64b11.jpg
 
Last edited:
Maybe it’s because most of my queries are informational in nature but Siri works about 50% of the time. The transcription is correct but it either brings up the wrong information or just points me to a web search. If I ask the same question with the same wording to google Assistant it works about 95% of the time.
 
Gosh I wish my Siri success rates where anywhere near that. On HomePod, most all questions have to repeat, and I eventually give up and do it on my phone. Same goes for the watch. I'm sure Google Assistant would be much better but for me the Apple Ecosystem is more important. And Google Assistant can't integrate with Apple Home and other Apple specific tech. If the difference between Assistant and Siri continue to increase, Apple core users may start considering abandoning parts of the Apple ecosystem such as Homekit. The hope is that its behind now, but Apple will only continue to improve it slowly but surely where is Google tends to go faster but can get bored with some things and not fully flesh things out. Will be interesting to see how it all turns out! I feel bad for Amazon who started with such a bang but will be pretty impossible to compete with Google or Siri in the long run.
 
Test is not relevant for me because it still can't recognize my accent properly. Google Assistant is the very best.
 



Five months after performing a test that put the smart speakers of multiple companies in the spotlight to determine how well they performed in various categories, Loup Ventures is back today with an IQ test focused entirely on digital AI assistants. To get the necessary results, the researchers asked Siri, Google Assistant, Alexa, and Cortana 800 questions each on a smartphone, and compared their findings to a previous AI test held in April 2017.

siri-iphone-x.jpg

For Siri in the new test, Apple's AI helper understood 99 percent of the queries and answered 78.5 percent of them correctly. That's an improvement on a similar AI-focused test from April 2017 (66.1 percent of 800 questions answered correctly). While Loup Ventures looked at similar methodologies when testing smart speakers in February, the researchers explain that it's "not worthwhile to compare" the results across these tests since "the use cases differ greatly between digital assistants and smart speakers."

This is particularly true for Siri on HomePod, which performs well in certain areas but is largely limited to the amount of actions it can perform on the speaker itself. This led Apple's HomePod to become relegated to the "bottom of the totem pole" in an AI assistant performance test during Loup Venture's smart speaker research in February, with Siri answering 52.3 percent of 782 total questions correctly, across the same five categories as the new test.

Loup Ventures grades each digital assistant on two metrics: "Did it understand what was being asked?" and "Did it deliver a correct response?" Questions came from five categories, including Local (example: "Where is the nearest coffee shop?"), Commerce ("Can you order me more paper towels?"), Navigations ("How do I get to uptown on the bus?"), Information ("Who do the Twins play tonight?"), and Command ("Remind me to call Steve at 2pm today").

loup-ventures-july-test.jpg

Charts via Loup Ventures


Questions were asked of Siri on an iPhone running iOS 11.4, Google Assistant on a Pixel XL, Alexa on the iOS app, and Cortana on the iOS app. Siri's best category was Command (90 percent of questions answered correctly), outperforming all rivals when asked to control aspects of the iPhone, smart home products, Apple Music, and more. Following Command, Siri performed well in Local (87 percent), Navigation (83 percent), and began dipping in Information (70 percent) and Commerce (60 percent).
Google Assistant was the top digital assistant in all categories except Command, with Loup Ventures particularly liking Google's "featured snippets" feature that reads off search results of voice queries and is often "exactly what you're looking for." Both Alexa and Cortana were lesser performers in the test due to the iOS app for each limiting what the assistants can do on an iPhone, unlike Siri's ability to perform tasks all over iOS and not just in one app.

In total, Google Assistant answered 85.5 percent of the 800 questions asked correctly and understood all of them, compared to Siri's 78.5 percent answered correctly and 11 misunderstood. Alexa correctly answered 61.4 percent and misunderstood 13, while Cortana was the "laggard" and correctly answered 52.4 percent and misunderstood 19.

loup-ventures-july-test-2.jpg

Over the 15 month period since April 2017, Siri improved by 13 percentage points, with Loup Ventures pointing out that it was "impressed with the speed at which the technology is advancing" for most of the assistants. The researchers went on to explain that many of the issues they had last year were erased by "improvements to natural language processing and inter-device connectivity."

Loup Ventures also sees more improvements coming down the line, particularly for Siri. This fall, the digital assistant will gain new abilities with Apple's upcoming "Siri Shortcuts" feature in iOS 12, which will let users create automations between apps that can be easily triggered through voice commands. The researches say they are "eager to test that feature" when it launches within iOS 12, likely coming in September.

Article Link: Siri on iOS Answered 78.5% of Queries Correctly in Latest Test, Trailing Google Assistant at 85.5%

SIRI sucks..i guess all of our expectations have descended into the abyss. We should expect, no demand more. A teillion dollar company should be able to solve this and get to excellence in 90 days
 
Last edited by a moderator:
Alexa is the user interface to a much more powerful system. Question answering is one thing, but Alexa controls for all devices in your home, online shopping, etc. Also, the cost for entry is very low. $20 or so, depending on the current sale, for an Echo Dot. So Alexa is found in many more households, or as in my case, in many more rooms of the same house.

You're correct on price-point, but I'm not sure what "more powerful system" you're talking about however, as each Amazon, Google, and Apple have their own home systems, which to my knowledge are the same for the most part, except that you can tell Alexa to order something (from Amazon only) for you. The hardware in most Alexa powered devices is essentially "junk". A 5 year old smartphone has more high-end components than the Amazon stuff.
 
Ok. Have a nice day.

Thanks, I will.

I'm going to pretend that this wasn't all dripping with sarcasm and MacRumors was, just for a moment, a wonderfully polite place :)
[doublepost=1532532996][/doublepost]
My grandfathers Jitterbug fliphone comprehends and answers 100% of all questions asked of it when you push the big red personal assistance button.

And it employs a staff of friendly helpful humans who are happy to get you whatever you need when you push it. I was joking when I started this, but it's actually pretty nice, now that I think about it.

throw an army of low paid workers at things and you can do tech pretty well. There was a startup that was doing that to reduce poly count on CAD models for use in Unity and such engines, claimed a "tech process" but really had a bunch of low paid 3D modelers doing the work manually. I believe they eventually failed. Jet did that in a way though too by not having an Amazon like logistics infrastructure but just having people buy all the orders from competitors and drop ship to customers at a loss, given the Walmart acquisition that strategy seemed to have worked for them (although admittedly they had an end game as they felt they only needed economy of scale to be able to compete with Amazon and deliver profitability). I know there are other examples but those are the ones that jump to mind.
 
Guess it's just me. I still cant get Siri to setup a simple calendar appointment properly. I find it faster and more accurate to just create it myself.

I must use the wrong phrasing or something...
 
How can people actually defend this?

-6ieLCh5-LyQMHi2SawpiIC0V-QkhSnMX4RjOR9WhMQ.png
I just tried this on my phone. It set an alarm for 10 minutes from the time I asked.

In other words, it worked. I so seldom use Siri for anything that I usually don’t even try, based on iPhone 4s experience with it. But it had no trouble with me setting this alarm. I tried setting it at 10:34.

upload_2018-7-25_10-42-46.png
 
  • Like
Reactions: ErikGrim
They aren't that far off as is, as you can see by the tests. Google has years of customer data to help with this stuff, fortunately Apple does not. I think most are totally fine with Apple taking this approach. I am actually surprised Google isn't much further ahead at this point, considering the amount of data they have.

The fact that Google isn't much further ahead than they are is a quite heartening concept to me. I think about potential scenarios like China state sponsoring a Chinese firm to develop their own AI based upon all of the data the Chinese government has access to from its full population of users. Without privacy protections in place and PR pushback (which has reigned in even Google from what they *could* be doing) if pure access to data were the sole arbiter of success no one in a western democracy could compete.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.