Sidebar Sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

Apple Preparing for Upcoming Siri Onscreen Awareness Feature With New iOS 18.2 API for Developers

MacRumors

MacRumors

macrumors bot
Original poster
Apr 12, 2001
68,106
38,858


Apple is working on more advanced Siri functionality as part of its Apple Intelligence feature set, and to prepare, it has been providing developers with App Intent APIs so apps will be ready for the new capabilities.

Apple-Intelligence-General-Feature-2.jpg

With the latest wave of betas, Apple has a new API that lets developers make onscreen content in their apps available to Siri and Apple Intelligence. From Apple's documentation:
When a user asks a question about onscreen content or wants to perform an action on it, Siri and Apple Intelligence can retrieve the content to respond to the question and perform the action. If the user explicitly requests it, Siri and Apple Intelligence can send content to supported third-party services. For example, someone could view a website and use Siri to provide a summary by saying or typing a phrase like "Hey Siri, what's this document about?"
Click to expand...
In the iOS 18.2 beta, ChatGPT integration with Siri allows users to ask questions about photos and documents, such as PDFs and presentations, and get information about them. You can, for example, ask Siri "what's in this photo?" and Siri will take a screenshot and hand it over to ChatGPT. ChatGPT then relays what's in the image, and the same feature works for PDFs and other documents.

It does not seem that the iOS 18.2 ChatGPT integration is the onscreen awareness functionality that Apple has planned for Siri, but it could be related. Apple describes onscreen awareness as the ability for Siri to understand and take action on things on the screen. If someone texts you an address, for example, you'll be able to say "Add this address to their contact card," and Siri will do it. This functionality is not available in iOS 18.2, and ChatGPT is limited to assessing screenshots, but it is somewhat confusing.

Onscreen awareness, like personal context and in-app actions, is a feature that Apple has planned for Siri, but it's probably not something that we're getting this year. Many of the Siri features are coming in a future version of iOS 18, and Bloomberg's Mark Gurman has said we can expect to see them in iOS 18.4, an update set to be released in the spring of 2025.

While there are multiple Siri features that won't be coming until next year, Apple is providing developers with APIs in advance so that developers have several months to prepare and so the features are ready for the public when the updates actually come out.

Article Link: Apple Preparing for Upcoming Siri Onscreen Awareness Feature With New iOS 18.2 API for Developers
 
Article Link
https://www.macrumors.com/2024/11/04/apple-onscreen-awareness-ios-18-2-api-developers/
  • Wow
Reactions: UpsideDownEclair
The best Siri feature Apple could introduce is if you could completely extirpate it from your devices.
 
  • Like
Reactions: childu
If developers have to actively build compatibility with the API, I think AI awareness is going to be pretty limited. What incentive does a developer have to pass details outside of their own app? Usually they only want data flow to go the other way: INTO the app.
 
  • Like
Reactions: Rhhk
Hold on: in order for Siri to be able to see on-screen content it would need the developers' permission?? I hope I got this wrong.
 
Worth noting that this is already supported in first party apps.

For example, if you're looking at a photo in Photos, or a webpage in Safari, you can say "Hey Siri send this to Johnny" and it will share the photo or webpage in a new iMessage with Johnny.

This new API presumably just extends that functionality to third party apps?
 
This already works for many years when you want to create a reminder in a first party app for something that is on screen.

It’s very handy but the fact it works in Mail but not WhatsApp is very annoying. A feature like this needs to work system wide.

I hope it does and that developers can add additional smarts without leaving it disabled. Because many devs will never bother.
 
From the A.I features released so far, I don’t think this is going to be anywhere close to useful.
The models are trash

The thing with AI.. or any new product.. is it have to work 90% of the time for people to pick it up. Like chatgpt, it might not be helpful for maybe 5% of the time, but it’s worth my time trying to use it.
When you have such an awkward opt-in development model, that covers a small percentage of use cases, after a few failed tries, I will never use screen aware features. It waste my time rather than actually help me out. The screen awareness feature should work without much developer input, that’s going to work on a lot more cases.
 
Is it just me or is the sheer quantity of features with every update getting a little overwhelming?

I like updates to my software but I find it hard to follow all the things that are there. Even when I read the update log I'll usually end up knowing that I probably won't use it anyways.

Too many buttons to press, too many settings to find somewhere on that tiny device...

Am I alone in this?
 
d3vilsadvocate said:
Is it just me or is the sheer quantity of features with every update getting a little overwhelming?

I like updates to my software but I find it hard to follow all the things that are there. Even when I read the update log I'll usually end up knowing that I probably won't use it anyways.

Too many buttons to press, too many settings to find somewhere on that tiny device...

Am I alone in this?
Click to expand...
Yes you are, nothing is overwhelming, just have to to keep up
 
Siri still doesn’t know my Scottish accent. It’s absolutely useless and doesn’t work with me. After 10 years they’ve had plenty of time to fix Siri not understanding accents. If I put an American accent on it understands me clearly. Rubbish.
 
  • Like
Reactions: Rooster 🐓 and Pruus
A horrid long winded way to do a circle to search.

Google already nailed this feature with long hold on home bar, asking Siri is moronic when a gesture should be enough.
 
Marketed as feature, but this is a back door for entities to know when the phone is unlocked and the user is online to make screenshots and record activities.

My cousin works for engineering he knows there is a special department called "AP" where this "feature" was suggested and forced upon to the software department right under Craig.
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back