You Can Now Get Visual Intelligence on iPhone 15 Pro – Here's How

[szw-mapple fan]

Visual intelligence, what a joke. Unless I am missing something, all this feature does is upload an image to either ChatGPT or Google image search, depending on the button. What amazing things you are doing apple. These billions invested in ai are paying off!!

Unless you are trying to reverse search the visual via google or ask a question about it, it actually does some processing on device. For example, creating calendar events from a poster, identifying animals/plants, selecting text, summarizing book chapter etc.

 

[HiVolt]

Its a huge disappointment if you ask me.

It cannot identify people at all!, like well known actors. both the image search or ask chatgpt.

 

[BigDO]

So wait, Apple Visual Intelligence is literally a shortcut to either ChatGPT or Google?

Just wow.

 

[bhirt37]

You’re missing something. It’s in-device processing.

Is it on device? When I tried it, I was prompted, asking if it could upload the picture to chatgpt or google, depending on what button I pressed. The macrumors article also seems to say that as well. I would love to hear more from you. The last thing I want to do is spread misinformation.

Everything I pointed my camera at only gave me 2 options. Ask which uploads to chatgpt and search which uploads to google. Can you give me an example of how to experience on device processing?

 

[oink_oink]

Won't be long now before iPhone is just a hollow case with Google phone inside. 🤣

 

[Howard2k]

Is it on device? When I tried it, I was prompted, asking if it could upload the picture to chatgpt or google, depending on what button I pressed. The macrumors article also seems to say that as well. I would love to hear more from you. The last thing I want to do is spread misinformation.

Everything I pointed my camera at only gave me 2 options. Ask which uploads to chatgpt and search which uploads to google. Can you give me an example of how to experience on device processing?

It is both on and off. It has some initial capabilities that are on device "What is this?" "That's a squirrel" and then it will offload to PCC (Apple's cloud AI) and ChatGPT, Google etc. But my understanding is that it does initial work locally, and then will only go to PCC, ChatGPT etc., if it cannot do the work locally.

So even if it's off to Google to find store opening hours, the preference is to keep as much work local.

I could be wrong, but that's my understanding, and that's how Apple describes their architecture.

 

[mlayer]

I've used Visual Intelligence since it became available in the beta. It works other than recognition of people. I imagine that will come later as it may also involve integration with Photos.

 

[SMH4KIDIOT]

It is both on and off. It has some initial capabilities that are on device "What is this?" "That's a squirrel" and then it will offload to PCC (Apple's cloud AI) and ChatGPT, Google etc. But my understanding is that it does initial work locally, and then will only go to PCC, ChatGPT etc., if it cannot do the work locally.

So even if it's off to Google to find store opening hours, the preference is to keep as much work local.

I could be wrong, but that's my understanding, and that's how Apple describes their architecture.

That's not how it works. Things that require OCR are processed on device like recognizing websites, phone numbers addresses, dates, translate etc. Location based recognition uses Maps data, all other requests are either sent to Apple PCC, Chatgpt and Google.

The OCR part, iPhones going back to iPhone 11 can do. Apple called it Live Text.
The landmark and plant recognition iPhones back to 11 can do. Apple called it Visual Look Up.

What visual intelligence does is bring those capabilities to a live camera view instead of inside the Photos app. The interesting thing is that you can get all those features of Visual Intelligence if you just download the google lens app and you don't need iPhone 15 Pro or iPhone 16 to get them.

 

[Howard2k]

That's not how it works. Things that require OCR are processed on device like recognizing websites, phone numbers addresses, dates, translate etc. Location based recognition uses Maps data, all other requests are either sent to Apple PCC, Chatgpt and Google.

The OCR part, iPhones going back to iPhone 11 can do. Apple called it Live Text.
The landmark and plant recognition iPhones back to 11 can do. Apple called it Visual Look Up.

What visual intelligence does is bring those capabilities to a live camera view instead of inside the Photos app. The interesting thing is that you can get all those features of Visual Intelligence if you just download the google lens app and you don't need iPhone 15 Pro or iPhone 16 to get them.

Are you sure that's not how it works? I believe those things are now front ended by Apple Intelligence, if I understand correctly, and then offloaded where needed. Again, I could be wrong, but that's the architecture that Apple describes.


EDIT: Actually they mention this in their press release:

Apple Intelligence features expand to new languages and regions today

Apple Intelligence is expanding to even more people around the world.

www.apple.com

Apple Intelligence marks an extraordinary step forward for privacy in AI and is designed to protect users’ privacy at every step. It starts with on-device processing, and for requests that require access to larger models, Private Cloud Compute extends the privacy and security of iPhone into the cloud to unlock even more intelligence.

 

[aevan]

MRF before Visual Intelligence on 15 Pro: They are gatekeeping this for the 16! They want me to sell my 15 Pro and upgrade!

MRF after Visual Intelligence on 15 Pro: Who needs this?

 

[SMH4KIDIOT]

Are you sure that's not how it works? I believe those things are now front ended by Apple Intelligence, if I understand correctly, and then offloaded where needed. Again, I could be wrong, but that's the architecture that Apple describes.


EDIT: Actually they mention this in their press release:

Apple Intelligence features expand to new languages and regions today

Apple Intelligence is expanding to even more people around the world.

www.apple.com

That's just a generic Apple statement about Privacy. Now i don't have an iPhone 15 or 16 to test it but if you do, you can request the Intelligence report to see what requests are on device vs off device. People who have done that have shown what gets handled on device vs off device.

Writing Tools:

• On-device: Proofread, rewrite, friendly, professional, concise
• PCC: Summary, key points, list, table, describe your change
• ChatGPT: Compose

Mail:

• On-device: Email preview summaries, Priority emails
• PCC: Email summarization, smart reply

Messages:

• On-device: Message preview summaries, Smart reply, Genmoji generation

Siri:

• On-device: (I was able to ask about emails and calendar events)
• ChatGPT: Any ChatGPT requests (will inform you before sending to ChatGPT)

Safari:

• PCC: Web page summaries

Notes:

• PCC: Audio recording summaries

Photos:

• On-device:
  • Intelligent search (after indexing)
  • Clean up (after downloading the clean-up model)

Notifications/Focus:

• On-device: Notification summaries, Reduce interruptions focus

Image Playground:

• On-device: Image generation (after image model is downloaded)

Visual intelligence is straightforward as it tells you when it is searching with ChatGPT or Google. The location features like restaurant and business lookup are maps based as several examples have shown. If i had a device that supports Apple Intelligence and Visual Intelligence, i would test it and report back but i don't. If you do, test them offline and online and perhaps use the Intelligence report to see what requests were done on device vs sent off device. I'm very curious myself. Most of my own findings are from my general knowledge and experience on how these things work and information i have gotten from other users.

 

[Howard2k]

That's just a generic Apple statement about Privacy. Now i don't have an iPhone 15 or 16 to test it but if you do, you can request the Intelligence report to see what requests are on device vs off device. People who have done that have shown what gets handled on device vs off device.



Visual intelligence is straightforward as it tells you when it is searching with ChatGPT or Google. The location features like restaurant and business lookup are maps based as several examples have shown. If i had a device that supports Apple Intelligence and Visual Intelligence, i would test it and report back but i don't. If you do, test them offline and online and perhaps use the Intelligence report to see what requests were done on device vs sent off device. I'm very curious myself. Most of my own findings are from my general knowledge and experience on how these things work and information i have gotten from other users.

Yeah I don't either Hopefully someone can try it out and see what the report looks like.
Additionally it would be trivial to disable the radios, try the features, and see which ones work and don't work too.

Seems to me they'd need to wrap all the features together to decide where to handle them though - on device, PCC, 3rd party cloud - so I think it makes sense that there is still the requirement for on-device processing in any case, both for interpretation of the task and the performance of simple tasks (like text summarization based on image) and it makes sense that it's the on-device Apple Intelligence that is doing that.

 

[johnnytravels]

More and more I’m beginning to realize Apple is just a “design” company. Great at making things look pretty and consistent, but really bad at doing anything “smart”.

I think this is an interesting observation.
Now that that they can’t make purpose-built things anymore, they start lagging behind the tech world.
Ironically, the iPhone would have been their own downfall then, as it was detrimental from unleashing the internet from its place at home (and then, from its occasional mobile use in wifi connected laptops) to an omnipresent force. This was what fuelled the transformation from discrete objects to streams (of entertainment and information).
Now there is no way to globally return from an all in one streaming device to an array of discrete things that mirror discrete objects of consumption.

Another reason for Apple being so behind is its orientation towards directing money more towards stock holders than r&d. And where they have invested in r&d, they have mostly tried to build discrete things, and those things have failed (from that charging mat, to the Vision Pro, to that car).

 

[Gregintosh]

This is a completely useless feature. And no reason why it's limited to iPhone 15 Pro and 16 series only. Another example of Apple artificially nerfing older phones to encourage upgrades. If they actually did any on-device processing or some sort of internal/custom algorithm I would understand the requirements but they do not.

You can literally get the same function with the ChatGPT app on ANY iPhone by snapping a picture with the app. Or opening the Google app to do an image search. All this function is doing is giving you an interface to use those two.

Here's a replacement for Apple Visual Intelligence for any iPhone:

Add a ChatGPT shortcut on the Lock Screen dashboard. Press it to open ChatGPT, click the + button and hit "Take a picture" and you're done. Just an extra click or two but will work on iPhone 11+ (or whatever models ChatGPT app supports).

 

[Gregintosh]

MRF before Visual Intelligence on 15 Pro: They are gatekeeping this for the 16! They want me to sell my 15 Pro and upgrade!

MRF after Visual Intelligence on 15 Pro: Who needs this?

Yes, because at first we thought the Apple feature actually did something on its own, and not just be a shortcut to two existing functions/3rd parties that we could have all along on any phone.

 

[chriscl]

Not in Europe

Working for me, here in Germany - iPhone 15 Pro.

I just set it up this morning and had a play around, and… yeah. It is a bit limited, in that it just seems to be sending whatever you have in the „camera“ view to (either) ChatGPT or Google. I am unsure what improvements or enhancements there are, over having to use either the Google app or ChatGPT app - other than, of course, having to have those specific apps installed (and I have neither….)

I need to have a proper play with it, to see if it can do anything earth-shattering!

 

[svish]

Happy to see that it has been made available on the 15 Pro models. Have not yet updated my phone. Will update and try it out soon.

 

[Abadgermac]

Working in UK.

 

[Ctrlos]

Is it on device? When I tried it, I was prompted, asking if it could upload the picture to chatgpt or google, depending on what button I pressed. The macrumors article also seems to say that as well. I would love to hear more from you. The last thing I want to do is spread misinformation.

Everything I pointed my camera at only gave me 2 options. Ask which uploads to chatgpt and search which uploads to google. Can you give me an example of how to experience on device processing?

Take a picture of something in a foreign language and it translates it instantly, assuming you have the language downloaded in the translate app before you travelled (pretty basic prep IMO)

 

[Ctrlos]

Working for me, here in Germany - iPhone 15 Pro.

I just set it up this morning and had a play around, and… yeah. It is a bit limited, in that it just seems to be sending whatever you have in the „camera“ view to (either) ChatGPT or Google. I am unsure what improvements or enhancements there are, over having to use either the Google app or ChatGPT app - other than, of course, having to have those specific apps installed (and I have neither….)

I need to have a proper play with it, to see if it can do anything earth-shattering!

It’s not that the ‘app’ itself is earth shattering or anything, just that it’s a quick button press away from any screen. It’s terribly convenient.

Complaining it offloads things to ChatGPT is kind of like complaining you have to rely on a search engine because Safari cannot search the web by itself. It’s not a weakness.

 

[d4cloo]

you can also just turn on the camera in ChatGPT and ask.

 

[Parowdy]

Kind of strange they made sure to get Apple Intelligence on the 16e, which they have had trouble getting onto anything at all, but couldn't get MagSafe on there, which they've managed to do for years.

Strange the things they chose to prioritize as selling points this year.

You seem to forget that the 16e is the „default“ straight-from-Apple iPhone around the WORLD now.
Apple knows exactly how much wireless charging is used on their older and lower tier iPhones outside of the US and figured they can cheap out on it because the majority of world wide customers won’t need or miss it.
AI is just a marketing strategy for the western world, especially the US. Nothing else to it.

 

[Parowdy]

I'm curious why Apple didn't also allow the enabling of Visual Intelligence through Shortcuts.

lol. unrelated „fun“ fact: before the iPhone 15 Pro there was an action for the ring/mute switch in Shortcuts available for all iPhone. After the 15 Pro was introduced ONLY action button iPhones could use that action which is one of these things nobody talked about but infuriated some people, including me, maybe obviously, because it makes zero sense, none.

 

[The Game 161]

It’s quite surprising that the iPhone 16 Pro doesn’t allow you to set the action button to Visual Intelligence.
Or maybe it doesn't show on mine, since AI is OFF?

not sure why people would want it to when it's done via the camera button.

 

[TLewis]

Also, the Google app has had this feature for some time. I’m really not sure why Apple implemented the feature as they’re using Google for searching. This feels like they were desperate for a “new” cool feature and just did a UI hack to connect to Google.