iOS 15's Live Text Feature Lets You Digitize Written Notes, Call a Number on a Sign, Translate a Menu, and Much More

[MacRumors]

In iOS 15, Apple is introducing a new feature called Live Text that can recognize text when it appears in your camera's viewfinder or in a photo you've taken and let you perform several actions with it.

For example, Live Text allows you to capture a phone number from a storefront with the option to place a call, or look up a location name in Maps to get directions. It also incorporates optical character recognition, so you can search for a picture of a handwritten note in your photos and save it as text.

Live Text's content awareness extends to everything from QR codes to emails that appear in pictures, and this on-device intelligence feeds into Siri suggestions, too.

For instance, if you take a picture that shows an email address and then open the Mail app and start composing a message, Siri's keyboard suggestions will offer up the option to add "Email from Camera" to the To field of your message.

Other Live Text options include the ability to copy text from the camera viewfinder or photos for pasting elsewhere, share it, look it up in the dictionary, and translate it for you into English, Chinese (both simplified and traditional), French, Italian, German, Spanish, or Portuguese.

It can even sort your photos by location, people, scene, objects, and more, by recognizing the text in pictures. For example, searching for a word or phrase in Spotlight search will bring up pictures from your Camera Roll in which that text occurs.

Live Text works in Photos, Screenshot, Quick Look, and Safari and in live previews with Camera. In the Camera app, it's available whenever you point your iPhone's camera at anything that displays text, and is indicated by a small icon that appears in the bottom right corner whenever textual content is recognized in the viewfinder. Tapping the icon lets you tap recognized text and perform an action with it. A similar icon appears in the Photos app when you're viewing a shot image.

In another neural engine feature, Apple is introducing something called Visual Look Up that lets you take photos of objects and scenes to get more information from them. Point your iPhone's camera at a piece of art, flora, fauna, landmarks, or books, and the Camera will indicate with an icon that it recognizes the content and has relevant Siri Knowledge that can add context.

Since Live Text relies heavily on Apple's neural engine, the feature is only available on iPhones and iPads with at least an A12 Bionic or better chip, which means if you have an iPhone X or earlier model or anything less than an iPad mini (5th generation), iPad Air (2019, 3rd generation), or iPad (2020, 8th generation), then unfortunately you won't have access to it.

The iOS 15 beta is currently in the hands of developers, with a public beta set to be released next month. The official launch of iOS 15 is scheduled for the fall.

Article Link: iOS 15's Live Text Feature Lets You Digitize Written Notes, Call a Number on a Sign, Translate a Menu, and Much More

 

[farewelwilliams]

implemented better than android

 

[ruka.snow]

Let's see them digitise doctors notes and prescriptions.

 

[LukeHarrison]

Since Live Text relies heavily on Apple's neural engine, so the feature is only available on iPhones and iPads with at least an A12 Bionic or better chip, which means if you have an iPhone X or earlier model or anything less than an iPad mini (5th generation), iPad Air (2019, 3rd generation), or iPad (2020, 8th generation), then unfortunately you won't have access to it.

Forgive me if I'm wrong, but wasn't the A11 the first chip with a Neural Engine? So surely they could make this work on the X (and even the 8 and 8 Plus) if they REALLY wanted to.

 

[haruhiko]

I remember using an app called World Lens in 2013 or so which did live translation through the camera. Helped me a lot when I traveled to various countries in Europe. Was it acquired by Google later? Anyway it’s great that Apple now has the function back and even better.

 

[Frustratedperson]

Forgive me if I'm wrong, but wasn't the A11 the first chip with a Neural Engine? So surely they could make this work on the X (and even the 8 and 8 Plus) if they REALLY wanted to.

They don’t want to. Hence why they aren’t enabling it for the intel macs. Just a cash grab as they hope everybody will sell their newish intel Mac, which they paid handsomely for, at a huge loss and then go and pay handsomely again for a new Apple silicon Mac. Apple isn’t the company from heaven as most seem to think.

 

[EmotionalSnow]

Forgive me if I'm wrong, but wasn't the A11 the first chip with a Neural Engine? So surely they could make this work on the X (and even the 8 and 8 Plus) if they REALLY wanted to.

The later phones have an improved Neural Engine. I'm guessing that these are required in order to ensure good performance.

 

[Unggoy Murderer]

They don’t want to. Hence why they aren’t enabling it for the intel macs. Just a cash grab as they hope everybody will sell their newish intel Mac, which they paid handsomely for, at a huge loss and then go and pay handsomely again for a new Apple silicon Mac.

I'm sure Apple could get it running on the A11, however people apparently forget there will be trade-offs to enable that to happen. What if enabling it on the A11 halved the battery life, would you want that, would that be a trade-off you'd be willing to make? Or perhaps the performance of the phone drops by 25%, how about that?

Apple can't (and shouldn't be expected to) support older hardware with every single new feature that's released. Apple do a good job of supporting older hardware far better than the competitors do (the iPhone 6s, a 2015 phone, gets iOS 15), they'll get the benefit of at least some new features and improvements.

I'm sure some people will want to upgrade their hardware to support the latest software additions, but I (and I'm sure like a lot of other folks like me with newish machines), won't be bothered enough. Nice features, but I won't lose sleep over not having them.

Apple isn’t the company from heaven as most seem to think.

Of course they're not - Apple is a money making enterprise.

 

[haruhiko]

They don’t want to. Hence why they aren’t enabling it for the intel macs. Just a cash grab as they hope everybody will sell their newish intel Mac, which they paid handsomely for, at a huge loss and then go and pay handsomely again for a new Apple silicon Mac. Apple isn’t the company from heaven as most seem to think.

Haha it seems that it was you who thought Apple is the company from heaven and now you’re disappointed?

Yeah the A11 has the neural engine, just like your 1993 Intel Pentium PC also has a CPU. It doesn’t mean anything. Probably the performance will be so bad if they give it to the iPhone X?

Talking about iPhone X, this phone easily gets hot and eats battery like nothing. Now with the battery degraded it’s even worse. I only use it as a back up phone but even as a back up phone the battery life is not that good.

 

[Googlyhead]

The old World Lens app (and subsequent iterations after Google acquired it) were able to both capture and translate text in real-time using substantially slower hardware.

Live text would just be this, without the translation (whether on device or not). I can’t see why the neural engine would be an absolute requirement, even if you try to make a ‘performance’ argument.

Forgive me if I'm wrong, but wasn't the A11 the first chip with a Neural Engine? So surely they could make this work on the X (and even the 8 and 8 Plus) if they REALLY wanted to.

The later phones have an improved Neural Engine. I'm guessing that these are required in order to ensure good performance.

 

[Unggoy Murderer]

The old World Lens app (and subsequent iterations after Google acquired it) were able to both capture and translate text in real-time using substantially slower hardware.

Live text would just be this, without the translation (whether on device or not). I can’t see why the neural engine would be an absolute requirement, even if you try to make a ‘performance’ argument.

Try using Word Lens for any sustained period - it absolutely hammered your battery. And just to be clear, that isn't a criticism of Word Lens - it was an exceptional app for its time, it just couldn't take advantage of optimised silicon.

The performance argument is *the* argument. Apple more than likely could implement it on all hardware, but what it it caused your battery life to be half of usual, or Safari and camera apps to be considerably slower while it runs CPU-bound neural networks on every image downloaded, or every camera frame?

Also worth noting, text recognition isn't a trivial task, especially with handwriting.

 

[Jarman74]

I remember using an app called World Lens in 2013 or so which did live translation through the camera. Helped me a lot when I traveled to various countries in Europe. Was it acquired by Google later?

Correct - and the original functionality has been baked into the Google Translate app.

 

[I7guy]

They don’t want to. Hence why they aren’t enabling it for the intel macs. Just a cash grab as they hope everybody will sell their newish intel Mac, which they paid handsomely for, at a huge loss and then go and pay handsomely again for a new Apple silicon Mac. Apple isn’t the company from heaven as most seem to think.

Ok it’s a cash grab…now what?

 

[insomniac86]

Forgive me if I'm wrong, but wasn't the A11 the first chip with a Neural Engine? So surely they could make this work on the X (and even the 8 and 8 Plus) if they REALLY wanted to.

From memory I recall reading something about the A11 Bionic being very limited in what it could do. For example 3rd party apps couldn’t utilise it.

EDIT: I also believe it was something like 600 Billion, 1 trillion to 11 trillion operations per sec from the A11 neural engine to the A13.

 

[matrix07]

Stoked that my iPad Air 3 is compatible 👍

 

[Just sayin...]

This (and tags on notes) is an updated feature that I will use a LOT. Thanks for this, Apple!

 

[jahsavi]

Seems like a key feature that will be in the Apple AR glasses and it’s being used in iOS 15. Very exciting!

 

[jonnysods]

Well this will incredibly handy!!

 

[PBG4 Dude]

The feature is so quick! Just long-tap an image, then tap on “Show Text” and boom!

 

[citysnaps]

Forgive me if I'm wrong, but wasn't the A11 the first chip with a Neural Engine? So surely they could make this work on the X (and even the 8 and 8 Plus) if they REALLY wanted to.

Not wrong, and no doubt along with many hundreds of other features if they REALLY wanted to. The real reality is, that all companies, even those of Apple's immense size, have limitations of what could be done at a particular point in time, given finite resources and time schedules.

Just because Apple is one of the largest and most successful companies in the world does not mean that everything under the Sun that could be done, can be done, due to finite resources.

 

[Abazigal]

Forgive me if I'm wrong, but wasn't the A11 the first chip with a Neural Engine? So surely they could make this work on the X (and even the 8 and 8 Plus) if they REALLY wanted to.

And I am sure world hunger would be ended if all the countries in the world really wanted to.

Your point being?

 

[Corsig]

This feature is turning out to be quite surprising. Looking forward to getting my hands on it. Really liking the translation of foreign text.

 

[Gherkin]

implemented better than android

Not disagreeing, but how so? I have an iPhone but use a lot of Google apps so I've had these abilities for years now. Why is the Apple one better?

 

[Apple_Robert]

This new feature appears to work in the same way that the app “TextSniper” does, which I bought a few months back. The app works great.

 

[LordVic]

Not disagreeing, but how so? I have an iPhone but use a lot of Google apps so I've had these abilities for years now. Why is the Apple one better?

You've been here long enough to know that there's a large enough cohort of posters who only post stuff like this as fast as they can without putting any thought behind it.

there's a term for them, but we're not allowed saying it

Have used Google Lens, and looking forward to testing this feature out on my iPhone. But people jumping in before testing it out and claiming "Apple did it better!" already before this thing is in the wild is just pure fandom.