Craig Federighi Explains How Scribble Feature in iPadOS 14 Was Developed

[MacRumors]

Scribble is one of the most notable new features in iPadOS 14, and in a new interview with Popular Mechanics, Apple SVP of software engineering Craig Federighi has revealed more details about how it was developed and how it works on iPad.

With Scribble in iPadOS 14, you can use an ‌Apple Pencil‌ to write in any text field, and the written text is then converted to typed text. The feature means you can use the ‌Apple Pencil‌ without having to swap over to a keyboard when you need to compose an email, write a Calendar event, visit a URL, send an iMessage, or conduct a search.

Federighi explained to Popular Mechanics how Scribble's early development relied on extensive data gathering involving analyzing how people from all over the world write things down.

"When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things," says Craig Federighi, senior vice president of software engineering at Apple. "We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation."

That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple's tech, static examples weren't enough. They needed to see the strokes that formed each letter. "If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written."

The job of integrating stroke-based recognition with character and word prediction is all done on-device, something that machine learning and raw processing power makes possible, explained Apple's software chief.

"It's gotta be happening in real time, right now, on the device that you're holding," Federighi says. "Which means that the computational power of the device has to be such that it can do that level of processing locally."

Subscribe to the MacRumors YouTube channel for more videos.​

Thanks to Scribble and iPadOS 14, an iPad‌ can be used almost entirely with touch and tools, and there's little need to type. If you're someone who prefers writing by hand, ‌Scribble‌ introduces tools that only enhance the experience. Check out the hands-on video above to see Scribble in action.

Article Link: Craig Federighi Explains How Scribble Feature in iPadOS 14 Was Developed

 

[ouimetnick]

It’s amazing how much more there is to this than just meets the eye. I generally don’t think about on device processing and learning. I’ve been using Scribble since iPadOS 14 came out and it’s really useful for converting notes into text.

 

[Harvey Zoltan]

I would not be surprised if it's Apple Pencil support and not touch support that comes to the AS Macs.

 

[VZvL]

Sounds familiar, Apple has been doing pretty much the same thing on the Mac for a while. Anyone remember "Ink"

 

[bob122989]

Newton technology all the way. I’ve been surprised they haven’t had it on iPhone and iPad since day one.

 

[chucker23n1]

I would not be surprised if it's Apple Pencil support and not touch support that comes to the AS Macs.

It would have the advantage of not requiring larger touch surfaces. However, it comes with some similar issues. For example, can they make it work with the nanotexture?

Cause having the BTO options be "I want Pencil support" vs. "I want a matte screen" is hella confusing.

Sounds familiar, Apple has been doing pretty much the same thing on the Mac for a while. Anyone remember "Ink"

Newton technology all the way. I’ve been surprised they haven’t had it on iPhone and iPad since day one.

Yeah, I wonder how much of Inkwell is still in this.

 

[1applerules1]

I turned off scribble 2 days into iPadOS 14. Caused more trouble than what it’s worth.

 

[rfog]

It easy to say, you English speaking (and writing) people, but lack of Spanish, French, and other common languages makes it completely unusable.

And please, don't say it "recognize" Spanish with English keyboard. Not accents, not ñ and if there is a near misstyped in English language, it changes. For example, "culo" (bottom, butt), is converted in "cult".

 

[hagar]

Great, now bring it to other languages than English and Chinese. I had to wait 5 years for QuickType in Dutch.

 

[rfog]

They do a Keynote. All is magic. All is the best of the best, put in the meantime they put under the carpet the not so fancy and c00l stuff. Did they had any problem to say "scribble English only"?

 

[simmepi]

Great, now bring it to other languages than English and Chinese. I had to wait 5 years for QuickType in Dutch.

Same for Swedish. I’m expecting QuickPath support to arrive in, oh, 2025 perhaps?

 

[rfog]

Same for Swedish. I’m expecting QuickPath support to arrive in, oh, 2025 perhaps?

The thing that makes me very angry with this is if the not-present feature in your language won't be in other equivalent platforms/tools/whatever, could be reasonable not be in macOS/iOS, but say, handwriting recognition in non-English languages is present since Windows XP and since Samsung Galaxy Note 1. Swipe is present in Android since ages.

And now comes the Fringe-Man, saying their swipe-changed-name or they handwritten recognition-changed-name is the best of the best, followed by a herd or brainless nerds jumping happy Apple has invented another c00l thing.

Said that, I'm an Apple fanboy: 3 iMacs, 2 Macbook Pro, 2 iPhone, 3 Apple Watches, 3 iPads... but what is true, is true.

 

[Harvey Zoltan]

What I was thinking was that touch/pencil support would be not so much for the AS laptops but possibly the new iMac. If the stand was something like the new Magic Keyboard for the iPad Pro it would be possible to position the screen for drawing and touch input (like that Microsoft thing). This would make sense to me but maybe lack of user demand and cost of doing this make it unlikely.

 

[rfog]

What I was thinking was that touch/pencil support would be not so much for the AS laptops but possibly the new iMac. If the stand was something like the new Magic Keyboard for the iPad Pro it would be possible to position the screen for drawing and touch input (like that Microsoft thing). This would make sense to me but maybe lack of user demand and cost of doing this make it unlikely.

I have been a hardcore user of Surface devices (from Surface 1 to 4, then I got tired of that crap) and it is very fatigue-tired to use pencil in screen in a laptop. You end with your arm very tired. I don't expect Apple will release MAC with touch-screen.

What they could release is a "handwritting" pad at keyboard side or with a new "magic unicorn" trackpad. (Said that, sometimes I've handwritten in a trackpad -it is not much reliable but can be done).

 

[Harvey Zoltan]

I have been a hardcore user of Surface devices (from Surface 1 to 4, then I got tired of that crap) and it is very fatigue-tired to use pencil in screen in a laptop. You end with your arm very tired. I don't expect Apple will release MAC with touch-screen.

What they could release is a "handwritting" pad at keyboard side or with a new "magic unicorn" trackpad. (Said that, sometimes I've handwritten in a trackpad -it is not much reliable but can be done).

You may well be right but many tech commentators seem to be convinced that touch screen Macs are coming. Hopefully not too much longer before we know for sure.

 

[farewelwilliams]

inb4 people say "steve jobs rolling in his grave" and bringing up the stylus quote...

 

[Scottsoapbox]

After two decades of never using a pencil because of computers, using a computer to not use a keyboard seems like a good idea too late.

Maybe the children can still be saved, but it's too late for me...

 

[rfog]

You may well be right but many tech commentators seem to be convinced that touch screen Macs are coming. Hopefully not too much longer before we know for sure.

We all expect to know what is in Apple Silicon. Me the first one.

The clue for touch screen is the new iconography and separation of screen items in Bug, sorry Big Sur, but I'm still dubious.

Do you know what is my wet dream? A MacBook convertible, 13", with touch screen. Then I will sell all my iPads/MBP and use only that for all. And an iPhone with stylus. I tried those in Windows, but they work fine as laptop but are crap when you convert into tablet.

 

[nylonsteel]

a great feature
however
aapl should have came up with this years ago

​

 

[Arran]

Newton technology all the way. I’ve been surprised they haven’t had it on iPhone and iPad since day one.

Pity they didn’t use Newton’s “puff of smoke” animation when scratching out a word.

It made fixing mistakes very satisfying.

 

[chucker23n1]

inb4 people say "steve jobs rolling in his grave" and bringing up the stylus quote...

The stylus quote was clearly a shorthand of "who wants to rely on a stylus exclusively to interact with the UI, when you already have ten fingers?". Using a stylus to write handwritten text or draw something is quite different from that.

 

[holzlondon]

I think the topic of languages is vitally important. Other language support is something that really does take too long for apple to bring out. I work about 60-70% in English, but use German a lot as well, and I am always annoyed when a great feature works only in one of the languages I work in.

Also, I hope they bring it to work properly in Pages, and Word (maybe Microsoft has to do something to make it work).

It seems to work fine when you're entering a text field somewhere, but sometimes, I would like to use script-to-text in a real document...

 

[rfog]

The stylus quote was clearly a shorthand of "who wants to rely on a stylus exclusively to interact with the UI, when you already have ten fingers?". Using a stylus to write handwritten text or draw something is quite different from that.

What he didn't though, or if he thought it, he ignored, is some people have fingers like blood sausage, and what is works, exude grease like it even if you follow the Apple Watch washing hands rules, and half hour after you washed your hands, and ten minutes using your iPad, you start seeing your own fingerprints through the screen.

 

[ArtOfWarfare]

You may well be right but many tech commentators seem to be convinced that touch screen Macs are coming. Hopefully not too much longer before we know for sure.

Everyday, thousands of MacBook screens are touched and people are bewildered that nothing happens. Touchscreens have been standard for over a decade now - it’s bizarre that there are computers costing thousands of dollars that don’t have touchscreens.

The fact that it’d be used so little didn’t stop Apple from including the touchbar - why on earth haven’t they made their products “Just Work” by including a touchscreen yet?

Seriously - have any kid who hasn’t used a Mac before use a MacBook for a few hours. I guarantee they’ll touch the screen at least once and consider it broken when nothing happens. Literally every phone, tablet, and non-Apple-or-budget laptop has a touchscreen - it’s just a given that a MacBook should, too.

 

[PickUrPoison]

I have been a hardcore user of Surface devices (from Surface 1 to 4, then I got tired of that crap) and it is very fatigue-tired to use pencil in screen in a laptop. You end with your arm very tired. I don't expect Apple will release MAC with touch-screen.

What they could release is a "handwritting" pad at keyboard side or with a new "magic unicorn" trackpad. (Said that, sometimes I've handwritten in a trackpad -it is not much reliable but can be done).

Actually I think they intend for an iPad to be the companion device. As in Sidecar.

Agree touch screen/pencil is very unlikely on Mac. The light pen interface was investigated in the 70s, maybe the 60s, and though absolute cursor positioning has its advantages compared to relative positioning, “gorilla arm” is real. It’s very fatiguing, as you say.