Become a MacRumors Supporter for $25/year with no ads, private forums, and more!

Craig Federighi Explains How Scribble Feature in iPadOS 14 Was Developed

MacRumors

macrumors bot
Original poster
Apr 12, 2001
50,443
11,833


Scribble is one of the most notable new features in iPadOS 14, and in a new interview with Popular Mechanics, Apple SVP of software engineering Craig Federighi has revealed more details about how it was developed and how it works on iPad.


With Scribble in iPadOS 14, you can use an ‌Apple Pencil‌ to write in any text field, and the written text is then converted to typed text. The feature means you can use the ‌Apple Pencil‌ without having to swap over to a keyboard when you need to compose an email, write a Calendar event, visit a URL, send an iMessage, or conduct a search.

Federighi explained to Popular Mechanics how Scribble's early development relied on extensive data gathering involving analyzing how people from all over the world write things down.
"When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things," says Craig Federighi, senior vice president of software engineering at Apple. "We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation."

That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple's tech, static examples weren't enough. They needed to see the strokes that formed each letter. "If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written."
The job of integrating stroke-based recognition with character and word prediction is all done on-device, something that machine learning and raw processing power makes possible, explained Apple's software chief.
"It's gotta be happening in real time, right now, on the device that you're holding," Federighi says. "Which means that the computational power of the device has to be such that it can do that level of processing locally."

Thanks to Scribble and iPadOS 14, an iPad‌ can be used almost entirely with touch and tools, and there's little need to type. If you're someone who prefers writing by hand, ‌Scribble‌ introduces tools that only enhance the experience. Check out the hands-on video above to see Scribble in action.

Article Link: Craig Federighi Explains How Scribble Feature in iPadOS 14 Was Developed
 
Last edited:
  • Like
Reactions: Ravi_MR and kazmac

chucker23n1

macrumors 601
Dec 7, 2014
4,636
5,826
I would not be surprised if it's Apple Pencil support and not touch support that comes to the AS Macs.

It would have the advantage of not requiring larger touch surfaces. However, it comes with some similar issues. For example, can they make it work with the nanotexture?

Cause having the BTO options be "I want Pencil support" vs. "I want a matte screen" is hella confusing.

Sounds familiar, Apple has been doing pretty much the same thing on the Mac for a while. Anyone remember "Ink" ;)

Newton technology all the way. I’ve been surprised they haven’t had it on iPhone and iPad since day one.

Yeah, I wonder how much of Inkwell is still in this.
 
  • Like
Reactions: Aston441
Comment

rfog

macrumors newbie
Jul 23, 2018
9
9
Netherlands
It easy to say, you English speaking (and writing) people, but lack of Spanish, French, and other common languages makes it completely unusable.

And please, don't say it "recognize" Spanish with English keyboard. Not accents, not ñ and if there is a near misstyped in English language, it changes. For example, "culo" (bottom, butt), is converted in "cult".
 
Comment

rfog

macrumors newbie
Jul 23, 2018
9
9
Netherlands
They do a Keynote. All is magic. All is the best of the best, put in the meantime they put under the carpet the not so fancy and c00l stuff. Did they had any problem to say "scribble English only"?
 
  • Wow
  • Like
Reactions: Dopestar and Nica
Comment

rfog

macrumors newbie
Jul 23, 2018
9
9
Netherlands
Same for Swedish. I’m expecting QuickPath support to arrive in, oh, 2025 perhaps?

The thing that makes me very angry with this is if the not-present feature in your language won't be in other equivalent platforms/tools/whatever, could be reasonable not be in macOS/iOS, but say, handwriting recognition in non-English languages is present since Windows XP and since Samsung Galaxy Note 1. Swipe is present in Android since ages.

And now comes the Fringe-Man, saying their swipe-changed-name or they handwritten recognition-changed-name is the best of the best, followed by a herd or brainless nerds jumping happy Apple has invented another c00l thing.

Said that, I'm an Apple fanboy: 3 iMacs, 2 Macbook Pro, 2 iPhone, 3 Apple Watches, 3 iPads... but what is true, is true.
 
  • Disagree
Reactions: droidgod
Comment

Harvey Zoltan

macrumors member
May 14, 2018
65
105
Brisbane
What I was thinking was that touch/pencil support would be not so much for the AS laptops but possibly the new iMac. If the stand was something like the new Magic Keyboard for the iPad Pro it would be possible to position the screen for drawing and touch input (like that Microsoft thing). This would make sense to me but maybe lack of user demand and cost of doing this make it unlikely.
 
Comment

rfog

macrumors newbie
Jul 23, 2018
9
9
Netherlands
What I was thinking was that touch/pencil support would be not so much for the AS laptops but possibly the new iMac. If the stand was something like the new Magic Keyboard for the iPad Pro it would be possible to position the screen for drawing and touch input (like that Microsoft thing). This would make sense to me but maybe lack of user demand and cost of doing this make it unlikely.

I have been a hardcore user of Surface devices (from Surface 1 to 4, then I got tired of that crap) and it is very fatigue-tired to use pencil in screen in a laptop. You end with your arm very tired. I don't expect Apple will release MAC with touch-screen.

What they could release is a "handwritting" pad at keyboard side or with a new "magic unicorn" trackpad. (Said that, sometimes I've handwritten in a trackpad -it is not much reliable but can be done).
 
  • Like
Reactions: twocents
Comment

Harvey Zoltan

macrumors member
May 14, 2018
65
105
Brisbane
I have been a hardcore user of Surface devices (from Surface 1 to 4, then I got tired of that crap) and it is very fatigue-tired to use pencil in screen in a laptop. You end with your arm very tired. I don't expect Apple will release MAC with touch-screen.

What they could release is a "handwritting" pad at keyboard side or with a new "magic unicorn" trackpad. (Said that, sometimes I've handwritten in a trackpad -it is not much reliable but can be done).
You may well be right but many tech commentators seem to be convinced that touch screen Macs are coming. Hopefully not too much longer before we know for sure.
 
  • Like
Reactions: Anox
Comment

rfog

macrumors newbie
Jul 23, 2018
9
9
Netherlands
You may well be right but many tech commentators seem to be convinced that touch screen Macs are coming. Hopefully not too much longer before we know for sure.

We all expect to know what is in Apple Silicon. Me the first one.

The clue for touch screen is the new iconography and separation of screen items in Bug, sorry Big Sur, but I'm still dubious.

Do you know what is my wet dream? A MacBook convertible, 13", with touch screen. Then I will sell all my iPads/MBP and use only that for all. And an iPhone with stylus. I tried those in Windows, but they work fine as laptop but are crap when you convert into tablet.
 
Comment

chucker23n1

macrumors 601
Dec 7, 2014
4,636
5,826
inb4 people say "steve jobs rolling in his grave" and bringing up the stylus quote...

The stylus quote was clearly a shorthand of "who wants to rely on a stylus exclusively to interact with the UI, when you already have ten fingers?". Using a stylus to write handwritten text or draw something is quite different from that.
 
Comment

holzlondon

macrumors newbie
Sep 28, 2020
1
1
I think the topic of languages is vitally important. Other language support is something that really does take too long for apple to bring out. I work about 60-70% in English, but use German a lot as well, and I am always annoyed when a great feature works only in one of the languages I work in.

Also, I hope they bring it to work properly in Pages, and Word (maybe Microsoft has to do something to make it work).

It seems to work fine when you're entering a text field somewhere, but sometimes, I would like to use script-to-text in a real document...
 
  • Like
Reactions: Anox
Comment

rfog

macrumors newbie
Jul 23, 2018
9
9
Netherlands
The stylus quote was clearly a shorthand of "who wants to rely on a stylus exclusively to interact with the UI, when you already have ten fingers?". Using a stylus to write handwritten text or draw something is quite different from that.

What he didn't though, or if he thought it, he ignored, is some people have fingers like blood sausage, and what is works, exude grease like it even if you follow the Apple Watch washing hands rules, and half hour after you washed your hands, and ten minutes using your iPad, you start seeing your own fingerprints through the screen.
 
  • Like
Reactions: amartinez1660
Comment

ArtOfWarfare

macrumors G3
Nov 26, 2007
8,927
4,787
You may well be right but many tech commentators seem to be convinced that touch screen Macs are coming. Hopefully not too much longer before we know for sure.

Everyday, thousands of MacBook screens are touched and people are bewildered that nothing happens. Touchscreens have been standard for over a decade now - it’s bizarre that there are computers costing thousands of dollars that don’t have touchscreens.

The fact that it’d be used so little didn’t stop Apple from including the touchbar - why on earth haven’t they made their products “Just Work” by including a touchscreen yet?

Seriously - have any kid who hasn’t used a Mac before use a MacBook for a few hours. I guarantee they’ll touch the screen at least once and consider it broken when nothing happens. Literally every phone, tablet, and non-Apple-or-budget laptop has a touchscreen - it’s just a given that a MacBook should, too.
 
Comment

PickUrPoison

macrumors G3
Sep 12, 2017
8,022
10,429
Sunnyvale, CA
I have been a hardcore user of Surface devices (from Surface 1 to 4, then I got tired of that crap) and it is very fatigue-tired to use pencil in screen in a laptop. You end with your arm very tired. I don't expect Apple will release MAC with touch-screen.

What they could release is a "handwritting" pad at keyboard side or with a new "magic unicorn" trackpad. (Said that, sometimes I've handwritten in a trackpad -it is not much reliable but can be done).
Actually I think they intend for an iPad to be the companion device. As in Sidecar.

Agree touch screen/pencil is very unlikely on Mac. The light pen interface was investigated in the 70s, maybe the 60s, and though absolute cursor positioning has its advantages compared to relative positioning, “gorilla arm” is real. It’s very fatiguing, as you say.
 
Last edited:
Comment
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.