Craig Federighi explains how the Apple Pencil Scribble feature in iPadOS 14 was developed

One of the tentpole features of iPadOS 14 is a new feature called Scribble, which automatically converts handwritten notes from Apple Pencil into typed text. Speaking to Popular Mechanics, Apple’s senior vice president of software engineering Craig Federighi offered additional details on how the feature works.

The iPadOS 14 Scribble feature allows Apple Pencil users to handwrite into almost any text field, such as the address bar in Safari or the compose field in the Messages app. The handwriting is then automatically converted into typed text, essentially meaning that users don’t have to switch back and forth between using the keyboard and Apple Pencil for input.

Federighi explained to Popular Mechanics that the basis of the Scribble feature is data gathering. During the feature’s development, Apple found people all over the world and had them write things with Apple Pencil:

“When it comes to understanding (handwriting) strokes, we do data-gathering. We find people all over the world, and have them write things,” says Craig Federighi, senior vice president of software engineering at Apple. “We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation.”

That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple’s tech, static examples weren’t enough. They needed to see the strokes that formed each letter. “If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written.”

As with most machine learning features of Apple products, all of the processing is done on-device, which presents another set of challenges. “It’s gotta be happening in real time, right now, on the device that you’re holding,” Federighi said. “Which means that the computational power of the device has to be such that it can do that level of processing locally.”

If you want to get a look at the iPadOS 14 Scribble feature in action, you can check out our full video with it and over 50 other new iPadOS 14 features and changes right here.