Latest ARKit demo shows off 3D finger painting through augmented reality

We’ve already seen dozens of examples of Apple’s new ARKit framework in use from navigation enhancements to the Falcon 9 landing, and now developer Osama Abdel-Karim has shown off augmented reality finger printing with a 3D twist.

The latest ARKit demo makes it possible to simply draw in the air with your finger as if you were holding a pen, and ink realistically appears through software as if it were on the surface. After your illustration is complete, the demo optionally lets you add an effect that brings the AR drawing to life as a 3D object.

The 3D finger painting demo uses a combination of Apple’s ARKit framework and an iOS 11 tool called Vision that tracks the fingertip through software to create the realistic finger painting effect. The developer writes more about the process:

One of the cool libraries that Apple introduced in iOS 11 is Vision Framework. It provides some computer vision techniques in a pretty handy and efficient way. In particular, we are going to use the object tracking technique. Object tracking works as follows: First, we provide it with an image and coordinates of a square within the image boundaries for the object we want to track. After that we call some function to initialize tracking. Finally, we feed in a new image in which the position of that object changed and the analysis result of the previous operation. Given that, it will return for us the object’s new location.

A full detailed write-up including source code further explains how the demonstration was created. See it in action below:

Apple first unveiled ARKit as part of iOS 11 for iPhone and iPad back in June at WWDC 2017. So far we’ve only seen hobby projects of how the framework can be used in the real world. When iOS 11 ships later this fall, we’re expecting to see tons of ARKit applications from launch partners including IKEA.