Vision Pro accessory could turn any flat surface into a virtual trackpad, with gesture support

An Apple patent granted last week described a wide range of potential Vision Pro accessories, and another specifically describes a hardware device designed to turn your desk – or other flat surface – into a virtual Magic Trackpad, with full gesture support.

At first glance, the patent is a somewhat odd one: using one piece of physical hardware to help emulate a virtual version of another piece of physical hardware, but we can see some potential benefits to Apple’s approach …

Vision Pro accessories

The more general patent describes a modular approach to adding hardware capabilities to a headset like Vision Pro.

These include additional cameras, for an even wider field of view, and a range of different sensors to add capabilities to the headset […]

Given Apple’s strong health focus, it’s not surprising that some of the proposed accessories are health sensors of various kinds […] Apple also describes fashion accessories.

Virtual trackpad

To me, the most exciting possibility for this technology is to replace a Mac and one more external monitors with a headset, either for use when traveling, or even permanently.

I’ve been experimenting with a Meta Quest app that allows multiple virtual Mac monitors – of which I’ll write about more in a separate piece in the next day or two – and have found that while controllers and hand gestures do work, they are no substitute for a Magic Keyboard. I’ve therefore been using the headset with a physical keyboard and trackpad.

Apple’s proposed approach would seem to eliminate the need for this, by turning any flat surface – from a desk to an airline tray table – into a virtual trackpad, with full support for gestures.

While this may be possible simply using the Vision Pro cameras to detect hand gestures, the patent says that this may not be reliable. It suggests that cameras in an external devices, placed on the surface, may do a better job.

Notably, [a] device may be better able to detect taps on the surface as it is also located on the surface and therefore sensors may have a clear line of sight to the location of the tap. In contrast, [another] device may resort to depth analysis to determine if the object has moved in such a way along the z-axis to qualify as a tap on the surface. In some embodiments, the set of one or more criteria includes a requirement that the object is a valid object. For example, an object is a valid object when the object is a digit of a hand. In some embodiments, an object is a valid object when the object is a writing instrument (e.g., pen, pencil) or a stylus.

But the gist of it appears to be that a camera on the flat surface will be better able to detect a gesture like a trackpad tap than cameras mounted in the headset.

The patent illustrations show a small box on the table, and show it detecting other Magic Trackpad-style gestures, like using thumb and forefinger to rotate a photo:

In the above illustration, the box is a physical Vision Pro accessory, while the trackpad is virtually displayed in the headset.

What’s the benefit over a physical trackpad?

This does raise one obvious question: If you’re going to have to buy and carry a physical piece of hardware to emulate a virtual trackpad, why not simply carry a trackpad instead?

This question isn’t directly addressed in the patent, but the illustration does suggest that the hardware device may be smaller than a trackpad. It also describes how it might track a “writing instrument” as well as a hand, so it may be that the accessory has a more flexible role than just emulating a trackpad.

Will we see Vision Pro accessories at launch?

Another big question we noted last time is: Does Apple have unannounced Vision Pro accessories up its sleeve, or is this something for a future model? We’re still at least five months away from launch, likely longer, so there is certainly plenty of time for more announcements by Apple.

Via Patently Apple