A new Apple patent application describes a way to use gaze tracking to video AR experiences using a ‘head-mounted display’…

Apple is widely believed to be working on an augmented reality headset generally referred to as Apple Glasses. The patent application is in relation to videoing the entire experience, namely the real-world environment over-layed with the AR content.

Rather than have a camera lens physically move to track your eye movements, the patent describes using multiple image sensors, and then using a mix of sensor-selection and digital zoom to capture what you’re looking at.

Head-mounted displays are used to provide virtual reality, augmented reality, and/or mixed reality experiences for users. Video from a virtual reality or mixed reality experience may be recorded for later playback or analysis. Gaze-tracking sensors may be used to provide an indication of where a subject person is currently looking […]

The subject matter described in this specification can be embodied in systems that include one or more gaze-tracking sensors, one or more image sensors, and a processing apparatus configured to: access gaze data captured using the one or more gaze-tracking sensors; apply a temporal filter to the gaze data to obtain a smoothed gaze estimate; determine a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view of the one or more image sensors; access a frame of video captured using the one or more image sensors; and apply signal processing to the frame of video based on the region of interest to obtain an enhanced frame of video.

The text notes that tracking eye movements in this way could result in jerky video footage, so it includes an option for smoothing it.

Moving the recording box with the gaze center point can result in a jittery video. Some implementations run a smoothing or stabilization algorithm on top of gaze center points to avoid jitter in recorded video and get a more pleasant recording.

Most of the speculation about Apple Glasses has been assuming that they are effectively an iPhone accessory, with the AR content wirelessly transmitted from the phone. However, Patently Apple notes that the patent is broader than that.

Apple notes that smartglasses will be able to work with an iPhone, an iPad, a MacBook, a desktop computer, a smart TV, a smart speaker and more.

Recent reports have suggested that the earliest launch date for an Apple Glasses product would be 2022, and quite possibly later than that.

One report says Apple believes this type of device will replace smartphones in around a decade, though personally I would see Apple Glasses as a replacement for my Apple Watch, not my iPhone.

Concept image: Martin Hajek/iDropNews

FTC: We use income earning auto affiliate links. More.

Apple July 4 sale Adorama

Check out 9to5Mac on YouTube for more Apple news:

You’re reading 9to5Mac — experts who break news about Apple and its surrounding ecosystem, day after day. Be sure to check out our homepage for all the latest news, and follow 9to5Mac on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our exclusive stories, reviews, how-tos, and subscribe to our YouTube channel

About the Author

Ben Lovejoy

Ben Lovejoy is a British technology writer and EU Editor for 9to5Mac. He’s known for his op-eds and diary pieces, exploring his experience of Apple products over time, for a more rounded review. He also writes fiction, with two technothriller novels, a couple of SF shorts and a rom-com!

Ben Lovejoy's favorite gear