Comment: As Apple moves into 3D sensing, what are VR/AR/MR and what are Apple’s plans? [Poll]

Today’s KGI report is just the latest piece of evidence that Apple has serious plans to somehow work with 3D space. Acquisitions, supply chain reports, patents and even comments by Tim Cook all add up to a definite interest in something in the 3D space. The question is: what?

There’s a lot of confusion surrounding terminology, so we thought it would be a good idea to start by distinguishing between the various terms being thrown around in this space – Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) – before discussing Apple’s interest in 3D …

I’ll drop the caps now.

Virtual reality

Virtual reality is a completely simulated environment. The Oculus Rift is the best-known example of this, where a headset completely isolates you from actual reality and immerses you in a totally invented one. While this is associated with large and relatively expensive headsets, even placing your iPhone into a Google Cardboard housing is virtual reality.

VR might be realistic, aiming to fool your eyes into thinking you’re in a real environment, like a flight simulator or war game, or it may be cartoon-like. Either way, it’s all virtual reality.

Augmented reality

Augmented reality mixes the real world with simulated views. Typically, it overlays stuff on reality, as in the image at the top of the piece. As another example, I have an iPad app called Star Tracker HD. Hold it up to the night sky, and you see the actual stars on the screen, but it overlays the camera view with the names and drawings of constellations, names of planets, galaxies and so on.

AR can be used for both practical purposes and entertainment. The best-known example of AR is Pokémon Go, which overlays cartoon-style characters on the real world.

Mixed reality

Mixed reality is nothing more than a marketing term created by Microsoft for its HoloLens. Really, it’s just augmented reality.

FPV

There’s one further term that sometimes gets thrown into the mix: First Person View (FPV). This is most commonly used to describe headsets that allow you to see the view from a remote camera, usually a drone. By seeing what the drone sees through a headset, and sometimes controlling the drone with head movements, it provides a more immersive experience.

Where is Apple headed?

Interestingly, while Apple is usually very reluctant to comment on its future plans, it has given us a pretty clear steer on its 3D interests. The first clue came during an earnings call in July of last year, when Cook said that Pokémon Go showed that augmented reality can be really great, and that the company sees great opportunities for it.

In an ABC News interview last September, Cook said that while VR can be really cool, it is less appealing than AR.

Virtual reality sort of encloses and immerses the person into an experience that can be really cool, but probably has a lower commercial interest over time. Less people will be interested in that.

He was even more dismissive of VR in another interview a month later.

VR, I think, has some interesting applications, but I don’t think it’s a broad-based technology like AR,” Cook explained. “Augmented reality will take some time to get right, but I do think that it’s profound. We might … have a more productive conversation, if both of us have an AR experience standing here, right? And so I think that things like these are better when they’re incorporated without becoming a barrier to our talking. … You want the technology to amplify it, not to be a barrier.

And just this month, Cook was again clear that it was AR, not VR, that excites him.

I’m excited about Augmented Reality because unlike Virtual Reality which closes the world out, AR allows individuals to be present in the world but hopefully allows an improvement on what’s happening presently. Most people don’t want to lock themselves out from the world for a long period of time and today you can’t do that because you get sick from it. With AR you can, not be engrossed in something, but have it be a part of your world, of your conversation. That has resonance.

So whatever Apple’s plans, we can be pretty confident that its offerings will be in the field of augmented reality. The specifics, of course, are a matter for speculation. While the KGI note references the possibility of using the front camera and a new 3D sensor as a way to generate a 3D selfie for use in games, this appears to be merely an example rather than a specific plan.

The iPhone and iPad are already suitable platforms for AR technology, but we’ve also heard reports of a wearable glasses product and use of AR in the maybe-someday Apple Car project.

How excited are you about Apple’s plans for AR? Please take our poll, and share in the comments any ideas you have about how Apple might use the technology.