Apple Has Reignited the Augmented Reality Discussion.

Hello dear readers, and welcome to the Friday post. You'll notice a couple of changes in the next couple of weeks. First, starting today all posts will be available to subscribers only. Subscriptions are still free (and there will always be a free tier), but folks who don’t subscribe will only see the first few paragraphs. 

My reason for the shift is simple: to help prevent my work being hoovered up by AI companies. More and more writers are adopting this same strategy and 404 Media has an excellent post on their decision to do so. I’m not saying that my work is comparable to theirs - it’s not - but I spend a lot of time researching, writing, and editing Design Flaw and I’d rather my hard work not be used to further the AI-fueled enshittification of the internet.

For you podcast dorks, I’m planning to add audio versions of my posts, probably starting next week. I was going to record this week’s but I’ve been fighting a cold and am definitely not audio-ready.

And now, on with your regularly-scheduled programming.

This week I had two students ask my thoughts on augmented reality (AR) and UX design. AR is hardly new, but last week’s launch of the Apple Vision Pro has brought the topic to the forefront. The nearest Apple Store is more than an hour away from me, so I haven’t tried a Vision Pro yet. But I have scheduled a demo for this weekend and will report back on how it goes. For today, I’ll focus on AR more generally.

First off, let’s clear up some terms.

Augmented Reality (AR) integrates digital content into the real world, enhancing users' perception by overlaying graphics, text, or animations onto the physical environment. Think smartphone apps or Meta Ray-Ban.

Virtual Reality (VR) immerses users entirely into digital environments via specialized equipment such as VR headsets, isolating them from the physical world to interact solely with the virtual realm. Meta Quest and HTC Vive are popular examples.

Mixed Reality (MR) blends aspects of both AR and VR, enabling digital content to interact with and appear as part of the real world, facilitating simultaneous interaction with both digital and physical elements. Mixed-reality devices typically employ advanced technologies like depth-sensing cameras and spatial mapping to seamlessly merge the two realities. Microsoft HoloLens is a good example.

There's a lot of overlap among the three and I think it’s fair to argue that the Vision Pro is more MR than AR. Forbes says much the same and suggests that Apple is letting itself off the hook: “By using the term 'Spatial Computing,' Apple doesn’t let itself get boxed into industry preconceptions and future devices might be closer to AR glasses.”

The lines are blurry here, so for the sake of clarity I’m going to simply use “augmented reality” unless the subject is obviously VR or MR. (I could use the more generic "extended reality" (XR) but it's not commonly used and we already have enough alternate realities in this article.)

No matter what you call it, this tech isn’t as new as you think, having been invented in 1968 by a Harvard scientist named Ivan Sutherland. AR was first used commercially (and by “commercially” I mean “for ads”) in 2008. Most of us think of these technologies in terms of games or entertainment – Pokemon Go, anyone? Also, theme parks have embraced AR in a big way.

Apple vs the Rest

Among the major players there are two basic device types: VR-style goggles and glasses-type devices. Meta Quest and Vision Pro are in the former category, Microsoft HoloLens and Meta Ray-Bans fall into the latter. (Also Google Glass, but I'm leaving that one alone.)

True to form for Apple, there's something extra, and that extra is EyeSight. While EyeSight is a really neat feature, the Vision Pro is still a closed device much like the Quest.

The HoloLens and Meta Ray-Ban on the other hand, are basically regular glasses with some augmented reality features built in. While you're not going to get a full-on virtual reality experience with these devices, you also aren't going to get sick.

Both Apple and Meta pitch their AR devices for work as well as play, but office work is the primary focus. In comparison, the HoloLens works much better in industrial and lab environments where being (almost) completely detached from your environment can be incredibly dangerous.

Industry Leads the Way

Like many other technologies, augmented and mixed reality have found a home - and I would argue the most viable use cases, at least for now - in industrial applications. It's being used in architecture, manufacturing, maintenance, and more. For example, AR applications can allow engineers and designers to visualize systems in three dimensions.

This means that they can see how different parts will fit and operate before building any physical prototypes. To use the aviation industry as an example, designers can project digital models of aircraft parts onto a physical space, allowing them to verify the design, assess of spatial requirements, and identify potential issues in the early stages of the design process. This means less rework, lower costs, and faster design cycles.

Maintenance technicians can also use AR in their by superimposing digital information and step-by-step instructions on their field of view while working on equipment. It also enables remote assistance, where experts can remotely guide on-site technicians through complex procedures by sharing what they see in real-time and annotating on-site tech's view with instructions.

Many industries are experiencing a shortage of experienced techs, so remote assistance can reduce repair times while minimizing stress and travel time for senior technicians.

Design Opportunities Ahead

Putting aside the Vision Pro hype, there are certainly some opportunities for UX designers. As usual, Apple is taking a different approach, pitching a device that works nicely across both work and consumer applications. (Yes, technically Meta is doing the same, but no one wants to hang out with weird legless avatars of their co-workers. Even Zuckerberg seems to have figured this out.)

We're going to see significant new applications as this tech continues to improve; expect increasing adoption in manufacturing, logistics, medicine, education, and more. While I'm not necessarily convinced that augmented reality is going to replace our screens, I do think we're going to see adoption accelerate, especially as business grapples with staffing issues.

For those of used to designing web pages and mobile apps, augmented reality requires some pretty big changes in thinking and approach. For starters, designing for the 3D world means that designers now have a Z axis to deal with. There are also lots of new interaction types: eye movement tracking, thumb controls, reach depth, and head tilt are just a few.

I haven't seen this widely discussed, but I also think that AR can have substantial benefits to people with disabilities. Giving people access to a broader collection of interactions opens some exciting possibilities.

Designers can get started with extended reality applications without dropping $3,000 on a headset. Both Microsoft and Apple have extensive design guidelines for their products, including Figma design kits.

The Caveats

Augmented reality is pretty nifty and has lots of real-world applications, but as the old saying goes, there's no such thing as a free lunch. Many people - myself included - are susceptible to motion sickness and will be unable to use many of these applications. Even watching a 3D movie gives me a headache or nausea, so wearing a full headset is probably not going to be better.

I'm skeptical of some of Apple's purported use cases. It's telling that most of the Vision Pro marketing imagery shows single people; there are no families watching a movie or friends catching a game. Putting aside the cost (outfitting a family of four would cost close to $15,000), what gets lost when people share an activity in the same room and yet are physically disconnected?

A recent Stanford study hints at an answer: these experiences can, in fact disrupt social connection:

Augmented reality and virtual reality also open up new avenues for abuse and increased safety concerns. It's important for designers and product managers to consider how our applications can be misused and include appropriate safety measures.

Apple has definitely brought AR back to the forefront, and in a big way. It will be at least a few years before things really shake out, but if anyone can make it stick, it's Apple. They tend to produce quality products with a high level of fit and finish. The possibilities are there, but so are the problems. As the devices get smaller, lighter, and cheaper, it will be up to us to to build safe, usable applications that blend the real world with the virtual.