Video passthrough for mixed reality headsets should be studied more | Stanford researchers

Virtual reality headsets are notorious for making many people who try them nauseous.

The mixed reality headsets with video passthrough — which allow you to see through a goggles screen to the outside world — don’t have as severe a problem when it comes to motion sickness, according to a study by Stanford University‘s Jeremy Bailenson and ten other researchers. But it can still cause visual impacts and some simulator sickness.

And there should be more study of the effects of passthrough on people, the report said.

“We conclude that the passthrough experience lends itself to many applications, but will also likely cause visual aftereffects, lapses in judgments of distance, induce simulator sickness, and interfere with social connection,” the report said. “We recommend caution and restraint for companies lobbying for daily use of these headsets, and urge scholars to rigorously and longitudinally study this phenomenon.”

As technology giants such as Apple and Meta spearhead the production of headsets designed to immerse users in mixed reality experiences, the paper sheds light on the psychological ramifications of relying on these devices for extended periods. The Apple Vision Pro debuts on Friday.

Millions of people will soon be spending hours each day relying on cameras and screens to
show them the surrounding world. Apple, Meta and other companies are mass-producing
headsets that block out light from the real world, and instead rely on passthrough video as an
enabling technology for mixed reality, the report said.

The authors, a diverse group with extensive experience across various passthrough headsets, sought to document their experiences to provide valuable insights for scholars, industry leaders, and organizations.

Mixed Reality (MR) headsets seamlessly blend virtual content with the physical world. They’re becoming the center of attention in the XR world with devices such as the Meta Quest 3 and Apple Vision Pro hitting the market.

The latest iterations focus on passthrough video, a technology where users are presented with real-time, stereoscopic, high-resolution video of the world inside the headset, effectively blocking out the real world. The paper delves into the temporal nature of passthrough usage, highlighting its shift towards being a medium for everyday activities.

Technological advancements and limitations

While the newest passthrough headsets boast advancements such as high-quality cameras, screens, and low latency, the authors acknowledge certain limitations. Parameters such as field of view, distortion, latency, and resolution still fall short of human vision standards. Despite these drawbacks, the paper underscores the potential of passthrough for extended use, both indoors and outdoors.

Prior to crafting the paper, the authors engaged in extensive passthrough usage across various headsets, including the Apple Vision Pro, Quest Pro, Quest 3, Varjo XR-3, and Night Vision Goggles.

The authors designed an Institutional Review Board (IRB)-approved protocol to systematically understand the technology. Each author spent approximately 140 minutes over two or three sessions using the Meta Quest 3, resulting in a total of 30 sessions. Activities varied, including estimating distances, walking outdoors, playing games, and engaging in conversations.

Insights from passthrough experiences

The authors describe their varied passthrough experiences, illustrating the peculiarities of relying on this technology. Activities like eating meals proved challenging due to distortion, while public spaces presented a sense of diminished presence for individuals in the background.

Motor tasks, such as pressing buttons in elevators, became more complex. The abrupt transitions from bright to dark scenes were particularly jarring.

Psychological implications and recommendations

Drawing from their experiences, the authors caution against the unrestricted daily use of passthrough headsets, highlighting potential visual after effects, lapses in judgments of distance, simulator sickness, and interference with social connections. The paper concludes with a call for scholars to rigorously and longitudinally study the psychological consequences of prolonged passthrough usage.

As these headsets become more integral to daily life, the study serves as a timely exploration of the impact of this transformative technology on our psychological well-being and everyday activities.

Besides Bailenson, the authors include Brian Beams, James Brown, Cyan DeVeaux, Eugy Han, Anna C. M.
Queiroz, Rabindra Ratan, Monique Santoso, Tara Srirangarajan, Yujie Tao and Portia Wang.

This work is unfunded, and the authors have no conflict of interest.

While using passthrough, a person does not see light from the real world, but instead relies on stereoscopic, color, high resolution, low latency, real-time video of the world which is displayed on small
screens inside a headset, the study noted.

There have been thousands of studies in psychology, communication, and human-computer interaction which study human behavior in MR, but that research has tended to focus specifically on the virtual content.

Simply put, there is a dearth of research focusing on passthrough as a medium one uses to perform everyday activities while viewing and navigating the real world. Because the newest headsets are light, the cameras and screens are high quality, and the overall system latency is low, passthrough can now be easily used for hours at a time, indoors and outdoors. This change in the temporal nature of passthrough usage necessitates careful consideration of this technology’s implications.

A person wearing a modern passthrough MR must instead rely on stereoscopic, color, high resolution, low latency, real-time video of the world. Eating a meal is strange due to distortion of the food. In public, people in the background seem less present. Motor tasks such as pressing a button in an elevator become challenging. Changes from bright to dark scenes are particularly jarring.

Headsets that utilize passthrough do their best to replicate the sights from the real world, but of course none are equal to actual human vision. Seeing the world through a clipped window can be challenging.

In summary, while the technology improves with every new headset and software update, passthrough falls far short of the human visual system–they are slower, grainier, and distorted, and cut off a large chunk of one’s field of view.

Distortion has also been shown to produce nausea, oculomotor discomfort, and disorientation. Moreover, this distortion from passthrough can impact the perception of one’s own body, by inducing the feeling that certain body parts are mislocated. Distance estimation is also worse when wearing a headset.

Simulator sickness

The researchers’ field notes showed that a majority of passthrough sessions caused simulator sickness symptoms, ranging from symptoms of eye strain, nausea, dizziness, and headache. In general, the 11 authors spend a lot of time each week in various MR headsets.

For over half of the group, who typically do not easily succumb to simulator sickness, to do so is quite notable, especially given that individual sessions were typically less than an hour.

One of the most accepted theories of simulator sickness in head-mounted displays is the sensory conflict
theory, under which scholars argue that users may experience sensations of nausea, dizziness, stomach awareness, head fullness, and sweating as a result of mismatches between the visual system, vestibular system, and nonvestibular proprioceptors.

There are several factors that contribute to simulator sickness, related to the user, such as age and gender; related to the experience, such as the locomotion type and duration of the content; and related to the system, such as field of view, latency, and resolution.

Simulator sickness changes with experience; short exposure to the same VR application on two separate days can reduce simulator sickness by 35-40% over time; with continued reduction over several subsequent exposures.

The researchers said that passthrough video will be the norm for MR headsets over the next few years. Whether or not this technology becomes pervasive, or just a flash in the pan similar to three-dimensional
television, remains to be seen. Passthrough will enable a number of useful MR experiences, and will allow fully immersed virtual reality users to quickly check-in with the real world without having to remove their headsets.

Researchers have also made compelling arguments for clinical use. For example, passthrough should be more effective than corrective eyewear, given that the entire depth range of the real world is displayed on the same focal plane, especially for presbyopes who struggle to focus on nearby objects.

On the other hand, the researchers urge caution to the companies pioneering this industry and invite researchers to examine topics that will help guide development and use of the technology in safe and responsible ways. Previous research on prism glasses and long-term headset use suggests that there will be consequences to using passthrough as an everyday medium, and simply put, there has not been any direct research on this topic, the report said.

Scholars should focus on how passthrough changes affect, cognition, and behavior during use,
but also on aftereffects.

For example, there are likely to be developmental issues. According to a 2021 survey, 17%
percent of children between the ages of eight and 18 own a VR headset and about one in twenty-five child users are donning headsets every day. And there isn’t any research on kids and MR.

Meta recently lowered its age limit from 13 to 10. And Apple is explicitly advertising that people can use their headsets “all day long,” and their safety guidelines will remain unknown until the official release of the headset. Meta has clear and useful health and safety guidelines, but they are designed for problems which might occur during use.

In other words, they offer specific strategies to avoid collisions and manage simulator sickness, but do not offer insights regarding long-term passthrough usage. Even the safety guidelines encourage extended use, urging users to start with thirty-minute sessions, but then to “increase the amount of time using your Meta Quest gradually as you grow accustomed to the experience.” Although Meta acknowledges the distortions in color and space perception in passthrough mode, it is unclear how a consumer should take action on this advice, the report said.

The report suggests creating guidelines for the amount of time people use passthrough each day, and to create schedules that incorporate breaks, take context and location into account, and put other guard-rails in place.

“Given that these strategies have failed epically with smartphones, we are not optimistic. If Apple and Meta create fantastic MR content that utilizes passthrough, people will most likely use it often,” the report said.

The report also suggested users undergo training when using the headsets for the first time. For example, soldiers spend dozens of hours learning how to use night vision goggles before putting them to use in the field, not just one time when they first use the goggles.

The tech is going to get better. But in the meantime, millions of people will walk around, cut off from light from the real world, instead seeing family, friends, cars, pets, and sharp knives through imperfect video, the report said. The researchers are also curious about the impact of social absence while wearing headsets.

“We believe that passthrough adoption for near-term MR use is very likely, but of course there is always uncertainty in predicting the future. It may be difficult to imagine the world portrayed by the movie Ready Player One, where everyone emulates George Stratton, Thad Starner, and Steve Mann, wearing headsets all day long in their public and private lives. Few people can even fathom a norm in which face-to-face interaction becomes largely mediated by passthrough headsets. But the largest technology companies are telling us, very transparently, that they are building this world. We should listen to them,” the report said.