Creating Full Sensory Experiences: The Future of AR/VR/MR/XR

Author:
Anne Corning

Humans experience the world through five senses: sight, hearing, touch, smell, and taste. Typically, sight takes priority as the first and most important source we turn to for information about our environment. While there can be some cultural variations around the globe, on average 80% of human sensory impressions are gathered through our eyes.1 Virtual reality (VR) is called an “immersive” technology, because VR device displays fill the entire field of view of the wearers, thereby immersing them in a complete visual experience.

Other sensory input can also be a powerful component of our experiences—for example, the smell of fresh mown grass may recall childhood memories, or the sound of a baby crying evokes anxiety. Most VR devices now enhance the visual experience by adding sound, using built-in speakers to provide auditory information such as background music and sound effect cues that match what is happening visually.

Some augmented- (AR) virtual- (VR) and mixed-reality (MR) device makers have also added haptic elements to the user experience using integrated gloves, suits, or other touch devices than enable a user to “feel” resistance and vibrations as they interact with virtual objects. Industry experts predict that taste will also become an integral part AR/VR/MR experiences at some point in the next decade.

Now, designers are working on the next sensory innovation: smell. The term Extended Reality (XR) is now being used to describe the future of interactive, multi-sensory virtual experiences (“XR” is also commonly used to refer to all types of AR/VR/MR in the aggregate). This post will take a look at the coming wave of scent-enhanced AR/VR/MR, or XR.

History of Sensory Virtual Reality

The concept of adding multiple sensory inputs to a user’s (or audience’s) experience is not a new one. Back in 1955, filmmaker Morton Heilig described his vision of a multi-sensory theater experience. "If we're going to step through the window into another world," he said, "why not go the whole way?" He built a prototype and in 1962 introduced Sensorama. It was a mechanical device that displayed stereoscopic 3D images in a wide-angle view and incorporated body tilting, stereo sound, wind, and aromas. Heilig—sometimes called the Father of Virtual Reality—created short films for the device to display including a bicycle ride through Brooklyn.

 

Schematic drawing (left) and marketing promotion (right) for 1962’s Sensorama virtual reality device.

Inventors tried a few other approaches to get scent to catch on in the movie industry. In 1959, AromaRama technology wafted scent through the air conditioning system of a theater showing Behind the Great Wall. That same year, a famous (infamous) attempt was made with Smell-o-Vision technology, pumping scents into the audience from vents below the theater seats. It was used for one movie, and one movie only, Scent of Mystery.

 

"The Process to end all Processes"

Decades later, in 1984, Six Flags Power Plan amusement park in Baltimore premiered what was billed as the first “4D” movie experience. Its Sensorium, a theater with rumbling seats and scents, closed along with the park facility five years later. Small theaters boasting similar 4D experiences (or 4DX) can still be found here and there within theme parks or museum exhibits. (A handful of Regal 4DX theaters remain in various locations across the United States – whether these theaters survive the pandemic remains to be seen.)

Extended Reality and Olfactory Engineering

What makes scent experiences possible is the science of olfactory engineering. It is remarkably complex to analyze a particular scent at the molecular level, break it down into its chemical components and proportions, and then replicate human response to various molecular combinations.2 Scientists have been working for decades to understand how to design both scents and scent delivery systems that will realistically enhance the virtual experience for users. Details such as the volume and concentration of each scent emitted by a system, timing, and refresh rates all affect the user’s perception and experience.3

In the last several years, as XR technologies have evolved, design engineers have continued to explore the idea of incorporating smell into the user experience. Recent innovations include various new scent devices, from face masks to amulets to booths:

  • OVR Technology has developed a device that aims to recreate a real-world scent experience using optical engineering algorithms. With a library of 250 different scents so far, their device fits across the user’s face just below a VR headset, connected via USB. It emits 0.1 millisecond bursts of scent and can change between scents in 20 milliseconds to create a sense of immediacy for the wearer. Called the ION, OVR’s device is aimed at training applications initially (e.g., in the military), and potentially gaming and retail applications down the road.4 

 

  
OVR Technology’s ION sensory unit (left) and shown in use with an HTC Vive headset (right). (Images © OVR Technology)

  • Japanese company Vaqso has taken a simpler approach, releasing a scent unit that they say can be used with any head-mounted device (HMD), using a Velcro strap and USB connection. The unit contains five cartridges of pre-mixed scents. Their collection includes one scent called Zombie.

 

 
The Vaqso scent unit attaches to a VR headset and offers five scent compartments to coordinate with a game experience. (Image © Vaqso)

  • Brooklyn-based start-up company Feelreal has developed a mask unit that clips onto a VR headset to provide an integrated sensory experience for games and movies or as an aromatherapy and medication tool. The mask delivers not only scents (via exchangeable cartridges), but also vibration, heat, wind (puffs of air), and mist to enhance the user’s experience. Each cartridge holds nine individual aroma capsules from Feelreal’s collection of 255 scents. 

However, Feelreal has hit a product release delay as their vapor-based capsule system has recently come under the purview of the U.S. Food & Drug Administration along with vapor-based scented e-cigarette and “vaping” devices, although Feelreal uses only food-grade scent components for user safety.5

 

The Feelreal mask unit shown with four types of VR devices; it clips onto the headset, and connects via Bluetooth. (Image © Feelreal, Source)

  • Taking a different approach from the liquid scent-based products above, start-up Aromajoin uses solid state scent cartridges encased in a tennis-ball sized device (the Aroma Shooter) that can be tethered to a VR headset, or even worn as amulet around the neck (the “Aroma Shooter Mini,” slated for release in 2021) allowing you to carry your digital scent device wherever you go.

 

 
The Aroma Shooter is a free-standing, hexagon-shaped scent-delivery unit. (Image © Aromajoin)

  • Eschewing face masks and devices entirely (apart from the standard VR headset), Netherlands-based SENSIK has created a sensory pod. It’s an enclosed booth that envelops the user with temperature, air, sound, scent, and visual stimuli to create a fully immersive virtual experience—sort of like being in a VR isolation tank.

The SENSIK sensory pod experience.

The Future Quality of XR Devices and Experiences

Industry prognosticators believe that sensory integration into XR and other digital platforms will take off in the next decade. By 2030 we may regularly experience scents during movies, feel the weight and motion of virtual objects, and wear earphones that translate languages in real time. AR devices may even be able to connect to human thoughts.6

But no matter what sensory technologies may be added to our devices, the first imperative for XR device makers is still the visual aspect of the user experience. XR headsets deliver immersive experiences thanks to significant and ongoing hardware and software development on the part of designers and manufacturers. In the R&D lab and in production, these devices demand rigorous testing and quality assurance to achieve the intended user experience.

Poorly designed headsets can deliver poor quality images and even cause issues like motion sickness if image alignment, clarity, and responsiveness are not perfectly coordinated. Display images must also be defect-free at high resolution. Because they are viewed so close to the eye, the tiniest blemish or irregularity in an XR display becomes glaringly obvious, detracting from the user experience.

XR Display Metrology Solution

The Radiant Vision Systems AR/VR Lens system accommodates the diverse viewing geometries of XR devices with a front-located lens aperture, which enables the imaging system’s entrance pupil to be positioned inside headsets at the same location as a human eye. Applying wide-field-of-view (FOV) optics, the system captures an entire display up to 120° horizontal and 80° vertical in a single measurement.

Used with Radiant’s TT-ARVR™ Software and a ProMetric® Imaging Photometer or Colorimeter, the AR/VR Lens enables device developers and manufacturers to evaluate critical visual performance qualities such as brightness, color, contrast, uniformity, and sharpness of projected images.

 

The AR/VR Lens replicates the position and size of the human pupil within a near-eye device (left), to capture the user’s full field of view. TT-ARVR Software enables precise analysis of display performance such as the MTF measurement shown here (right).

 

CITATIONS

  1. Why Good Vision is So Important”, Zeiss. (Retrieved July 23, 2020)
  2. Keller, A., et al., “Reverse-engineering human olfactory perception from chemical features of odor molecules”, Science, February 24, 2017. DOI: 10.1126/science.aal2014
  3. Zybura, M. and Eskeland, G., “Olfaction for Virtual Reality”, University of Washington Industrial Engineering, Winter 1999
  4. McKinnon, A., “Olfactory Engineering: the Future of Multisensory VR Could Be Scent-Based”, VR Scout, June 15, 2020
  5. Hayden, S., “Feelreal VR Scent Mask Hits Roadblock Amidst Crackdown on Flavored Vaping Products”, Road to VR, Jan 2, 2020.
  6. Bayern, M., “Internet of Senses is on the horizon, thanks to AR and VR”, TechRepublic, December 10, 2019 
     
radiant vision system wechat