Augmented Medicine: How 3D AR is Changing Healthcare

Augmented reality (AR) falls into the category of “spatial computing”—a merger of digital and physical space. Nowhere does this concept hold greater potential for life-changing applications than in medicine. Use of AR, along with virtual (VR) and mixed reality (MR), in the healthcare industry is projected to reach a global market size of US$ 7.05 billion by 2026, growing at an explosive 28.3% compound annual growth rate (CAGR),¹ including hardware and software. AR alone is projected to be a US$ 1.2 billion market in 2020, growing to $5.1 billion in 2025.²

Augmented reality is helping to advance safety and efficiency in medicine and healthcare, from both the provider and the patient sides. It has already led to changes in areas such as minimally invasive surgery, medical education, diagnostics, medication adherence, and disease detection. One of the biggest breakthroughs provided by new AR applications is their ability to present three-dimensional visualizations using the data from traditional 2D medical imaging technologies like ultrasound, X-ray, and MRI (magnetic resonance imaging). It stands to reason: human bodies are 3D objects, so being able to view a 3D holographic representation of a patient’s body, organs, or other internal structures is a big leap forward for diagnosis and treatment.

Patient-Focused AR Applications

Augmenting Vision. Patients with retinitis pigmentosa, a degenerative eye disease that causes poor vision, can be aided by AR glasses. This inherited disease leads to difficulty with peripheral vision and seeing in low-light conditions, and cannot be corrected with glasses, contact lenses, medication, or surgery.

But a new study out of the University of Southern California found that AR glasses can improve patient mobility by 50% and their grasp performance (ability to grasp a specific item even when it’s located behind other items) by 70%. This solution focuses on AR’s capabilities as “assistive technology” to enhance—not replace—the natural human sense of sight, using brighter colors to spotlight objects and obstacles in the patient’s environment.

Take Your Meds. Anyone who’s pickup up a prescription lately is familiar with the long, printed handouts covered with fine print about the use, administration, effects, side-effects, and other information about each and every medication. Who can read and absorb all of that?! Augmented reality is helping improve patients' understanding and retention of important drug knowledge by providing 3D models of medication mechanisms and how they work inside the body.

Medication adherence (taking medication as prescribed) is one of the most important factors for successful treatment. The Centers for Disease Control and Prevention estimate that non-adherence causes 30%-50% of chronic disease treatment failures, and 125,000 deaths per year.³ New AR apps have been designed to help patients adhere to their medication regimen (e.g., with medication reminders or visual support for a family member who needs to administer an injection), including one using the Microsoft Hololens AR device.

A new mHealth initiative, just launched in the UK, is using AR and gaming techniques to help young patients with asthma learn to use their inhalers properly. While still early, the application—called MySpira, developed by Orbital Media and the University of Suffolk, showed that improvement in the children's medication management effectivness by 26% over just watching a video, and 70% over paper handouts.

 

The MySpira AR app that helps kids with asthma learn to use their inhaler correctly. (Image Source)

Provider-Focused AR Applications

Ultrasonic and Radiographic Training. Ultrasound scanning, X-rays, and MRI are among the most valuable tools in the medical toolkit. By projecting ultra-high-frequency sound waves (ultrasound), X-rays (radiography and computed tomography), and magnetic and radio waves (MRIs), medical providers can “see” inside the body for diagnosis, assessing a patient’s condition, and planning treatments or surgeries.

The medical imaging industry has focused tremendous effort on improving these technologies to enhance the quality and readability of images. At the same time, physicians, radiologists, and other imaging technicians have worked diligently to develop their skills at interpreting images. Nevertheless, errors happen. One study estimated that 3%-5% of errors are due to cognitive or perceptual errors by the technician across all imaging modalities.⁴

One challenge for pathologists, radiologists, and other professionals who read medical images is that they typically have to look at a series of 2D scans/films and then integrate these multiple images in their mind to visualize the 3D reality of the organ, tumor, or other image on film. The power of AR applications are their ability to integrate image data to render a complete 3D holographic, interactive image of the patient. This not only helps practicing professionals, but has been useful in training imaging technicians how to perform diagnosis from the start.

In one study, ultrasound students used a tablet-based AR system to visualize a patient’s heart. Using the system “gives the student quick insight into which anatomical structures they are viewing even if their ability to interpret diagnostic images is not yet fully formed…”⁵ and can help speed up the typically year-long training period.

 

Using smart glasses, a radiologist can view and interact with CT, MRI, and ultrasound scans in three dimensions. (Image Source)

Heart to Heart. Cardiologists at Boston’s Beth Israel Deaconess Medical Center are working on using AR to help surgeons visualize scarring in the heart as they perform surgery. Imagery from cardiac MRI scans is projected as a 3D holograph of myocardial scarring onto a glass screen that’s worn like a mask on the surgeon’s face. During the procedure, the surgeons can interact with the data without touching the screen or a mouse, helping to maintain a sterile environment. So far tested only on animal patients, this AR imaging technique will next be tested for treatments of arrhythmia by merging scar information with electrophysiology data.

Image-Guided Surgery Goes 3D. Many minimally invasive procedures use image-guided systems to help physician visualize and perform treatment inside the body through a tiny incision, thus avoiding the risks of open surgery. New AR/MR technology is poised to take the images currently displayed on 2D screens into the 3D realm to improve a surgeon’s ability to “see” what they’re doing during a procedure. Philips and Microsoft have partnered to use the HoloLens 2 holographic computing platform to enable hands-free, heads-up interaction with 3D digital objects during an operation.

 

AR systems can improve workflow and procedural navigation in the cath lab: physicians wearing Microsoft Hololens 2 visors can view and interact with 3D holograms above the patient on the table, manipulating the image with only voice and hand gesture commands to maintain a sterile environment. (Image Source)

Device Quality and Efficacy Is Critical

When people’s health and lives are on the line, any AR, VR, or MR systems used in a diagnostic or treatment settings need to be extremely reliable. Any information displayed on-screen must be readable and clear to the user while not interfering with necessary visibility of the surrounding environment. To ensure the quality and performance of AR/VR/MR displays such as headsets, goggles, and glasses that may be used for medical purposes, the hardware requires precise functional testing that comes as close as possible to replicating a user’s visual experience. Variations in luminance and color, contrast against the background environment, and other issues with display uniformity and clarity are unacceptable. Because images are viewed close to the eye in these systems, even small defects such as a few dead pixels are magnified and may impact visualization.

Radiant Vision Systems has developed an AR/VR Lens solution that features a unique optical design specially engineered for measuring near-eye displays (NEDs), such as those integrated into VR, AR, and MR headsets. The lens design simulates the size, position, and field of view of the human eye to accurately capture and evaluate the what a user will see.

 

Radiant ProMetric® Imaging Colorimeter with the AR/VR Lens, shown with TT-ARVR™ Analysis Software.

The aperture of the AR/VR Lens is 3.6mm—replicating the size of a human pupil—and is located on the front of the lens, enabling the imaging system’s entrance pupil (where light is received) to be positioned within NED headsets at the same location as the human eye. Paired with a Radiant ProMetric^® Imaging Photometer or Colorimeter, the AR/VR Lens captures high-resolution images of displays inside the headset for pixel and sub-pixel measurements to detect defects and anomalies that may impact visual quality.

Radiant’s TrueTest™ Software and our TT-ARVR™ module provide the leading display test algorithms, with the capability to sequence tests for rapid evaluation of all relevant display characteristics in a matter of seconds, applying selected tests to a single image captured by the camera. Standard tests in TrueTest include luminance, chromaticity, contrast, uniformity, mura (blemishes), pixel and line defects, and more. Unique tests for AR/VR display analysis include:

• Modulation transfer function (MTF) to evaluate image clarity, based on Slant Edge Contrast ISO 12233
• Image Distortion (for distortion characterization)
• Image Retention/Sticking
• Factory distortion calibration of the lens solution to normalize wide field-of-view images for testing
• With spatial x,y positions reported in degrees (°)

These new AR healthcare initiatives outlined above, and many others like them, represent the best combined efforts of medical professionals and technology developers working together to realize the promise of augmented reality to help enhance people’s lives. 

To learn more about quality measurement for AR/VR/MR devices, read the whitepaper “Enabling Display Measurement within Augmented & Virtual Realty Headsets”.

 

 

 

CITATIONS

1. AR/VR in Healthcare Market By Technology (AR, VR), By Component (Hardware, Software, Services), By Application (Surgeries, Simulation & diagnostics, Patient care management, Education, others), By End user And Segment Forecasts, 2016-2026, Report published by Reports and Data, May 15, 2019. 
2. “Augmented Reality in Medicine and Healthcarre", ThinkMobiles, retrieved 9/6/2019. 
3. “Why You Need to Take Your Medications as Prescribed or Instructed”, U.S. Food & Drug Administration, Retrieved 9/6/19. 
4. Shavel, T., “Augmented Reality is Taking Over Radiology Training”, Imaging Technology News, February 27, 2019. 
5. “How Augmented Reality Helps Train Ultrasound Technicians”, MedCognition, April 1, 2019.