AR/VR in the OR – Surgical Applications of Augmented and Virtual Reality

When it comes to futuristic medicine some people may think of Star Trek’s tricorder scanners and hypospray injectors, or recall Dr. McCoy’s horror at twentieth-century medicine.¹ Here in the third decade of the 21st century, we may not have versions of those devices yet, but medical care on Earth now incorporates some fairly advanced technologies—from endoscopic surgeries to proton-beam treatments for cancer. 

Recently, augmented reality (AR) has joined the ranks of futuristic medical technologies with applications that range from training and education to patient behavioral health support, surgery, and diagnostic imaging. In fact, the adoption of AR/VR in healthcare is predicted to increase, with the market for these technologies growing tenfold from $1.6 billion in 2020 to $16.2 billion by 2027.² 

Representation of a surgeon consulting patient diagnostic scans with an AR/VR device in the operating room.

Already, the benefits of AR/VR for medical uses are evident: “The efficiency of healthcare processes such as anatomy, operative services, and medical training procedures can be enhanced by using augmented reality technologies. Also, continuous improvements have been observed in the surgical environment and procedures with the application of augmented reality technologies. Moreover, implementing augmented reality technologies in rehabilitation services gives a more controllable environment where users can interact and enhance the recovery process.”³

Spotlighting Surgical AR Applications

Every surgical procedure carries some degree of risk. Any time our protective outer layers of skin are cut, or our internal body structures and organs are exposed, infection can occur. After a major surgery, healing can be slow before a patient recovers full function and vigor. Once healed, scar tissues may remain and can sometimes hinder future activity. Therefore, any techniques that can help minimize, shorten, or eliminate the need for invasive surgical procedures is of interest to medical professionals and patients alike.

In addition to their potential for operating-room use, AR/VR systems are also being developed and used for activities that support diagnosis and for pre-surgical preparation and training.⁴  For example, Chicago-based company Immersive Touch^® provides an AR/VR platform that integrates with standard diagnostic imaging data such as CT and MRI scans. Physicians can use the platform to view patient-specific anatomy and pathology in 3D, plan surgical approaches within an immersive virtual environment, and rehearse specific surgical procedures or practice on a specific medical device before operating on a patient.

A cardiothoracic surgeon analyzes a patient's broken ribs in a 3D virtual model and maps out an upcoming surgery on the patient-specific model.

Superheroes in the Operating Room

Neurosurgeons at Baltimore’s eminent Johns Hopkins University School of Medicine have successfully used augmented reality in surgery. The first AR-enabled procedure was performed in June 2020, when six screws were placed in a patient’s spine for spinal fusion surgery to alleviate sever pain. The second use of surgical AR was to remove a cancerous tumor from a patient’s spine.⁵

In both of these breakthrough procedures, doctors used an AR headset with a see-through display that projects images of the patient’s internal anatomy. The internal images of bones and other tissues were created from CT scans—essentially giving surgeons the superpower of X-ray vision to see inside a patient’s body. One significant benefit of using the AR technology is the way it displays images in a doctor’s direct view, so there’s no need to look away from the patient to examine digital scans on a separate screen.

Augmented reality technology guides a spine neurosurgeon at Johns Hopkins to a successful patient treatment outcome.

Similarly, surgeons at the UK’s St. Mary’s Hospital and Imperial College London have used the Microsoft HoloLens to “overlay 3D digital models of CT scans onto patients in the operating theatre to assist them in understanding the patient’s unique anatomy of blood vessels, tissue and bones prior to incision.”⁶

Connecting Remote Clinical Teams

During the pandemic, many people have experienced virtual meeting and AR remote collaboration platforms, or participated in a virtual consult with their medical provider. Similar remote connectivity using AR can work in the operating theater as well. 

A surgical team in Barcelona used Vuzix M400 Smart Glasses in the operating room in collaboration with Spanish mobile telecommunications operator Vodafone España. A gastric surgeon performed a sleeve gastrectomy procedure on a patient while streaming the surgery live, connected to a team of doctors in Miami, FL, who attended the surgery virtually. This enabled collaboration and consultation between remote clinicians while reducing in-person exposure and minimizing the need for surgical personal protective equipment that was in short supply due to COVID-19.⁷

Wearing the Vuzix M400 Smart Glasses, clinicians collaborating on medical treatments via video and audio stream in real-time to remote audiences and experts, reducing exposure and conserving PPE during the COVID-19 pandemic. (Image Source)

Smile: AR/VR for Dental Surgeries 

Interestingly, dental implantology and orthognathic surgery have been the most frequent applications of AR/VR in medicine so far. “Virtual planning improved the accuracy of inserting dental implants using either a statistic guidance or dynamic navigation. In orthognathic surgery, prediction planning and intraoperative navigation are the main applications of virtual reality.”⁸

Virtual reality has also been used to enhance education and training for oral and maxillofacial surgery by recreating the environment and allowing students to practice procedures without touching live patients. Incorporating “haptic feedback provides an additional layer of immersive reality to improve manual dexterity and improve clinical training.”⁹

Device Quality and Efficacy Is Critical

When people’s health and lives are on the line, any AR, VR, or MR systems used in diagnostic or treatment settings need to be extremely reliable. Any information displayed on screen must be readable and clear to the user while not interfering with necessary visibility of the surrounding environment. 

To ensure the quality and performance of AR/VR/MR displays such as headsets, goggles, and glasses that may be used for medical purposes, the hardware requires precise functional testing that comes as close as possible to replicating a user’s visual experience. Variations in luminance and color, contrast against the background environment, and other issues with display uniformity and clarity are unacceptable. Because images are viewed close to the eye in these systems, even small defects such as a few dead pixels are magnified and may impact visualization.

Radiant Vision Systems has developed an AR/VR Lens solution that features a unique optical design specially engineered for measuring near-eye displays (NEDs), such as those integrated into VR, AR, and MR headsets. The lens design simulates the size, position, and field of view of the human eye to accurately capture and evaluate what a user will see.

Paired with a Radiant ProMetric^® Imaging Photometer or Colorimeter, the AR/VR Lens captures high-resolution images of displays inside the headset for pixel and sub-pixel measurements to detect defects and anomalies that may impact visual quality.

Radiant’s ProMetric® Imaging Colorimeter with the AR/VR Lens, shown with TT-ARVR™ analysis software.

These new AR healthcare initiatives outlined above, and many others like them, represent the best combined efforts of medical professionals and technology developers working together to realize the promise of augmented reality to help enhance people’s health and save lives.

While industry experts are generally optimistic, a few aspects of applying AR to routine medical care still need to be addressed. 9  Dr. McCoy might have some reservations about current medical practices, but he might be relieved to see we’re making progress towards more advanced technological approaches:

 

CITATIONS

1. Star Trek: The Voyage Home, Paramount Pictures, 1986, Directed by Leonard Nimoy. 
2. Augmented and Virtual Reality in Healthcare Market Size by Technology (Augmented Reality, Virtual Reality), by Component (Hardware, Software, Services), by Application (Surgery, Training and Education, Behavioral Therapy, Medical Imaging)...Forecast 2021-2027. Global Market Insights, August 2021. 
3. “Healthcare Augmented & Virtual Reality Market revenue to cross UDS $16.2 Bn by 2021; Global Market Insights, Inc.” GlobalNewswire, August 23, 2021.
4. Vavra, P, et al., “Recent Development of Augmented Reality in Surgery: A Review.” Journal of Healthcare Engineering Volume 2017, Article ID 4574172, August 21, 2017. DOI:10.1155/2017/4574172
5. “Johns Hopkins Performs its First Augmented Reality Surgeries in Patients,” NeuroLogic, Johns Hopkins Medicine, February 16, 2021.
6. “Augmented Reality in Healthcare: How AR is Revolutionizing Healthcare,” Reydar, December 1, 2020.  
7. “Barcelona Hospital Surgeons Now Using Vuzix M400 Smart Glasses to Perform Assisted Gastrointestinal Surgeries on Patients.” Display Daily, June 30, 2020
8. Ayoub, A. and Pulijala, Y., “The application of virtual reality and augmented reality in Oral & Maxillofacial Surgery.” BMC Oral Health Volume 19, Article number: 238, November 8, 2019.
9. Ibid.
10. Vavra, P, et al., “Recent Development of Augmented Reality in Surgery: A Review.” Journal of Healthcare Engineering Volume 2017, Article ID 4574172, August 21, 2017. DOI: 10.1155/2017/4574172