MIT Develops AR System That Gives Doctors X-Ray Vision During Ultrasound Scans
Researchers at the Massachusetts Institute of Technology have developed a new augmented reality system that could transform the way medical professionals perform and interpret ultrasound scans — giving users what amounts to real-time X-ray vision into the human body.
The system, called AR-VIU (Augmented Real-Time Volumetric Imaging in Ultrasound), combines a compact 3D ultrasound probe with an AR headset to overlay a live three-dimensional representation of internal tissue directly over the patient’s body. Rather than staring at a flat 2D image on a monitor and mentally reconstructing what the tissue looks like in three dimensions — a skill that takes years to master — clinicians can simply look at the patient and see inside.
The Problem With Traditional Ultrasound
Interpreting ultrasound images has always been one of medicine’s more demanding cognitive tasks. A trained sonographer must take a series of two-dimensional slices and mentally assemble them into an accurate three-dimensional picture of what lies beneath the skin. Get it wrong and the consequences can be serious — a misplaced biopsy needle, a missed abnormality, a delayed diagnosis.
“It’s a difficult skill to master, and there are long learning curves,” says Jason Hou, an MIT graduate student and lead author of the research. “The hardest thing is this mental tomography bottleneck where you’re trained to reconstruct the 2D slices in your 3D mental space. That is a cognitive burden that can lead to inaccuracies in scanning.”
How AR-VIU Works
The MIT system uses a compact ultrasound probe — roughly the size of a deck of cards — that captures real-time 3D data and streams it into Unreal Engine, a professional computer graphics platform. The engine converts the raw ultrasound data into a precise volumetric rendering, which is then displayed through an AR headset worn by the clinician.
The result is a live, three-dimensional image of the tissue being scanned, superimposed directly over the patient’s body at the correct location. By tilting their head or moving around the patient, the clinician can view the internal structure from different angles — something that is simply not possible with traditional 2D ultrasound.
The probe uses fewer ultrasound elements than conventional 3D imaging systems, making it less expensive to build and less power-hungry to operate — an important consideration for wider clinical adoption.
Tested on Experts and Novices Alike
The MIT team tested AR-VIU on 18 participants — nine experienced ultrasound professionals including sonographers and physicians, and nine people who had never used ultrasound before. Each participant performed identification and targeting tasks using four different imaging approaches, including traditional 2D ultrasound, 3D imaging on a standard screen, and the new AR system.
The results were striking. AR-VIU significantly improved performance across all users, but the effect was especially pronounced among novices. Those with no prior ultrasound experience performed nearly as well as seasoned experts when using the AR system — a gap that widened considerably when both groups reverted to traditional 2D imaging.
Most novice participants said they preferred the AR approach, describing it as more intuitive and easier to understand. Many expert users, while more accustomed to conventional methods, acknowledged clear advantages for specific tasks such as needle placement during biopsies.
“Overlaying images with the anatomy and providing 3D visual context makes ultrasound significantly easier for novices to understand,” says Shrihari Viswanath, co-lead author of the study.
Clinical and Training Implications
The potential applications of AR-VIU extend across two distinct areas of healthcare. In clinical settings, the technology could improve accuracy in procedures that rely on precise ultrasound guidance — biopsies, catheter placements, and cardiac imaging among them. In training environments, it could dramatically accelerate the development of competent ultrasound practitioners, reducing the years typically required to master the mental reconstruction skills the technology renders unnecessary.
“For training, this could make ultrasound more intuitive and more understandable. On the clinical side, it could be less time-consuming, more accurate, and also give health care providers more peace of mind. They wouldn’t have to wonder if they missed anything,” says Canan Dagdeviren, associate professor of media arts and sciences at MIT and senior author of the study.
The research has been published in Nature Communications Engineering. The team is continuing to improve the resolution of the imaging system and conducting further trials to validate the accuracy of AR-VIU ahead of potential clinical deployment.