Augmented reality lets doctors see under a patient’s skin

3D models from CT and MRI scans are projected onto patients as they move, for accuracy in surgery, rehab, and education.

ProjectDR is an augmented reality system that allows medical images such as CT scans and MRI data to be displayed directly on a patient’s body. University of Alberta

By Katie Willis, University of Alberta January 24, 2018

The system, called ProjectDR, allows medical images such as CT scans and MRI data to be displayed directly on a patient’s body in a way that moves, as the patient does.

“We wanted to create a system that would show clinicians a patient’s internal anatomy within the context of the body,” explained Ian Watts, a computing science graduate student and the developer of ProjectDR.

The technology includes a motion-tracking system using infrared cameras and markers on the patient’s body, as well as a projector to display the images. But the really difficult part, Watts explained, is having the image track properly on the patient’s body even as they shift and move. The solution: custom software written by Watts that gets all of the components working together.

ProjectDR is an augmented reality system that allows medical images such as CT scans and MRI data to be displayed directly on a patient’s body in a way that moves as the patient does.

ProjectDR was developed by graduate students Ian Watts and Michael Fiest, supervised by Pierre Boulanger, Cisco Chair in Healthcare Solutions and professor in the Faculty of Science, and by Greg Kawchuk, professor in the Faculty of Rehabilitation Medicine. UAlberta Science. Published on Youtube Jan 23, 2018

Vast applications
“There are lots of applications for this technology, including in teaching, physiotherapy, laparoscopic surgery and even surgical planning,” said Watts, who developed the technology with fellow graduate student Michael Fiest.

ProjectDR also has the capacity to present segmented images—for example, only the lungs or only the blood vessels—depending on what a clinician is interested in seeing.

For now, Watts is working on refining ProjectDR to improve the system’s automatic calibration and to add components such as depth sensors. The next steps are testing the program’s viability in a clinical setting, explained Pierre Boulanger, professor in the Department of Computing Science.

ProjectDR demonstrated on a mannequin. University of Alberta

Next steps
“Soon, we’ll deploy ProjectDR in an operating room in a surgical simulation laboratory to test the pros and cons in real-life surgical applications,” said Boulanger.

“We are also doing pilot studies to test the usability of the system for teaching chiropractic and physical therapy procedures.” added Greg Kawchuk, a co-supervisor on the project from the Faculty of Rehabilitation Medicine. Once these pilot studies are complete, the research team expects the deployment of the system in real surgical pilot studies will quickly follow.

Watts is co-supervised by Boulanger, Cisco Chair in Healthcare Solutions and professor in the Faculty of Science, and by Kawchuk, professor in the Faculty of Rehabilitation Medicine.

ProjectDR was presented last November at the 23rd ACM Symposium on Virtual Reality Software and Technology, VRST 2017 in Gothenburg, Sweden.

Source University of Alberta

ProjectDR: Augmented Reality System for Displaying Medical Images Directly onto a Patient, Ian Watts, Pierre Boulanger, Gregory Kawchuk. 23rd ACM Symposium on Virtual Reality Software and Technology. Gothenburg, Sweden. November 8-10

The McGill University Health Centre uses augmented reality in ENT surgery. New technology using augmented reality and mixed-reality is assisting surgeons with planning and real-time positioning in patients who require ear, nose and throat surgery. The state-of-the-art technology, known as Targeted Guided Surgery (TGS), aims to improve patient outcomes and safety in this complex surgery, which is performed in close proximity to the optic nerve and the brain. The McGill University Health Centre (MUHC) is the first institution in North America to use augmented and mixed reality in sinus surgery. Published on Vimeo  May 30, 2017

Surgery Pad – mobile medical augmented reality App for the Apple iPad. This video demonstrates “Surgery Pad”, an augmented reality application for the Apple iPad that allows a surgeon to view into a patient. The application was developed at the German Cancer Research Center in Heidelberg. mbits imaging GmbH. Published on Youtube Jan 4, 2012

Also see
CT and MRI Scans Projected onto Patients for Accurate Surgeries, Medical Education in medGadget
What lies beneath: Made-in-Edmonton augmented reality system lets doctors see under a patient’s skin in CBC
Cisco Canada funds UofA health research chair in IT World Canada
ProjectDR allows doctors to “see into” patients’ bodies in New Atlas
MUHC takes augmented reality into the operating room McGill University Health Centre
Augmented reality helps surgeons ‘see through’ tissue to reconnect blood vessels in Imperial College London
Mobile Medical Augmented Reality App for the Apple iPad – Interview with Prof. Hans-Peter Meinzer in Medical Augmented Reality

MOBILITY MENU
   403-240-9100
Call 403-240-9100