A team of researchers at the Rhode Island Hospital has incorporated and reimagined the way 3D technology, virtual reality and augmented reality can be used in medical settings.
The Emergency Medical HoloLens project emerged from work pioneered by Derek Merck, currently associate professor of emergency medicine at the University of Florida; Scott Collins, technical lead of CT and 3D technology services at the RIH and Leo Kobayashi ’94 MD ’98, principal investigator of the project and director of research and innovation at the Lifespan Medical Simulation Center, as well as University students. The team focused on creating an augmented reality representation — a holoimage — of patient anatomy. These holoimage representations are hologram-like projections that are fixed in space, said Owen Leary ’18 MD ’24, one of the student creators of the project and a current clinical research program coordinator in the Department of Neurosurgery at the RIH.
People using the HoloLens wear a visor-like technology that can project a patient anatomy model to a fixed location in their surroundings, Collins said. “It is basically an augmented reality device you can wear on your head.”
After the team acquired the first of two HoloLenses from Microsoft, the researchers overlaid real patients’ anatomy and pathology onto human patient simulations using the device, Kobayashi wrote in an email to The Herald.
For one experiment, Collins designed an augmented reality model using a covered cardboard box, blocks that served as obstacles within the box and trace amounts of hidden blue fluid. After putting on the HoloLens, students were responsible for removing the liquid using only a syringe, sensory touch and an augmented reality representation of the box’s contents. This task was intended to be a simplified simulation of a kind of intubation process, in which doctors insert tubes into patients’ throats to drain harmful fluids.
In a different project focused on virtual reality, Collins used an Oculus Rift to design a virtual reality anatomical model using images from real patients. Collins hoped to teach students to use the technology and then put together an exhaustive anatomical library, but doctors’ unfamiliarity with the VR technology may be why the project has not taken off within the medical community.
The team partnered with the University library to preserve and disseminate the holoimages to the public through the Brown Digital Repository. The BDR is a digital library that contains documents such as students’ honors theses, doctoral dissertations and University archives. One part of this repository is the Open Data Collection, mostly consisting of scientific work that has been made available for access and reuse at no cost, said Andrew Creamer, the scientific data management specialist for the University’s library. Creamer facilitated the collaboration between the research team and the library. The partnership began three years ago when the Rockefeller Library obtained the Sidney E. Frank Digital Studio as a 3D printer facility.
Leary,first starting on the holoimage project as an undergraduate, wrote descriptions of all of the holoimage files currently on the BDR to make them easily accessible to website browsers.
But when creating these files on the repository, he had to consolidate the files into a uniform, accessible, file format. “The philosophy is that because most Brown research is publicly funded …(it) should be freely available to the public and tax payers to utilize those works,” Creamer said.
The use of 3D technology at hospitals extends beyond the holoimage project. At RIH, researchers have also consulted 3D technology to better understand how to treat tumors.
Unlike an X-Ray which is a 2D image, patient CT and MRI scans are 3D images. Still, “when the radiologist looks at these 3D-images, they click through slices … to see 2D images,” Leary said. “It is possible to see the image as 3D through segmentation, though.” Segmentation is a process by which researchers highlight a portion of a 2D cross section of an MRI or CT scan and select a section of interest to revert into 3D.
Developing these segments can assist researchers determine the specific shape of the tumor they are targeting, which allows physicians to develop a plan for the safest and most effective treatment. “This helps us make sure we hit what we want and don’t hit what we don’t want. You don’t want to cook a hole in someone’s bowel in the process (of treating their tumor),” Collins said.
Similarly, the holoimage project team hoped to better the treatment of patients by taking their technology out of the lab and applying it to “help patients and family members understand complex … pathology” prior to operations, Kobayashi wrote. “Unfortunately, the team was unable to secure funding to conduct the research.” Still, their work has been primarily an open educational resource.
The team does not have access to the statistical influence of their work and does not know if others have utilized their projects. “We don’t even know if our own medical school is using it,” Creamer said.
Still, he hopes that these augmented and virtual anatomical representations will help medical students, and even high schoolers, as the technology could allow movement away from dissecting real animals, saving students from exposure to chemicals and limiting the “murdering of frogs,” Creamer said.
