Consultant surgeon Jag Dhanda stands before you in the operating theatre, performing a procedure. At each step, he explains what he’s doing in detail: where he’ll make his incision, how he’ll preserve the nerve. It feels like you’re in the room with him – but you aren’t. You’re watching an immersive recording he’s prepared earlier through a virtual reality headset. These VR training videos are just one example of how extended reality is being used to enhance surgeons’ capabilities. And they couldn’t have come at a better time.
Today’s surgeons are expected to get to grips with an overwhelming number of skills and to learn modern procedures that are far more complicated than those of the past, says former surgeon and founder of extended reality (XR) healthcare training company, Osso VR, Justin Barad. Yet our current system doesn’t allow them enough time and hands-on practice to do so.
Thanks to high non-clinical workloads and staff shortages, trainees often struggle to get access to the OR to observe, assist and build up experience. And, after their formal training is done, that’s it – they’ll be expected to operate on live patients. Even as a practicing surgeon, you might attend a workshop relating to a new procedure but wouldn’t necessarily have a chance to hone those skills before heading into theatre. Having access to immersive, interactive training materials without needing to set foot in the operating room could help train more surgeons to a high standard, much quicker. Once in theatre, extended reality tools can help make surgery safer, too – such as enhancing a surgeon’s view of a patient’s anatomy so they can operate with greater precision. And these solutions are already being rolled out in universities and hospitals around the world.
Immersive learning
What if trainees could watch a procedure being performed in front of them, as many times as they needed to, without needing to physically be in the operating room? It’s what Dhanda, who is the director of extended reality education company VRiMS, has made possible by creating the world’s largest immersive training library. To access it, you put on a VR headset that’s synched to a 360º video of a procedure being performed on a cadaver.
You can see the operation happening in 3D, right in front of you, as if you’re in the room yourself. And you can see it from different angles via multiple views that are overlaid onto the video, too. “You would never normally see that little blood vessel, but you can in an overlay,” says Dhanda.
Interactive elements can also be included, such as narration of what’s happening and quiz questions that pop up intermittently. These can be tweaked depending on the audience: a medical student would get a different set of questions than an experienced surgeon watching to brush up their memory.
These videos allow trainees to gain a familiarity with different procedures as if they were observing them in the operating room. Medical students often have to wait until their third or fourth year before they can get into the theatre, and they won’t necessarily get to engage in the specialties they’re interested in. Even the most experienced surgeons will have operations they rarely perform and could use a refresher on.
And while the technology doesn’t allow you to participate in the virtual procedure – you can’t manipulate any objects or change the course of what’s happening – the immersive experience also makes it easier to retain knowledge, Dhanda explains. “If I train you to do something with a VR headset, you’re 16 times more likely to remember it than reading it in a book.”
Simulating surgery
It’s possible to actually practice surgery in virtual reality, too. These solutions also use a VR headset that’s synced up to training materials, only they’re high-fidelity simulations rather than pre-recorded videos. Osso VR is one of the companies that makes them. “Everything we do is a fully interactive simulation,” says Barad. Here’s how it works: you put on a headset that can track the 3D position of your hands and choose your content. Then, you can interact with it either by holding hand controllers that give haptic feedback or moving your hands as you would normally (though you’d lose the feedback in this case).
As with Dhanda’s library, each video offers varying levels of guidance. If it’s your first time with a particular simulation, it will tell you what to do. But if you’ve got a lot of experience, you can do the operation from memory. For some procedures, there’s a zoom-in function so you can see a particular area blown up, in a way that wouldn’t be possible in real life.
In validation studies, Barad shares that people using Osso’s technology had a 200–300% increase in performance outcomes. In one such trial, published in the journal Surgical Education, a group of participants trained for a particular procedure using Osso to a set level of proficiency, while another followed the traditional route – learning from lecture materials, though taking as much time as they needed. Both groups performed the operation in a test environment and were graded. Those trained with VR outscored their peers in all assessments – and in four out of five categories, the difference was significant.
This also speaks to the impact of having standardised assessments for surgery, like benchmarks or scores for different skills. Currently, there’s little to none of this, Barad explains. “It is very subjective. It’s, ‘I feel ready to do this’, or ‘I feel that someone else is ready to do it’.”
In the above study, confidence in one’s skill wasn’t enough: the lecture materials group studied until they felt capable yet were outperformed by those who had to meet a set standard. “It’s not just the training that’s so valuable,” Barad says. “It’s that assessment component: having objective insight into how ready you are to do something.”
Extended reality in the OR
And then there’s being in the operating theatre itself, standing over a patient. Here, extended reality can help surgeons plan and execute complex procedures with greater precision. At the University of Alabama at Birmingham, MD and assistant professor in the Department of Orthopaedic Surgery, Scott Mabry, is using mixed reality to perform total hip replacements.
First, he takes a CT scan of the hip ahead of surgery, which is then modelled into a 3D image. With the help of an engineer working for the company who manufactures the technology, Mabry then plans each step of the surgery in precise detail. He can map out the exact position and sizing for the replacement hip socket, for example.
The image is synched to a VR headset. When he puts it on, he can see a 3D hologram of the patient’s anatomy. It can be viewed in different modes, to show him how things should look at each stage of the procedure. Mabry can check these visuals at any point during the operation via voice command or by bringing his wrist into view and tapping at a virtual watch, which brings up a menu display. Here, he can pull information like a patient’s x-rays into his field of vision, too. All without physically touching anything.
In hip replacements, screws are used to secure the implant. Because of the precise visualisation this tool allows, “You can put in these great screws, you know exactly where they’re going into the bone,” says Mabry. This enhances the fixation of the implant, which means that patients can get back on their feet sooner after the operation, he adds.
For complex cases where you can’t use the regular landmarks you would in surgery, for instance if a patient had plates or screws where a round hip socket would naturally be, the impact of this technology could be significant. Mabry can see this approach being used for other joint replacement procedures, too, like shoulder or ankle replacements. “Anytime you’re planning a surgery with three dimensions, then you need to see all of those views. Having this headset is very beneficial.” He estimates that around 5–10% of hospitals in the US are currently using, or at least dabbling with, extended reality tools like his. Other solutions for surgery are currently being investigated in research, including those that assist with the placement of screws during orthopaedic trauma surgery and allow for enhanced visualisation of anatomy.
What’s next for VR?
“We’re seeing a lot more uptake and exploration of VR in today’s health systems,” says Barad. “And this is not only for surgeons, but the technology can be used to upskill wider members of care teams, too,” he adds. And because all you need to participate is a headset and access to the training materials, these solutions could be scaled globally. Dhanda is presently working to share his training programme with hospitals across Africa and at the time of writing, a roll-out across every medical school in the UK had just gone underway.
While there’s still room for improvement with the technologies themselves – for example, Mabry’s hologram can be thrown off if you don’t position a screw exactly right during an operation – Mabry, Dhanda and Barad all agree that they’re set to take off within the next few years. “I think it’s one of the biggest inflection points in healthcare, outside of major discoveries like antibiotics,” says Barad.
