Extended Reality for the Operating Room (XR4OR)

At XR4OR, Philippe C. Cattin presented his research on virtual reality (VR) planning for complex surgical interventions using the Specto Volume Rendering application. He also demonstrated how these planning tools can be transferred into the operating room. While modern surgical loupes reduce eye strain and improve ergonomics during procedures, they also limit the field of view, making it difficult for surgeons to maintain an overview of the entire surgical scene. To address this limitation, integrating augmented reality (AR) into surgical loupes offers a promising solution. AR overlays can provide real-time guidance – such as anatomical landmarks, navigation cues, segmented tumors, or hemodynamic data – without requiring the surgeon to look away from the operative field. By seamlessly blending essential information into the surgeon’s line of sight, AR-enhanced loupes can improve situational awareness, support better decision-making, and ultimately contribute to safer and more efficient surgeries. However, no medically certified AR-enhanced surgical loupes are currently available on the market. This needs to be addressed soon by the MedTech Industry. Despite their potential, deploying VR and AR technologies for routine hospital use remains challenging. Beyond their excellent visual experience, AR/VR systems must prove their clinical effectiveness and efficiency. Medical professionals are only likely to adopt these tools broadly if they demonstrably save time in their already busy schedules. One major barrier is the time-consuming and often unreliable setup process. Connectivity issues with AR/VR glasses, frequent system reboots, and general instability make these solutions impractical for daily clinical use – issues that may be tolerated in research settings, but not in routine care. Another obstacle to clinical integration is the lack of standardized interaction interfaces across different AR/VR hardware manufacturers. While laptops universally rely on keyboards and mice, AR/VR devices use a wide array of input methods. Some manufacturers rely on hand controllers with varying designs and button layouts; others use hand tracking with different gesture sets. These inconsistencies hinder seamless adoption and training, further slowing down the integration of AR/VR into clinical workflows.

This presentation highlighted recent advances by my group on the interactive rendering of medical images for simulation, planning and guidance of surgical operations, with a specific focus on mixed reality displays. We have shown how advanced volumetric rendering techniques can help visualize 4D echography images, how segmented volumes can be interactively volume-rendered in virtual reality to improve depiction of the anatomy for surgical planning and how these interactive rendering techniques can be leveraged to simulate surgical procedures for the training of residents. Following the presentation, a demo of our VR interactive volume rendering system was setup for other seminar participants to try.

As a researcher in Human-Computer Interaction, the focus of my work is on innovative and multimodal interaction techniques in different application domains. The domain of the operating room (OR), with its phases of training, preoperative, interoperative, and postoperative activities, offers a multitude of possibilities to be supported by interactive technology to make the procedures more efficient and successful, to improve patient safety and well-being as well as the work conditions of surgeons and OR staff. Nowadays, many digital systems already support the work in the OR, such as by providing image data during surgery. However, the interaction with these systems is often cumbersome, as it takes place as a secondary task next to the primary operation task in sterile conditions and with usually busy hands. Thus, solutions for the interactions with the increasing number of systems are needed that fit this specific and safety-critical environment, align well with the main task of the operation, and do not increase workload. A rise of systems that use extended reality (XR) and artificial intelligence (AI) to provide more extensive assistance to surgeons and medical staff will also set new requirements for the interaction with technology during training, preoperative planning, during surgery in the operating room, and postoperative care. A well-balanced design of embodied, multimodal implicit and explicit XR interactions that also consider a trustful and sensible human-AI collaboration is needed. During the Dagstuhl Seminar “Extended Reality for the Operating Room” we fruitfully discussed this highly interdisciplinary, applied, and dynamic field of research from many angles, compiling visions for smooth, intelligent, and effective XR systems of the future. I am interested in working on a design space for XR interaction in the OR, putting together the dimensions that need to be taken into account when designing for this domain, which include the different phases, tasks and goals, the user roles as well as (explicit and implicit) input and output facilities. Especially the integration of implicit interaction, such as through the tracking of eye gaze and biosensing for input, or the inclusion of additional modalities for output, such as haptic feedback, are promising paths to facilitate and enrich XR interactions in the OR. Nowadays, many parallel systems and screens exist that are not integrated. The usability of used XR systems, such as HMDs, still needs to be improved, and interactions lack standards, so they may be challenging to learn and remember. Thus, there is a great demand for standardization and integration and the design of personalized and context-aware, more intelligent interactive assistance systems.

During the Dagstuhl Seminar, I focused on developing research project ideas and exploring directions for future grant proposals. I also contributed to drafting a joint white paper that brought together perspectives from different participants to outline shared research priorities. In addition, I met several researchers whose work intersects with mine, leading to new academic connections that may support future collaboration. The seminar provided time and space for thoughtful exchange, helping identify common interests and potential pathways for joint work in global health.

Extended Reality in the operating room may enhance surgical precision, efficiency, and patient outcomes. Key aspects are the accurate registration and display of multimodal imaging data and the transfer of the preoperative planning to the surgical site. XR may provide real-time 3D overlays of patient anatomy, helping surgeons navigate complex procedures with greater accuracy. The holographic visualization of critical structures, like blood vessels, tumors, or nerves, may reduce surgical risks. Transferring preoperative planning to the surgical site allows for real-time navigation of procedures. The surgeon receives intraoperative guidance by superimposing 3D images of the planned surgical result on the region of interest. In reconstructive oral and maxillofacial surgery cases, complex preoperative planning is performed, and surgical guides and patient-individual implants are manufactured. Therefore, adherence to the surgical plan is crucial for avoiding intra- and postoperative complications. Incorporating automated segmentation and analysis of large amounts of patient data further helps predict surgical risks and machine learning models may suggest surgical approaches to improve the precision of a surgical procedure. A second key aspect besides the availability of information is its individual display, e.g., on head-mounted devices, to avoid distraction and work continuously without breaking focus. User profiles for all staff members in the OR may improve the distribution of information in the OR and avoid redundant information.

The LARACROFT platform introduces an augmented reality (AR) framework tailored for remote collaboration in laparoscopic surgery. Designed to operate within the spatial and cognitive constraints of the operating theater, the system supports real-time interaction between local and remote surgeons via a shared AR workspace. It integrates hand tracking, video passthrough, and gaze-directed, foot-triggered input mechanisms to enable effective control without compromising sterility or workflow. In a pilot study, medical students and mentors engaged in standardized surgical tasks, revealing key insights into the impact of AR-supported telecollaboration on motor coordination and communication. Observed limitations, including latency and display resolution, highlighted current technological thresholds in surgical AR. However, participants reported enhanced engagement and situational awareness, pointing toward the platform’s potential for scalable deployment in surgical education and support.

XR has the potential to be used in a variety of ways for the OR, such as medical device navigation, visualization of internal structures, identification of cancerous tissue, pre-procedure or intra-procedure planning, information and image display, telemedicine and communication, and many more. Rather than focus here on a specific use case, I offer some thoughts and questions about how we might enable XR’s eventual adoption and use in the OR.

There are cases where XR use by patients shows a lot of promise and success (for example, pain distraction, neurological rehabilitation, PTSD). XR technology has advanced considerably in recent years but to my knowledge, it does not yet have widespread adoption in the OR, at least not for physician users. Why?

When researching, developing, and commercializing new technologies for the OR, we face the challenge of how to integrate those technologies into physician workflows. If a new technology takes too much time to set up for each patient, it becomes an opportunity cost with less time available to treat additional patients. If using the technology is too complex, it ends up not being used. For example, some physicians have told me that even though they have access to some sophisticated medical device navigation systems, setup takes long enough that it is not worth the time or complexity.

Also, for an organization to invest in a technology commercialization effort, there must be reasonable evidence that the technology (compared to standard of care or other less disruptive technologies) will improve patient outcomes, enable physicians to do things they otherwise could not do, improve hospital throughput, generate profit for the hospital, or provide some other business benefit.

As researchers and technologists focused on improving patient care with XR, how might we develop XR to enable its use without workflow disruption? For example, what can be done to make setup quick and efficient? How can technology be made robust to failure or emergency situations (e.g., if an emergency arises, can the physician immediately remove a headset and revert to standard of care?)? How can ergonomics be improved, especially for long procedures?

These considerations often end up being procedure-specific because different procedures have different workflow requirements. For complex procedures, a lot of time and mental effort can be spent on communication between physicians and OR staff, so that physicians can see what they need to see. One could imagine developing procedure-specific conversational AI models for a physician to more easily interact via voice, without requiring very precise wording, or without requiring challenging hand gestures or controllers. Procedure-specific inference models may also be able to understand procedure workflow and show “just-in-time” information to the physician, reducing communication overhead.

There are likely many other workflow-related improvements that can be made, and it would be interesting to explore how our research can support physicians and staff in the OR without negatively impacting their workflow.

XR, in a strict sense, only encompasses the mediation of virtual, computer-generated content for all human senses i.e., visual overlays, haptic or acoustic amplifications. However, XR alone is just one of several technologies (AI, image processing, 3D-printing, diagnostics) driving the progress in medicine and surgery and a focus solely on XR without including those other domains will rather hinder than facilitate XRs uptake in the OR. In general, it is important not to ask ourselves “What can XR do for the surgeon?”, but rather “What problems does the surgeons currently have? Where are the limitations of how they can help the patient?” Using this approach, we will discover where XR really brings a benefit and is not just pushed from a technology enthusiastic perspective. We will further discover how XR fits best in to the supporting ecosystem of advanced technologies (AI, image processing, 3D-printing, diagnostics). By facilitating synergies, and win-win-situations, we will foster the transfer into the OR. The following use case shall demonstrate my perspective:

1. 1.

   Patient with Face/Head-Trauma. When the patient arrives in OR the surgeon could get an overlay of the MRI on the patient using AR. This is not a very new use case, but everything one could achieve solely with XR. By broadening the scope of involved technologies, a much more advanced scenario is imaginable. By 3D-scanning the patients head in the OR an AI could calculate how the bone fragments have moved since the MRI was taken and can visualize this updated 3D-model to the surgeon via an AR headset. Another AI, could deduce how the bone fragments have to be put together stepwise, which implants are necessary for this, how standard implants have to be bend and what custom implants need to be 3D-printed. Once this is determined 3 paralyzed processes are imitated (1) the printing of the custom implants; (2) A nurse or a resident surgeon is adapting the shape of the standard implant using a template of the target shape visualized through and AR display. An AI checks in the background the forming process and provides and audio-visual feedback when the correct shape is reached; (3) The surgeon gets support through the AR glasses by visualising the positions of the bone fragments and the instructions of how they have to be put together using the implants. During the entire surgery the visual guidance is updated based on the tracked deformation of the patient’s soft tissue. Another option for this use case for severe trauma or in case of tumour resection is, that the AI calculating the reconstructions steps also uses pictures form the patient before the incident.

2. 2.

   Tracking for AR guidance, VR training and automated surgery. In the future OR, every step of the surgery is tracked. This data can be used to generate training scenarios in VR simulation but also to teach an AI to perform surgery autonomously. Further, it allows to gather the various existing individual surgical techniques and may allow to find the objectively best procedures. In AR supported surgical guidance applications this knowledge is used for generating the AR instructions.

3. 3.

   XR for Telesurgery. In the future telesurgery will not be performed by the surgeon looking at mono- or stereoscopic images of a live feed. Instead, a detailed and complete 3D representation of the patenting will be simulated visually, haptically, auditory and olfactory. This allows the surgeons to perform the surgery on a virtual patient whilst their movements are tracked and translated to according movements of the surgery robot.

4. 4.

   Miscellaneous Use Cases. (1) AI based detection of a sudden vessel rupture, with AR visualization of the rupture spot. (2) AI based in-situ detection of tumours with AR visualisation and guidance. (3) Integrating of health monitor data and other patient data into a single cockpit that is visualized in AR for the surgeon during surgery.

In this talk, I presented our experience at Universidad Carlos III de Madrid introducing Augmented Reality (AR) in the Operating Room (OR). Our initial open-source solution combined a 3D-printed reference, a surgical guide, and an application running on a smartphone, offering a versatile approach for visualizing anatomical models augmented over 3D-printed phantoms or directly onto patients during surgery. With this straightforward method, we successfully displayed essential surgical planning information in training sessions for sacral neurostimulation and during craniosynostosis and microtia surgeries. Smartphones and tablets offer numerous advantages, such as affordability, ease of introduction into sterile environments via transparent bags, and the possibility of sharing the AR visualization among multiple users. However, when porting our AR solutions to Head-Mounted Displays (HMDs) like Microsoft Hololens2, the interaction with holograms and the immersive experience greatly improved. Despite current HMD limitations, this approach offers significant benefits for the OR, including natural hand-gesture interactions and enhanced 3D perception. Our recent work integrating Hololens2 with the 3DSlicer platform has opened new possibilities, combining the advantages of AR visualization with the computational and algorithmic capabilities available in 3DSlicer. These findings were demonstrated after the talk, utilizing a 3D-printed craniosynostosis patient model to visualize AR content both on a tablet and via Hololens2.

The main outcome for me for this seminar is a much better understanding of the needs of the medical professionals in the OR, in terms of things they can and cannot do, and their requirements of supporting technology like XR. Discussions with doctors and XR researchers led to new ideas for visual representations of relevant data in an integrated context that can be used inside the OR for different kinds of procedures.

Several aspects are important to consider in order to advance the use of extended (XR) or mixed reality (MR) in the operating room (OR). When thinking about MR, we mostly focus on the visual aspects, but MR can also augment or integrate other senses, such as audio or haptic, which can also support a procedure. There have been approaches to use audio or tactile information to guide interventions (e.g., [3] or [5]). Furthermore, also for surgery planning, haptic props have been shown as a promising approach to interacting with image data (e.g., [1, 2] or [4]). Multimodal feedback might also be particularly interesting in a situation where a specialist is joining remotely and is virtually present in the room. Subtle on-site occurrences, like a sound from a device or someone coming into the OR asking a question, might not be picked up and transmitted by the collaboration system by default but distract the team on-site. This could lead to confusion as the reason for the team’s attention shift is unclear to the remote specialist. Therefore, it is important to consider what and how it is transmitted in these remote collaboration settings. To create a better sense of co-presence, a system should also not simply represent the position the remote person placed themselves but maybe also add multimodal feedback for the on-site personnel, e.g., heat patches imitating the body heat someone standing there would radiate. Likewise, it might be worthwhile to explore which kind of user representations support remote collaboration best, as humans pick up subtle cues without conscious processing that influence the relationships. For example, our breathing and heart rate are perceived and can synchronise, leading to a feeling of closeness and allowing for an understanding of the other person’s state in real settings, which are details that current avatar representations omit. Additionally, as avatar representations can also be a source of distraction, investigating which aspects of the user representation are beneficial for use in a high-stress environment and which add stress. The aspects of this talk were intensely discussed in the context of remote collaboration for surgical contexts, but might also play a role in robotics-assisted surgery or remote robotic surgery. It is worthwhile to investigate how the humans operating them can be present in the OR, and, potentially, the results can also be transferred to robotics representations to increase awareness and collaboration with it.