Source-linked AI summary
EndoLive: Real-Time Style Transfer for Endoscopic Endonasal Skull Base Surgical Video
Griffin Hurt, Calvin Brinkman
TL;DR
Cadaveric practice differs from live surgery in tissue appearance and bleeding, although cadaveric models are widely used for training complex EESB procedures. EndoLive combines ConStructS and HyPER-GAN, training on unpaired cadaveric and live frames before distilling translated pairs for real-time inference. EndoLive achieves a gap close of 64% and, while performing preprocessing, reaches 55 frames per second, above the 30 frames per second needed for a real-time video stream.
Problem
Cadaveric practice differs from live surgery in tissue appearance and bleeding, although cadaveric models are widely used for training complex EESB procedures.
Source: §1 · p. 2Method
Results
EndoLive achieves a gap close of 64% and, while performing preprocessing, reaches 55 frames per second, above the 30 frames per second needed for a real-time video stream.
Source: §5.2 · p. 6, §5.3 · p. 6Takeaways & Limitations
The authors conclude that EndoLive can produce high-quality synthetic endoscope images and run in real time on desktop, laptop, and mixed-reality headset devices.
Source: §6 · p. 6Neither underlying architecture is temporally aware, leaving long surgical videos susceptible to anatomy-color or generated-blood changes appearing and disappearing.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
Complex surgical procedures around critical anatomy, such as the endoscopic endonasal skull base surgery, requires significant practice and training on the part of the surgeon before they are allowed to perform the operation on a live patient. This training in typically done in cadaveric specimens, due to them containing the same critical structures as a living human. However, cadavers are not a perfect 1-to-1 substitute for a living patient. The dead and preserved tissues of a cadaver are colored completely differently than a living human, and -- without complex and expensive pumping systems -- do not bleed in the same way. As a result, identifying the critical pieces of anatomy that make this procedure so complex can be quite different in a live case than in a surgeon's cadaveric practice. This paper presents EndoLive, a framework for real-time style transfer between cadaveric endoscopic video and living human endoscopic video. Our method combines the ConStructS GAN model for realistic style transfer for surgical applications, with the HyPER-GAN model that can learn complex translations and perform them in real-time. We train EndoLive on unpaired cadaveric and live images taken from an endoscope, and test the trained model with cadaveric video, on a variety of devices. Experimental results demonstrate that EndoLive can perform cadaveric-to-live translation at speeds well above the minimum necessary for real-time, while maintaining semantic consistency of critical anatomical structures. Our source code is available at https://github.com/griffhurt/endolive.
1. Introduction
EESB surgery requires extensive training, but cadaveric practice differs from live surgery in tissue appearance and bleeding. EndoLive proposes combining two models for real-time cadaveric-to-live endoscopic video translation, with spatial-computing use motivating fast inference.
1. Introduction
- Some surgeons spend up to 9 years after medical school training in the skull-base approach before independent practice.Hands-on training often occurs under senior surgeons in operative settings, while cadaveric dissection is the most common model-based tool.Source: §1 · p. 2
- Cadaveric models support surgical training but lack active bleeding and breathing perception, and perfusing fresh cadavers adds substantial cost.They can provide gross-anatomy estimates and help surgeons develop spatial reasoning skills.Source: §1 · p. 2
- Spatial-computing surgery applications require extremely low glass-to-glass latency, making fast inference and high frame rates necessary.Sterility constraints prevent surgeons from physically adjusting surrounding monitors, while virtual displays can be manipulated within a head-mounted display.Source: §1 · p. 2
- EndoLive combines ConStructS and HyPER-GAN for real-time cadaveric-to-live endoscopic video style transfer.The authors investigate accuracy and efficiency on desktop hardware and an Apple Vision Pro spatial-computing device.Source: §1 · p. 2
2. Related Work
Prior work spans surgical image translation, semantic consistency, photorealistic rendering, and real-time patch-based translation. EndoLive draws on these approaches to address endoscopic surgical video style transfer.
2. Related Work
- Style transfer has broad computer-vision applications, while this work focuses on real-time translation for surgical use.The authors note that these scopes have been studied independently, but their particular combination has received little attention.Source: §2 · p. 2
2.1.1. Laparoscopic image-to-image translation
- A laparoscopic translation study extended CycleGAN with a structural-similarity loss and produced synthetic images with quality akin to real surgical images.The synthetic data was intended for tasks including image processing in deep-learning networks.Source: §2.1.1 · p. 3
2.1.2. Exploring semantic consistency
- ConStructS uses patch and semantic losses to preserve semantic consistency in unpaired one-way translation, but was not designed for real-time inference.Its encoder-decoder approach was developed to generate training data for deeper models.Source: §2.1.2 · p. 3
2.2.1. Enhancing photorealism enhancement
- Enhancing Photorealism Enhancement uses simulation-only G-buffer metadata to produce photorealistic translations, but is not real-time and is impractical for cadaveric surgery.Figure 2 shows example cadaveric and live images from the EndoLive dataset.Source: §2.2.1 · p. 3, Figure 2 · p. 3
2.2.2. Patch-based translation
- HyPER-GAN uses patch-based training on images already translated by another model to enable high-quality image translation in real time.This design lets it be applied to pre-existing style-transfer models and use cases.Source: §2.2.2 · p. 3
3. Data
The study constructs an endoscopic dataset from live EESB surgery and cadaveric trials. It excludes non-endoscopic portions and standardizes the extracted images for training.
3. Data
- The dataset draws on endoscopic videos recorded in pilot tests and ongoing skull-base surgery research because existing endoscopic datasets typically lack cadaveric images.The source videos came from a previous study and ongoing research.Source: §3 · p. 3
3.1. Data Cleaning
- Segments filmed outside the nasal cavity were removed because cadaveric and live patient faces differ visually and surgeons are not attending to the endoscopic feed then.The excluded segments include closing and tool changes.Source: §3.1 · p. 4
3.2. Data Preprocessing
- The dataset contains 19,619 live and 17,151 cadaveric images extracted at 30 frames per second from 5.45 and 4.76 hours of video, respectively.Cadaveric frames were cropped to the endoscope feed, and all images were resized to 512 by 512 pixels.Source: §3.2 · p. 4
4. Methods
EndoLive uses ConStructS to translate unpaired cadaveric frames and then distills its outputs into HyPER-GAN for real-time inference. The implementation modifies data handling, while both underlying models lack temporal awareness.
4. Methods
- ConStructS runs at roughly 10 fps on dual NVIDIA RTX A5000s, so EndoLive uses its unpaired translations to create pairs for HyPER-GAN training and real-time inference.HyPER-GAN avoids the G-buffer semantic information required by EPE and uses a smaller architecture to reduce GPU memory use.Source: §4 · p. 4
4.1. Modifications to the HyPER-GAN Code
- OpenCV video I/O was moved to separate producer-consumer threads, and image normalization and axis permutation were moved from CPU to GPU.The authors note that video-reading overhead would be absent when frames come directly from a capture card.Source: §4.1 · p. 4
4.2. Note on Temporal Stability
- Neither model is temporally aware, so long videos may show anatomy color and generated blood appearing and disappearing.Testing showed overall good temporal stability, which the authors associate with training on many similar successive frames and similar cadaveric and live anatomy.Source: §4.2 · p. 5
5. Experiments
Experiments evaluate EndoLive on unseen cadaveric frames, measuring image quality and translation speed across hardware. EndoLive closes much of the KID gap and runs above 30 frames per second even on the Apple Vision Pro with preprocessing.
5.1. Testing Data
- The test set contains 10,800 frames from three minutes of cadaveric video, with frames unseen during training despite showing the same anatomy.Source: §5.1 · p. 5
5.2. Quality evaluation
- EndoLive closes 64% of the KID gap, versus 75% for base EPE without HyPER-GAN; it also outperforms base ConStructS and HyPER-GAN EPE.The HyPER-GAN model trained on ConStructS closed more of the gap than ConStructS alone; the authors suggest a possible role for pretrained VGG-16, but call that explanation speculative.Source: §5.2 · pp. 5–6, Figure 4 · p. 6
- HyPER-GAN trained on ConStructS closes more of the KID gap than ConStructS alone, though the authors describe the reason as speculative.They suggest pretrained VGG-16 may help the model capture semantic similarity between paired live and cadaveric images.Source: §5.2 · p. 6
5.3. Speed evaluation
- While performing preprocessing, the system reaches 55 frames per second, well above the 30 frames per second needed to keep up with a real-time video stream.Figure 5 reports average translated frames per second for the RTX A5000, M3 Max processor, and M2 processor, with and without preprocessing.Source: §5.3 · p. 6, Figure 5 · p. 6
6. Conclusion
The authors conclude that EndoLive produces high-quality cadaver-to-live endoscopic translations in real time across desktop, laptop, and mixed-reality headset hardware. They identify formal surgeon evaluation and testing other endoscopic procedures as future work.
6. Conclusion
- EndoLive performs real-time cadaver-to-live endoscope video translation with image quality comparable to established high-performing style-transfer models.The authors say the framework runs on desktop computers, laptops, and mixed-reality headsets.Source: §6 · p. 6
- The authors say EndoLive could increase the realism of cadaveric trials and potentially reduce the learning curve for surgeons transitioning to living patients.Source: §6 · p. 6
6.1. Future Work
- Future work includes a formal study with surgeons to assess effects on critical-anatomy identification and cadaveric training, and applying EndoLive to other endoscopic procedures.Source: §6.1 · p. 6