Source-linked AI summary
Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery
Yuehao Wang, Yonghao Long, Siu Hin Fan, Qi Dou
TL;DR
The paper addresses deformable tissue reconstruction from single-viewpoint stereo endoscopic video, where non-rigid motion, tool occlusion, and limited 3D clues complicate existing approaches. It uses dynamic neural radiance fields with mask-guided ray casting and stereo-depth guidance, achieving superior reconstruction across robotic-surgery scenarios.
Problem
Reconstructing deformable surgical tissues from single-viewpoint stereo video is important but challenged by non-rigid deformation, tool occlusion, and limited 3D clues.
Method
The framework uses canonical and time-dependent neural radiance fields, mask-guided ray casting, stereo depth-cueing ray marching, and depth-supervised optimization.
Results
The method achieves superior performance across robotic-surgery scenarios including large elastic deformations and tissue cutting.
Takeaways & Limitations
Neural rendering provides a demonstrated pathway for reconstructing dynamic, occlusion-affected surgical scenes from single-viewpoint binocular images.
Abstract
from arXiv · showhide
Reconstruction of the soft tissues in robotic surgery from endoscopic stereo videos is important for many applications such as intra-operative navigation and image-guided robotic surgery automation. Previous works on this task mainly rely on SLAM-based approaches, which struggle to handle complex surgical scenes. Inspired by recent progress in neural rendering, we present a novel framework for deformable tissue reconstruction from binocular captures in robotic surgery under the single-viewpoint setting. Our framework adopts dynamic neural radiance fields to represent deformable surgical scenes in MLPs and optimize shapes and deformations in a learning-based manner. In addition to non-rigid deformations, tool occlusion and poor 3D clues from a single viewpoint are also particular challenges in soft tissue reconstruction. To overcome these difficulties, we present a series of strategies of tool mask-guided ray casting, stereo depth-cueing ray marching and stereo depth-supervised optimization. With experiments on DaVinci robotic surgery videos, our method significantly outperforms the current state-of-the-art reconstruction method for handling various complex non-rigid deformations. To our best knowledge, this is the first work leveraging neural rendering for surgical scene 3D reconstruction with remarkable potential demonstrated. Code is available at: https://github.com/med-air/EndoNeRF.
1 Introduction
Surgical scene reconstruction from stereo endoscopic video is important for clinical and robotic applications but remains difficult because scenes deform, viewpoints are sparse, and tools occlude tissue. The paper addresses these challenges by adapting neural rendering with dynamic representations, mask-guided ray casting, and stereo-depth strategies.
- Surgical reconstruction supports intra-operative navigation, augmented reality, simulation, immersive education, and robotic surgery automation.
- Dynamic tissue deformation, topology changes, sparse viewpoints, and tool occlusion create major reconstruction challenges.Sparse viewpoints limit 3D clues, while instruments reduce reconstruction completeness.
- Earlier SLAM-based methods fuse stereo depth maps but commonly assume static scenes or exclude surgical tools, limiting practical use.Recent methods combine tool masking, stereo depth estimation, and related reconstruction components.
- The proposed framework represents dynamic surgical scenes with neural implicit fields and addresses occlusion and weak single-view geometry using three stereo-aware strategies.These strategies are mask-guided ray casting, depth-cueing ray marching, and depth-supervised optimization.
2 Method
The method reconstructs deformable surgical tissues from single-viewpoint stereo video using dynamic neural radiance fields, while addressing tool occlusion and weak single-view 3D cues.
- 2.1 Overview of the Neural Rendering-based Framework: Each training iteration samples a frame, casts tool-guided rays, marches points using stereo depth, renders color and occupancy, and optimizes the fields.The pipeline uses stereo depth cues during sampling and jointly supervises rendered color and optical depth.
- 2.2 Deformable Surgical Scene Representations: The framework represents deformable surgical scenes with a canonical neural radiance field and a time-dependent displacement field.Both fields are implemented as multilayer perceptrons; the displacement maps space-time coordinates to deformation relative to the canonical field.
- 2.3 Tool Mask-Guided Ray Casting: Tool masks assign zero sampling probability to instrument pixels, directing ray casting toward tissue regions instead.Importance maps initialize tissue pixels to one and tool pixels to zero, with scaling for tissue areas frequently occluded across frames.
- 2.4 Stereo Depth-Cueing Ray Marching: Stereo depth-cueing ray marching concentrates sampled points near tissue surfaces rather than relying solely on hierarchical stratified sampling.A Gaussian-like impulse is centered at each pixel’s estimated depth, and its width is controlled by ξ.
- 2.5 Optimization for Deformable Radiance Fields: Statistical depth refinement identifies high-residual stereo-depth pixels as corrupt and replaces them with smoother predicted depths before continued optimization.This procedure is intended to alleviate artifacts caused by fitting fuzzy-pixel and specular-highlight errors.
3 Experiments
Experiments on six DaVinci robotic prostatectomy clips evaluate reconstruction under tissue deformation, topology changes, tool occlusion, and imperfect stereo depth. The method shows stronger qualitative and quantitative performance, while depth-related modules each address distinct reconstruction failures.
- Experimental Setup: The evaluation uses 807 frames from six single-viewpoint stereo clips spanning traction, tissue pushing and pulling, and tissue cutting.The clips last 4–8 seconds at 15 fps and include non-rigid deformation and tool occlusion.
- Qualitative Results: The method reconstructs water-tight tissues during large tool-induced deformations and captures continuous per-frame changes over consecutive pulling.The comparison method fails to fully track these deformations and produces holes and noisy points.
- Qualitative Results: During tissue cutting with topology changes, the method tracks detailed cutting procedures and recovers tool-occluded tissue regions.Mask-guided ray casting and the interpolation property of neural implicit fields jointly support recovery of hidden tissues.
- Quantitative Results: The method outperforms E-DSSR by ↑16.433 PSNR, ↑0.295 SSIM and ↓0.342 LPIPS on the dataset.Removing the neural displacement field causes a noticeable performance drop.
- Ablation Study: Without depth-supervised loss, the pipeline cannot learn correct geometry from single-viewpoint input.The ablation study evaluates depth-supervised loss, depth refinement, and depth-cueing ray marching.
- Ablation Study: Disabling depth refinement introduces abrupt artifacts, while depth-cueing ray marching further reduces artifacts, especially at boundary points.The refinement module addresses corruption in stereo depth estimation.
4 Conclusion
The paper presents a neural rendering framework for dynamic surgical-scene reconstruction from single-viewpoint binocular images. It combines dynamic neural radiance fields, mask-guided ray casting, and stereo depth priors to address tissue deformation and tool occlusion, with superior performance demonstrated across robotic-surgery scenarios.
- The framework represents surgical scenes with a dynamic neural radiance field to address complex tissue deformation and tool occlusion.
- Mask-guided ray casting handles tool occlusion, while a stereo depth prior addresses the limited 3D information of the single-viewpoint setting.
- Superior performance is achieved across robotic-surgery scenarios including large elastic deformations and tissue cutting.
- The authors identify potential for NeRF-based reconstruction techniques to inspire robotic-surgery scene understanding and downstream clinical-oriented tasks.