Source-linked AI summary
Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints
David Recasens, José Lamarca, José M. Fácil, J. M. M. Montiel, Javier Civera
TL;DR
Monocular endoscopic reconstruction must handle deforming, weakly textured, and poorly illuminated in-body scenes. Endo-Depth-and-Motion combines self-supervised depth, photometric camera tracking, and volumetric fusion to produce dense reconstructions and motion estimates, with high-quality and competitive results reported on Hamlyn.
Problem
In-body monocular videos make camera-motion estimation and dense scene reconstruction difficult because cavities can deform and images can lack texture or stable illumination.
Method
The pipeline predicts self-supervised depth, uses it as pseudo-RGBD keyframes for photometric pose tracking, and fuses registered depth maps volumetrically.
Results
The paper reports high-quality Hamlyn reconstructions, competitive depth performance against relevant baselines, and satisfactory TSDF fusion performance.
Takeaways & Limitations
Self-supervised depth, photometric tracking, and volumetric fusion form a promising pipeline for dense monocular reconstruction and motion estimation in endoscopies.
Abstract
from arXiv · showhide
Estimating a scene reconstruction and the camera motion from in-body videos is challenging due to several factors, e.g. the deformation of in-body cavities or the lack of texture. In this paper we present Endo-Depth-and-Motion, a pipeline that estimates the 6-degrees-of-freedom camera pose and dense 3D scene models from monocular endoscopic videos. Our approach leverages recent advances in self-supervised depth networks to generate pseudo-RGBD frames, then tracks the camera pose using photometric residuals and fuses the registered depth maps in a volumetric representation. We present an extensive experimental evaluation in the public dataset Hamlyn, showing high-quality results and comparisons against relevant baselines. We also release all models and code for future comparisons.
I. INTRODUCTION
Monocular endoscopic reconstruction and camera localization are challenging because in-body scenes violate assumptions common to 3D vision, including rigidity, texture, and stable illumination. The paper targets dense cavity reconstruction and accurate ego-motion using depth networks, photometric tracking, and volumetric fusion.
- In-body endoscopic images challenge state-of-the-art 3D vision because illumination can be insufficient or unstable and texture can be scarce.
- The paper targets dense 3D reconstructions of in-body cavities and accurate camera ego-motion from monocular endoscopic sequences.
- Endo-Depth-and-Motion creates pseudo-RGBD keyframes with depth networks, estimates motion photometrically, and fuses registered keyframes volumetrically.
- The approach reports high-quality reconstructions on the Hamlyn dataset and competitive results against relevant baselines.
II. RELATED WORK
Prior depth and motion methods often rely on sensors, rigidity, texture, or stable correspondences that are problematic in endoscopic imagery. Endo-Depth-and-Motion instead combines self-supervised depth with photometric odometry and pseudo-RGBD keyframes for monocular reconstruction.
- Deep depth networks evolved from supervised training with depth sensors toward self-supervised methods based mainly on multi-view photometric consistency.
- Self-supervised depth learning avoids requiring additional sensors, sophisticated renderings, or domain transfer for in-body monocular reconstruction.
- Multi-view depth-and-motion methods face drastic illumination changes, weak texture, deformations, tool insertions, fluids, and small camera motions in in-body images.
- The proposed combination uses photometric odometry with self-supervised Endo-Depth predictions as pseudo-RGBD keyframes.
- Earlier intracorporeal SLAM methods include filtering, nonlinear tracking and mapping, map densification, and deformable approaches based on rigidity or deformation models.
III. OVERVIEW
Endo-Depth-and-Motion processes monocular endoscopic video in three stages: predict keyframe depth, estimate frame motion relative to keyframes, and fuse keyframe depths into a volumetric model.
- A deep neural network predicts pixel-wise depth for selected keyframes from the monocular endoscopic video.
- Camera motion relative to the closest keyframe is estimated by minimizing photometric error with image pyramids and robust error functions.
- Keyframe depth maps are fused into a TSDF-based volumetric representation.
IV. ENDO-DEPTH
Endo-Depth predicts dense depth maps using a self-supervised Monodepth2-style network trained with photometric and smoothing losses. The photometric loss warps source images using depth and camera transformations, while stereo or monocular training supplies the source views.
- The depth model uses a U-Net with a ResNet18 encoder and maps an image I to its depth map D=f(I;θ).
- Training minimizes a weighted sum of photometric loss L_p and smoothing loss L_s in a self-supervised manner.
- The photometric loss compares each target pixel with the minimum appearance residual across warped source images and combines reprojection error with SSIM.
- Source images can come from stereo pairs, adjacent monocular frames, or both training setups.
- The transformation T_t't converts points from the target frame to the source frame, using calibrated stereo motion or a learned monocular-view transformation.
V. PHOTOMETRIC TRACKING
The method tracks camera motion between a keyframe and current frame by minimizing photometric error using dense depth predicted by Endo-Depth. A coarse-to-fine strategy and robust residual handling improve optimization while the system estimates six-degree-of-freedom pose updates.
- The tracker estimates the relative transformation Tck between a target keyframe and current source frame using dense keyframe depth.The keyframe depth is predicted by Endo-Depth, and corresponding pixels are aligned through photometric optimization.
- Photometric tracking minimizes color differences between corresponding keyframe and warped-frame pixels in a Lucas–Kanade-style nonlinear least-squares problem.Correspondences are obtained by unprojection and projection, with the warped color sampled from the current frame.
- The optimizer first estimates rotation at the coarsest pyramid level, then estimates the full six-degrees-of-freedom pose at finer scales.Lie algebra updates parameterize local motion, while the coarse-to-fine procedure improves convergence and resilience to motion blur.
- The pose update combines a global transformation with the exponential mapping of local Lie-algebra updates, Tck = expSE(3)(ψ)T0ck.Gauss–Newton optimization converges in a few steps according to the method description.
- The tracker does not model occlusions or illumination changes, relying on their small effect in the narrow-baseline setup and saturating residuals to reduce outlier influence.The saturation threshold γ is determined experimentally.
VI. VOLUMETRIC RECONSTRUCTION
Registered pseudo-RGBD keyframes are fused into a volumetric implicit surface representation using a truncated signed distance function.
- The pipeline fuses registered pseudo-RGBD keyframes into a TSDF-based implicit surface representation.The scene is divided into voxels, each storing a cumulative signed distance to the closest surface truncated at a specified depth.
VII. EXPERIMENTAL RESULTS
The evaluation uses challenging intracorporeal Hamlyn sequences containing weak textures, deformations, reflections, surgical tools, and occlusions.
- The experiments use 21 Hamlyn videos with diverse image resolutions and calibration parameters.The selected sequences depict intracorporeal scenes with weak textures, deformations, reflections, surgical tools, and occlusions.
A. Endo-Depth
Endo-Depth is evaluated on challenging Hamlyn endoscopic sequences using self-supervised training configurations, cross-validation, and comparisons with supervised and multi-view baselines. Stereo or monocular-stereo training provides strong depth performance and stable global scale, while the method also supports dense reconstruction across deformable scenes.
- Evaluation setup: The evaluation uses 21-fold cross-validation across Hamlyn videos, grouping sequences with similar resolution and camera intrinsics for training.Test frames with low-quality ground truth were removed, and standard depth-network metrics were reported.
- Training modalities: Stereo loss achieves the best performance in most sequences because stereo pairs avoid illumination changes and scene deformation.Monocular training is additionally affected by learning relative camera motion.
- Scale stability: Endo-Depth trained with monocular-stereo or stereo losses learns a stable global scale without depth scaling, with average RMSE differences below 1 cm from scaled evaluation.The stable scale is reported as important for reducing trajectory drift.
- Baseline comparison: Endo-Depth is compared with supervised LapDepth and multi-view IsoNRSfM on Hamlyn depth reconstruction.IsoNRSfM was re-run with an implementation that produced better results than the referenced prior work.
- Dense reconstruction: The method evaluates dense depth on challenging videos containing deformations, low texture, and tool intrusion, while reporting results across 21 Hamlyn sequences.The authors emphasize dense whole-image depth, discontinuity handling, and single-shot prediction as advantages for deformable mapping.
- Efficiency: Endo-Depth processes one test image in approximately 15 ms on an Nvidia RTX 2080Ti GPU and 55 ms on an AMD Ryzen 9 3900X CPU.The reported timings are for depth prediction alone.
B. Camera Tracking
The camera-tracking evaluation compares the proposed photometric approach with ICP and hybrid photometric-geometric alternatives. It also studies keyframe creation ratios and reports per-frame computation time on GPU and CPU.
- Tracking comparisons: Photometric tracking is compared with point-to-point ICP, point-to-plane ICP, and hybrid photometric-geometric residuals using optimal keyframe creation ratios.Different keyframe creation ratios are also evaluated qualitatively in Fig. 4.
- Efficiency: The Python implementation requires approximately 300 ms per frame on an Nvidia RTX 2080Ti GPU and 700 ms on an AMD Ryzen 9 3900X CPU.These are the reported computation times for the tracking implementation.
C. Volumetric Fusion
The pipeline fuses predicted depth maps from registered Hamlyn keyframes into TSDF volumetric reconstructions. Qualitative results show reconstructions across multiple videos, although ground-truth scenes and camera trajectories are unavailable for direct accuracy assessment.
- C. Volumetric Fusion: TSDF fusion combines predicted depth maps from registered keyframes into volumetric 3D reconstructions.The representation stores a truncated signed distance function over voxels and updates it using predicted depth maps.
- C. Volumetric Fusion: Ground-truth scenes and camera trajectories are unavailable, so reconstruction accuracy is assessed qualitatively.
- C. Volumetric Fusion: Volumetric fusion of a 1200-frame reconstruction takes approximately 7 s on an AMD Ryzen 9 3900X CPU.
VIII. CONCLUSIONS AND FUTURE WORK
The paper presents Endo-Depth-and-Motion as a combination of self-supervised depth learning, photometric odometry, and volumetric fusion for endoscopic reconstruction and motion estimation. Its evaluation reports competitive depth performance and satisfactory tracking and fusion, while identifying deformation modeling as future work.
- VIII. CONCLUSIONS AND FUTURE WORK: Endo-Depth-and-Motion combines self-supervised depth learning, photometric odometry, and volumetric fusion for endoscopic videos.
- VIII. CONCLUSIONS AND FUTURE WORK: Endo-Depth achieves competitive performance against IsoNRSfM and supervised LapDepth on Hamlyn sequences.
- VIII. CONCLUSIONS AND FUTURE WORK: Endo-Depth retains its performance without per-image scaling, indicating that stereo losses can learn the scene’s global scale.
- VIII. CONCLUSIONS AND FUTURE WORK: The photometric tracker estimates camera motion relative to pseudo-RGBD keyframes, and TSDF fusion produces satisfactory reconstructions.
- VIII. CONCLUSIONS AND FUTURE WORK: Future work will incorporate deformation models into the scene representation.