Source-linked AI summary
Dense Depth Estimation in Monocular Endoscopy with Self-supervised Learning Methods
Xingtong Liu, Ayushi Sinha, Masaru Ishii, Gregory D. Hager, Austin Reiter, Russell H. Taylor, Mathias Unberath
TL;DR
The paper addresses dense depth estimation for monocular endoscopy without manual labels, CT scans, or anatomy and shading models. It trains a CNN using sparse geometric supervision from monocular videos and multi-view stereo, achieving submillimeter cross-patient error and outperforming recent natural-video self-supervised methods.
Problem
Dense endoscopic depth estimation lacks practical ground truth, while photometric-constancy assumptions in generic self-supervised methods do not hold for endoscopy.
Method
The method trains a convolutional network with sparse SfM-derived supervision and geometric loss functions, requiring monocular endoscopic images without manual labels or CT data.
Results
0.40 (±0.18) mm average residual error was achieved across four testing patients, with submillimeter errors for all testing reconstructions.
Takeaways & Limitations
The approach generalizes across patients and endoscope cameras while avoiding photometric constancy and additional imaging modalities during training and application.
Takeaways & Limitations
The method depends on SfM producing reasonable reconstructions and camera poses, so it does not apply when SfM fails to do so.
Abstract
from arXiv · showhide
We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires monocular endoscopic videos and a multi-view stereo method, e.g., structure from motion, to supervise learning in a sparse manner. Consequently, our method requires neither manual labeling nor patient computed tomography (CT) scan in the training and application phases. In a cross-patient experiment using CT scans as groundtruth, the proposed method achieved submillimeter mean residual error. In a comparison study to recent self-supervised depth estimation methods designed for natural video on in vivo sinus endoscopy data, we demonstrate that the proposed approach outperforms the previous methods by a large margin. The source code for this work is publicly available online at https://github.com/lppllppl920/EndoscopyDepthEstimation-Pytorch.
I. INTRODUCTION
Endoscopic navigation needs accurate, dense 3D reconstruction from video, but endoscopy challenges conventional reconstruction and supervised learning. The paper motivates a self-supervised approach that integrates multi-view stereo information without relying on photometric constancy.
- Clinical motivation: Computer vision navigation using intra-operative endoscopic video can integrate easily into clinical workflows without additional hardware.Such systems provide anatomical and positional information that helps surgeons avoid nearby critical structures.
- Reconstruction challenge: Accurate registration requires sufficiently dense intra-operative 3D reconstructions, which are difficult because of specular reflectance, changing appearance, and tissue deformation.
- Existing reconstruction methods: Traditional SfM and SLAM can reconstruct 3D structure and camera poses, but sparse features in endoscopic anatomy produce uneven reconstructions that may impair registration.
- Learning-based methods: Dense supervised depth learning is difficult because endoscopic images lack directly corresponding dense ground truth, often motivating synthetic data generated from patient-specific CT scans.
- Self-supervised learning gap: Generic self-supervised depth methods are poorly suited to endoscopy because photometric constancy and reliable global-scale cues are unavailable.
- Paper motivation: The proposed direction integrates SfM reconstructions and camera poses into network training, using geometric relationships that do not assume lighting constancy.
II. METHODS
The method trains convolutional networks for dense monocular-endoscopy depth estimation using sparse self-supervisory signals derived from SfM. Its training inputs combine video geometry and sparse maps, while custom layers transform predictions for loss computation.
- Training framework: The training framework derives sparse self-supervisory signals from SfM applied to monocular endoscopic video sequences.
- Training inputs: Training uses endoscopic frames, camera poses and intrinsics, sparse depth maps, sparse soft masks, and sparse flow maps.
- Network architecture: Custom Depth Scaling, Depth Warping, and Flow from Depth layers convert monocular depth predictions into forms suitable for loss calculation.
A. Training Data
Training data are generated automatically from calibrated endoscopic videos through undistortion, SfM reconstruction, visibility processing, and projection into sparse depth, flow, and soft-mask representations. These signals provide sparse geometric supervision while weighting reconstruction reliability.
- Data generation: The fully automated data pipeline takes endoscopic and calibration videos and can in principle run online by replacing SfM with SLAM-based methods.
- Preprocessing: Video frames are undistorted, then SfM estimates a sparse reconstruction, camera poses, and point visibility; filtering removes extreme reconstruction outliers.
- Sparse depth maps: Sparse depth maps project SfM 3D points into each frame and anchor the scale of monocular depth predictions to the SfM results.
- Sparse depth maps: The projected depth for each visible 3D point is its frame-relative z-axis component, while its 2D projection identifies the corresponding image location.
- Sparse representations: Visibility indicators mark whether each SfM point is visible in a frame, and regions without projected points receive zero values in the sparse representations.
- Sparse flow maps: Sparse flow represents the 2D projected movement of the reconstruction between two frames and incorporates their camera poses.
- Sparse soft masks: The sparse soft mask selects valid sparse signals and downweights less reliable regions, with reliability related to the number of frames used to triangulate each point.
B. Network Architecture
The training architecture uses a two-branch Siamese network that combines single-frame depth predictions with sparse SfM-derived signals and geometric transformations between frames. During application, only the trained single-frame branch predicts dense depth up to a global scale.
- Training architecture: Training uses two frames from the same video sequence as inputs to a self-supervised two-branch Siamese network.The frame interval is constrained to provide sufficient overlap between the selected frames.
- Application phase: At application time, the trained single-frame architecture predicts a dense depth map that is accurate up to a global scale.The two-branch training structure is not required for single-frame inference.
- Training architecture: The pipeline supplies endoscopic frames, camera poses and intrinsics, sparse depth maps, sparse masks, and sparse flow maps to the network.Custom differentiable layers convert predictions into forms needed for loss computation.
- Monocular depth estimation: The monocular depth module is a modified DenseNet that outputs one-channel depth with linear activation and nearest-neighbor upsampling.These changes adapt the architecture to depth prediction and reduce checkerboard artifacts.
- Custom layers: The Depth Scaling Layer matches predicted depth scale to SfM depth using the sparse depth map as an anchor.The prediction is correct only up to a scale before this matching step.
- Custom layers: The Flow from Depth Layer converts scaled depth into dense 2D flow using relative camera poses and camera intrinsics.This produces flow representations for comparison with sparse SfM flow.
- Custom layers: The Depth Warping Layer warps a depth map from frame k into frame j after modifying depths for the viewpoint change.Bilinear sampling generates the dense depth map viewed from frame j.
C. Loss Functions
The loss design combines scale-invariant agreement with sparse SfM flow and geometric consistency between depth predictions from two frames. Together, these losses use sparse supervision while extending constraints to regions without sparse annotations.
- Loss design: The method introduces novel losses that exploit sparse SfM supervision and enforce geometric consistency between depth predictions from two frames.The two objectives are combined into the overall training loss.
- Sparse Flow Loss: Sparse Flow Loss minimizes differences between predicted dense flow maps and corresponding sparse SfM flow maps.Because it compares 2D pixel movement, the loss is scale-invariant and avoids imbalance from arbitrary SfM scales.
- Depth Consistency Loss: Depth Consistency Loss constrains two independently predicted depth maps after warping one prediction into the viewpoint of the other.It supplies information for regions where sparse annotations are unavailable.
- Overall loss: The overall loss combines flow and consistency terms as L(j, k) = λ1Lflow(j, k) + λ2Lconsist(j, k).The weights λ1 and λ2 control the contributions of the two objectives.
A. Experiment Setup
Experiments use rectified sinus endoscopy videos from patients and cadavers, with leave-one-out training and evaluation against CT-derived surfaces and SfM-derived sparse depth. Accuracy is assessed through registered point-cloud residuals and scale-invariant depth metrics on valid sparse locations.
- Dataset and splits: The dataset contains 10 rectified sinus endoscopy videos from 8 patients and 2 cadavers, totaling approximately 30 minutes.The videos were acquired with different endoscopes under an IRB-approved protocol.
- Dataset and splits: Leave-one-out experiments use data from 7 of 8 patients for training and evaluation on the held-out patient.This supports cross-patient assessment of generalization.
- CT-based evaluation: One evaluation compares point clouds from predicted depths with corresponding surface models generated from patient CT data.Twenty frames with sufficient anatomical variation are selected per testing patient.
- CT-based evaluation: The CT-based accuracy estimate is the average Euclidean residual error over closest point pairs after point-cloud registration.Registered point clouds are compared with CT surface models using IMLOP.
- SfM-based evaluation: SfM-based evaluation uses test frames with valid camera poses and rescales predictions to match corresponding sparse depth maps.Only scale-invariant metrics are used because SfM has arbitrary global scale.
- SfM-based evaluation: SfM metrics are evaluated only at valid sparse-depth positions and corresponding prediction locations.The reported metrics include Absolute Relative Difference and Threshold values at σ = 1.25, 1.252, and 1.253.
- Data sparsity: Sparse reconstructions contain 4687 ± 6276 points on average, with 1518 ± 1280 projected points per image and 1.85 ± 1.56% valid pixels after downsampling.These figures characterize the sparsity of the supervision available to the method.
B. Cross-patient Study
The cross-patient study evaluates generalization by leaving each of four patients out during training. All testing reconstructions achieved submillimeter residual errors, with a 0.40 (±0.18) mm average across patients.
- Four leave-one-out experiments left out Patients 2, 3, 4, and 5 during training for evaluation.
- The qualitative comparison includes testing and training images, depth maps, and reconstructions from the proposed method and two competing methods.Predictions were re-scaled with sparse depth maps and normalized using the same maximum depth values for visualization.
- All testing reconstructions achieved submillimeter residual errors across the four testing patients.
- 0.40 (±0.18) mm was the average residual error across testing frames from all four testing patients.
C. Comparison Study
The comparison study evaluates the proposed method against Zhou et al. and Yin et al. using qualitative, CT-registration, and SfM-related comparisons. The proposed method outperformed both alternatives by a large margin, including lower residual error on Patient 4.
- The proposed method outperformed Zhou et al. and Yin et al. by a large margin in depth-prediction comparisons.
- 0.38 (±0.13) mm was the proposed method’s average residual error on Patient 4, compared with 1.77 (±1.19) mm for Zhou et al. and 0.94 (±0.36) mm for Yin et al.
- The comparison used four trained proposed-method models and gave the competing methods access to all patient data except Patient 4 during training.This evaluation therefore favored the competing methods in terms of training exposure.
- Photometric-consistency losses used by the competing methods are challenged by jointly moving camera and illumination in monocular endoscopy.
- The CT-registration visualization shows the proposed reconstruction adhering to patient contours with few outliers, unlike the competing reconstructions.
D. Ablation Study
The ablation study compares training with SFL alone against combined SFL and DCL. Combining both losses improved residual error and produced slight improvements on SfM-related metrics.
- 0.38 (±0.13) mm was the combined SFL-and-DCL model’s average residual error, versus 0.47 (±0.10) mm for SFL alone.
- The combined SFL-and-DCL model achieved absolute relative difference 0.13, compared with 0.14 for SFL alone.
- Threshold metrics at σ = 1.25, 1.252, and 1.253 were 0.85, 0.98, and 1.00 with both losses, versus 0.81, 0.98, and 1.00 with SFL alone.
- DCL provided additional guidance where SfM reconstructions were inaccurate around specularity or missing near boundaries.
IV. DISCUSSION
The method does not require labeled data and generalizes across endoscopes and patients, but its training depends on SfM reconstructions and camera poses and its output has only global scale.
- The method does not require labeled data for training and generalizes well across endoscopes and patients.
- Training relies on reconstructions and camera poses from SfM, so the method does not apply when SfM cannot produce reasonable results.
- Large systematic SfM errors across a large portion of highly dynamic data may cause the method to fail.
- Dense depth maps are produced only up to a global scale, requiring additional application-phase information when global scale is needed.
V. CONCLUSION
The work presents self-supervised dense depth estimation for monocular endoscopy without anatomy or shading models. It uses monocular videos with sparse multi-view stereo supervision and achieves strong cross-patient and comparative results.
- The method trains convolutional neural networks for dense depth estimation in monocular endoscopy without a priori anatomy or shading models.
- It requires monocular endoscopic videos and a multi-view stereo method during training, rather than manual labeling or patient CT scans.
- The approach generalizes across patients, achieving submillimeter residual errors when trained on small amounts of unlabeled data from other patients.
- On in vivo sinus endoscopy data, it outperforms two recent self-supervised depth estimation methods by a large margin.