Source-linked AI summary
Self-Supervised Monocular Depth and Ego-Motion Estimation in Endoscopy: Appearance Flow to the Rescue
Shuwei Shao, Zhongcai Pei, Weihai Chen, Wentao Zhu, Xingming Wu, Dianmin Sun, Baochang Zhang
TL;DR
Endoscopic brightness fluctuations violate the brightness-constancy assumption underlying monocular self-supervised depth and ego-motion estimation. The paper introduces appearance flow and a unified framework to model these variations, and reports large-margin gains across several datasets, with identified artifacts and scale ambiguity remaining limitations.
Problem
Endoscopic brightness fluctuations from illumination changes, non-Lambertian reflections, and interreflections undermine the brightness constancy assumption used for self-supervised depth and ego-motion estimation.
Method
The paper introduces appearance flow and a unified framework with structure, motion, appearance, and correspondence modules to reconstruct appearance and calibrate brightness.
Results
The unified framework outperforms competing self-supervised approaches by a large margin on the SCARED, EndoSLAM, SERV-CT, and Hamlyn2 datasets.
Takeaways & Limitations
Appearance flow relaxes brightness constancy by representing geometric and radiometric transformations, supporting robust endoscopic depth and ego-motion estimation.
Takeaways & Limitations
Depth predictions show slight artifacts in oversaturated regions, and the framework retains the intrinsic scale ambiguity of monocular self-supervised estimation.
Abstract
from arXiv · showhide
Recently, self-supervised learning technology has been applied to calculate depth and ego-motion from monocular videos, achieving remarkable performance in autonomous driving scenarios. One widely adopted assumption of depth and ego-motion self-supervised learning is that the image brightness remains constant within nearby frames. Unfortunately, the endoscopic scene does not meet this assumption because there are severe brightness fluctuations induced by illumination variations, non-Lambertian reflections and interreflections during data collection, and these brightness fluctuations inevitably deteriorate the depth and ego-motion estimation accuracy. In this work, we introduce a novel concept referred to as appearance flow to address the brightness inconsistency problem. The appearance flow takes into consideration any variations in the brightness pattern and enables us to develop a generalized dynamic image constraint. Furthermore, we build a unified self-supervised framework to estimate monocular depth and ego-motion simultaneously in endoscopic scenes, which comprises a structure module, a motion module, an appearance module and a correspondence module, to accurately reconstruct the appearance and calibrate the image brightness. Extensive experiments are conducted on the SCARED dataset and EndoSLAM dataset, and the proposed unified framework exceeds other self-supervised approaches by a large margin. To validate our framework's generalization ability on different patients and cameras, we train our model on SCARED but test it on the SERV-CT and Hamlyn datasets without any fine-tuning, and the superior results reveal its strong generalization ability. Code will be available at: \url{https://github.com/ShuweiShao/AF-SfMLearner}.
1. Introduction
Endoscopic scenes challenge monocular self-supervised depth and ego-motion estimation because sparse features and severe brightness variation undermine conventional assumptions. The paper introduces appearance flow and a unified framework to address these issues, reporting strong results across multiple datasets.
- Motivation: Endoscopic images contain sparse, unevenly distributed key points, limiting conventional multiview stereo reconstruction.The paper connects this difficulty to poor reconstruction quality in endoscopic scenes.
- Motivation: Directly applying autonomous-driving self-supervised methods remains challenging because endoscopic videos violate the brightness constancy assumption.Brightness fluctuations arise from irradiance variation, reflections, and interreflections.
- Method: Appearance flow models brightness-pattern variation and induces a generalized dynamic image constraint combining geometric and radiometric transformations.Its field can vary between nearby pixels, relaxing the brightness constancy requirement.
- Method: The unified framework combines structure, motion, appearance, and correspondence modules to reconstruct appearance and calibrate image brightness.The design also includes a residual-based smoothness loss and an auxiliary loss.
- Results: Detailed experiments show that the designed components improve depth and ego-motion estimation accuracy.The paper evaluates the framework on endoscopic datasets and compares it with competing self-supervised approaches.
- Results: The unified framework outperforms competing self-supervised approaches by a large margin on SCARED, EndoSLAM, SERV-CT, and Hamlyn2.These datasets span the primary evaluation and cross-dataset settings reported in the introduction.
2. Related work
Prior work spans supervised depth prediction, warping-based self-supervision, and endoscopic methods using SfM, PSNR, or affine brightness transformations. This paper instead proposes appearance flow and a direct unified framework that does not require CT scans or SfM preprocessing.
- Supervised depth estimation: Supervised monocular depth methods achieved strong estimates, but accurate large-scale RGB-D datasets are difficult to collect for endoscopic scenes.Sensor noise and tissue non-Lambertian reflections complicate dataset construction.
- Self-supervised estimation: Self-supervised methods learn depth through warping-based view synthesis using separate DepthNet and PoseNet models.Later approaches added geometric priors and mechanisms for edge cases such as motion and occlusion.
- Endoscopic methods: Endoscopic self-supervised methods have used SfM-generated sparse supervision, PSNR objectives, and affine brightness transformers.These approaches represent successive attempts to address supervision or photometric robustness.
- Proposed approach: The proposed method introduces appearance flow and a robust unified self-supervised framework for endoscopic depth and ego-motion estimation.Its appearance-flow formulation addresses brightness variation beyond preceding methods.
- Proposed approach: The method is direct and requires neither CT scans nor multiview stereo algorithms such as SfM.This distinguishes it from approaches that rely on external anatomical data or preprocessing pipelines.
3. Methodology
The methodology replaces brightness constancy with appearance flow and a generalized dynamic image constraint, then integrates these ideas into a unified framework for endoscopic depth and ego-motion estimation. The framework combines geometric, radiometric, correspondence, and regularization components to reconstruct appearance while addressing solution ambiguity.
- Self-supervised estimation: Self-supervised view synthesis estimates depth and ego-motion by warping source frames using predicted depth and relative motion, with appearance differences providing supervision.Depth back-projects target pixels into 3D, ego-motion projects them into another image plane, and differentiable inverse warping synthesizes the source view.
- Brightness inconsistency: Brightness constancy becomes unreliable in endoscopy because light-source motion, non-Lambertian reflection, and interreflection create severe brightness fluctuations that bias supervision and degrade estimates.These effects motivate constraints that include radiometric transformations in addition to geometric transformations.
- Appearance flow: Appearance flow describes radiometric transformation between frames, enabling a generalized dynamic image constraint that combines geometric and radiometric transformations.When appearance flow is zero, the generalized constraint reduces to the conventional brightness-constancy constraint.
- Ill-posed problem: The generalized constraint introduces greater solution indeterminacy because depth, ego-motion, and unconstrained appearance flow can jointly satisfy the constraint without physical consistency.The method therefore requires appropriate regularization to constrain solutions, including smoothness considerations around depth and appearance-flow discontinuities.
- Unified framework: The unified framework contains structure, motion, appearance, and correspondence modules to estimate depth and ego-motion, calibrate brightness, register frames, and filter invisible pixels.The correspondence module uses registration to emphasize irradiance variation and produces a visibility mask for occluded or out-of-view points.
- Unified framework: An auxiliary optical-flow-based loss regularizes the appearance-flow network, with optical flow improving appearance-flow prediction during reconstruction.The reconstructed image is obtained using optical flow and a spatial transformer.
4. Experiment
Experiments evaluate the framework across multiple endoscopic datasets, depth and ego-motion tasks, and cross-dataset settings. The results show improved depth estimation, ablation gains from the proposed components, and strong generalization without fine-tuning.
- Datasets and protocols: Experiments use SCARED, EndoSLAM, SERV-CT, and Hamlyn datasets for depth, ego-motion, and cross-dataset evaluation.SCARED provides the main depth experiments; EndoSLAM supports ego-motion evaluation, while SERV-CT and Hamlyn assess generalization.
- Evaluation design: The evaluation includes extensive SCARED depth experiments, EndoSLAM qualitative depth results, and cross-dataset testing using SCARED-trained models.Models trained on SCARED are tested on SERV-CT and Hamlyn without fine-tuning, while ego-motion is evaluated on SCARED and EndoSLAM.
- Depth results: The framework significantly outperforms compared methods on SCARED depth metrics, especially Sq Rel under severe brightness fluctuations.Sq Rel is sensitive to large depth errors, and the paper attributes the improvement to coping with illumination variation, non-Lambertian reflection, and interreflection.
- Cross-dataset generalization: Models trained on SCARED achieve superior SERV-CT and Hamlyn results without fine-tuning, indicating generalization across patients and cameras.Only the test-frame resolution is adjusted before inference.
- Ablation study: Adding the correspondence module improves evaluation performance, while combining appearance flow and regularization produces the best ablation results.The ablations identify appearance flow with appropriate regularization as key to improving depth estimation accuracy.
- Ego-motion results: The method achieves lower ATE than several self-supervised methods on SCARED and exceeds Endo-SfM and SC-SfMLearner in most EndoSLAM cases.On EndoSLAM, low ATE on organs excluded from training is reported as evidence of organ adaptability and generalization.
5. Conclusion
The framework addresses severe interframe brightness fluctuations in endoscopic scenes through appearance flow and a unified self-supervised approach. Experiments across multiple datasets demonstrate efficacy and generalization, while the method remains limited by oversaturation artifacts and monocular scale ambiguity.
- Contributions: Appearance flow relaxes the brightness constancy assumption by modeling severe interframe brightness fluctuations through a generalized dynamic image constraint.The constraint incorporates variations in brightness patterns in endoscopic scenes.
- Contributions: The unified framework targets robust monocular depth and ego-motion estimation in minimally invasive surgery environments.It is presented as having higher immunity against brightness variations than competing self-supervised methods.
- Experimental validation: Experiments on SCARED, EndoSLAM, SERV-CT and Hamlyn illustrate the method's efficacy and generalization ability.The cited conclusion reports evaluation across multiple datasets.
- Limitations and future work: Oversaturated regions produce slight depth artifacts because information is lost, while stronger smoothness can create oversmooth structural boundaries.The authors propose increasing the smoothness-loss weight as a possible mitigation but note its boundary-side effect.
- Limitations and future work: The method has intrinsic scale ambiguity when predicting depth maps and camera poses from monocular self-supervised learning.Future work considers object size or tracked endoscope displacement or velocity to recover absolute scale.