Source-linked AI summary
Stereo Correspondence and Reconstruction of Endoscopic Data Challenge
Max Allan, Jonathan Mcleod, Congcong Wang, Jean Claude Rosenthal, Zhenglei Hu, Niklas Gard, Peter Eisert, Ke Xue Fu, Trevor Zeffiro, Wenyao Xia, Zhanshi Zhu, Huoling Luo, Fucang Jia, Xiran Zhang, Xiaohong Li, Lalith Sharan, Tom Kurmann, Sebastian Schmid, Raphael Sznitman, Dimitris Psychogyios, Mahdi Azizian, Danail Stoyanov, Lena Maier-Hein, Stefanie Speidel
TL;DR
Dense depth estimation for endoscopic scenes is constrained by limited high-quality surgical datasets and difficult ground-truth acquisition. This paper presents the SCARED challenge dataset, evaluates stereo-reconstruction submissions, and reports dataset and pipeline issues; the overall winner is selected by the lowest mean error across two test datasets.
Problem
Accurate surgical-image depth estimation depends on high-quality datasets, but few realistic surgical datasets exist because ground-truth acquisition is difficult.
Method
The paper presents structured-light stereo data from porcine cadavers, evaluates submitted reconstruction methods on two test datasets, and documents dataset-related issues and additional methods.
Results
The overall challenge winner is the method with the lowest mean error across the two test datasets, submitted by Trevor Zeffiro of Rediminds Inc.
Takeaways & Limitations
SCARED provides a high-quality surgical dataset for developing and evaluating endoscopic depth-estimation methods.
Abstract
from arXiv · showhide
The stereo correspondence and reconstruction of endoscopic data sub-challenge was organized during the Endovis challenge at MICCAI 2019 in Shenzhen, China. The task was to perform dense depth estimation using 7 training datasets and 2 test sets of structured light data captured using porcine cadavers. These were provided by a team at Intuitive Surgical. 10 teams participated in the challenge day. This paper contains 3 additional methods which were submitted after the challenge finished as well as a supplemental section from these teams on issues they found with the dataset.
I. INTRODUCTION
Surgical-scene reconstruction supports computer-assisted surgery applications, but accurate depth estimation depends on suitable data and remains challenging at camera frame rates. SCARED addresses limited surgical datasets by providing structured-light data from porcine cadavers.
- Surgical-scene reconstruction supports SLAM for augmented reality, perception in automation and safety systems, and diagnostic tools.
- Stereo depth estimates match pixels between calibrated camera views and triangulate them, using either classical methods or deep learning.These approaches require no scene modification and can use a simple recording system attached to the endoscope.
- Real-time reconstruction at the camera frame rate is challenging and requires specialized processing hardware.
- Few surgical depth datasets exist because obtaining ground-truth data is difficult; existing datasets are often small or use phantoms or unrealistic environments.
- SCARED addresses this gap with a high-quality dataset built from porcine cadavers and structured-light projection.
II. DATA
SCARED provides structured-light stereo data collected with a da Vinci Xi across seven training and two test datasets. The challenge evaluates mean absolute depth error, while submitted pipelines address stereo geometry, depth completion, and dataset limitations.
- II. DATA: SCARED contains 7 training datasets and 2 test datasets captured using a da Vinci Xi surgical robot.Each dataset represents one porcine subject and contains 4–5 keyframes.
- II. DATA: Keyframes capture unique views with projected structured-light patterns and dense stereo reconstructions.Robot pose measurements and scene-static assumptions support approximate depth maps for intermediate frames.
- II. DATA: Depth maps are created by recording a white-light stereo image, projecting 10-bit Gray-code patterns, and reconstructing the scene while it remains still.The projector provides a 1280 × 720 pattern and has a 15 × 3 × 9 cm form factor.
- A. Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute: The Fraunhofer pipeline analyzes stereo geometry, rectifies images, removes depth outliers, and completes missing depths with a neural surface-optimization process.Its homography is initialized over 15–25 frames and then treated as static for the sequence.
B. University of Western Ontario
The University of Western Ontario submission combines multi-scale cost-volume filtering with post-processing and a learned correction for stereo rectification errors.
- B. University of Western Ontario: The method uses multi-scale cost-volume filtering with mean-squared-error loss and a guided median filter for post-processing.
- B. University of Western Ontario: A regression model is trained to correct rectification errors.
- B. University of Western Ontario: The Norwegian University of Science and Technology submission estimates disparity variationally at the coarsest multiscale-pyramid level and upsamples it with a modified bilateral filter.
D. Rediminds Inc.
The Rediminds submission trains PSMNet with Gaussian-based interpolation to fill holes in training data, while other submissions use unsupervised or multi-scale disparity-learning strategies.
- D. Rediminds Inc.: Rediminds trains a Pyramid Stereo Matching network using Gaussian-based interpolation to remove holes in the training data.
- II. DATA: The dataset collection uses a projector to cast Gray-code patterns into the endoscope’s view for depth-map creation.
- The unaffiliated submission trains a DNN without supervision to predict disparity and reprojects it using stereo calibration parameters.
- The Harbin Institute of Technology submission uses a six-level UNet with multi-scale losses for appearance matching, disparity smoothness, and left-right consistency.
- The Shenzhen Institute submission predicts disparity directly with a modified VGG encoder, skip connections, and a dual-branch decoder, followed by neighborhood smoothing.
H. Johns Hopkins University
Three submitted methods applied distinct deep-learning approaches to endoscopic depth estimation: PSMNet with filtered and reconstructed training data, CycleGAN-based synthesis, and UNet regression with smoothness regularization.
- Johns Hopkins University: PSMNet training was improved by filtering inaccurate pixels with SGBM and reconstructing training images using ICP-generated point clouds.An L1 loss regularized the results.
- Shenzhen Institute for Advanced Technology: CycleGAN synthesized a depth map directly from a rectified stereo pair through a two-stage process.
- Heidelberg University: UNet was trained with mean absolute error loss and smoothness regularization for depth estimation.
IV. POST-CHALLENGE METHODS
Three post-challenge methods explored alternative stereo architectures and dataset handling, including pretrained matching, hierarchical matching, and quantized cost-volume optimization.
- Scope: The three methods were submitted after the challenge and therefore were excluded from the final challenge ranking.
- UCL: Deep Pruner: Deep Pruner used Sceneflow pretraining, manual ground-truth outlier removal, stereo rectification, and augmentation, while excluding data affected by high calibration errors.The method ignored interpolation sequences and datasets 4 and 5, leaving 25 samples for allocation.
- UCL: HSM: HSM followed the same processing pipeline as Deep Pruner but used a batch size of 2 and 350 training epochs.
- University of Bern: A University of Bern method used parallel left-right disparity prediction with photometric losses and a quantized 3D cost volume.
- University of Bern: 40% fewer disparities reduced inference time by 40% to 79.1ms without loss of accuracy.Quantization also reduced memory use, enabling larger batch sizes or higher training resolution.
V. RESULTS
Evaluation used two test datasets with five keyframes each and measured per-pixel mean absolute depth error in millimeters, masking pixels without ground truth during interpolation.
- Evaluation protocol: Two test datasets each contained five keyframes, with interpolation sequences between keyframes 1–4 and a single image at keyframe 5.
- Evaluation protocol: The metric was mean absolute error in mm for depth measurements at each pixel.
- Evaluation protocol: Pixels without associated ground truth were masked during interpolation, and frames with ground truth in less than 10% of pixels were discarded.
- Results: Overall errors for test dataset 1 were reported in Fig. II, with per-frame motion errors shown in Figs. 3 and 4.
- Results: Overall errors for test dataset 2 were reported in Fig. III, with per-frame motion errors shown in Figs. 5 and 6.
C. Overall
The challenge winner was determined by the lowest mean error across both test datasets, while supplemental work addressed errors arising from forward-kinematics-based point-cloud projection.
- C. Overall: Trevor Zeffiro of Rediminds Inc. won the challenge by achieving the lowest mean error across the two test datasets.Jean-Claude Rosenthal received second place.
- VI. Supplemental material on dataset: The original dataset projected point clouds using forward kinematics, making their accuracy dependent on robot positioning, noise, and video synchronization.
- VI. Supplemental material on dataset: The proposed correction pipeline detected and matched SIFT keypoints, estimated endoscope pose with a perspective-n-point formulation, and reprojected the point cloud into the image.
1) Calibration Errors:
The datasets contain calibration inaccuracies and timing offsets that affect stereo rectification and alignment between video and interpolated depth data.
- Calibration Errors:: Calibration files for datasets 4 and 5 produce large rectification errors, with matched features failing to lie on common scanlines.Fundamental-matrix-only rectification was adequate, suggesting the main error lies in the camera intrinsics.
- Calibration Errors:: Video frames and interpolated ground-truth depth images are not time synchronized, with the video lagging the depth sequences.
3) Ground Truth-RGB misalignment:
The paper reports ground-truth/RGB misalignment and presents error visualizations and aggregate depth-error tables across both test datasets.
- Ground Truth-RGB misalignment:: Ground-truth and RGB data are misaligned in datasets 8 and 9.
- Ground Truth-RGB misalignment:: Table II reports mean absolute depth error in millimeters for test dataset 1 and identifies the best challenge-period method in bold.
- Ground Truth-RGB misalignment:: Figures 3 and 4 plot per-frame errors for test dataset 1 across keyframes 1–4, using mean absolute or mean per-pixel error descriptions.
- Ground Truth-RGB misalignment:: Table III reports mean absolute depth error in millimeters for test dataset 2 and identifies the best challenge-period method in bold.
- Ground Truth-RGB misalignment:: Figures 5 and 6 plot mean per-pixel errors for test dataset 2 across keyframes 1–4.
- Ground Truth-RGB misalignment:: Figure 7 contrasts provided-calibration rectification, estimated-extrinsic rectification, and visual-match-based uncalibrated rectification for dataset 4.
- Ground Truth-RGB misalignment:: Figure 8 shows a dataset 9 ground-truth depth map shifted relative to RGB, with corresponding disparity differences also visible in a network output.