Source-linked AI summary
Self-Supervised Siamese Learning on Stereo Image Pairs for Depth Estimation in Robotic Surgery
Menglong Ye, Edward Johns, Ankur Handa, Lin Zhang, Philip Pratt, Guang-Zhong Yang
TL;DR
In vivo surgical data may lack known depth labels, while depth estimation is needed for surgical scenes. This paper trains an autoencoder with stereo-pair reconstruction and extends it to a Siamese architecture; on in vivo videos, the Siamese model achieved the best mean SSI among the compared approaches. The label-free training is presented as applicable to large-scale in vivo video processing where depth labels are unavailable.
Problem
Known depth labels were unavailable for the in vivo surgical data, motivating depth estimation that can be trained without them.
Source: RESULTS · p. 2Method
An autoencoder predicts per-pixel disparity, while a spatial transformer reconstructs an image from its stereo counterpart; a Siamese extension shares weights across both views and adds disparity consistency.
Source: MATERIALS AND METHODS · pp. 1–2Results
The Siamese architecture achieved the best mean SSI, and both deep learning approaches outperformed ELAS and SPS on the test set.
Source: RESULTS · p. 2Takeaways & Limitations
The authors state that training without known depth labels supports applicability to large-scale in vivo video processing where such labels are unavailable.
Source: DISCUSSION · p. 2
Abstract
from arXiv · showhide
Robotic surgery has become a powerful tool for performing minimally invasive procedures, providing advantages in dexterity, precision, and 3D vision, over traditional surgery. One popular robotic system is the da Vinci surgical platform, which allows preoperative information to be incorporated into live procedures using Augmented Reality (AR). Scene depth estimation is a prerequisite for AR, as accurate registration requires 3D correspondences between preoperative and intraoperative organ models. In the past decade, there has been much progress on depth estimation for surgical scenes, such as using monocular or binocular laparoscopes [1,2]. More recently, advances in deep learning have enabled depth estimation via Convolutional Neural Networks (CNNs) [3], but training requires a large image dataset with ground truth depths. Inspired by [4], we propose a deep learning framework for surgical scene depth estimation using self-supervision for scalable data acquisition. Our framework consists of an autoencoder for depth prediction, and a differentiable spatial transformer for training the autoencoder on stereo image pairs without ground truth depths. Validation was conducted on stereo videos collected in robotic partial nephrectomy.
MATERIALS AND METHODS
The framework predicts per-pixel disparity from an RGB image and uses a spatial transformer to reconstruct its stereo counterpart for self-supervised training. A Siamese extension estimates disparity from both images with shared weights and adds left-right consistency to the loss.
MATERIALS AND METHODS
- The autoencoder predicts disparity from one stereo image, and a spatial transformer uses its counterpart to reconstruct that image for error-based training.The framework uses stereo pairs and camera parameters, without requiring ground-truth depth labels in the training procedure described here.Source: MATERIALS AND METHODS · p. 1
- The basic network uses a modified DeConvNet autoencoder and an L1 reconstruction-error loss.Fully connected layers are removed to reduce parameters; batch normalization, rectified linear units, and a final convolutional layer produce the disparity map.Source: MATERIALS AND METHODS · p. 1
- The Siamese network predicts disparity from both stereo images using autoencoders with shared weights, and includes left-right disparity consistency in its loss.The stated loss weights are 0.5 for each image loss and 1.0 for the consistency loss.Source: MATERIALS AND METHODS · p. 2
RESULTS
On held-out in vivo surgical video, the Siamese architecture achieved the best mean SSI among the compared approaches. The evaluation used rectified stereo pairs without ground-truth depth labels.
RESULTS
- The Siamese architecture achieved the best mean SSI, and both deep learning approaches outperformed ELAS and SPS on the test set.The evaluation compares image reconstruction quality using SSI because ground-truth depth labels were unavailable for the in vivo surgical data.Source: RESULTS · p. 2
- Testing used a separate video sequence containing 3,000 stereo pairs, with SSI calculated for reconstructions from both camera views.The training dataset contained 20,000 rectified stereo pairs randomly sampled across 11 in vivo video sequences from robotic partial nephrectomy.Source: RESULTS · p. 2
- At 192×96-pixel resolution, depth estimation took approximately 7 ms per image after training.Training lasted 40 epochs and took approximately 8 hours for the basic architecture and 18 hours for the Siamese architecture.Source: RESULTS · p. 2
DISCUSSION
The paper presents self-supervised depth-estimation frameworks for surgical images, extending a basic network to a Siamese network that improves generalisability. The authors report accurate depth estimation and performance above standard stereo-matching approaches, with training that does not require known depth labels.
DISCUSSION
- The basic depth-estimation network combines an autoencoder with a spatial transformer, and the paper extends it to a Siamese network.The Siamese architecture estimates depth from both images in a stereo pair.Source: DISCUSSION · p. 2
- Figure 2 presents example results from the basic and Siamese architectures, with higher image intensity indicating closer distance to the cameras.Table 2 reports SSI measures of image-reconstruction quality; higher mean values indicate better performance.Source: Figure 2 · p. 2, Table 2 · p. 2
- The Siamese extension improves model generalisability, and the reported approach outperforms standard stereo-matching approaches on in vivo robotic-surgery videos.The framework does not require known depth labels during training, supporting its stated applicability to large-scale in vivo video processing where such labels are unavailable.Source: DISCUSSION · p. 2