Source-linked AI summary
Deep Learning and Conditional Random Fields-based Depth Estimation and Topographical Reconstruction from Conventional Endoscopy
Faisal Mahmood, Nicholas J. Durr
TL;DR
Colonoscopy can miss clinically relevant lesions, and estimating mucosal topography is difficult with a small, deformable endoscopic scene. The paper develops a CNN-CRF that learns monocular depth from synthetic endoscopy data and uses the estimates to reconstruct colon-surface topography. It validates the approach on synthetic, phantom, porcine, and human endoscopy data, while identifying artifacts and ground-truth acquisition errors as limitations.
Problem
Colonoscopy can miss more than 20% of polyps that should be removed, while existing topography-estimation methods rely on geometric assumptions or prior information.
Method
A joint CNN-CRF learns unary and pairwise depth potentials from synthetic endoscopy images with ground-truth depth, then reconstructs mucosal topography from a single image.
Results
The method estimates depth on synthetic and real endoscopy data and uses the resulting depth maps to reconstruct colon-surface topography.
Takeaways & Limitations
The approach provides a foundation for integrating topographic information into computer-aided lesion detection, segmentation, and classification.
Takeaways & Limitations
The method can produce artifacts from specular reflections, non-dominant inverse-intensity cues, and pairwise similarities, while CT-based ground truth also contains registration and reconstruction errors.
Abstract
from arXiv · showhide
Colorectal cancer is the fourth leading cause of cancer deaths worldwide and the second leading cause in the United States. The risk of colorectal cancer can be mitigated by the identification and removal of premalignant lesions through optical colonoscopy. Unfortunately, conventional colonoscopy misses more than 20% of the polyps that should be removed, due in part to poor contrast of lesion topography. Imaging tissue topography during a colonoscopy is difficult because of the size constraints of the endoscope and the deforming mucosa. Most existing methods make geometric assumptions or incorporate a priori information, which limits accuracy and sensitivity. In this paper, we present a method that avoids these restrictions, using a joint deep convolutional neural network-conditional random field (CNN-CRF) framework. Estimated depth is used to reconstruct the topography of the surface of the colon from a single image. We train the unary and pairwise potential functions of a CRF in a CNN on synthetic data, generated by developing an endoscope camera model and rendering over 100,000 images of an anatomically-realistic colon. We validate our approach with real endoscopy images from a porcine colon, transferred to a synthetic-like domain, with ground truth from registered computed tomography measurements. The CNN-CRF approach estimates depths with a relative error of 0.152 for synthetic endoscopy images and 0.242 for real endoscopy images. We show that the estimated depth maps can be used for reconstructing the topography of the mucosa from conventional colonoscopy images. This approach can easily be integrated into existing endoscopy systems and provides a foundation for improving computer-aided detection algorithms for detection, segmentation and classification of lesions.
I. INTRODUCTION
Colonoscopy can miss clinically relevant lesions, while conventional computer-vision approaches face the colon’s monocular, deformable, dynamic setting. Measuring surface topography could improve lesion assessment and support computer-aided analysis without relying only on color and texture.
- Clinical motivation: More than 20% of clinically relevant polyps may be missed during colonoscopy, and non-polypoid lesions are especially difficult to screen.Missed lesions are associated with a substantial share of colorectal cancers detected after optical colonoscopy.
- Clinical motivation: Chromoendoscopy increases lesion contrast by encoding surface topography as color, but it doubles procedure time and requires specialized training.
- Computational opportunity: Computational colon-topography measurement could amplify lesion contrast, assist geometric classification, augment conventional images, and support lesion detection and examination-coverage metrics.
- Technical challenge: Colonoscopy challenges depth estimation because a monocular camera, close lighting, wide field of view, motion, limited working space, non-uniform texture, and deformable tissue violate conventional computer-vision assumptions.
- Technical challenge: Prior learning-based methods can require patient-specific texture, color, and shape, while earlier virtual-colonoscopy training data lacked realistic optical properties and real-endoscopy validation.
- Technical challenge: Graphical models are relevant because CNN-based pixel-wise approaches can lack spatial consistency, including smooth transitions between neighboring predictions.
B. Contributions and Significance
The paper addresses monocular endoscopy depth estimation with a CNN-CRF trained on realistic synthetic and virtual endoscopy data with ground-truth depth. Its datasets combine complementary topographic detail and validation sources spanning synthetic, phantom, porcine, and human endoscopy.
- Motivation: Patient-specific low-level texture limits generalization, motivating depth estimation from more transferable visual cues in endoscopic images.
- Method: The method formulates continuous monocular endoscopy depth estimation as a CRF learning problem implemented with a CNN.
- Validation: Validation spans digital synthetic-colon data, a silicone phantom, and porcine optical endoscopy registered with CT for ground-truth depth.
- Training data: Over 100,000 texture-free synthetic colon images provide ground-truth depth from an anatomically realistic virtual endoscope.
- Training data: CT-reconstructed silicone-colon phantom data adds realistic high-spatial-frequency topography that the texture-free synthetic colon lacks.
- Validation: Human endoscopy images were used for qualitative evaluation after graph-based in-painting partially removed specular reflections.
- Dataset design: The study excludes CTC training data because cleaning, resolution, post-processing, and non-polypoid-lesion miss-rate limitations reduce its practical suitability.
B. Deep Learning with Conditional Random Fields
The method estimates continuous depth from monocular endoscopy by combining CNN-based unary regression with CRF pairwise smoothing. The network learns these terms jointly and solves depth inference in closed form.
- B. Deep Learning with Conditional Random Fields: The continuous CRF models depth with unary superpixel regression and pairwise smoothness between similar neighboring superpixels.The unary term uses CNN parameters γ, while the pairwise term uses β.
- B. Deep Learning with Conditional Random Fields: The fully convolutional network produces spatial convolution maps that support superpixel-wise depth prediction.The architecture pools convolutional maps into superpixel feature vectors before the fully connected unary layers.
- B. Deep Learning with Conditional Random Fields: The CRF loss minimizes the negative log likelihood of the conditional depth distribution during training.The optimization uses stochastic-gradient backpropagation for the unary and pairwise parameters.
- B. Deep Learning with Conditional Random Fields: Neighboring-superpixel similarities use intensity differences and grayscale histograms as pairwise cues.These similarities are expressed in a general ℓ2 form.
- B. Deep Learning with Conditional Random Fields: Depth inference has a closed-form solution for the maximum a posteriori problem on a new endoscopic image.The formulation uses the graph Laplacian and related matrix notation to simplify optimization.
C. Adversarial Training for Domain Adaptation
The domain-adaptation process transforms real endoscopy images into a synthetic-like representation before depth estimation. Porcine optical views are registered with CT-based virtual endoscopy to obtain ground-truth depth.
- C. Adversarial Training for Domain Adaptation: Adversarial training bridges synthetic and real image domains by transforming real endoscopy images into synthetic-like representations.The setup trains a discriminator and transformer network, with an ℓ1 self-regularization term.
- C. Adversarial Training for Domain Adaptation: Porcine colon validation registers optical endoscopy views with CT-reconstructed virtual endoscopy views to derive ground-truth depth.The CT reconstruction used 720 projections, and only registered regions were compared.
A. Experimental Setup
The experiments use superpixelized virtual endoscopy data and separate training, validation, and test partitions. Training proceeds through synthetic pretraining and colon-phantom fine-tuning.
- A. Experimental Setup: The implementation oversegments each virtual endoscopy image into SLIC superpixels and assigns corresponding ground-truth depth to each superpixel.The networks were implemented with VLFeat Mat-ConvNet in MATLAB 2017a and CUDA 8.0.
- A. Experimental Setup: Table II reports performance evaluation for synthetic Blender-generated endoscopy data.The supplied table passage identifies the dataset evaluated but does not provide its numerical entries.
- A. Experimental Setup: The data split uses 55% for training, 40% for validation, and 5% for testing.The network was pretrained on synthetic colon data and fine-tuned on colon phantom data.
B. Quantitative Evaluation Metrics
The evaluation uses three standard monocular-depth metrics: relative error, root mean square error, and average log10 error.
- B. Quantitative Evaluation Metrics: Relative Error is one of the metrics used to evaluate monocular depth estimation.
- B. Quantitative Evaluation Metrics: Root Mean Square Error is one of the metrics used to evaluate monocular depth estimation.
- B. Quantitative Evaluation Metrics: Average log10 Error is one of the metrics used to evaluate monocular depth estimation.
C. Comparative Analysis
The paper compares CNN-CRF depth estimation with a non-CRF FCN regression model, while noting that direct comparison with prior endoscopy methods is not possible because datasets and evaluation methods differ.
- Direct comparison with existing endoscopy depth-estimation work is not possible because the datasets and evaluation methods differ.
- Qualitative comparisons include predicted depth and topographical reconstructions for monocular endoscopy images.
- Performance evaluation is reported separately for colon-phantom virtual endoscopy data and real porcine endoscopic data.
- The CNN-CRF method is compared with an FCN regression model that replaces the CRF loss layer with least-squares regression.This comparison is intended to assess the benefit of incorporating a graphical model.
D. Results with Synthetic Colon and Phantom Virtual Endoscopy Data
On held-out synthetic-colon and silicone-phantom images, the CNN-CRF network improved accuracy across every reported metric for images resembling its training data.
- The CNN-CRF method improved accuracy on every reported metric for 10,000 randomly selected test images similar to the training data.Testing used synthetic-colon and silicone-colon-phantom images excluded from training.
E. Results with Porcine Colon Real Endoscopy Data
Real-data evaluation used registered porcine-colon endoscopy and CT views to obtain ground-truth depth, followed by comparison with an assumption-based monocular reconstruction method.
- 1,460 porcine-colon images were evaluated using depth maps registered between optical endoscopy and CT reconstruction.Only registered regions were compared after applying the same filtering pipeline to both depth maps.
- The proposed reconstruction was compared with a tubular-assumption-based 3D monocular reconstruction method using the same input image and reconstruction from that prior work.
F. Qualitative Results with Real Human Endoscopy Data
On real human colonoscopy images, the network produced coarse depth maps that matched intuitive image cues and supported surface-topography reconstruction, while the paper identifies several artifacts and testing-error sources.
- Qualitative results: The network regressed coarse depth maps from NIH and MICCAI colonoscopy images that matched intuitive cues in the images.
- Qualitative results: The estimated depth maps were used to reconstruct the topography of the colon surface and compared with an assumption-based monocular reconstruction.
- Conclusions: The CNN-CRF architecture was trained on synthetic data and validated on real porcine colon tissue with CT-registered optical endoscopy.The authors describe the approach as potentially useful for lesion detection, segmentation, and classification algorithms.
- Limitations: The method can produce artifacts from specular reflections, non-dominant inverse-intensity cues, and pairwise similarities.
- Limitations: Testing errors also arose from CT reconstruction, filtering, streaking around metallic fiducials, scaffold rotation, and virtual-to-optical endoscopy registration.
- Future work: Future work targets broader synthetic medical-image generation and using depth estimation as an additional cue for other endoscopy applications.