Source-linked AI summary
Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration
Taylor L. Bobrow, Mayank Golhar, Rohan Vijayan, Venkata S. Akshintala, Juan R. Garcia, Nicholas J. Durr
TL;DR
Quantitative evaluation of colonoscopy computer vision remains difficult because real videos lack ground-truth annotations. This paper introduces C3VD and a video-based 2D-3D registration method to pair clinical-colonoscope footage with 3D ground truth. Temporal information improves registration accuracy over single-frame registration, and the dataset supports evaluation of reconstruction components and full methods.
Problem
Colonoscopy computer-vision methods are difficult to evaluate quantitatively because ground-truth annotated data are lacking.
Source: §1 · p. 1Method
The paper registers clinical-colonoscope video sequences to known 3D models by comparing depth-transformed keyframes with rendered views using edge alignment.
Source: §5 · pp. 9–10Results
Video-based registration improves accuracy over single-frame registration, and the proposed edge-based loss outperforms NCC and DICE while costing less computation.
Source: §5 · pp. 9–10Takeaways & Limitations
C3VD supports evaluation of pixel-level reconstruction components and full SLAM methods, including techniques for identifying missed regions.
Source: §5 · p. 10Registration requires precise pose measurements from a robotic-arm setup, limiting scope range and motion; the phantom models are static unlike flexible, dynamic colonic tissue.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a Colonoscopy 3D Video Dataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at durr.jhu.edu/C3VD.
1. Introduction
Colonoscopy computer-vision research lacks ground-truth data for quantitative benchmarking, while existing endoscopy datasets often trade imaging realism against label availability. C3VD addresses this gap with registered clinical-colonoscope videos and paired 3D and pixel-level ground truth.
1. Introduction
- Missed lesions remain a screening concern: visible-polyp detectors require lesions to enter the field of view, and missed regions were estimated at 10% of colon surface.The introduction also reports that an estimated 22% of precancerous lesions go undetected during screening.Source: §1 · p. 1
1.1.1. Endoscopy reconstruction datasets
- Existing reconstruction datasets span rendered, model-based, and animal-tissue data, but synthetic rendering has limits in simulating real optics and illumination.Real-image datasets also have limited suitability for colonoscopy, and one dataset with a clinical colonoscope lacks pixel-level ground truth.Source: §1.1.1 · p. 2
1.1.2. Registering real and virtual endoscopy images
- Registering endoscopic frames to a known surface model enables derivative ground truth, but conventional feature matching is challenged by weak corners and specular reflections.Prior approaches include fiducials, manual virtual-camera alignment, and calibrated camera-scanner setups.Source: §1.1.2 · p. 2
1.2. Contributions
- The paper presents C3VD as an HD clinical-colonoscope video dataset with 10,015 frames paired with pixel-level labels and video-level 3D ground truth.Labels include depth, surface normals, occlusion, optical flow, and six-degree-of-freedom pose; videos also have surface models and coverage maps.Source: §1.2 · p. 3
- The dataset contribution includes a 2D-3D video-registration algorithm that compares GAN-estimated depth with rendered views along a measured camera trajectory using an edge-based loss.The dataset, colon model, phantom molds, and registration algorithm are publicly available.Source: §1.2 · p. 3
2. Methods
The method combines phantom production, video acquisition with a known trajectory, and registration to a ground-truth 3D surface model.
2. Methods
- The protocol first produces high-fidelity phantom models.These models provide the recording target for the dataset.Source: §2 · p. 3
- Video sequences are recorded with a ground-truth trajectory.The method description places video and trajectory recording before registration.Source: §2 · p. 3
- The acquired video and trajectory sequences are registered with a ground-truth 3D surface model.This registration produces paired, pixel-level ground truth.Source: §2 · p. 3
2.1. Phantom model production
The phantom models are sculpted and molded to provide colon anatomy and varied surface appearance for video acquisition.
2.1. Phantom model production
- A complete colon model from sigmoid colon to cecum was digitally sculpted using anatomical imagery as reference.The authors report higher-frequency detail than in models derived from resolution-limited CT colonography.Source: §2.1 · p. 3
- Five colon segments were divided into three-part molds and cast in silicone.The molds were 3D printed at 16 micrometer resolution.Source: §2.1 · p. 3
- Silicone was manually applied in 5–12 layers with varying opacity to emulate patient-specific tissue features and vasculature at varying optical depths; surface lubricant simulated mucosal specularity at recording time.Source: §2.1 · p. 4
2.2. Data acquisition
A clinical colonoscope was mounted to a robotic arm to record HD video and pose while traversing static silicone colon models.
2.2. Data acquisition
- The Olympus colonoscope was rigidly mounted to a robotic arm, and the phantom models were held static during recording.Video was captured in uncompressed HD format.Source: §2.2 · p. 4
- For each segment, 4–8 programmed poses were connected by robotic-arm interpolation through the colon.The arm had 10 micrometer repeatability.Source: §2.2 · p. 4
- The robotic arm pose log was recorded at 63 Hz alongside video from the clinical colonoscope.The video processor output was captured using an HD frame grabber.Source: §2.2 · p. 4
2.3. Registration pipeline overview
The pipeline aligns colonoscopy imagery with rendered views of a 3D model by converting sampled video frames to depth and optimizing a shared model transform.
2.3. Registration pipeline overview
- The pipeline samples model-transform predictions, compares target-depth frames with rendered views along measured camera poses, and updates the transform with an evolutionary optimizer.The unknown phantom pose is represented as a rigid transform with rotation and translation components.Source: §2.3 · p. 4
2.3.1. Data preprocessing
- Preprocessing samples keyframes, synchronizes video and pose by maximizing optical-flow and pose-displacement correlation, converts arm poses through hand-eye calibration, and manually aligns the initial transform.The calibration relates robotic-arm poses to colonoscope camera poses.Source: §2.3.1 · p. 4
2.3.2. Target depth estimation
- A cGAN predicts target depth for each keyframe from synthetic training pairs, with three BSDFs used for domain randomization and a model plus fourth BSDF reserved for validation.The training set contains 1,000 synthetic colonoscopy image-depth pairs.Source: §2.3.2 · pp. 4–5
2.3.3. Rendering depth frames
- A spherical intrinsic model represents the colonoscope’s approximately 170° field of view; rendered depth frames at camera poses are compared with target depths.The camera model shifts pixels relative to the optical center, accounts for lens misalignment, and relates distorted coordinates to ray directions.Source: §2.3.3 · p. 5
2.3.4. Edge loss & optimization
- To reduce scale inconsistency in learned depth, the method blurs Canny contours and minimizes 1 minus their averaged overlap across frame pairs with CMA-ES.The similarity score ranges from 0 to 1 and is maximal when rendered and target edges are perfectly aligned.Source: §2.3.4 · pp. 5–6
2.4. Simulated screening colonoscopies
The simulated screening recordings provide video with coarse pose and 3D surface models, including sequences acquired by a gastroenterologist.
2.4. Simulated screening colonoscopies
- Four simulated screening colonoscopy sequences were recorded with coarse six-degree-of-freedom pose and 3D surface models.A trained gastroenterologist withdrew the scope from the cecum while video and pose were recorded.Source: §2.4 · p. 6
- The ground-truth colon model cross-section depicts a sculpted model whose reference-image-based detail is higher resolution than CT colonography volumes.Source: Figure 5 · p. 6
- The scope’s electromagnetic sensor recorded pose at 40 Hz, and temporally neighboring poses were linearly interpolated when tracking failed.The sensor was rigidly affixed to the scope tip.Source: §2.4 · p. 6
3. Experiments and results
The section describes registration implementation, quantifies accuracy on synthetic sequences, and assesses alignment on real recorded videos.
3.1. Implementation
- The implementation generated depth targets with a trained network, reused extracted target edges, and ran CMA-ES with 100-sample populations; evaluations averaged 4.0 milliseconds per keyframe.Rotation and translation bounds were scaled to a uniform sampling space.Source: §3.1 · p. 7
3.2. Evaluation with synthetic data
- At K=5 keyframes, registration reached 0.321-millimeter translational and 0.159-degree rotational accuracy, improving 55.6% and 60.4%, respectively, over single-frame registration.K=5 was used in remaining experiments to balance accuracy with computation time.Source: Figure 6 · p. 7, §3.2 · p. 7
- Depth-domain registration was an order of magnitude more accurate on average than color-domain registration.The figure also shows registration error decreasing as the number of keyframes increases.Source: Figure 6 · p. 7, §3.2 · p. 9
- Rotational errors were 0.117, 0.063, and 0.070 degrees for simple, moderate, and complex trajectories; translational errors were 0.150, 0.070, and 0.089 millimeters, respectively.Source: §3.2 · p. 9
3.3. Evaluation with real video sequences
- On real recorded videos, the pipeline aligned sample frames with target frames throughout the video, assessed qualitatively because the ground-truth model transform was unknown.Source: Figure 7 · p. 8, §3.3 · p. 9
4. Dataset and distribution format
The dataset pairs registered video frames with pixel-level ground truth and provides sequence-level geometry and coverage information, alongside simulated screening videos.
4. Dataset and distribution format
- The 22 registered sequences contribute 10,015 frames with paired depth, surface normals, occlusion, and optical flow ground truth.Source: §4 · p. 9
- Each sequence also includes camera pose, a 3D mesh, and a coverage map marking mesh faces as observed or unobserved.Source: §4 · p. 9
- Four simulated screening sequences add 20,058 frames with ground-truth camera poses in homogeneous form.Source: §4 · p. 9
5. Discussion and conclusion
The discussion presents sequence-based 2D-3D registration as more accurate than single-frame alignment and uses it to build C3VD, a real-colonoscope dataset with registered ground truth. It describes benchmarking uses and notes limitations in robotic acquisition and dataset diversity.
5. Discussion and conclusion
- Video-based registration improves accuracy over single-frame registration; edge alignment further improves accuracy, and its loss outperforms NCC and DICE at lower computational cost.Depth-domain comparison addresses errors caused by specular reflections and surface textures.Source: §5 · pp. 9–10
- Moderate and complex trajectories outperform simple ones; with K=5 keyframes fixed, motion and edge-feature diversity are possible explanations.Simple trajectories mainly translate along the colon axis, while moderate and complex trajectories also include lateral translation, pitch, and yaw variation.Source: §5 · p. 10
- C3VD pairs real HD clinical-colonoscope videos with registered ground truth, models tubular phantom lumens, and covers the full angular field of view.The dataset uses an omnidirectional camera model and includes depth, surface normal, occlusion, and optical-flow labels.Source: §5 · p. 10
- C3VD supports evaluation of SLAM components, full reconstruction methods, missed-region detection, and algorithms on realistic-motion screening sequences.Its 3D assets can also support synthetic training data, and its mold files and fabrication protocols enable phantom model production and modification.Source: §5 · p. 10
- Robotic-arm pose measurement limits scope range and motion, while static phantoms do not reproduce flexible, dynamic tissue or frequent scope contact.Small hand-eye calibration errors can also accumulate into camera-trajectory drift.Source: §5 · p. 10
- C3VD remains limited in diversity and scope, with coverage metrics needing more trajectories, colon shapes, and varied missed regions.The discussion also identifies loose stool and debris as challenges and proposes broader imaging and tissue conditions, including alternative endoscopes, NBI, and chromoendoscopy.Source: §5 · pp. 10–11