Source-linked AI summary
A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images
David Vázquez, Jorge Bernal, F. Javier Sánchez, Gloria Fernández-Esparrach, Antonio M. López, Adriana Romero, Michal Drozdzal, Aaron Courville
TL;DR
Colonoscopy screening can miss polyps, and visual malignancy assessment requires histological analysis, motivating better decision support. The paper introduces an extended, annotated benchmark and FCN baselines for endoluminal scene segmentation, which outperform prior results without post-processing and have potential for nearly real-time DSS use.
Problem
Colonoscopy is limited by missed polyps and inability to assess polyp malignancy visually during the procedure, while few studies address endoluminal scene segmentation.
Method
The paper combines two large colonoscopy datasets, adds annotations for lumen and specular highlights, and trains fully convolutional networks as segmentation baselines.
Results
The FCN baselines outperform previously published endoluminal scene-segmentation results by a large margin without further post-processing.
Takeaways & Limitations
The benchmark, standard splits, metrics, code, and data support further research toward more accurate colonoscopy decision support systems.
Takeaways & Limitations
The dataset lacks non-polyp frames, so polyp detection is reformulated as polyp localization.
Abstract
from arXiv · showhide
Colorectal cancer (CRC) is the third cause of cancer death worldwide. Currently, the standard approach to reduce CRC-related mortality is to perform regular screening in search for polyps and colonoscopy is the screening tool of choice. The main limitations of this screening procedure are polyp miss-rate and inability to perform visual assessment of polyp malignancy. These drawbacks can be reduced by designing Decision Support Systems (DSS) aiming to help clinicians in the different stages of the procedure by providing endoluminal scene segmentation. Thus, in this paper, we introduce an extended benchmark of colonoscopy image, with the hope of establishing a new strong benchmark for colonoscopy image analysis research. We provide new baselines on this dataset by training standard fully convolutional networks (FCN) for semantic segmentation and significantly outperforming, without any further post-processing, prior results in endoluminal scene segmentation.
1 Introduction
Colonoscopy screening is limited by missed polyps and delayed malignancy assessment, motivating decision support systems for endoluminal scene analysis. The paper addresses limited colonoscopy segmentation research by introducing an extended benchmark and FCN baselines.
- Colonoscopy screening can miss small, flat, or fold-hidden polyps, while malignancy is determined only after histological analysis.
- Decision support systems could assist clinicians by detecting, segmenting, and assessing polyp malignancy during colonoscopy.
- Few existing optical-colonoscopy studies address endoluminal scene segmentation, despite broader progress of CNNs in medical-image segmentation.
- Endoluminal scene segmentation supports navigation, highlights potential lesion areas, and can use specular highlights to reduce false-positive polyp detection.
- The paper contributes an extended colonoscopy-image benchmark and new FCN baselines that outperform prior endoluminal scene-segmentation results without post-processing.
2 Endoluminal scene segmentation benchmark
The benchmark combines existing colonoscopy datasets, extends their annotations to additional scene classes, and defines patient-separated splits and segmentation metrics. It also supports polyp localization under an overlap-based criterion.
- Dataset: EndoScene combines CVC-ColonDB and CVC-ClinicDB into 912 images from 44 video sequences acquired from 36 patients.
- Dataset: The annotations cover polyps, mucosa background, lumen, specular highlights, and void regions corresponding to black frame borders.
- Dataset: The dataset is split into training, validation, and test sets containing 60%, 20%, and 20% of images, with patients kept in a single set.
- Metrics: Segmentation is evaluated with Intersection over Union and per-pixel accuracy, including mean per-class IoU across each set.
- Polyp localization: A polyp is considered localized when its IoU with the associated ground-truth mask exceeds 0.5, averaged independently across polyps and then per set.
3 Baseline
The baseline uses a fully convolutional network for per-pixel colonoscopy scene prediction. Its design restores spatial resolution with upsampling and skip connections, while training includes augmentation and imbalance handling.
- FCNs extend CNNs to semantic segmentation by producing per-pixel predictions with upsampling layers that recover input spatial resolution.
- Skip connections link downsampling and upsampling paths to recover fine-grained information lost through pooling.
- The implementation uses FCN8 trained with stochastic gradient descent and RMSProp, with validation-based early stopping and dropout regularization.
- Training uses random crops and experiments with rotations, zooming, shearing, and elastic transformations for data augmentation.
- Void pixels are excluded from loss and metric calculations, while median-frequency balancing is applied in some experiments to address class imbalance.
4 Experimental results
The experiments assess augmentation, class formulation, and FCN8 against prior methods for endoluminal segmentation and polyp localization. FCN8 improves overall segmentation and localization, although specular-highlight segmentation remains weaker than traditional methods.
- 4.1.1 Influence of data augmentation: Warping, rotation, and zoom improved polyp segmentation, while shearing could produce unrealistic polyp deformations.Warping applies small local elastic deformations intended to represent realistic shape variation.
- 4.1.1 Influence of data augmentation: Combining all augmentation techniques increased mean IoU by 4.51% and global mean accuracy by 1.52%.The evaluation was performed on the validation set.
- 4.1.2 Influence of number of classes: 3-class segmentation achieved IoU higher than 50% for both polyp and lumen, while the best polyp segmentation occurred in the 2-class setting.Adding underrepresented classes such as lumen and specular highlights makes optimization more difficult.
- 4.1.3 Comparison to state-of-the-art: FCN8 with augmentation improved IoU by 15% for background, 29% for polyps, 18% for lumen, and 14% in mean IoU over prior methods.Mean accuracy also improved by 14%, while traditional methods remained better for specular highlights.
- 4.1.3 Comparison to state-of-the-art: A 2-class model without additional augmentation achieved the best mean IoU of 72.74% and mean accuracy of 94.91%.This model ignored the specular-highlights class while retaining strong polyp segmentation.
- 4.1.3 Comparison to state-of-the-art: Qualitative results included clean polyp masks but also missed or under-segmented polyps, while lumen and mucosa segmentation remained decent or good.The reported failure examples involved a small missed polyp and an under-segmented polyp.
- 4.2 Polyp localization: At a 50% IoU threshold, FCN8 variants increased polyp localization rate by 40% over the previously published method.Models with better polyp segmentation also showed better polyp localization.
5 Conclusions
The paper introduces an extended benchmark with annotations and standard splits and metrics, alongside FCN baselines that outperform prior results without post-processing. The authors position these resources and nearly real-time processing as groundwork for future colonoscopy decision-support systems.
- 5 Conclusions: The benchmark adds annotations for polyps, mucosa, lumen, and specular highlights, with standard training, validation, and test splits.The dataset was intended to support comparison and was planned for public release upon acceptance.
- 5 Conclusions: Standard metrics were defined to help speed research in endoluminal scene segmentation.The metrics accompany the benchmark as a basis for comparison.
- 5 Conclusions: Fully convolutional network baselines outperformed previously published results by a large margin without further post-processing.The pipeline was also extended to polyp localization because the dataset lacked non-polyp frames.
- 5 Conclusions: FCN models were described as nearly real-time and therefore having potential for inclusion in future colonoscopy DSS.The conclusion links performance and processing speed to possible future use in decision support.
- 5 Conclusions: The paper calls for larger labeled datasets and architectures better suited to colonoscopy data, while making code and data publicly available.These resources are presented as a first step toward more accurate DSS and broader research contribution.