Source-linked AI summary
A multi-centre polyp detection and segmentation dataset for generalisability assessment
Sharib Ali, Debesh Jha, Noha Ghatwary, Stefano Realdon, Renato Cannizzaro, Osama E. Salem, Dominique Lamarque, Christian Daul, Michael A. Riegler, Kim V. Anonsen, Andreas Petlund, Pål Halvorsen, Jens Rittscher, Thomas de Lange, James E. East
TL;DR
Automated polyp detection and segmentation lack rigorous evaluation on large, multicentre datasets with accessible pixel-level annotations. This paper presents PolypGen, a six-centre dataset curated and reviewed by experienced gastroenterologists, and technically validates segmentation methods, finding stronger performance from deeper residual architectures but reduced performance on sequence data and small polyps.
Problem
Existing public colonoscopy datasets often lack pixel-level annotations, impose access barriers, or contain data from few centres, limiting rigorous generalisability assessment.
Method
The paper constructs and quality-assures a diverse multicentre dataset, using annotations reviewed by six senior gastroenterologists, and evaluates segmentation architectures with boundary-sensitive metrics.
Results
DeepLabV3+ with ResNet101 achieved the highest single-frame DSC of 0.82, while sequence C6 out-of-sample testing reached a highest DSC of 0.65.
Takeaways & Limitations
Residual networks, atrous spatial pyramid pooling, and deeper backbones improved performance on unseen centres, while small polyps remained difficult to segment.
Takeaways & Limitations
Annotation accuracy, reliability, and completeness are subjective, and ambiguous annotations were mostly removed rather than having expert disagreement quantified.
Abstract
from arXiv · showhide
Polyps in the colon are widely known cancer precursors identified by colonoscopy. Whilst most polyps are benign, the polyp's number, size and surface structure are linked to the risk of colon cancer. Several methods have been developed to automate polyp detection and segmentation. However, the main issue is that they are not tested rigorously on a large multicentre purpose-built dataset, one reason being the lack of a comprehensive public dataset. As a result, the developed methods may not generalise to different population datasets. To this extent, we have curated a dataset from six unique centres incorporating more than 300 patients. The dataset includes both single frame and sequence data with 3762 annotated polyp labels with precise delineation of polyp boundaries verified by six senior gastroenterologists. To our knowledge, this is the most comprehensive detection and pixel-level segmentation dataset (referred to as \textit{PolypGen}) curated by a team of computational scientists and expert gastroenterologists. The paper provides insight into data construction and annotation strategies, quality assurance, and technical validation. Our dataset can be downloaded from \url{ https://doi.org/10.7303/syn26376615}.
Background
Colorectal cancer is a major health burden, and most cases arise through the progression of benign adenomas or serrated polyps. Existing datasets limit rigorous development and generalisability because many lack pixel-level annotations, have restricted access, or represent only a few centres.
- About 1.3 million new colorectal cancer cases are detected worldwide annually, with about 51% mortality and CRC ranking third in cancer mortality.
- Approximately 90% of colorectal cancers result from the slow transformation of adenomas or serrated polyps, although only a minority of these lesions progress to cancer.
- Assessing malignant potential is particularly challenging for lesions smaller than 10 mm, while complete boundary delineation is needed for complete resection.
- Human limitations contribute to miss rates of 26% for adenomas, 9% for advanced adenomas, and 27% for serrated polyps.
- Public datasets often lack pixel-level ground truth, require prolonged or uncertain access approval, or contain images from only one or a few centres.
Methods
PolypGen was constructed as a multicentre dataset combining colonoscopy images and video sequences from six centres, with heterogeneous sampling and expert-reviewed annotations. The dataset includes varied polyp sizes, positive and negative sequences, and quality-assured pixel-level labels for generalisability assessment.
- Data collection: Data were collected from six centres across Egypt, France, Italy, Norway, and the United Kingdom, involving more than 300 patient videos or frames.Each centre handled its own ethical, legal, and privacy requirements, including consent, ethics review, and anonymisation procedures.
- Sampling strategy: Positive sequences contain polyp and nearby normal-mucosa frames, whereas negative sequences come from videos with confirmed normal mucosa or regions away from polyps.This design preserves realistic polyp appearance and disappearance while providing negative cases that may resemble polyps, including reflections and stool-covered mucosa.
- Sampling strategy: The dataset combines 1,537 single frames with 2,225 sequence frames, releasing 3,762 positive frames across the six centres.Single-frame sampling included different viewpoints, while sequence sampling captured visible and invisible polyp frames with minimal gaps.
- Dataset composition: The dataset includes small, medium, and large annotated polyps, with size-based counts reported for each centre and examples spanning flat lesions, large lesions, and dye-assisted resection.Polyp sizes were categorised as small (≤100 × 100 pixels), medium (>100 × 100 and ≤200 × 200 pixels), or large (≥200 × 200 pixels).
- Annotation and quality assurance: Annotations were produced by three experienced researchers and revised, cross-validated, and independently reviewed by six senior gastroenterologists.A common manual-annotation protocol was used to reduce heterogeneity in segmentation boundaries, with centre experts participating in review.
- Annotation and quality assurance: Quality assurance rejected 30.5% of 2,213 non-ambiguous annotations, leaving 1,537 accepted frames after review.A separate ambiguous batch had a 34.17% rejection rate during its second review.
Data Records
PolypGen provides an openly structured, anonymised dataset organized into positive single-frame and sequence data, alongside negative sequence samples and supporting quality-review information.
- Dataset organization: The released dataset adds sixth-centre data and positive and negative frames beyond the EndoCV2021 training subset.The current release does not prescribe training and test splits, leaving users free to choose their strategies.
- Folder structure: Positive data are organized by single-frame centre splits and sequence frames, while negative data contain only sequence samples.The folder structure separates the main dataset into Positive and Negative folders, with centre-wise subdivisions for single frames.
- Quality review: Annotation review materials include expert review counts, centre labels, and checks for ambiguous annotations across centres C1–C6.The review figure also reports accepted and rejected frames and categorizes flat and protruded polyps.
- Data representation: The dataset includes sample images and folder metadata, with image sizes ranging from 384×288 to 1920×1080 pixels.Masks retain the original image dimensions, and full anonymisation means gender and age information are not provided.
Technical Validation
Technical validation benchmarks established segmentation CNNs on centre C6 out-of-sample data using overlap, precision-recall, distance, and boundary metrics. DeepLabV3+ with ResNet101 generally performed best, while smaller polyps remained substantially harder to segment.
- Validation design: 1449 single frames from centres C1–C5 were used for training and validation, while centre C6 supplied 88 single frames and 432 sequence frames for out-of-sample testing.The split was 80% training and 20% validation, with C6 representing a different population and endoscopy device.
- Evaluation metrics: The evaluation used Jaccard Index, DSC, F2-score, precision, recall, accuracy, average Hausdorff distance, ASD, and NSD.Fβ scores summarize precision and recall, while boundary metrics assess distances between predicted and reference segmentation borders.
- Compared models: Popular segmentation architectures included FCN, U-Net, PSPNet, and DeepLabV3, with out-of-sample results reported for both single-frame and sequence data.These models were benchmarked as established semantic segmentation methods for polyp segmentation.
- Results: DeepLabV3+ with ResNet101 achieved the strongest overall performance, including DSC 0.82 and dAHD 9.67 on single-frame data.DeepLabV3+ with ResNet50 offered the second-best inference speed at 47 FPS with DSC = 0.81 and dAHD = 9.95.
Discussion
Performance varied substantially with data format, architecture, boundary metric, and validation-centre choice. Sequence data and difficult visual cases exposed lower segmentation performance, while residual and deeper architectures improved generalisability.
- Data-format effects: DSC exceeded 0.80 for ResNetUNet and DeepLabV3+ with ResNet101 on single frames but was around 0.70 on sequence data.The sequence dataset contained nearly five times as many frames, with greater viewpoint, size, and image-quality variation.
- Efficiency and boundary metrics: 87 FPS was achieved by ResNetUNet with ResNet34, while its DSC remained close to DeepLabV3+ with ResNet50.ResNetUNet also produced lower MASD than DeepLabV3+ for both single data (35.83 vs 41.04) and sequence data (27.10 vs 28.19).
- Architecture and polyp size: Baseline vanilla UNet, PSPNet, and FCN8 produced sub-optimal results, whereas residual networks, atrous spatial pyramid pooling, and deeper backbones improved generalisability.Small polyps of ≤100×100 pixels generally performed poorly, while medium and large polyps were captured more effectively.
- Validation effects: Validation-data choice substantially affected network performance even for architectures that performed optimally on this dataset.The centre-wise evaluation used training on four centres, validation on an individual centre, and testing on an out-of-sample centre.
- Qualitative error patterns: Qualitative results linked better single-frame performance to clear, lifted polyps distinct from the background, while networks often missed sessile or flat polyps in sequences.Strong local illumination and polyps blending into the mucosa were associated with worse single-frame results.
Limitations of the dataset
The dataset includes both polyp and nearby non-polyp images, but annotation quality remains subjective and ambiguous annotations were mostly removed. Future versions aim to quantify disagreement among experts.
- Annotation limitations: Annotation accuracy, reliability, and completeness are subjective to the annotators, and ambiguous annotations were mostly removed.Future versions will aim to quantify the level of disagreement among experts for each frame.
Usage Notes
The released dataset is intended for educational, research, and commercial use under CC-BY licensing. Samples are organised into positive and negative categories, including single and sequence frames for different modelling needs.
- Licence: The dataset is released under a Creative Commons CC-BY licence for educational, research, and commercial purposes.Users must credit the creators by citing the paper and acknowledging them.
- Sample organisation: Positive samples are divided into single frames and sequence frames, while users may select samples according to their method.Fully convolutional networks can use positive samples, whereas recurrent methods may use positive and negative sequence data.
- Qualitative examples: Figure 9 presents 12 best-performing and 12 worst-performing frames for both single and sequence data based on DSC.The figure is intended to show qualitative examples at the upper and lower ends of performance.
Code availability
Code is available for evaluating generalisability and conducting centre-based split analyses, alongside benchmark code with the paper’s segmentation training and validation split.
- Generalisability analysis: The EndoCV2021 polyp detection and segmentation repository provides inference code for centre-based split analysis to evaluate generalisability.This code is available at the GitHub repository identified in the paper.
- Benchmarking: The PolypGen-Benchmark repository provides benchmark code with the paper’s training and validation split for segmentation.Method implementations are also available through GitHub repositories listed in Table 1.
Author contributions statement
The statement assigns primary responsibility for conceptualizing, coordinating, collecting, curating, annotating, analyzing, and writing the work to S. Ali, with other authors contributing clinical, annotation, validation, and manuscript expertise.
- S. Ali conceptualized, initiated, and coordinated the work while leading data collection, curation, annotation, analysis, and most writing.
- T. de Lange contributed to the introduction, clinical correctness, sequence descriptions, and manuscript feedback.
- D. Jha and N. Ghatwary assisted with data annotation and parts of the technical validation.
- Several additional authors contributed directly to the work.
Competing interests
The authors disclose advisory, ownership, speaking, and executive relationships for several contributors, while all other authors report no known relevant conflicts.
- J. E. East reports clinical advisory board roles and speaker fees, including ownership in Satisfai Health.
- A. Petlund is identified as CEO and T. de Lange as chief medical scientist at Augere Medical.
- All other authors declare no known competing financial interests or personal relationships that could have influenced the reported work.