PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation

Debesh Jha, Pia H. Smedsrud, Dag Johansen, Thomas de Lange, Håvard D. Johansen, Pål Halvorsen, Michael A. Riegler

arXiv:2107.12435v1cs.CV

TL;DR

The paper addresses missed colorectal abnormalities by extending ResUNet++ for automatic polyp segmentation with CRF and TTA. Evaluated across six public datasets, the extensions improved performance on same-dataset and cross-dataset evaluations, with ResUNet++ plus CRF achieving DSC 0.9293 and mIoU 0.8898 on CVC-ClinicDB.

  • Problem

    Colonoscopy has a high overall miss-rate, motivating computer-aided systems that can identify overlooked regions and help detect and characterize colorectal lesions.

  • Method

    The paper extends ResUNet++ with Conditional Random Field and Test-Time Augmentation, using horizontal and vertical flips and evaluating six public image and video datasets.

  • Results

    Across same-dataset and cross-dataset evaluations, CRF and TTA improved polyp-segmentation performance; ResUNet++ with CRF achieved DSC 0.9293 and mIoU 0.8898 on CVC-ClinicDB.

  • Takeaways & Limitations

    The results support ResUNet++ with CRF for image datasets and combinations involving TTA for mixed and cross-dataset evaluations, including robust cross-dataset performance from ResUNet++ plus TTA.

  • Takeaways & Limitations

    The study is retrospective and uses curated data, while images were resized to 256 × 256, causing information loss that may affect performance.

Abstract

from arXiv · show

Colonoscopy is considered the gold standard for detection of colorectal cancer and its precursors. Existing examination methods are, however, hampered by high overall miss-rate, and many abnormalities are left undetected. Computer-Aided Diagnosis systems based on advanced machine learning algorithms are touted as a game-changer that can identify regions in the colon overlooked by the physicians during endoscopic examinations, and help detect and characterize lesions. In previous work, we have proposed the ResUNet++ architecture and demonstrated that it produces more efficient results compared with its counterparts U-Net and ResUNet. In this paper, we demonstrate that further improvements to the overall prediction performance of the ResUNet++ architecture can be achieved by using conditional random field and test-time augmentation. We have performed extensive evaluations and validated the improvements using six publicly available datasets: Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, ETIS-Larib Polyp DB, ASU-Mayo Clinic Colonoscopy Video Database, and CVC-VideoClinicDB. Moreover, we compare our proposed architecture and resulting model with other State-of-the-art methods. To explore the generalization capability of ResUNet++ on different publicly available polyp datasets, so that it could be used in a real-world setting, we performed an extensive cross-dataset evaluation. The experimental results show that applying CRF and TTA improves the performance on various polyp segmentation datasets both on the same dataset and cross-dataset.

I. INTRODUCTION

Colorectal polyp segmentation is motivated by substantial colonoscopy miss-rates and challenging lesion variability. The paper extends ResUNet++ with CRF and TTA, evaluates it across six datasets, and emphasizes robustness, generalization, and overlooked flat or sessile polyps.

  • 6-27% is the estimated adenoma miss-rate, with small, sessile, and flat polyps most often missed.
  • Polyp variability in shape, size, color, and appearance, together with image artifacts, complicates segmentation.
  • CADx segmentation can focus analysis on lesions, reduce manual intervention and segmentation errors, and support clinical workflow.
  • The paper extends ResUNet++ with Conditional Random Field and Test-Time Augmentation to improve polyp-segmentation prediction performance.
  • The evaluation spans six public image and video datasets and includes cross-dataset testing to assess generalizability.
  • The authors separately selected and released 196 flat or sessile polyps from Kvasir-SEG for challenging-polyp evaluation.

II. RELATED WORK

Prior work spans handcrafted, CNN, pretrained, FCN, GAN, and Mask R-CNN approaches for automated polyp detection and segmentation. The section identifies weak cross-dataset evaluation and limited universal-model development as central gaps.

  • Prior polyp-segmentation methods have used edge or texture analysis, CNNs, pretrained networks, FCNs, GANs, and Mask R-CNN.
  • Existing studies report high performance using smaller training and test datasets, but evaluation designs vary substantially.
  • Minimal cross-dataset testing limits assessment of CADx generalizability across data sources.
  • Few studies design universal models for accurately segmenting polyps from different sources.
  • Single, small, imbalanced, and handpicked datasets make comparisons and performance conclusions difficult to interpret.
  • The paper addresses these gaps through experiments on multiple colonoscopy image and video datasets, mixed-center training, and testing on unseen datasets.

III. THE RESUNET++ ARCHITECTURE

ResUNet++ is an encoder-decoder segmentation network derived from ResUNet and U-Net. Its design combines residual learning, squeeze-and-excitation blocks, ASPP, attention blocks, and skip connections.

  • ResUNet++ combines residual, squeeze-and-excitation, ASPP, and attention blocks within an encoder-decoder architecture.
  • The encoder uses squeeze-and-excitation blocks, while ASPP replaces the ResUNet bridge and attention blocks are added to the decoder.
  • Additional skip connections connect encoder residual units to decoder attention blocks for feature propagation.
  • The architecture uses encoder filter levels [32, 64, 128, 256, 512] and reverses them in the decoder.
  • The network contains one stem block, three encoder blocks, ASPP modules, three decoder blocks, and a sigmoid-based output stage.
  • Residual connections route input information directly to outputs, supporting gradient flow during training.

B. Squeeze and Excitation block

The architecture uses channel recalibration, multi-scale contextual processing, decoder attention, and CRF refinement to improve feature selection and polyp-mask quality.

  • B. Squeeze and Excitation block: Squeeze-and-excitation blocks recalibrate channel-wise feature responses by modeling interdependencies between channels.
  • B. Squeeze and Excitation block: The SE operation uses global average pooling and sigmoid gating to amplify effective feature maps and suppress irrelevant ones.
  • C. Atrous Spatial Pyramid Pooling: ASPP uses parallel atrous convolutions to capture contextual information at multiple scales between encoder and decoder stages.
  • D. Attention Units: The decoder attention block highlights relevant information and filters irrelevant information in feature maps.
  • E. Conditional Random Field: CRF models shape, region connectivity, and contextual information to produce more refined polyp outputs.

F. Test Time Augmentation

Test-time augmentation modifies each test image, averages predictions across augmented versions, and uses horizontal and vertical flips in this study. The experiments evaluate segmentation across six datasets using overlap and pixel-level metrics.

  • Test-Time Augmentation: TTA creates multiple modified versions of each test image and averages their predictions into the final output.This study uses horizontal and vertical flips for test-time augmentation.
  • Datasets: The experiments use six segmented-polyp datasets that differ in image counts, resolutions, capture devices, and segmentation-mask accuracy.The datasets include Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, ETIS Larib Polyp DB, CVC-VideoClinicDB, and ASU-Mayo Clinic.
  • Evaluation: DSC measures similarity with ground truth, while IoU measures overlap between the predicted and ground-truth polyp masks.The study also calculates mIoU across semantic classes and uses precision and recall to indicate over-segmentation and under-segmentation.

C. Data Augmentation

The study augments training data extensively and applies test-time augmentation only during evaluation. Models are trained with standardized preprocessing and optimization procedures before comparing ResUNet++ variants across datasets.

  • Data Augmentation: Training uses numerous geometric, color, distortion, dropout, and blur augmentations, while validation remains unaugmented and TTA is applied during testing.Images are resized to 256×256 pixels, and training, validation, and testing splits generally follow an 80:10:10 ratio.
  • Implementation: All models are implemented with Keras and TensorFlow and trained for 300 epochs with early stopping and stochastic gradient descent with warm restarts.Experiments run on a Volta 100 Tensor Core GPU in an Nvidia DGX-2 system.
  • Evaluation Design: The study compares ResUNet++ with CRF, TTA, and their combination on same-dataset, mixed-dataset, and cross-dataset evaluations.The authors also compare the proposed approaches with recent state-of-the-art methods, while noting that testing mechanisms differ across studies.

A. Results comparison on Kvasir-SEG dataset

On Kvasir-SEG, the proposed ResUNet++ variants outperform the original ResUNet++ in quantitative, qualitative, and ROC-curve evaluations. The reported comparisons attribute these gains to TTA, CRF, and their combinations.

  • The proposed methods outperform ResUNet++ on Kvasir-SEG across quantitative, qualitative, and ROC-curve evaluations.The qualitative examples include polyps that are usually missed during colonoscopy, with predicted masks closely matching ground truth.
  • Applying TTA, CRF, or their combinations improves the reported Kvasir-SEG results.
  • The qualitative comparison covers UNet, ResUNet, ResUNet++, and the proposed models on commonly missed polyps.The caption reports high similarity between ground-truth and predicted masks for the proposed models.
  • Cross-dataset evaluation shown in Figure 5 tests a model trained on CVC-ClinicDB against Kvasir-SEG.
  • The evaluation also includes ROC curves for proposed models on Kvasir-SEG and models trained and tested on CVC-ClinicDB.

D. Results comparison on ETIS-Larib Polyp DB

On ETIS-Larib Polyp DB, ResUNet++ achieves the strongest DSC and mIoU among the compared variants, while the combined approaches differ only marginally. The dataset-specific result contrasts with broader gains reported elsewhere, although ResUNet++ with TTA has higher precision than ResUNet++.

  • The ETIS-Larib results are compared directly with ResUNet++ rather than UNet and ResUNet because ResUNet++ had already shown superior performance on Kvasir-SEG and CVC-ClinicDB.
  • ResUNet++ achieves a maximum DSC of 0.6364 and mIoU of 0.7534 on ETIS-Larib Polyp DB.These values are reported as improvements over SOTA of 0.84% in DSC and 18.64% in mIoU, respectively.
  • Differences among ResUNet++, ResUNet++ + CRF, ResUNet++ + TTA, and ResUNet++ + CRF + TTA are marginal on ETIS-Larib.
  • ResUNet++ records recall of 0.6346, slightly higher than the proposed methods, while ResUNet++ + TTA has higher precision.
  • The architecture’s performance is data specific: combined approaches outperform SOTA over five independent datasets, but ResUNet++ performs better than the combinations on ETIS-Larib.ETIS-Larib contains 196 images, with 156 used for training.

E. Results on Kvasir-Sessile

The proposed combinations were evaluated on sessile-polyp, video, and mixed-dataset settings. Results varied by dataset, with annotation quality and dataset composition affecting measured performance.

  • Kvasir-Sessile: 0.5042 DSC and 0.6606 mIoU were obtained by ResUNet++ with TTA on the Kvasir-Sessile dataset.The dataset contains small, diverse images and few training samples.
  • CVC-VideoClinicDB: 0.8811 DSC and 0.8739 mIoU were obtained by ResUNet++ with CRF on CVC-VideoClinicDB.The dataset includes 11,954 polyp and normal video frames, although its masks are not pixel perfect.
  • ASU-Mayo ClinicDB: 0.8850 DSC and 0.8635 mIoU were obtained by ResUNet++ with CRF on ASU-Mayo ClinicDB.ASU-Mayo contains 18,781 polyp and non-polyp frames.
  • Mixed-dataset evaluation: 0.5084 DSC and 0.6859 mIoU were obtained on CVC-ColonDB using ResUNet++ with TTA after mixed-dataset training.Using the same mixed training setup, ResUNet++ with CRF and TTA obtained 0.3603 DSC and 0.6468 mIoU on CVC-VideoClinicDB.
  • Annotation effects: Evaluation scores on CVC-VideoClinicDB can be lowered by differences between predicted masks and the dataset’s imperfect oval or circular ground truth.The model was trained on datasets with more precise annotations.

I. Cross-dataset result evaluation on Kvasir-SEG

Cross-dataset experiments trained models on Kvasir-SEG and tested them on five independent datasets. TTA performed best on image datasets, while combining ResUNet++, CRF, and TTA performed best on video datasets.

  • Kvasir-SEG training: 0.6817 average best mIoU and 0.4779 average best DSC were obtained when training on Kvasir-SEG and testing across five independent datasets.The evaluation covered both image and video datasets.
  • Cross-dataset comparison: ResUNet++ with TTA performed better on the cross-dataset image evaluations.The combined ResUNet++, CRF, and TTA approach performed best on the video evaluations.
  • Evaluation conditions: Cross-dataset scores were affected by differences in ground-truth shape and class balance between training and test datasets.Kvasir-SEG contained pixel-segmented polyps, whereas some test datasets contained oval-shaped masks and many non-polyp frames.
  • CVC-ClinicDB training: Training on CVC-ClinicDB produced the same pattern: TTA led on image datasets, while ResUNet++ with CRF and TTA led on video datasets.The best model values were similar on CVC-VideoClinicDB and ASU-Mayo Clinic.
  • Additional comparison: 0.6522 mIoU was achieved when training on CVC-ClinicDB and testing on ETIS-Larib, the highest value in the cited comparison.

VI. DISCUSSION

The discussion finds that CRF and TTA improve ResUNet++ across same-, mixed-, and cross-dataset settings, but the strongest combination depends on the dataset type. The evaluation also highlights generalization, computational size, and data-dependent limitations.

  • Overall findings: CRF and TTA improved ResUNet++ performance on same datasets, mixed datasets, and cross-datasets.
  • Generalization pattern: ResUNet++ with TTA was most generalizable on still images, while ResUNet++ with CRF and TTA performed best on video datasets.
  • Discrimination: All proposed models achieved AUC values greater than 0.93 for distinguishing polyp from non-polyp images.
  • Model complexity: ResUNet++ requires fewer parameters than models using pre-trained encoders, while its performance gain compensates for increased training time.
  • Evaluation strategy: Cross-data testing uses one training dataset and several datasets from different centers or scope manufacturers to assess generalization.The authors present this as an initiative toward improving generalizability.
  • Data dependence: Different algorithms performed well on different datasets, with CRF generally stronger on video and many image datasets while TTA remained competitive.The discussion also reports no significant improvement for some methods on other datasets.

C. Challenges

The paper addresses difficult polyp appearances and clinical variability, while acknowledging that retrospective, curated data and image resizing limit clinical interpretation. The proposed system nevertheless reports improved performance and detection of small and flat polyps.

  • Segmentation challenges: ResUNet++ with CRF and TTA still struggles on images affected by preparation quality, camera angle, superfluous information, and varying morphology.It nevertheless performs considerably better than the previous model and another SOTA algorithm.
  • Missed-polyp detection: Small and flat polyps are clinically relevant targets because they are often missed during endoscopic examinations.The model is described as performing well on both flat and small polyps.
  • Clinical evaluation: The study is retrospective, and prospective clinical evaluation on full colonoscopy videos is identified as essential.The experiments use curated data rather than prospective full-colon examinations.
  • Method limitations: The study resized all images to 256 × 256, which may lose information and affect performance.Further code optimization may also improve performance.
  • Study design: ResUNet++ was combined with CRF and TTA and evaluated on six public datasets, including cross-dataset generalizability analyses.
  • Reported performance: The comprehensive evaluation reported good performance for ResUNet++ with CRF on image datasets and ResUNet++ with TTA or CRF plus TTA on mixed and cross-datasets.
Loading 2107.12435v1…