Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning

• 20% of polyps are missed during colonoscopy, and overlooked polyps remain a persistent problem despite improved endoscopic tools.Source: §I · pp. 1–2
• Public colonoscopy datasets are scarce, while annotation requires expertise and must capture variability in polyp size, shape, texture, color, orientation, and patient characteristics.Source: §I · p. 2
• Automated polyp systems should provide robust performance, high accuracy, real-time processing, and clinically interpretable outputs.Source: §I · p. 2
• Detection and localisation support surveillance and polyp-load assessment, whereas pixel-wise segmentation enables boundary delineation during procedures and ablations.Source: §I · p. 2
• The paper proposes ColonSegNet, an efficient encoder-decoder segmentation architecture designed for real-time colonoscopic polyp processing.Source: §I · p. 2
• It benchmarks state-of-the-art methods on Kvasir-SEG for detection, localisation, and segmentation using standard metrics and analyses best- and worst-performing cases.Source: §I · p. 2

• CNN-based methods have become the dominant approach for automated polyp detection after earlier handcrafted color- and texture-based methods.Source: §II · p. 2
• Existing studies include region-based CNNs, YOLO-v2, artificial intelligence systems, and generative approaches for polyp detection and localisation.Source: §II · pp. 2–3
• Segmentation research includes SegNet, dilated convolution networks, ResUNet++, DoubleUNet, boundary-aware networks, and other biomedical segmentation approaches.Source: §II · p. 3
• Kvasir-SEG provides annotated masks and bounding boxes, making it suitable for detection, localisation, and segmentation among publicly available endoscopic datasets.Source: §II · p. 3
• Benchmarking is needed because existing literature does not comprehensively compare detection, localisation, segmentation, and inference time across methods.Source: §II · p. 3
• The paper introduces a reproducible automatic polyp benchmark using the publicly available Kvasir-SEG dataset.Source: §II · p. 4

III. MATERIALS – DATASET

• Kvasir-SEG contains 1072 polyp images with segmentation masks and bounding boxes for detection, localisation, and segmentation evaluation.Source: §III · p. 4
• The dataset images range from 332 × 487 to 1920 × 1072 pixels and include 700 large, 323 medium, and 48 small polyps.Source: §III · p. 4
• Figure 1 presents sample images alongside ground-truth masks in the second column and corresponding detection bounding boxes in the third column.Source: §III · p. 4
• The study uses baseline object-detection and localisation methods together with deep learning-based segmentation methods.Source: §III · p. 4

IV. METHOD

• Detection methods predict the object class and regress bounding boxes, while segmentation methods classify the object class for each image pixel.Source: §IV · p. 4

A. Detection and localisation baseline methods

• Detection methods combine backbones, feature networks, and prediction heads; the benchmark includes EfficientDet, Faster R-CNN, YOLOv3, YOLOv4, and RetinaNet.The methods span one-stage and two-stage detectors with CNN backbones, feature pyramids, and bounding-box prediction heads.Source: A. Detection and localisation baseline… · p. 5

B. Segmentation baseline methods

• Segmentation baselines include UNet, PSPNet, DeepLabV3+, ResUNet, ResUNet++, DoubleUNet, HRNet, and UNet-ResNet34 with varied encoder-decoder designs.These architectures use mechanisms including pyramid pooling, atrous separable convolutions, residual blocks, attention, squeeze-and-excitation, and high-resolution representations.Source: B. Segmentation baseline methods · p. 5

C. ColonSegNet

• ColonSegNet uses two encoder blocks and two decoder blocks with residual blocks, squeeze-and-excitation, and skip connections for multi-scale segmentation features.Its decoder produces a binary segmentation mask through a 1 × 1 convolution and sigmoid activation.Source: C. ColonSegNet · pp. 5–6

2) Data Augmentation:

• The preprocessing pipeline addresses the substantial data requirements of supervised deep learning through computational augmentation.Manual data collection, curation, and annotation require significant clinical and computational resources.Source: 1) Data Augmentation · p. 6
• The models use horizontal and vertical flips, random rotation, random scaling, and random cropping to increase training variation.These augmentations computationally increase the number of training samples.Source: 1) Data Augmentation · p. 6
• All images are normalized and resized to 512×512 before entering the models.Normalization subtracts the image mean and divides by the standard deviation.Source: 1) Data Augmentation · p. 6

V. RESULTS

• The results section presents quantitative and qualitative findings after introducing the evaluation metrics and experimental setup.Source: §V · p. 6

A. Evaluation metrics

• The evaluation uses Average Precision and IoU for object detection and localisation.IoU measures bounding-box overlap, while AP summarizes the precision-recall curve.Source: 1) Detection and localisation task · p. 6

1) Detection and localisation task:

• AP is averaged over 11 IoU levels from 0.25 to 0.75 in increments of 0.05.This aggregates precision-recall performance across the specified overlap thresholds.Source: 1) Detection and localisation task · p. 6

2) Segmentation task:

• The segmentation evaluation reports Dice coefficient, Jaccard coefficient, precision, recall, overall accuracy, and frames per second.Jaccard coefficient is also termed IoU, and FPS measures inference time during testing.Source: 2) Segmentation task · pp. 6–7
• Frames per second is included to assess the clinical applicability of segmentation methods through test-time inference speed.Source: 2) Segmentation task · pp. 6–7
• Dice coefficient is defined using true positives, false positives, and false negatives, while accuracy also includes true negatives.The metric definitions use tp, fp, tn, and fn for pixel-level evaluation.Source: 2) Segmentation task · p. 7

B. Experimental setup and configuration:

• The experiments resized Kvasir-SEG images to 512×512, used 880 images for training and 120 for validation, and evaluated inference on NVIDIA GTX2080Ti.Training used multiple frameworks and GPUs, with extensive empirical hyperparameter searches for detection, localisation, and segmentation.Source: B. Experimental setup and configuration · p. 7

1) Detection and localisation:

• 180 FPS with AP 0.80 and IoU 0.81 made ColonSegNet nearly four times faster than YOLOv4 while retaining competitive detection and localisation scores.YOLOv4 with Darknet53 achieved AP 0.8513 and 48 FPS, whereas ColonSegNet achieved 180 FPS with AP 0.80 and IoU 0.81.Source: 1) Detection and localisation · p. 7

2) Segmentation:

• ColonSegNet used roughly six times less computational complexity than UNet-ResNet34 and about 0.75 times the size of conventional UNet.Its recall and overall accuracy were reported as close to those of UNet-ResNet34.Source: 2) Segmentation · p. 7
• 35 FPS and DSC 0.8757 made UNet-ResNet34 the accuracy leader, while ColonSegNet achieved 182.38 FPS for real-time segmentation.UNet-ResNet34 also achieved mIoU 0.8100, whereas ColonSegNet was selected for computational efficiency and speed.Source: 2) Segmentation · pp. 7, 10
• Nearly 11× and seven times faster than DeepLabv3+ with ResNet101 and ResNet34, respectively, ColonSegNet maintained competitive DSC with substantially larger networks.DeepLabv3+ with ResNet101 had more than 11 times as many trainable parameters, while DeepLabv3+ with ResNet34 had nearly eight times the computational complexity.Source: 2) Segmentation · p. 8

D. Qualitative evaluation

• Large polyps were detected and segmented with high-quality masks across the compared algorithms.The top-scored segmentation samples showed strong qualitative agreement across methods.Source: D. Qualitative evaluation · p. 10
• YOLOv4 often localized polyps more accurately than competing detectors in selected examples, while other methods missed small polyps or produced overly wide boxes.The qualitative comparison also includes missed detections and false identification of stool and polyp regions.Source: D. Qualitative evaluation · p. 10
• Small flat polyps, inclined views, and saturated regions adversely affected segmentation quality, defining important difficult-case conditions.These cases were identified among the bottom-scored samples in Figure 5(c).Source: D. Qualitative evaluation · p. 11
• ColonSegNet produced shapes similar to ground truth and comparatively improved predictions on almost all worse-performing samples.Some outliers remained among its predictions on the top-scored images.Source: D. Qualitative evaluation · p. 11

Benchmark scope

• The segmentation benchmark includes popular methods selected for medical-image relevance, speed, accuracy, or combinations of these criteria.Examples include UNet, Faster R-CNN, YOLOv3, PSPNet, FCN8, DoubleUNet, DeepLabv3+, and YOLOv4.Source: §VI · p. 11

Detection and localisation

• YOLOv3 with Darknet53 achieves the best mIoU, whereas RetinaNet with ResNet101 has competitive average precision but nearly 5% lower IoU than YOLOv4 and YOLOv3-spp.Source: §VI · p. 11
• YOLOv4 with Darknet53 and CSP provides the preferred detection-localisation trade-off between speed, average precision, and IoU.The authors attribute this balance to Cross-Stage-Partial connections and CIoU loss for bounding-box regression.Source: §VI · p. 11

Semantic segmentation

• ColonSegNet achieves the highest segmentation speed at 182.38 FPS and improves over the evaluated methods in semantic segmentation accuracy.UNet with ResNet34 and DeepLabv3+ are identified as strong competitors across several evaluation metrics.Source: §VI · p. 11

Robustness

• Segmentation robustness is limited by lighting-dependent texture similarity, sessile or flat polyps, and small polyps relative to image size.The authors suggest more varied samples, spatial pyramid layers, and context-aware artifacts or shape information as possible remedies.Source: §VI · p. 11

Study limitations

• The retrospective study design, image resizing, and empirically selected hyperparameters constrain validation and may affect algorithm performance.The authors state that clinical studies are needed for real-world validation and that hyperparameters could be further optimized.Source: §VI · p. 11

Overall findings

• ColonSegNet detects and localises polyps at 180 FPS and segments them at 182.38 FPS on Kvasir-SEG.The evaluated algorithms also show strong performance across average precision, IoU, DSC, precision, recall, F2-score, and FPS.Source: §VII · p. 11

Benchmark contribution

• The benchmark provides reproducible comparisons using established metrics and qualitative failure cases across variable polyp sizes and image resolutions.Source: §VII · pp. 11–12

Clinical relevance

• The reported methods show potential for clinical use in polyp detection, localisation, and segmentation, while further work should improve accuracy, speed, and robustness.Source: §VII · p. 12

Future work

• Future improvements include data augmentation, fine-tuning, artifact handling, and models that account for parameter counts required by clinical systems.Source: §VII · p. 12