PaperlayerView exact revision on arXiv

Source-linked AI summary

TRUE-Colon: Exposing a Consistent Transfer Asymmetry in Real-Time Polyp Detection

Sebastian Doerrich, Andreas Franz Schwab, Francesco Di Salvo, Shyam Nandan Rai, Hanh Huyen My Nguyen, Christian Ledig

arXiv:2608.13711v1eess.IVcs.CVcs.LG

TL;DR

Curated lesion-centric benchmarks inadequately represent full colonoscopy procedures, limiting deployment-relevant evaluation of CADe systems. TRUE-Colon evaluates four detectors across curated benchmarks and 60 full procedures, finding that procedure-trained models transfer better while RT-DETR leads on sensitivity and temporal detection quality.

  • Problem

    Curated lesion-centric benchmarks under-represent long negative stretches and procedure artifacts, limiting evidence about CADe deployment-relevant performance.

  • Method

    TRUE-Colon evaluates four real-time detectors across curated benchmarks and continuous full-procedure videos using localization, false-alert, latency, and temporal-reliability measures.

  • Results

    Procedure-trained models transfer well to curated benchmarks while curated-trained models generalize poorly to full procedures; RT-DETR leads on sensitivity, latency, and persistence, whereas convolutional detectors offer higher throughput.

  • Takeaways & Limitations

    Deployment-focused CADe training and evaluation should use full-procedure video and matched false-alert operating points rather than curated benchmarks alone.

  • Takeaways & Limitations

    Dataset differences in realism, scale, annotation, and negative-frame prevalence prevent attributing the transfer asymmetry to realism alone.

Abstract

from arXiv · show

Computer-aided detection (CADe) systems for colonoscopy promise to reduce clinical miss rates, yet reliable real-world deployment remains elusive. This translational gap stems in part from a structural flaw in model development: the reliance on curated datasets that under-represent the long negative stretches and procedure-related artifacts characteristic of routine examinations. Training and evaluating architectures strictly on these lesion-centric benchmarks creates an illusion of success, since such benchmarks cannot capture clinically crucial metrics. To expose this gap, we establish TRUE-Colon, a standardized benchmarking protocol that measures key deployment characteristics alongside localization accuracy, and evaluate four real-time architectures (Faster R-CNN, YOLOv8, YOLOv11, RT-DETR) across curated benchmarks (SUN, PICCOLO) and 60 unedited, full-length procedures (REAL-Colon). We observe a consistent transfer asymmetry: models trained strictly on curated clips suffer a severe performance collapse when evaluated on full procedures, whereas procedure-trained models substantially improve rejection of non-polyp content on REAL-Colon, and largely retain their accuracy on curated benchmarks. Beyond transferability, we find that the Transformer detector attains the strongest sensitivity and the earliest, most persistent detections, while the convolutional detectors stay competitive at a higher throughput. Together, these results indicate that both training and benchmarking for deployable CADe should shift from curated, lesion-centric clips toward full-procedure data and deployment-relevant operating points. Source code is available at https://github.com/sdoerrich97/true-colon.

1 Introduction

TRUE-Colon addresses the translational gap in real-time polyp detection by evaluating deployment-relevant behavior beyond localization accuracy. Across four detectors, it reveals asymmetric transfer from curated clips to full procedures and an accuracy-throughput trade-off between Transformer and convolutional models.

  • Motivation: Clinical miss rates reach up to 27% because transient, morphologically variable polyps are obscured by folds, debris, and changing illumination.These challenges motivate CADe systems as an automated safety layer during colonoscopy.
  • Translational gap: Curated lesion-centric datasets under-represent long negative stretches and procedure-related artifacts, limiting the clinical relevance of standard CADe evaluation.Despite high public-benchmark accuracy, this distribution mismatch contributes to restricted clinical deployment.
  • TRUE-Colon protocol: TRUE-Colon measures localization accuracy, false-alert burden, detection latency, and temporal reliability at a matched false-alert operating point.The protocol enables direct comparison across architectures using deployment-relevant metrics.
  • Transfer asymmetry: Curated-trained detectors collapse on full procedures, whereas procedure-trained models transfer well to curated benchmarks, identifying full-procedure data as the more informative deployment signal.This transfer asymmetry is observed consistently across four detectors.
  • Architecture comparison: At matched false-alert operating points, the Transformer leads in sensitivity, latency, and temporal persistence, while convolutional detectors remain competitive at substantially higher throughput.The findings expose an accuracy-versus-compute trade-off obscured by fixed-threshold comparisons.

2 Methodology

TRUE-Colon evaluates polyp detectors using deployment-relevant characteristics beyond localization accuracy, including false-alert burden, detection latency, and temporal reliability. Its cross-distribution design compares curated and procedure-level training and testing across four real-time architectures.

  • Deployment-oriented evaluation: TRUE-Colon reports localization accuracy, false-alert burden, detection latency, and temporal reliability; sequence evaluations define a positive frame as containing at least one annotated polyp.These characteristics complement standard object-detection metrics that do not capture continuous-video behavior.
  • Deployment-oriented evaluation: Localization accuracy uses IoU-matched predictions with mAP50 (IoU ≥0.50) and mAP50:95 averaged across IoU thresholds from 0.50 to 0.95.These metrics quantify spatial accuracy but not operational false alerts on negative video segments.
  • Deployment-oriented evaluation: Detector outputs become binary alerts at confidence threshold τ, with valid detections requiring confidence above τ and positive-frame overlap with an annotated polyp box.Positive and negative frames are then classified using valid detections to quantify alert behavior.
  • Deployment-oriented evaluation: Latency counts frames from annotated lesion onset to the first valid detection, while temporal reliability distinguishes any detection from persistence across at least half of visible frames.Latency reflects early-frame sensitivity for memoryless per-frame detectors and is reported at matched τ⋆; lesion-level temporal metrics require continuous videos with consistent lesion identifiers or equivalent visibility windows.
  • Cross-distribution evaluation: Cross-distribution transfer compares curated-to-curated, curated-to-procedure, procedure-to-procedure, and procedure-to-curated settings using Dcur for SUN/PICCOLO and Dproc for REAL-Colon.The design tests whether full-procedure training improves robustness on procedure-level data.
  • Model and training setup: The benchmark includes Faster R-CNN, YOLOv8-M, YOLOv11-M, and RT-DETR, trained with default recipes for 100 epochs with early stopping at patience 10 and fixed 640×640 inputs.Faster R-CNN uses Detectron2 with effective batch size 96; the other architectures use Ultralytics with effective batch size 208.

3 Experiments and Results

Experiments span curated SUN and PICCOLO benchmarks and 60 full procedures in REAL-Colon, revealing sharp curated-to-procedure transfer failure. Procedure-level evaluation further shows RT-DETR’s strongest fixed-threshold accuracy and sensitivity, alongside higher false-alert burden and a throughput–accuracy trade-off among YOLO detectors.

  • Dataset distributions: REAL-Colon comprises 60 full, unedited colonoscopy videos and maintains the highest proportion of negative frames (> 85%), unlike curated SUN and PICCOLO.REAL-Colon uses patient-level independence and a 10/2/3 train/validation/test video split per institution.
  • Dataset distributions: Lesion-level temporal metrics are computable only on REAL-Colon because SUN and PICCOLO lack continuous procedure recordings with consistent per-lesion identifiers.SUN supports frame-level alert metrics, whereas PICCOLO is limited to instance-level localization on still images.
  • In-domain baselines: On PICCOLO, RT-DETR reaches mAP50 = 0.321 versus 0.770 for YOLOv11, consistent with underperformance on the approximately 2,000-image training set.The passage attributes this result to Transformer detectors being more data-hungry and notes PICCOLO’s extremely small number of negative images.
  • Cross-distribution transfer: Curated-trained models degrade sharply on REAL-Colon: YOLOv11 falls from mAP50 = 0.724 on SUN to 0.164, while RT-DETR falls from 0.689 to 0.225.YOLOv11 trained on PICCOLO degrades further to mAP50 = 0.059, indicating insufficient exposure to non-polyp content.
  • Deployment-relevant behavior: At τ = 0.2 on REAL-Colon, RT-DETR achieves mAP50 = 0.488 and sensitivity = 0.720, but its FPR is 0.082 versus YOLO FPR ≤0.020.Fixed thresholds confound model quality with score calibration because architectures incur different false-alert burdens at the same threshold.
  • Deployment-relevant behavior: YOLOv8 reaches 46.8 frames, reflecting a throughput–accuracy trade-off; differences across n = 21 test lesions fall within seed variance and are trends rather than rankings.The passage characterizes the YOLO detectors as having markedly higher throughput.

4 Discussion and Conclusion

The discussion concludes that curated benchmarks overestimate clinical readiness, while full-procedure data better expose transfer, false-alert, latency, and temporal-reliability challenges. It also limits interpretation because dataset differences and lesion-size imbalance complicate attribution and generalization.

  • Clinical readiness: Curated benchmarks overestimate clinical readiness because performance drops sharply on continuous, artifact-heavy full procedures.The authors attribute this transfer asymmetry to limited exposure to prolonged negative segments and procedure-related artifacts.
  • Deployment requirements: Full-procedure video is required for deployment-oriented training and evaluation because curated benchmarks cannot measure false-alert burden, latency, or temporal reliability.The discussion identifies false-alert burden, time-to-first-detection, and temporal reliability as deployment-relevant characteristics.
  • Architecture trade-offs: RT-DETR achieves the lowest mean latency and highest persistence, while YOLO detectors remain close behind at markedly higher throughput.The authors frame this as an accuracy-versus-compute trade-off rather than a uniform advantage for either architecture family.
  • Limitations: Dataset differences in scale, annotation protocol, and negative-frame prevalence prevent attributing transfer asymmetry to realism alone.The authors state that lesion-centric benchmarks remain unable to measure false-alert burden, latency, or temporal reliability regardless of size.
  • Limitations: The curated-versus-procedure contrast may confound negative-frame prevalence with domain difference, and detection is dominated by medium and large lesions with near-zero AP on small lesions.The authors propose an in-domain control using REAL-Colon lesion-bearing frames alone and note that hard-negative mining remains unexplored.

Supplementary Material … A.2 Software

The supplementary material provides reproducibility details for the study, including dataset-use declarations, fixed computational environments, hardware specifications, and software configurations. It also documents additional evaluations and releases data splits and per-video frame ranges supporting the reported analyses.

  • Supplementary Material: The supplement details the computational setup, data splits, extended REAL-Colon analyses, and confirmation of the main-paper ranking.Section A covers reproducibility; Section B releases REAL-Colon frame ranges and the SUN case-level split; Section C extends evaluation across localization, operating points, temporal reliability, runtime, and ablations.
  • Data use declaration.: The study uses REAL-Colon, SUN, and PICCOLO under their publishers’ academic research and benchmarking terms.Ethics approvals and consent procedures are those reported in the original dataset publications.
  • A Computational Setup: All experiments used two fixed workstations with pinned drivers and dependencies to make reported statistics reproducible.Workstation 1 handled all REAL-Colon experiments and final evaluations, while Workstation 2 was used only for SUN and PICCOLO training.
  • Workstation 1 (primary):: Workstation 1 provided two Intel Xeon Gold 5416S CPUs, 64 threads, 503 GiB memory, two NVIDIA H100 NVL GPUs, and local NVMe storage.Each H100 NVL GPU had 95 830 MiB VRAM.
  • Workstation 2 (supplementary training):: Workstation 2 used an Intel Xeon W-2265 CPU with 24 threads, 125 GiB memory, and two NVIDIA RTX A5000 GPUs.Each RTX A5000 GPU had 24 564 MiB VRAM.
  • A.2 Software: Both systems ran Ubuntu 24.04.3 LTS with CUDA 12.9, while WS1 used driver 575.57.08 and WS2 used driver 580.95.05.Two Conda environments were maintained for the study.
  • A.2 Software: The shared Ultralytics environment used Python 3.10.19, ultralytics 8.3.232, and PyTorch 2.6.0+cu124 for YOLOv8, YOLOv11, and RT-DETR.The WS1-only Detectron2 environment used Python 3.10.19, pycocotools 2.0.10, Detectron2 0.6, and PyTorch 2.5.1+cu121 for Faster R-CNN.

B Dataset Partitioning Details … C.2 AFROC Operating Curves

The paper provides reproducible dataset partitions and extends REAL-Colon evaluation with stricter localization, recall, object-size, and AFROC analyses. RT-DETR leads localization and sensitivity, especially when false alerts are rare.

  • B Dataset Partitioning Details: Exact frame- and case-level assignments for REAL-Colon and SUN are released to make the Table 1 partition reproducible.
  • B.1 REAL-Colon: Per-Video Frame Ranges: Only inside-patient frame-ID intervals are retained for each of the 60 REAL-Colon videos; out-of-patient insertion and withdrawal frames are discarded.
  • B.2 SUN: Case-Level Split Assignment: SUN is split at the case level across all 113 cases, preventing patient-level leakage into evaluation folds.
  • C Extended Experimental Results: The extended REAL-Colon experiments report mAP75, AR at 1, 10, and 100 detections per image, and COCO-style AP by object size using three seeds.
  • C.1 Detection Accuracy on REAL-Colon: RT-DETR leads every localization threshold and recall budget, reaching mAP75 = 0.359 and AR100 = 0.672 versus YOLOv8 at 0.299 and 0.471.RT-DETR’s AR rises from 0.415 at AR1 to 0.672 at AR100, indicating additional lesion recovery as the recall budget grows.
  • C.1 Detection Accuracy on REAL-Colon: Average precision is driven by medium and large lesions because small-polyp AP is near zero for all detectors.
  • C.2 AFROC Operating Curves: AFROC curves compare lesion-level sensitivity with false-positive fraction using model-specific thresholds near FPR ≈4–5% and a shared τ = 0.2.
  • C.2 AFROC Operating Curves: RT-DETR retains the highest sensitivity at the lowest false-positive fractions, while YOLO detectors approach it as the false-positive budget increases.Its ranking advantage is therefore widest in the low-false-alert regime emphasized as clinically important.

C.3 Per-Seed Confidence Threshold Analysis

Model-specific confidence thresholds provide a more appropriate and stable comparison than a shared threshold or per-seed refitting. At its own threshold, RT-DETR avoids the precision loss seen at τ = 0.2 and remains the top detector by frame-level F1 and F2.

  • Threshold selection strategies: Model-specific thresholds are tuned once to FPR ≈4–5%, whereas a shared τ = 0.2 or independently refitted per-seed thresholds represent alternative selection strategies.The reported model-specific thresholds are Faster R-CNN 0.20, YOLOv8 0.06, YOLOv11 0.05, and RT-DETR 0.30.
  • Threshold selection strategies: At τ = 0.2, RT-DETR precision falls to 0.398 while recall reaches 0.720; at its own threshold, precision recovers to 0.577 at 0.651 recall.The shared threshold causes RT-DETR to over-fire, motivating detector-specific operating points.
  • Detector ranking: RT-DETR attains the best frame-level F1 (0.681) and F2 (0.662), ahead of YOLOv8 with F1 = 0.648 and F2 = 0.621.This ranking matches the main paper once each detector uses its model-specific operating point.

Results

Threshold selection materially affects detector comparisons: a shared threshold penalizes RT-DETR, whereas model-specific thresholds align detectors at comparable false-positive rates. At matched operating points, RT-DETR achieves the best frame-level F1 and F2, while per-seed refitting only reduces variance.

  • Threshold strategies: A shared confidence threshold of τ = 0.2 penalizes RT-DETR because it over-fires at low τ.Model-specific and per-seed thresholds provide alternative operating strategies.
  • Threshold strategies: Model-specific thresholds place every detector at a comparable false-positive rate.Detection-level precision and recall are reported under shared, model-specific, and per-seed threshold strategies.
  • Frame-level performance: At matched operating points, RT-DETR attains the best frame-level F1 and F2.Per-seed refitting only reduces variance.

C.4 Lesion-Level Consistency and Early Detection · C.5 Runtime and Real-Time Suitability · C.6 Ablation Studies

The sections assess lesion-level persistence and detection latency, real-time throughput, and YOLOv11 design choices under controlled operating conditions. RT-DETR provides the strongest lesion-level consistency and early detection, while all detectors meet real-time operation on an H100.

  • C.4 Lesion-Level Consistency and Early Detection: For 21 REAL-Colon lesions, the analysis tracks whether each lesion is detected, the fraction of visible frames detected, and latency to first detection.A lesion counts as detected within a 1, 3, or 5 s window when at least 15 frames are detected.
  • C.4 Lesion-Level Consistency and Early Detection: All four detectors eventually detect every lesion, but persistence and latency differ substantially across architectures.Lesion 14, a zero-shot SSL subtype absent from training, has the highest miss rate across all architectures.
  • C.4 Lesion-Level Consistency and Early Detection: 17.3/21 lesions exceed 50% frame persistence with RT-DETR, which also has the earliest first detection at 38.3 frames.Convolutional detectors exceed the 50%-persistence threshold on at most 16/21 lesions; Faster R-CNN’s first-detection latency is 63.9 frames.
  • C.5 Runtime and Real-Time Suitability: Throughput is measured at batch size 1 on one NVIDIA H100 NVL GPU for preprocessing, inference, post-processing, and both inference-only and end-to-end FPS.Real-time operation is conventionally defined as ≥30 FPS.
  • C.5 Runtime and Real-Time Suitability: 181.8 FPS is YOLOv8’s end-to-end throughput, compared with 163.9 FPS for YOLOv11, 51.0 FPS for RT-DETR, and 37.5 FPS for Faster R-CNN.All detectors exceed the 30 FPS threshold; RT-DETR carries the highest accuracy and costs roughly 3× the throughput of YOLOv11.
  • C.6 Ablation Studies: The YOLOv11 ablations vary input resolution, model capacity, negative-frame sampling, and optimizer under model-specific confidence thresholds.Thresholds are selected to produce comparable frame-level FPR under the IoU > 0 criterion.

Input Resolution … C.8 Qualitative Detection Examples

Across supplementary analyses, 640 × 640 provides YOLOv11-M’s best accuracy–throughput trade-off, while balanced negative sampling and larger capacity improve REAL-Colon accuracy. Open detectors broadly match commercial sensitivity and latency, but shared failures remain on challenging visual conditions.

  • Input Resolution: 640 × 640 gives YOLOv11-M the best accuracy-to-throughput trade-off and is the resolution used in the main paper.Scaling to 1024 × 1024 adds only +0.016 mAP50 while reducing throughput from 163.9 to 143.8 FPS.
  • Input Resolution: At 300×300, all four detectors remain close to the SSD-300 reference mAP50 of 0.338, with RT-DETR highest at 0.432.TPR and FPR in this baseline-matched comparison are observational because the original thresholds are unspecified.
  • Model Capacity, Negative Sampling, and Optimizer: Larger YOLOv11 capacity improves REAL-Colon mAP50 from 0.353 for S to 0.384 for M and 0.420 for L.The M variant is retained as the accuracy-to-throughput compromise matching the 640 × 640 operating point.
  • Model Capacity, Negative Sampling, and Optimizer: A balanced 1:1 negative ratio is best, reaching mAP50 = 0.401; starving the model of background raises the confidence threshold to τ = 0.27 and costs sensitivity.Negative sampling is especially relevant to full-procedure data because negative frames vastly outnumber positives.
  • C.7 Comparison with Commercial CADe Systems: Open-detector first-detection latency is 1.2–2.1 s at model-specific thresholds and nominal 30 fps, broadly within the commercial range of 0.6–1.5 s.The reported takeaway is that false-positive control offers the main remaining headroom.
  • C.8 Qualitative Detection Examples: All four architectures localize clearly visible polyps, but their predictions diverge mainly on flat, non-polypoid, and NBI frames.The qualitative comparison uses representative REAL-Colon test frames across varying visual conditions.
  • C.8 Qualitative Detection Examples: Shared failures are dominated by specular highlights, motion blur, instrument occlusion, and flat lesions, marking the main headroom for full-procedure detection.No architecture is robust to these frames, which also underlie near-zero small-polyp AP and low-persistence lesions.

D Evaluation Metric Definitions … D.4 Runtime Metrics

The paper defines complementary detection-, image-, frame-, and runtime-level metrics for evaluating localization, false-alert burden, operating characteristics, and real-time feasibility. These definitions distinguish box-level matching from frame-level classification and formalize FROC/AFROC and latency-throughput measures.

  • D.1 Detection-Level Metrics: Detection-level metrics match predicted and ground-truth boxes one-to-one using an IoU threshold, defining TP, FP, and FN while leaving TN undefined.Precision, recall, and mAP evaluate localization accuracy of individual bounding boxes.
  • D.1 Detection-Level Metrics: Sensitivity equals recall, while Fβ combines precision and recall; β = 2 prioritizes recall because missed polyps carry greater clinical risk than false alarms.The β parameter controls the relative weighting of precision and recall.
  • D.2 FROC and AFROC: FROC extends ROC to multiple detections per image by measuring sensitivity against the average number of false positives per image, or FPPI.Standard ROC assigns one decision per image and therefore does not distinguish one false alarm from multiple false alarms within an image.
  • D.2 FROC and AFROC: AFROC replaces unbounded FPPI with the false-positive fraction, the fraction of negative images containing at least one false-positive alert above threshold.This confines the curve to the unit square and supports scalar summaries such as area under the AFROC curve while retaining sensitivity.
  • D.3 Frame-Level Metrics: Frame-level evaluation collapses detections into a binary alert per frame at confidence threshold τ, creating defined TPframe, FNframe, FPframe, and TNframe categories.A frame is positive when it contains a detection exceeding τ and satisfying the stated ground-truth overlap condition.
  • D.3 Frame-Level Metrics: Frame-level TPR, FPR, and specificity are computed from the resulting confusion matrix as TP/(TP + FN), FP/(FP + TN), and TN/(TN + FP), respectively.These metrics quantify frame-level sensitivity, false-alert rate, and correct rejection of negative frames.
  • D.4 Runtime Metrics: End-to-end image latency is timg = tpre + tinf + tpost, and throughput is FPS = 1000/timg[ms].Real-time operation is conventionally defined as ≥30 FPS, corresponding to end-to-end latency below 33 ms.
Loading 2608.13711v1…