PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation

Siddharth Gupta, Jitin Singla

arXiv:2609.10495v1cs.CVcs.LG

TL;DR

Because ground-truth annotations are unavailable during real-time colonoscopy, polyp segmentation failures can go undetected. The paper proposes RBQE, which estimates reliability from agreement between a primary model and an independently trained referee, and finds that independent training is useful while cross-architecture agreement performs best. RBQE also supports selective rejection of low-agreement predictions, with limitations in referee availability, shared failure modes, and demonstrated scope.

  • Problem

    Ground-truth annotations are unavailable at inference, so unreliable automatic polyp segmentations cannot be directly assessed using standard metrics.

  • Method

    RBQE estimates segmentation reliability from agreement between a primary segmentation model and an independently trained referee on the same image.

  • Results

    SegFormer-B0 achieved ROC-AUC = 0.960, exceeding the same-architecture independent referee at ROC-AUC = 0.923; excluding empty-mask cases reduced these values to 0.876 and 0.783, respectively.

  • Takeaways & Limitations

    RBQE provides a reference-free, deployment-time reliability signal using one additional referee forward pass and can support selective prediction by rejecting low-agreement cases.

  • Takeaways & Limitations

    RBQE requires a suitable referee model, and independently trained models can share failures that agreement-based estimation cannot detect.

Abstract

from arXiv · show

In real-time colonoscopy, ground-truth annotations are unavailable at inference, so polyp segmentation models can fail silently. We propose Referee-Based Quality Estimation (RBQE), a reference-free framework measuring agreement between a primary segmentation model and an independently trained referee on the same image. RBQE is evaluated on a standardized 1,223-image external benchmark drawn from four public datasets, using four referee configurations chosen to separate two design axes: referee independence and architectural diversity. Using a common Agreement Dice descriptor, a same-architecture referee differing from the primary model only in random initialization already yields a useful reliability signal (ROC-AUC = 0.923), showing that independent training alone is sufficient. Cross-architecture referees improve further: SegFormer-B0 achieves the strongest performance (ROC-AUC = 0.960), significantly outperforming the same-architecture control and UNet++, and exceeding a representative Test-Time Augmentation baseline by 0.055 ROC-AUC under an identical protocol, whereas a prompt-coupled MedSAM referee underperforms despite maximal architectural diversity. Because empty-mask agreement is trivially separable, we also report a restricted evaluation excluding such cases: ROC-AUC falls to 0.876 (SegFormer-B0, 1,046 images) and 0.783 (same-architecture control, 975 images), yet RBQE's margin over both baselines widens on this identical subset. RBQE additionally increases the mean Dice of retained predictions as low-agreement cases are progressively rejected, supporting selective prediction, and requires only one additional deterministic referee forward pass at inference. Our study therefore supports cross-model agreement as a practical, interpretable reliability framework for automated polyp segmentation.

1. Introduction

Automatic polyp segmentation can fail under challenging imaging conditions, yet ground truth is unavailable during inference for detecting unreliable predictions. RBQE addresses this gap by estimating reliability from agreement between independently trained models, with independence and architectural diversity evaluated separately.

  • Motivation: 26% adenoma miss rate was reported across more than 15,000 tandem-colonoscopy procedures, highlighting variability in colonoscopy-based detection.The cited meta-analysis included 43 tandem-colonoscopy studies.
  • Motivation: Segmentation models fail under camera motion, poor illumination, artifacts, blur, small polyps, and domain shift, while inference lacks ground truth for direct quality measurement.Standard DSC and IoU therefore cannot flag unreliable predictions at deployment.
  • Proposed framework: RBQE estimates reliability from agreement between a primary segmentation model and an independently trained referee applied to the same image.The referee design separates training/output independence from architectural diversity.
  • Principal findings: ROC-AUC = 0.923 for the same-architecture control shows that independent training alone provides a meaningful reliability signal.The control differed from the primary model only in random initialization.
  • Principal findings: ROC-AUC = 0.960 for SegFormer-B0 was the strongest reported result, significantly outperforming the same-architecture control and UNet++.A prompt-coupled MedSAM referee underperformed despite maximal architectural diversity, indicating that diversity without output-level independence is insufficient.
  • Evaluation protocol: ROC-AUC changed from 0.960 to 0.876 for SegFormer-B0 and from 0.923 to 0.783 for the same-architecture control when empty-mask cases were excluded.The restricted protocol was introduced because empty-mask agreement is trivially separable, and RBQE’s margin over evaluated baselines widened on that subset.

2. Related Work

Related work includes single-model uncertainty and quality estimation, ensemble disagreement, reverse-classification approaches, and selective prediction. RBQE is positioned within agreement-based SQE while targeting independently trained, architecturally heterogeneous models without per-image retraining.

  • SQE motivation: Ground-truth annotations are unavailable during real-time inference, motivating Segmentation Quality Estimation without access to ground truth.This is the central deployment setting for no-reference SQE.
  • Single-model approaches: Single-model SQE methods use image and mask features, uncertainty, Bayesian approaches, or related model behavior to estimate segmentation quality.Existing methods include CNN-based predictors, predictive uncertainty, and Bayesian deep learning.
  • Agreement-based approaches: Deep ensembles estimate uncertainty from disagreement among same-architecture models, while reverse classification fits a new classifier per test case.Random initialization can create function-space diversity, but shared architectural bias may preserve correlated failures.
  • Selective prediction: Selective classification formalizes the risk–coverage trade-off between prediction coverage and error rate, which RBQE evaluates using its continuous agreement score.RBQE applies this framework beyond binary reliability estimation.

3. The RBQE Framework

RBQE compares primary and referee masks from the same image, extracts complementary agreement descriptors, and converts their consistency into a post-hoc reliability signal. The framework uses independently trained referees spanning same- and cross-architecture settings and handles degenerate masks explicitly.

  • Framework overview: RBQE measures agreement between two independently trained segmentation models applied to the same input image.The models may share an architecture while differing in initialization or may also differ in architecture and training procedure.
  • Framework overview: The pipeline generates the primary mask, obtains an independent referee prediction, extracts agreement features, and estimates reliability from cross-model agreement.This sequence leaves the deployed primary model unchanged.
  • Agreement descriptors: Agreement descriptors capture regional overlap, predicted lesion size, boundary similarity, and spatial localization.Agreement Dice and IoU measure overlap; Area Ratio, Boundary Agreement, and Centroid Distance provide complementary consistency views.
  • Agreement descriptors: Agreement Dice measures overlap between primary and referee masks, with larger values indicating stronger regional agreement.The descriptor uses foreground-pixel counts and mask intersection.
  • Agreement descriptors: Agreement IoU is retained for interpretability, but its monotonic relationship with Agreement Dice gives identical rank orderings and ROC-AUC values.It is not an independent source of evidence.
  • Agreement descriptors: Area Ratio evaluates predicted lesion-size consistency independently of spatial overlap, while Boundary Agreement emphasizes contour alignment.Area Ratio is insensitive to small spatial shifts; Boundary Agreement uses a 3-pixel correspondence tolerance.
  • Agreement descriptors: Centroid Distance measures spatial consistency between predicted masks using centroid separation normalized by the image diagonal.Smaller distances indicate greater localization agreement and are comparable across image resolutions.
  • Degenerate cases: Degenerate masks receive fixed agreement values, with Agreement Dice, Agreement IoU, and Boundary Agreement set to 0 and both-empty Area Ratio set to 1.0.These trivially separable cases motivate full and non-degenerate evaluation.

4. Experimental Protocol

The study evaluates RBQE on an external, multi-dataset benchmark using standardized failure definitions, metrics, baselines, and implementation settings. It also tests degenerate-case sensitivity and compares referee and competing quality-estimation configurations under matched protocols.

  • Datasets: The external benchmark combines four independent public datasets spanning varied image quality, lesion characteristics, and segmentation difficulty.The datasets contain 612, 380, 196, and 60 images, respectively, and were not used for model training.
  • Benchmark construction: RBQE evaluation uses a full standardized benchmark of 1,223 images and restricted referee-specific subsets excluding degenerate empty-mask cases.The restricted protocol addresses the artificial separability caused by agreement on empty masks.
  • Failure definition and metrics: Segmentation failures are detected as a binary quality-estimation problem using predicted-mask overlap with ground truth, with threshold sensitivity assessed around the primary failure definition.Dice-based quality is computed from the predicted mask and ground-truth mask, while RBQE itself produces a continuous reliability score.
  • Failure definition and metrics: ROC-AUC is the primary threshold-independent metric, with accuracy, precision, recall, bootstrap confidence intervals, and paired DeLong tests providing complementary evaluation.Confidence intervals use 1,000 image-level bootstrap resamples, and statistical significance uses p < 0.05.
  • Baselines: The comparison includes Test-Time Augmentation and morphology-based quality estimation baselines evaluated with the same failure definition, benchmark, ROC-AUC analysis, and statistical protocol.The baselines are also recomputed on the identical 1,046-image restricted subset used for SegFormer-B0.
  • Implementation: All referee architectures use the corresponding original or official implementations, standardized preprocessing, model-specific resizing, and consistent probability-map resampling before mask thresholding.The primary model is YOLOv8n-Seg; evaluated referee architectures include SegFormer-B0, UNet++, MedSAM, and an Independent YOLO Referee.

5. Results

RBQE uses cross-model agreement to estimate segmentation reliability without inference-time ground truth. Results show that independent training provides a useful signal, architectural diversity can improve it, and performance remains informative beyond trivially separable empty-mask cases.

  • Agreement descriptor evaluation: Agreement Dice and Agreement IoU achieved the strongest correlation with ground-truth DSC (ρ = 0.734), leading to Agreement Dice's adoption as the primary descriptor.Centroid Distance also performed strongly, whereas Area Ratio was near chance and was not recommended as a standalone descriptor.
  • Agreement descriptor evaluation: Strong agreement between independently trained primary and referee masks generally indicated successful segmentation, while substantial disagreement was associated with failure.RBQE derives reliability from consistency between independent predictions rather than repeated evaluation of one model.
  • Independence versus architectural diversity: 0.923 ROC-AUC was achieved by the same-architecture Independent YOLO Referee, while SegFormer-B0 reached 0.960 and UNet++ 0.938.SegFormer-B0 significantly outperformed both the Independent YOLO Referee (p < 0.001) and UNet++ (p = 4.29 × 10−3).
  • Independence versus architectural diversity: 0.863 ROC-AUC was achieved by MedSAM, which underperformed the same-architecture control because its prompt-derived output was coupled to the primary model.Architectural diversity improved the signal only when accompanied by genuine output-level independence.
  • Restricted evaluation: 0.876 ROC-AUC remained for SegFormer-B0 on the 1,046-image non-degenerate subset, compared with 0.960 on the full benchmark.The restricted protocol excludes trivially separable empty-mask cases, while the reported RBQE margin over evaluated baselines widens on this subset.
  • Comparison with existing no-reference SQE methods: 0.960 ROC-AUC for RBQE exceeded the protocol-matched TTA baseline by 0.055 and the morphology-based baseline by 0.033 on the full benchmark.When empty-mask cases were excluded, RBQE fell from 0.960 to 0.876, while morphology-based SQE fell from 0.927 to 0.709 and TTA from 0.905 to 0.621.

6. Discussion

Cross-model agreement provides a low-cost, reference-free reliability signal that distinguishes reliable from unreliable segmentations without inference-time ground truth. Its practical value extends from failure detection to selective prediction, while shared biases and output coupling remain important caveats.

  • Reliability signal: Cross-model agreement consistently distinguished reliable from unreliable segmentations across four referee configurations and a 1,223-image external benchmark without ground-truth annotations at inference.The evaluation also verified headline results on non-degenerate restricted subsets.
  • Mechanism: Agreement Dice and Agreement IoU were the strongest descriptors, followed by Centroid Distance and Boundary Agreement, because they measure spatial and geometric consistency between separately parameterized models.Shared dataset exposure and systematic biases can still produce correlated errors.
  • Comparison with existing approaches: Morphology-based descriptors capture geometric regularity but may miss incorrectly localized predictions, while single-model TTA remains vulnerable when the model is confidently wrong.This positions cross-model agreement as complementary to existing no-reference quality-estimation paradigms.
  • Clinical relevance: Rejecting the lowest-agreement half of predictions raises mean retained DSC from 0.731 to 0.911, and retaining only the top 10% raises it to 0.945.RBQE can therefore support tunable triage between automation coverage and diagnostic risk using one additional referee forward pass.

7. Limitations

The study’s evidence is bounded by its binary polyp-segmentation setting, referee and independence requirements, deterministic mask-based descriptors, retrospective benchmarks, and whole-prediction rather than localized reliability estimates.

  • Scope: RBQE was evaluated exclusively on binary polyp segmentation, so generalizability to multiclass segmentation and other medical imaging modalities remains unestablished.The limitation concerns both task type and imaging domain.
  • Referee availability and independence: Deployment in a new clinical setting requires training an additional referee or identifying a suitable pretrained model, unlike single-model uncertainty methods.The framework also assumes independently trained models fail in different ways, an assumption challenged by high-agreement failure cases.
  • Descriptor design: The deterministic geometric descriptors operate directly on binary masks and omit probabilistic outputs, feature-space similarity, and topology-aware representations.Those omitted representations may improve sensitivity to subtler failure modes, but their benefit is not established here.
  • Validation setting: The evaluation is retrospective, so prospective multicenter validation across broader acquisition hardware and patient populations is needed before routine clinical deployment.The public benchmarks may not capture the full variability of future clinical settings.
  • Localization: RBQE estimates reliability for a prediction as a whole but does not identify which region is unreliable.Localized pixel-level uncertainty maps are proposed as a possible complement.

8. Conclusion

RBQE estimates deployment-time segmentation reliability from agreement between an independently trained referee and the primary model. Independence alone is informative, while architectural diversity can improve performance, with practical efficiency and selective-prediction benefits.

  • RBQE provides reference-free, deployment-time reliability estimation without ground-truth annotations, quality-labelled training data, or changes to the deployed segmentation model.
  • ROC-AUC = 0.923 for the Independent YOLO Referee and 0.960 for SegFormer-B0 separate the contributions of independence and architectural diversity.The same-architecture referee differs only in initialization, whereas SegFormer-B0 is architecturally distinct.
  • ROC-AUC falls from 0.960 to 0.876 for SegFormer-B0 and from 0.923 to 0.783 for the Independent YOLO Referee when trivial empty-mask cases are excluded.On the identical restricted subset, RBQE’s margin over evaluated baselines widens rather than narrows.
  • RBQE outperforms morphology-based and TTA-based baselines by 0.033 and 0.055 ROC-AUC, supports selective prediction, and requires one additional deterministic forward pass.
  • The results support independently trained model agreement as a practical, interpretable, and computationally efficient reliability signal for automated polyp segmentation.The authors identify extension to multiclass and multimodality segmentation as future work.

Ethics statement

The study used publicly available, pre-existing, de-identified colonoscopy image datasets and involved no new human-participant data collection.

  • All colonoscopy image datasets were publicly available, pre-existing, and de-identified, so no new ethics approval was required.

Declaration of generative AI and AI-assisted technologies in the writing process

The authors used Claude to improve manuscript grammar, language, and readability, then reviewed and edited the resulting text.

  • Claude assisted with manuscript language and readability, while the authors retained responsibility for the publication’s content and results.

CRediT authorship contribution statement

Siddharth Gupta led the study’s conceptualization, methodology, software, analysis, investigation, and original drafting, while Jitin Singla contributed conceptualization, supervision, and review.

  • Siddharth Gupta handled conceptualization, methodology, software, formal analysis, investigation, and the original draft.
  • Jitin Singla contributed conceptualization, supervision, and writing review and editing.
Loading 2609.10495v1…