Source-linked AI summary
Reliable Benchmarking of Artifact Detection in Computational Pathology: A Reproducibility and Uncertainty Analysis
Konstantinos Moutselos, Ilias Maglogiannis
TL;DR
Small, imbalanced whole-slide benchmarks make quality-control comparisons difficult to interpret because sampling, training, partition, metric, and preprocessing variability are inadequately characterized. The paper proposes a four-axis reliability protocol and applies it to an independent reconstruction of a diffusion-based artifact detector. The central mechanism reproduces, but comparative claims remain unresolved within evaluation uncertainty.
Problem
Whole-slide quality-control benchmarks often have few independent slides, concentrated annotation, pooled ratio metrics, and a single inherited partition, limiting interpretability of reported differences.
Method
The protocol quantifies test-set sampling, training stochasticity, partition composition, and undocumented preprocessing, requiring claims to survive all four axes.
Results
The auxiliary contrastive term yields a small but real improvement that replicates across independently trained models, while comparative claims do not survive evaluation uncertainty.
Takeaways & Limitations
Small-cohort benchmarks support weaker conclusions than ordinary reporting implies, while the four checks are cheap enough to accompany evaluations on such resources.
Takeaways & Limitations
The protocol is demonstrated on one method and one benchmark, and the reconstruction is not the authors' code with residual unspecified implementation differences.
Abstract
from arXiv · showhide
Background and Objective: Quality control is a prerequisite for whole-slide image analysis, yet the benchmarks on which quality-control methods are compared share four properties that make their reported differences hard to interpret: few independent slides, annotation concentrated in a minority of them, pooled ratio metrics with no closed-form standard error, and a single inherited train/test partition. We propose a reliability protocol for such benchmarks. Methods: The protocol quantifies four sources of variability - test-set sampling, training stochasticity, partition composition, and undocumented preprocessing - a claim is reportable only if it survives all four; three of the four cost minutes of compute. We apply it to an independent reconstruction of a published diffusion-based artifact detector, evaluated on the original 24-slide partition and against a supervised baseline. Results: The method's central mechanism reproduces: the auxiliary contrastive term improves pooled F1 from 0.673 to 0.688 and replicates under a second seed (+0.0156, p = 0.031; +0.0190, p = 0.005), although it acts on pen marking rather than the artifact types cited to motivate it. Its comparative claims do not: differences between design variants, and against the supervised baseline, fall inside the uncertainty of the evaluation. Four of 24 slides carry 70% of scored annotated pixels, giving an effective sample size of 6.2, and the inherited partition sits at the 7th percentile. An unreported tissue-restriction step excludes 41.4% of out-of-focus annotation against 2.6% of air bubble; such a gate is confounded with blur by construction. Conclusions: Small-cohort benchmarks support far weaker conclusions than current reporting implies. The four checks are cheap enough to accompany any evaluation on such a resource and separate reproducible effects from differences the evaluation cannot resolve.
1 Department of Digital Systems, University of Piraeus, Piraeus, Greece
The paper concerns reproducibility, uncertainty quantification, benchmarking, computational pathology, quality control, and whole-slide imaging.
- The paper focuses on reproducibility and uncertainty quantification in computational pathology benchmarking.
- Its application area is quality control for whole-slide imaging.
1. Introduction
Artifact-detection benchmarks can appear precise while resting on few independent slides, uneven annotation, pooled metrics, and one inherited partition. The paper proposes and applies a four-axis reliability protocol, finding that one mechanism reproduces but comparative claims do not.
- Artifact benchmarks often report point estimates on a single small partition without intervals, repeated training, or assessment across slide draws.
- The independent unit is the slide rather than the pixel, while uneven annotation means a minority of slides governs pooled-statistic variance.
- The protocol quantifies test-set sampling, training stochasticity, partition composition, and undocumented preprocessing, requiring claims to survive all four axes.
- The auxiliary contrastive term improves detection across independently trained models and is not explained by test-set sampling, although it acts on a different artifact type than originally invoked.
- Differences between design variants and against a supervised baseline fall inside evaluation uncertainty, with baseline ordering reversing without statistical support.
- The 24-slide test partition has an effective sample size of approximately six, and a single pooled F1 interval spans nearly 0.4.
- A tissue restriction excludes 41% of out-of-focus annotation versus 3% of air bubble, confounding the gate with the artifact by construction.
- The protocol is demonstrated on one method and one benchmark, without a claim of empirical generality beyond that case.
2. Materials and methods
The methods define when a reliability protocol is warranted, quantify uncertainty from small and imbalanced benchmarks, and test training, partition, preprocessing, and sampling variability.
- The protocol is intended for benchmarks with few independent units, concentrated target signal, pooled ratio metrics, and a single fixed partition.
- The Kish effective sample size converts unequal annotation weights into the number of equally weighted units with the same variance.
- Pooled pixel-level metrics lack closed-form standard errors, so tests designed for means of independent observations do not apply.
- Paired differences can remain resolvable when absolute metric levels are not, because pairing removes dominant between-unit variance.
- Bootstrap intervals estimate movement under a different draw of units, while paired resampling uses the same units for both configurations.
- A second training seed tests whether a claimed effect exceeds retraining noise, but two runs support comparison rather than an uncertainty interval.
- Partition composition is surveyed by random draws from the eligible pool, computing annotation, class counts, and effective sample size without retraining.
- Undocumented preprocessing is tested because routine operations can interact with the measured phenomenon and may shift per-class results through non-uniform exclusion.
1. Report paired differences with intervals rather than levels. Where levels are reported,
The protocol requires benchmark claims to report uncertainty, effective sample size, seed repeats, per-class coverage, and partition composition rather than relying on pooled levels alone. Its reconstruction also exposes substantial ambiguity in preprocessing and data partitioning.
- Paired differences should carry intervals and an explicit statement that they are not comparable across studies.
- Effective sample size, seed repeats, per-class coverage denominators, and partition percentiles are required alongside headline metrics.These checks distinguish cohort limitations, stochastic variation, and denominator imbalance.
- The reconstruction trains every plausible reading of ambiguous design choices, producing five models that share the same data pipeline and inference procedure.Ambiguities include contrastive-head placement, artifact sampling, and minimum annotation overlap.
- The imposed tissue restriction is an undocumented addition that changes both which pixels are scored and the per-slide Otsu threshold.Without a restriction, background false positives dominate; the restriction is not neutral across artifact types.
- Per-slide F1 varies by more than twenty-fold, while per-type sensitivities differ seven-fold, making pooled ratios sensitive to benchmark composition.Pooled F1 sums counts across slides before taking one ratio rather than averaging per-slide F1 values.
3. Results
The contrastive mechanism reproduces under independent retraining, but design variants and comparisons with the supervised baseline are not statistically resolvable. The benchmark is also strongly affected by tissue restriction and slide composition.
- 0.6725 to 0.6881: adding the contrastive term improves pooled F1 by +0.0156, with 95% CI [+0.0035, +0.0339] and p = 0.031.The gain also appears on artifact-bearing slides, indicating improved segmentation rather than only fewer clean-tissue false alarms.
- +0.0190 pooled F1 improvement replicates under an independently trained second seed, with 95% CI [+0.0072, +0.0390] and p = 0.005.The effect is four to five times the seed-to-seed variability of retraining the same configuration.
- +0.0385 and +0.0325 pen-marking sensitivity gains are the reproducible per-type effects; out-of-focus and folding changes cross zero.The contrastive term acts on pen marking rather than the artifact types cited as its motivation.
- None of the ambiguous design choices differs significantly on the pooled metric, although variant families trade gains in pen marking and folding against losses in out-of-focus.Out-of-focus accounts for 53% of annotated pixels, so these per-type changes cancel in pooled F1.
- The supervised baseline leads on all four artifact types, while the pooled comparison is not statistically resolvable.Its per-type advantages include +0.454 for pen marking, +0.391 for air bubble, and +0.071 for out-of-focus.
- 41.4% of out-of-focus annotation is excluded by the tissue restriction versus 2.6% of air-bubble annotation, lowering measured out-of-focus sensitivity from 0.892 to 0.527.The exclusion is concentrated in heavily blurred slides rather than being a property of the class alone.
4. Discussion
The reproducible result is narrow: the contrastive term improves pooled F1, but pooled metrics conceal artifact-specific trade-offs and the benchmark cannot resolve most comparisons. The protocol’s broader contribution is to expose limits imposed by concentrated annotations, partition choice, undocumented preprocessing, and narrow scope.
- 4.1 Implications for the method: Pooled metrics can hide behaviorally important trade-offs: a design choice may improve one artifact subgroup while harming another.The paper identifies this as a general hazard of compositionally skewed benchmarks.
- 4.2 Implications for the benchmark: Four of 24 slides carry 70% of scored annotated pixels, yielding an effective sample size of 6.2 and a pooled-F1 interval of roughly ±0.19.The benchmark’s binding constraint is the scored ground-truth concentration rather than any evaluated method.
- 4.3 Generality and limits of the protocol: The protocol’s four checks are cheap enough for routine use and can eliminate expensive experiments when preliminary variability already defeats a comparison.The partition survey cost minutes of processor time and removed the case for a 30-GPU-hour experiment.
- 4.3 Generality and limits of the protocol: The study demonstrates the protocol on one method and one benchmark, so its frequency across methods and benchmarks remains unestablished.Generalizing the frequency requires applying the protocol across several benchmarks and methods.
5. Conclusions
The protocol shows that benchmark-level uncertainty, concentrated ground truth, and undocumented preprocessing constrain conclusions about artifact-detector performance. It reproduces the detector’s central mechanism but not its comparative claims, while remaining inexpensive to apply.
- Reliability protocol: Four checks quantify test-set sampling, training stochasticity, partition composition, and undocumented preprocessing, requiring claims to survive all four.Three checks cost minutes of processor time; the fourth requires additional training.
- Mechanism reproducibility: The reconstructed detector reproduces a small improvement from its auxiliary contrastive mechanism across independently trained models.The improvement acts on pen marking rather than the artifact types cited as its motivation.
- Comparative claims: Design-variant and supervised-baseline comparisons remain inside evaluation uncertainty, with baseline ordering reversing without statistical support.The protocol therefore separates a reproducible mechanism from unresolved comparative differences.
- Implications: The benchmark, rather than any evaluated method, is the binding constraint, so reported conclusions are weaker than ordinary reporting implies.The protocol is presented as a practical way to distinguish real effects from differences the evaluation cannot resolve.
- Benchmark uncertainty: Four of 24 slides concentrate scored ground truth enough to yield an effective sample size of six and a confidence interval of ±0.19 for any single pooled F1.The inherited partition ranks at the 7th percentile of effective sample size among partitions of the same cohort.
- Undocumented preprocessing: An undocumented tissue restriction excludes 41% of out-of-focus annotation versus 3% of air-bubble annotation, confounding the gate with the artifact.Because the gate thresholds image properties whose degradation defines out-of-focus tissue, no adjustment within that family separates them.
Figure captions
The figures document reconstruction choices, qualitative outputs, uncertainty in paired F1 differences, slide-level variability, and artifact detection patterns by component size and type.
- Figure 1: Fig. 1 maps the reconstruction pipeline and marks unspecified implementation choices, including tissue restriction and thresholding.Tissue restriction determines both scored pixels and the Otsu-based decision threshold.
- Figure 2: Fig. 2 compares reconstructed models with GrandQC on two test slides from the original publication.GrandQC outputs include a dark spot / foreign object class without a counterpart among the four evaluated types.
- Figure 3: Fig. 3 shows paired differences in pooled pixel-level F1 with 95% percentile intervals and reports two-sided paired permutation tests.The reported panel p-values are 0.031, 0.005, 0.884, 0.993, 0.907, 0.404, 0.983, 0.553, and 0.463.
- Figure 4: Fig. 4 explains that pairing narrows intervals when runs agree slide by slide, especially for closely related configurations.Intervals against a different method are wider than those against a closely related configuration.
- Figure 5: 80.8% of annotated area lies above 10^5 pixels and belongs almost entirely to out-of-focus and pen marking, so pooled per-type sensitivity covers different size ranges.Folding and air bubble have no components above 10^5 pixels.
Declarations
The declarations report authorship, funding, ethics, competing interests, data and code availability, and licensing conditions for derived outputs and source masks.
- Competing interests: The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.
- Data and code availability: The reconstruction, evaluation, resampling, partition-survey, and diagnostic code and derived outputs are available in a Zenodo record.Per-slide confusion counts and bootstrap replicates are included, allowing reported intervals to be recomputed without whole-slide images.
- Licensing and authorship: The code is released under MIT and derived data under CC BY 4.0, while the manuscript reports author contributions and LLM-assisted drafting followed by review.
- Data availability: GrandQC masks are not redistributed in the deposit and must be obtained from the cited record under their own licence.The deposit contains only counts derived from those masks.
- Code availability: The archive states that it is not a reference implementation of the reconstructed method.
- Funding: The work was conducted within Project PRODIGY with Greek and European funding, and computing resources came from the University of Piraeus.
- Ethics: The study used publicly available, de-identified TCGA whole-slide images and released annotations; no new human data were collected.The authors state that ethics approval was not required.