Reliable Benchmarking of Artifact Detection in Computational Pathology: A Reproducibility and Uncertainty Analysis

• The authors are affiliated with the Department of Digital Systems, University of Piraeus, Piraeus, Greece.Source: §1 · p. 1
• Keywords: reproducibility, uncertainty quantification, benchmarking, computational pathology, quality control, and whole-slide imaging.Source: §1 · p. 2

1. Introduction

• Artifact benchmarks are vulnerable because slides, not pixels, define independence; annotations are unevenly distributed; pooled ratios lack closed-form standard errors; and partitions are inherited once.These conditions make precise-looking second-decimal comparisons uninterpretable despite large pixel counts.Source: §1 · p. 3
• The protocol requires claims to survive test-set sampling, training stochasticity, partition composition, and undocumented preprocessing variability before becoming reportable.Three checks cost minutes of compute and the fourth requires one additional training run, without additional data or code beyond reproduction.Source: §1 · p. 4
• The auxiliary contrastive term produces a small detection improvement that replicates across independently trained models, is not an artifact of test-set sampling, and acts on a different artifact type than the motivating one.Source: §1 · p. 4
• Differences between design variants and the supervised baseline fall inside evaluation uncertainty, with baseline ordering reversing without statistical support.The study does not interpret this as a finding about the method because the benchmark cannot resolve these comparisons.Source: §1 · pp. 4–5
• The 24-slide test partition has an effective sample size of approximately six, and any single pooled F1 interval spans nearly 0.4.Ground truth is concentrated in a handful of slides, so another slide draw can move results more than reported literature differences.Source: §1 · p. 5
• A tissue-restriction gate excludes 41% of out-of-focus annotation versus 3% of air-bubble annotation, shifting per-type sensitivity by 0.36.Because the gate thresholds image properties degraded by out-of-focus tissue, it is structurally confounded with blur and cannot be repaired by threshold adjustment.Source: §1 · p. 5

1.1 Related work on artifact detection

• Supervised segmentation models learn artifacts directly from exhaustive pixel annotation; GrandQC, trained on 420 densely annotated slides, is the reference example and serves as our baseline.It pairs a learned tissue-detection module (Dice 0.957) with a seven-class artifact segmenter and provides quality-control masks for the whole of TCGA.Source: §1.1 · p. 6
• Generative detectors fit clean tissue and flag poorly reconstructed regions, avoiding artifact labels and potentially detecting unseen failures, but poor reconstruction is not equivalent to artifact status.This approach has been applied from general out-of-distribution detection to brain MRI and whole-slide pathology, including comparisons with autoencoder- and GAN-based alternatives [21,22,23].Source: §1.1 · p. 7

1.2 Evaluation and uncertainty practice in biomedical image analysis

• Reported-measurement uncertainty concerns how pooled results change across test samples, training seeds, partitions, and preprocessing, rather than how confident a model is in individual predictions.Pooled ratios lack closed-form standard errors, are not means of per-case values, and can be dominated by a small minority of unevenly annotated cases.Source: §1.2 · p. 8
• The introduction identifies evaluation design as the unresolved source of comparison reliability in whole-slide quality control, where small benchmarks are asked to distinguish methods trained on vastly larger corpora.The paper states that this question has not been posed for whole-slide quality control and that standard practices for quantifying such variability are absent.Source: §1.2 · pp. 8–10

1.3 The method and benchmark examined

• The reproduced method uses a pathology foundation model as a generative prior, diffusion reconstruction error for artifact heatmaps, LoRA fine-tuning on clean tissue, and an auxiliary contrastive objective.Three implementation choices are underspecified: the contrastive representation, artifact-patch sampling, and minimum annotation overlap; the evaluation reports latent separation, pixel-level F1, per-type sensitivity, and comparison with GrandQC.Source: §1.3 · p. 10
• The benchmark defines eight artifact classes but evaluates four—out-of-focus, pen marking, tissue folding, and air bubble—which the studied method adopts.Annotations are whole-slide scale but non-exhaustive; pen marking outside tissue is frequently unmarked.Source: §1.3 · p. 11
• 16 training and 24 test slides are selected from 42 usable cases, while the full 50-slide AIRAQC test set contains 25 slides with artifacts and 25 without.The studied method therefore partitions its evaluation set into training and test material.Source: §1.3 · p. 11
• Because the method is evaluated on more slides than it is trained on, the benchmark setup is unusual and shapes the study's subsequent analysis.The inherited partition is the original work's partition, which the study adopts.Source: §1.3 · p. 11

2.1 A reliability protocol for small-cohort benchmarks

• The reliability protocol is intended for benchmarks where small cohorts, concentrated signal, pooled ratios, and fixed partitions make apparently precise conclusions unsupported.Its four components are cheap relative to the training they accompany.Source: §2.1 · p. 11, §2.1.1 · p. 12

2.1.1 When a benchmark needs this

• A benchmark needs the protocol when slides, patients, or acquisitions—not pixels or patches—are the independent units and target annotations are concentrated in a minority.Pooled ratio metrics lack closed-form standard errors, while a single inherited partition makes results conditional on one split; the protocol is unnecessary for large, evenly sampled tests.Source: §2.1.1 · p. 12

2.1.2 Axis 1 — Test-set sampling

• Resampling test units with replacement and stratification yields percentile intervals for pooled ratios, while paired resampling and sign-flip permutation tests assess within-study differences.Pairing can make differences resolvable even when absolute levels are not, but comparisons with published figures cannot be paired without their per-unit counts.Source: §2.1.2 · p. 13

2.1.3 Axis 2 — Training stochasticity

• Retraining one configuration under a second seed tests whether an effect exceeds training noise, because test-set stability alone cannot establish design-level reproducibility.Two runs support comparison rather than an interval; an effect smaller than retraining variation belongs to a training run, not necessarily the design choice.Source: §2.1.3 · p. 14

2.1.4 Axis 3 — Partition composition

• A few hundred random partitions surveyed at fixed proportions quantify composition, target coverage, and effective sample size without retraining, locating the observed split by percentile.Substantial test-side variation shows that some uncertainty belongs to the particular draw, while an unfavorable effective-sample-size percentile widens intervals beyond the cohort’s inherent requirement.Source: §2.1.4 · pp. 14–15

2.1.5 Axis 4 — Undocumented preprocessing

• The preprocessing audit enumerates unspecified operations, tests defensible variants, measures metric spread, and checks whether exclusions affect subgroups unevenly.Unevenly excluded ground truth is not counted as false negatives, so resulting per-class figures use different denominators and cannot support unbiased class comparisons.Source: §2.1.5 · pp. 15–16
• Undocumented preprocessing can be confounded when a gate operates on a property altered by the phenomenon under study, producing non-uniform exclusion by construction.This axis measures consequences of an interpretive choice rather than chance and requires only re-evaluation under an alternative variant.Source: §2.1.5 · p. 16

2.1.6 Decision rule and reporting

• A quantitative claim is reportable only if it clears the paired resampling interval, retraining noise, partition dependence, and unspecified implementation choices.Claims that clear some but not all axes should be reported with the failed axis named, rather than withheld or asserted.Source: §2.1.6 · p. 16

1. Report paired differences with intervals rather than levels. Where levels are reported,

• The protocol requires paired differences with intervals, effective sample size, seed repeats, per-class denominator coverage, and partition percentiles, with claims reportable only after surviving all four axes.Levels must carry intervals and be explicitly identified as non-comparable across studies; null seed effects remain informative.Source: §1 · pp. 16–17

2.3 Data and ground truth

• The benchmark uses 16 training and 24 test slides drawn from 42 eligible slides, yet 16 plus 24 is 40 against a stated cohort of 50, leaving 10 unaccounted; evaluation covers four artifact classes and pools pixel-level metrics.Annotations include 24,440 pixels with an exact colour outside the four evaluated classes; metrics are computed at pixel level inside the refined coverage mask and pooled.Source: §2.3 · pp. 19–20

2.4 Instantiating the protocol

• The benchmark uses 24 inherited test-partition slides with heavily concentrated annotations; reported metrics are pooled ratios, and compared configurations differ in the second decimal place of pooled F1.In 20,000 replicates, slides are drawn with replacement.Source: §2.4 · pp. 20–21
• The protocol instantiation varies only the seed for matched-step comparisons, surveys 200 alternative partitions, and defines tissue restriction through patch and pixel-level masks.The tissue criterion is weaker than prescribed: reported denominator-only spread is a lower bound on total preprocessing variability, while the patch threshold is a tuned undocumented degree of freedom.Source: §2.4 · p. 21

2.5 Component-level analysis

• Per-slide F1 varies more than twenty-fold and per-type sensitivities seven-fold, motivating connected-component analysis because artifact type and scale are confounded at slide level.Each component is treated as one observation using its size, type, and predicted coverage within the refined metric mask.Source: §2.5 · p. 23

2.6 Supervised baseline

• GrandQC is compared using released masks on 21 of 24 slides, with paired scoring under the same coverage mask and measured label mapping.The three missing slides are clean frozen sections, so artifact slides remain represented while clean-slide false-positive-rate power is reduced.Source: §2.6 · pp. 23–24

2.7 Verification

• Three verification errors, including one that would have produced a plausible component-analysis result, were caught by asserting that per-slide counts re-pool to saved aggregates.Thresholds are stored with each result and evaluations can be replayed deterministically.Source: §2.7 · p. 24

3.1 The mechanism reproduces

• 0.6725 to 0.6881 pooled F1 follows adding the contrastive term, replicates under a second seed at +0.0190, and is confined to pen marking.The first-seed difference is +0.0156, 95% CI [+0.0035, +0.0339], p = 0.031; the second is p = 0.005, while out-of-focus and tissue folding cross zero.Source: §3.1 · p. 25

3.2 What does not survive

• 0.6761, 0.6727, and 0.6753 for alternative design choices do not differ significantly from the λ = 0 baseline, despite type-specific trade-offs.Variant B gains on pen marking and folding but loses on out-of-focus, which accounts for 53% of annotated pixels; pooled F1 hides these differences.Source: §3.2 · pp. 25–26

3.3 The supervised baseline

• The best configuration scored 0.688 (0.692 in the second seed); although the baseline leads on all four artifact types, pooled ordering is not statistically resolvable.Source: §3.3 · p. 27

3.4 What the benchmark can support

• 41.4% of out-of-focus annotation is excluded versus 2.6% of air-bubble annotation, reducing measured out-of-focus sensitivity from 0.892 to 0.527.The tissue restriction excludes 28.6% of all annotated pixels; the tissue mask accounts for 40.7 percentage points of the out-of-focus loss.Source: §3.4 · p. 29
• The coverage mask excludes 41.4% of pooled out-of-focus annotation, with exclusion tracking blur severity rather than class label; under fixed stain, scanner, and thresholds, retention ranges from 49.0% to 88.8%.Source: §3.4 · pp. 29–30
• 3.88 effective test-set samples and a 7th-percentile partition make benchmark estimates unstable rather than fixed properties of a method.The largest slide contains 45% of ground truth, while the benchmark’s estimator is unstable enough that absolute figures should not be treated as method properties.Source: §3.4 · pp. 30–31
• Only 96k air-bubble pixels across two slides are scored, with an interval of [0.002, 0.950], despite 15.6M training pixels for that type.The training partition therefore heavily represents a type that is effectively unmeasurable at evaluation.Source: §3.4 · p. 31

3.5 Scale as a hidden denominator

• 0.776, 0.538, 0.390, and 0.000 are recovered for out-of-focus, pen marking, folding, and air bubble, respectively, among 1,000–10,000-pixel components.At 100,000–1,000,000 pixels, out-of-focus scores 0.916 versus pen marking’s 0.486, while folding and air bubble have no components there.Source: §3.5 · pp. 31–32
• 0.1% of components above 100,000 pixels carry 80.8% of annotated pixels, so pooled per-type sensitivity measures different object-size ranges across artifact types.Out-of-focus and pen marking are measured mainly on very large regions, whereas folding and air bubble are measured on small ones, making their values non-comparable.Source: §3.5 · p. 32

4.1 Implications for the method

• A +0.016 to +0.019 pooled-F1 effect replicates under an independent seed, excludes zero, and is four to five times same-configuration run variation.It is the only claim that clears all four reliability axes, although the practical gain is small relative to a roughly nineteen-point metric interval.Source: §4.1 · pp. 32–33
• The contrastive term improves pen marking, while raw-latent placement also improves folding but degrades out-of-focus detection, effects pooled F1 can conceal.The result appears where the out-of-distribution signal is strongest, and the mechanism cannot be determined from this experiment.Source: §4.1 · p. 33

4.2 Implications for benchmark design and reporting

• Four of 24 slides carry 70% of scored pixels, yielding an effective sample size of 6.2 and a roughly ±0.19 interval for pooled F1.Across 200 random partitions, the test side varies substantially and the inherited partition lies at the 7th percentile of effective sample size.Source: §4.2 · pp. 34–35
• An unspecified tissue restriction can shift a per-type figure by 0.36, while cheap appearance heuristics remain structurally confounded with out-of-focus tissue.Broader saturation-and-luminance gating does not solve the problem because optical blur changes both properties through the same physical mechanism.Source: §4.2 · pp. 34–35
• Artifact size and type occupy largely separate ranges, so pooled per-type sensitivities compare different size distributions rather than equivalent difficulty.Reporting detected fraction by component-size band, component counts, and annotation size distribution makes results comparable across benchmarks at negligible additional computation.Source: §4.2 · p. 35

4.3 Generality and limits of the protocol

• The protocol is demonstrated on one method and benchmark, but resampling, seed repetition, and partition surveying transfer mechanically across domains.Auditing undocumented preprocessing requires domain knowledge; the transferable check is to ask whether the gated image property is altered by the phenomenon under study.Source: §4.3 · p. 36
• Three of four comparative claims did not survive on the single benchmark examined, but this case cannot establish how frequently published claims would fail.The authors frame the contribution modestly: where the four conditions hold, the checks are cheap enough to be routine and materially changed conclusions in this case.Source: §4.3 · p. 37

4.4 Limitations

• The reconstruction may differ from the original implementation, including a training set containing only a twenty-ninth of the reported clean patches because unspecified count parameters were unavailable.Residual undocumented choices cannot be enumerated, despite training documented ambiguities under every interpretation.Source: §4.4 · p. 37
• Seed variance used two runs of one configuration, partition variance was surveyed without retraining, and the preprocessing axis decomposed one restriction without testing an alternative.The reported spread is therefore a lower bound, and the recommended tissue segmentation was not tested for retaining blurred tissue.Source: §4.4 · p. 37
• The supervised baseline was unavailable for three clean frozen-section slides, reducing power for the clean-slide false-positive rate.This limitation specifically weakens that comparison rather than the entire evaluation.Source: §4.4 · p. 37
• The evaluation covers fifty slides from one retrospective archive, one stain, and one annotation source, so it does not address immunohistochemistry, prospective material, or annotation-team variability.Although the slides span four cancer types, contributing sites and scanning systems, the preparation and digitisation may not resemble routine clinical practice.Source: §4.4 · p. 38

Reliability protocol

• The protocol quantifies test-set sampling, training stochasticity, partition composition, and undocumented preprocessing, requiring claims to survive all four checks.Three checks cost minutes of processor time, while the fourth concerns training.Source: §5 · p. 38

Empirical findings

• The auxiliary contrastive term reproducibly improves performance, but design-variant and supervised-baseline comparisons remain unresolved by evaluation uncertainty.Its effect acts on pen marking rather than the artifact types motivating the method, and the baseline ordering reverses without statistical support.Source: §5 · p. 38

Benchmark limitations

• Four of 24 slides yield an effective sample size of six and ±0.19 uncertainty for pooled F1, while the inherited partition lies at the 7th percentile.This interval contains essentially every figure published on the resource.Source: §5 · pp. 38–39

Undocumented preprocessing

• An undocumented tissue restriction excludes 41% of out-of-focus annotation versus 3% of air-bubble annotation, distorting comparisons between artifact types.Because the gate measures image properties whose degradation defines out-of-focus tissue, gate and artifact are confounded by construction.Source: §5 · p. 39

Implications

• The checks preserve the benchmark and confirmed method while separating reproducible effects from differences the evaluation cannot resolve.They cost a small fraction of the training they accompany and generalize beyond this benchmark, method, or modality.Source: §5 · p. 39

Fig. 1

• Two undocumented choices—the artifact-patch sampling scheme and minimum annotation overlap—were ablated without measurable effect and omitted from the pipeline figure.Diamonds identify choices left unspecified in the published account, with italic annotations describing consequences measured in this study.Source: Figure captions · pp. 40–41
• The tissue-restriction choice has the largest measured consequence because it changes both which pixels are scored and the Otsu-derived decision threshold.The pipeline figure represents this restriction as one node even though it acts at two points.Source: Figure 1 · p. 40, Figure captions · p. 41

Fig. 2

• GrandQC comparisons require two readings on the test slides because excluding its unmatched dark-spot channel costs 0.074 on TCGA-69-7979 but is negligible on TCGA-86-8075.The reconstructed models emit binary artifact/not-artifact outputs, whereas GrandQC distinguishes classes.Source: Figure captions · p. 41, Figure 2 · p. 41

Fig. 3

• Only variant A separates from the basic model under both seeds, whereas paired differences against variant A are substantially narrower than comparisons with a different method.The paired permutation p-values are 0.031 and 0.005 for the two seed-supported separations, versus 0.404, 0.983, 0.553, and 0.463 for selected runs against variant A.Source: Figure captions · p. 42, Figure 3 · p. 42

Fig. 4

• F1 ranges from 0.03 to 0.95 across slides, while configurations nearly overlap within rows, making differences more precise than pooled performance levels.The pooled value is a ratio of summed counts dominated by a few large slides, so it does not represent a typical slide’s performance.Source: Figure captions · p. 43

Fig. 5

• Artifact types occupy different size ranges: 80.8% of annotated area lies above 10^5 pixels, almost entirely from out-of-focus and pen marking.Folding and air bubble have no components above 10^5 pixels, so pooled per-type sensitivity summarizes different size distributions for each type.Source: Figure captions · p. 44, Figure 5 · p. 44

Declarations

• The authors report no known competing financial interests or personal relationships that could have influenced the work.Source: Declarations · p. 45
• Evaluation outputs, bootstrap replicates, partition surveys, diagnostics, and reconstruction code are deposited at Zenodo, with parameters enabling deterministic replay.The deposit contains derived counts rather than redistributed GrandQC masks, and explicitly states that the reconstruction is not a reference implementation.Source: Declarations · p. 45
• Publicly available, de-identified TCGA images and released annotations were used; no new human data were collected, so ethics approval was not required.Source: Declarations · p. 46

Declaration of generative AI and AI-assisted technologies in the manuscript preparation

• A large language model assisted with drafting and revising the manuscript; the authors reviewed the content and retain full responsibility for it.Source: Declaration of generative AI and… · p. 46