PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation Choices Shape Biomedical ML Claims: A Pediatric Pneumonia Benchmark Case Study

Bhanu Prakash Vangala, Sowmya Guda, Latha Peddi, Navya Vangala

arXiv:2609.37848v1cs.CVcs.LG

TL;DR

Biomedical ML scores are often read as properties of models even though evaluation choices also shape them. This paper compares those effects in a controlled Kermany pediatric radiograph case study. Its findings show that protocol choices can matter as much as or more than architecture choice, so benchmark scores need their evaluation context.

  • Problem

    A headline biomedical ML score can be read as a property of the model even though it also depends on how the benchmark was evaluated.

    Source: §1 · p. 1

  • Method

    The paper conducts a controlled case study on the Kermany pediatric radiograph dataset, comparing architecture effects with changes in evaluation protocol.

    Source: §1 · p. 2

  • Results

    0.044 AUROC on average separates frozen from fine-tuned backbones, compared with a 0.026 spread among pretrained architectures under one fixed protocol; split, threshold, and file-only results also reveal evaluation-dependent differences.

    Source: §1 · p. 2, §4.2 · p. 6, §4.5 · p. 7

  • Takeaways & Limitations

    A benchmark score reflects both model and protocol, so the split, metric, threshold, uncertainty, and observed shift should travel with the score.

    Source: §5 · p. 8, §6 · p. 9

    This is a case study of one public pediatric radiograph dataset; the released metadata do not identify the source of the train-to-test shift, and the small test set limits architecture comparisons.

    Source: §6 · pp. 8–9

Abstract

from arXiv · show

Biomedical machine learning papers often compress model performance into one headline number. That number can look like a property of the model even when it depends strongly on how the benchmark was evaluated. We study this problem on the widely used Kermany pediatric chest radiograph dataset using nine image classifiers and a controlled evaluation protocol. Under the same protocol, the eight pretrained backbones differ by only 0.026 AUROC. In contrast, changing whether the backbone is frozen or fine-tuned changes AUROC by 0.044 on average, and changing the decision threshold changes balanced accuracy by 0.090 on average. The official test split is also measurably different from the training pool: a partition classifier distinguishes them at AUC 0.697, rising to 0.898 for normal radiographs. Most strikingly, a classifier using only file properties, with no image anatomy, reaches 0.992 balanced accuracy within the training pool but falls to 0.496 on the official test split. Validation-fitted thresholds and calibration also transfer imperfectly. These results show that a high benchmark score can support different conclusions when the split, training policy, threshold, metric, calibration, and uncertainty are not communicated with it. We end with a seven-item reporting recommendation in which each item is tied to an effect measured in the study

1 Introduction

This study asks whether a biomedical benchmark score reflects the model alone or the evaluation protocol. Using a controlled case study of the Kermany pediatric radiograph dataset, it compares architecture effects with protocol effects and links reporting recommendations to measured examples.

Motivation

  • The reported number is often treated as a model property even though it also depends on how the model was evaluated.If changing a protocol component shifts the score as much as changing architecture, the headline cannot be interpreted without the protocol that produced it.Source: §1 · pp. 1–2

Case study

  • The Kermany pediatric radiograph dataset is a useful case because it is public, widely reused, and distributed with fixed training and test splits.Published studies vary in split policy, backbone training, threshold selection, and calibration reporting.Source: §1 · p. 2

Main findings

  • 0.044 AUROC on average separates frozen from fine-tuned backbones, exceeding the 0.026 spread among pretrained architectures under one fixed protocol.The backbone-training effect reaches 0.103; the official test split also differs from the training pool, and file-only balanced accuracy falls from 0.992 in the training pool to 0.496 on the official test split.Source: §1 · p. 2

Contribution to responsible communication

  • The study is a controlled case study of how a shared benchmark can support different stories when its evaluation protocol is hidden.It connects each communication recommendation to a measured effect rather than offering a generic checklist.Source: §1 · p. 2

2 Background and Related Work

Prior work highlights shortcut signals and evaluation choices that complicate interpretation of biomedical ML performance. This study situates its controlled protocol comparison within those concerns and the reporting context of TRIPOD+AI.

Shortcut learning

  • High performance does not establish that a medical imaging model learned clinically meaningful evidence; predictive performance describes results, not the evidence used.Potential shortcuts include acquisition patterns, hospital identity, image processing, and text markers [Zech et al., 2018, DeGrave et al., 2021]; shortcut learning describes reliance on an easy correlated feature rather than the intended mechanism.Source: §2 · p. 2

Evaluation design

  • Small test sets, leakage, data reuse, class imbalance, and unstable rankings make medical AI evaluation sensitive to design choices [Varoquaux and Cheplygina, 2022, Roberts et al., 2021].Prior benchmark studies report that duplicates can change rankings and newly collected test sets can reveal drops hidden by original evaluations [Barz and Denzler, 2020, Recht et al., 2019].Source: §2 · p. 2

Thresholds and calibration

  • A model can retain ranking performance while producing poorly chosen decisions or misleading confidence values when validation and test distributions differ.Youden’s index selects an operating point with equally weighted sensitivity and specificity, while temperature scaling is a post hoc calibration method that often improves estimates on data drawn from the validation distribution [Guo et al., 2017].Source: §2 · p. 3

Reporting guidance

  • TRIPOD+AI asks prediction-model studies to describe evaluation data, discrimination, calibration, thresholds, and uncertainty.The paper frames the communication challenge as ensuring that a performance claim retains enough context when it travels outside the paper.Source: §2 · p. 3

3 Study Design

The study defines protocol components and evaluation procedures for comparing model, split, threshold, calibration, and distribution effects. It uses a patient-disjoint validation split, leaves the official test split untouched, and reports uncertainty alongside performance measures.

Dataset and split policy

  • The patient-disjoint split uses the released training pool for new training and validation partitions while leaving the official test split untouched.After removing exact byte-level duplicates, the partitions contain 4,420 training images, 770 validation images, and 618 official test images; filenames provide patient or study identifiers for grouping related images.Source: §3 · p. 3

Models and backbone training

  • The model comparison includes a scratch-trained small CNN and eight pretrained backbones, with the pretrained models fully fine-tuned under a shared protocol.To isolate backbone-training policy, each pretrained architecture is also evaluated with a frozen backbone and trainable classifier head.Source: §3 · p. 3

Decision threshold and metric

  • Balanced accuracy is emphasized for threshold-dependent comparisons, while AUROC and AUPRC summarize ranking performance.The official test set is 62.5% positive, so an always-positive classifier obtains 0.625 raw accuracy; the validation-selected threshold uses Youden’s J and is compared with 0.5 and a test-selected oracle.Source: §3 · p. 3

Calibration

  • Calibration is assessed by fitting one temperature on validation logits and measuring expected calibration error with 15 equal-width confidence bins.The study tests whether this validation-fitted correction improves calibration on the official test split.Source: §3 · pp. 3–4

Distribution shift test

  • A frozen ResNet-50 feature representation and logistic regression classify whether an image came from the training pool or official test split.The analysis uses five-fold out-of-fold predictions, a label-shuffling control, discriminator AUC, and proxy A-distance; values near 0 and 2 indicate difficult and easy partition separation, respectively.Source: §3 · p. 4

File-only shortcut test

  • The file-only shortcut test trains a gradient-boosted tree on file properties without image pixels or anatomical features.It compares five-fold cross-validation inside the training pool with evaluation after fitting on that pool and testing on the untouched official split.Source: §3 · p. 4

Target prevalence operating point

  • The target-prevalence operating point uses unlabeled target predictions and an externally specified prevalence, and is included only as a sensitivity analysis.The test-selected threshold is an oracle that cannot be used in a valid evaluation; the target-prevalence method’s results and limitations are reported separately.Source: §3 · p. 4

Uncertainty and statistical testing

  • The study reports means and standard deviations over three seeds where available, percentile-bootstrap AUROC intervals from 2,000 test-case resamples, and paired DeLong tests.It also reports precision-recall curves and AUPRC because class imbalance can make these summaries more informative for some comparisons.Source: §3 · p. 4

4 Results

Across this benchmark, evaluation choices changed the reported interpretation alongside architecture choice: protocol effects exceeded the narrow pretrained-backbone spread, and shortcuts and validation-fitted choices transferred imperfectly.

4.1 Architecture differences are small relative to several protocol effects

  • 0.026 AUROC separates the eight pretrained backbones under the reference protocol; their uncertainty intervals overlap substantially, while the scratch-trained CNN is weaker.Individual-model 95% bootstrap intervals are roughly ±0.02 AUROC, making three-decimal reporting without intervals potentially overprecise.Source: §4.1 · pp. 4–5, Table 1 · p. 5, Figure 1 · p. 5

4 Results

  • 0.044 mean AUROC change from backbone training policy and 0.090 mean balanced-accuracy change from threshold selection illustrate the measured effects of evaluation choices.Table 2 summarizes effects associated with protocol components; oracle values measure transfer gaps.Source: Table 2 · p. 5, §4.2 · p. 6, §4.5 · p. 7

4.2 Backbone training changes the conclusion more than architecture choice

  • 0.044 mean AUROC change, reaching 0.103, accompanies switching between frozen and fully fine-tuned backbones, and the direction varies across models.Some models improve after fine-tuning, while others change little or lose balanced accuracy; rankings do not simply transfer between policies.Source: §4.2 · p. 6

4.3 The official test split is measurably different from the training pool

  • 0.697 AUC distinguishes the official test split from the training pool, rising to 0.898 for NORMAL radiographs; the shift’s source is unidentified.The partition classifier reaches only 0.542 between the study’s train and validation partitions, and the duplicate audit found no near twin in the training pool for any test image.Source: §4.3 · p. 6, Figure 2 · p. 6

4.4 A file only predictor shows why pooling and re-splitting can be misleading

  • 0.992 balanced accuracy from file properties alone in the training pool falls to 0.496 on the untouched official test split.The classifier uses no radiograph pixels or anatomy; the authors interpret the result as a shortcut that does not survive the split change.Source: §4.4 · pp. 6–7, Figure 2 · p. 6

4.5 Thresholds and calibration fitted on validation do not fully transfer

  • 0.090 average balanced-accuracy change follows validation threshold selection, while validation-fitted temperature scaling shifts mean test ECE only from 0.172 to 0.169.Test-label oracle thresholding improves balanced accuracy by a further 0.083 on average; the target-prevalence sensitivity analysis raises it from 0.787 to 0.863, versus an unattainable test oracle of 0.870. ECE worsens in 9 of 26 runs after temperature scaling.Source: Figure 3 · p. 7, §4.5 · p. 7

5 From a Benchmark Score to a Responsible Claim

The section argues that benchmark scores require their evaluation context: results from different protocols answer different questions, and measured protocol effects motivate a compact reporting rule. Its scope is a cautious claim about the released labels, not clinical readiness or cross-hospital generalization.

5 From a Benchmark Score to a Responsible Claim

  • Published results under pooled and official-split protocols answer different questions, so their headline accuracies do not form a valid ranking.Table 3 places published results beside the study’s official-split rows while noting differences in split policy, metric, training, and operating point.Source: §5 · p. 8, Table 3 · p. 8
  • Seven reporting details should accompany benchmark scores: split, backbone training, threshold, class-imbalance-aware metric, calibration, uncertainty, and train–test shift.Each recommendation is tied to an effect measured in the study.Source: §5 · p. 8
  • The measured cost of omitted protocol details includes training, threshold, file-only shortcut, and train–test shift effects.The section notes that backbone training changes AUROC more than pretrained architecture spread, threshold selection changes balanced accuracy by even more, and the file-only shortcut drops across split policies.Source: §5 · p. 8
  • A strong discrimination score does not establish whether a model used clinically meaningful visual evidence.The supplementary Grad-CAM and SHAP audit summarizes attribution mass outside a body mask rather than treating a heatmap as causal explanation.Source: §5 · p. 8
  • The strongest supported claim is separation of the released labels on the official split under a stated protocol, not cross-hospital generalization, clinical utility, or triage readiness.The official split contains an undocumented distribution difference, so the score should travel with split, metric, threshold, uncertainty, and observed shift.Source: §5 · p. 8

6 Limitations and Conclusion

The conclusion is bounded by a single public pediatric radiograph dataset, an unexplained train–test shift, and a small test set. It reiterates that scores depend on protocol choices and directs readers to appendices for supporting tables and diagnostics.

6 Limitations and Conclusion

  • This is a case study of one public pediatric radiograph dataset, not a survey of biomedical benchmarks.The released metadata do not identify the train–test shift’s source, and the file-only experiment does not prove that a particular neural network uses the shortcut.Source: §6 · pp. 8–9
  • The small test set limits architecture comparisons, and the target-prevalence threshold is only a sensitivity analysis because reliable deployment prevalence may be unavailable.Additional robustness and reproducibility details appear in Appendices G and H.Source: §6 · p. 9
  • A benchmark score reflects model and protocol; training, split, threshold, calibration, and shift choices affect results as much as or more than architecture choice.Those conditions need to travel with the score.Source: §6 · p. 9

Appendix Guide

  • Appendices A–H provide complete model tables, transfer diagnostics, statistical tests, leakage controls, plots, attribution diagnostics, robustness checks, and reproducibility artifacts.The main text reports only columns needed for central comparisons.Source: Appendix Guide · p. 11

A Full Model and Protocol Results

The appendix tables document the scope and setup of the model and protocol comparisons. They provide full fine-tuned results, per-architecture frozen-versus-fine-tuned results, and threshold and calibration comparisons.

A Full Model and Protocol Results

  • Table 6 reports per-architecture default versus validation-selected thresholds and ECE before and after temperature scaling.The caption defines the default as t = 0.5 on uncalibrated scores and the tuned setting as a Youden-optimal threshold on temperature-scaled scores; lower ECE is better.Source: A Full Model and Protocol Results · p. 11, Table 6 · p. 12
  • Table 4 gives full fine-tuned model results on the held-out official test split, with mean and standard deviation over three seeds.The operating point and temperature were selected only on the patient-disjoint validation split.Source: Table 4 · p. 11, A Full Model and Protocol Results · p. 11
  • Table 5 compares frozen and fine-tuned results per architecture while holding architecture, schedule, split, and augmentation constant.The fine-tuned setting makes all weights trainable; the frozen setting is the linear-probe protocol.Source: Table 5 · p. 11, A Full Model and Protocol Results · p. 11

B Calibration and Operating Point Transfer

The section reports how operating-point and calibration behavior transfers to the official test split. Tables 7–8 provide numerical decompositions alongside the main visual result in Figure 3.

B Calibration and Operating Point Transfer

C Statistical Testing of Architecture Differences

Paired testing finds that most pretrained-backbone differences are not statistically distinguishable. The duplicate audit finds no official-test near twin at the primary threshold and cautions against coarse perceptual hashing.

C Statistical Testing of Architecture Differences

D Corpus Composition and Duplicate Audit

  • Figure 5 shows representative test images and class composition; class imbalance motivates balanced accuracy, while duplicate auditing is kept separate from shift analysis.Source: D Corpus Composition and Duplicate Audit · p. 13, Figure 5 · p. 13
  • At ρ ≥ 0.98, no official-test image has a near twin in training; at ρ ≥ 0.95, 14 of 624 test images are flagged for inspection.The audit also removes 26 exact byte-level duplicates from training and 6 from test before model training or evaluation.Source: D Corpus Composition and Duplicate Audit · pp. 13–14
  • A 64-bit difference hash at Hamming radius 5 flags approximately 93% of test images, likely because distinct chest radiographs share a similar global layout.The authors include this sensitivity check to avoid confusing an unsuitable duplicate detector with evidence of leakage.Source: D Corpus Composition and Duplicate Audit · p. 14

E Discrimination Curves and Error Structure

The discrimination curves support the section’s metric discussion without adding a separate result claim. The attribution audit instead illustrates that performance metrics and visual explanations answer different questions.

E Discrimination Curves and Error Structure

  • Figure 6 presents seed-0 ROC and precision-recall curves, with the three highest-AUROC architectures labeled; precision-recall complements ROC for class-imbalance-sensitive reporting [Saito and Rehmsmeier, 2015].Table 4 summarizes scalar performance, and the threshold error structure is not repeated here.Source: Figure 6 · p. 14, E Discrimination Curves and Error… · p. 14

F Supplementary Attribution Audit

  • Equation 9 summarizes outside-body absolute attribution mass: 17.0% (SD 7.7%) for predicted-normal versus 34.5% (SD 10.3%) for predicted-pneumonia radiographs, at mean confidence 0.997.The body mask is the largest bright connected component after Otsu thresholding.Source: F Supplementary Attribution Audit · p. 14
  • Grad-CAM and SHAP are descriptive diagnostics, not causal explanations; attribution values are not used to claim clinical validity.The audit illustrates that discrimination metrics do not reveal which image regions contribute to a prediction.Source: F Supplementary Attribution Audit · p. 14, Figure 7 · p. 15

G Seed Ensemble Robustness Check

Averaging predictions from the three trained seeds offers a variance-reduction check without changing the conclusion that protocol effects can exceed architecture differences. The appendix also describes artifacts intended to make reported values reproducible.

G Seed Ensemble Robustness Check

  • Averaging probabilities changes AUPRC modestly and does not alter the conclusion that protocol effects can exceed architecture differences.The ensemble gain is largest for weaker backbones and within noise for VGG-16; the table uses official-test AUPRC because the positive class is the majority.Source: G Seed Ensemble Robustness Check · p. 15, Table 9 · p. 15

H Reproducibility

  • The intended anonymized release includes patient-disjoint split lists, per-case probabilities, feature extractors, classifiers, and scripts to regenerate tables and figures.Stored outputs also permit recomputation of metrics, operating points, confidence intervals, calibration analyses, and statistical tests without retraining.Source: H Reproducibility · p. 15

Related papers

Loading 2609.37848v1…