PaperlayerView exact revision on arXiv

Source-linked AI summary

Scores That Hold, Benchmarks That Leak: Measuring Dataset Contamination in Public Brain-Tumor MRI Classification

Bhanu Prakash Vangala, Sowmya Guda, Latha Peddi, Navya Vangala

arXiv:2610.00421v1cs.CVcs.AI

TL;DR

Accuracy-focused evaluation leaves benchmark integrity—the independence of test data from training—unmeasured. The paper audits contamination across image, patient, and source layers, finding severe leakage even as deduplication barely changes scores.

  • Problem

    Accuracy alone does not measure whether test data are independent of training, so benchmark integrity requires direct assessment rather than inference from a stable leaderboard.

    Source: §I · p. 1

  • Method

    The paper defines duplicate, patient, and source-label contamination layers and measures them with model-independent audit primitives.

    Source: §I · p. 1

  • Results

    Contamination is severe across image, patient, and source layers, while removing leaked test images changes balanced accuracy by -0.009.

    Source: §IX · p. 10

  • Takeaways & Limitations

    Reported accuracy on these corpora alone should not be treated as evidence of tumor recognition; benchmark integrity should be measured directly, including with a header-only baseline.

    Source: §IX · p. 10

    The patient-leakage estimate covers only test images traceable to figshare, and the reported rates concern one benchmark family rather than public medical-imaging datasets generally.

    Source: §VII · p. 9

Abstract

from arXiv · show

Automated classification of brain tumors from MRI is a heavily published application of deep learning in medical imaging, with reported accuracies on public benchmarks routinely exceeding 98%. However, accuracy does not capture a critical dimension of benchmark quality: dataset integrity, defined as the independence of test from training data at the image, patient, and acquisition-source levels. We introduce a three-layer contamination framework comprising duplicate, patient, and source-label leakage to assess the public corpora on which this literature rests. We audit the three most widely used corpora against a chest-radiograph negative control and quantify each layer's effect on measured performance across nine architectures and three evaluation conditions. Contamination is severe at every layer: 28.8% of the dominant corpus's official test split has a near-twin in its own training split, a second corpus leaks 22.3% of its test images byte-identically, 95.5% of traceable test images share a patient with training, and file-header features containing no anatomy separate tumor from no-tumor at 0.959 balanced accuracy, at parity with fine-tuned ResNet backbones. The unexpected result is that removing every identified leaked test image leaves balanced accuracy essentially unchanged: stable performance after deduplication does not establish benchmark integrity. Our findings establish dataset integrity as a distinct, measurable axis of benchmark quality that a stable leaderboard cannot certify. For biomedical research, reported accuracy on these corpora alone does not establish that a model has learned to recognize tumors rather than exploit dataset-specific cues. We release the contaminated-file lists, recovered patient identifiers, and deduplicated splits.

I. INTRODUCTION

The introduction argues that accuracy-focused evaluation does not establish whether test data are independent of training at the image, patient, and acquisition-source levels. It presents a three-layer audit and controlled re-evaluation to measure contamination and its relationship to benchmark scores.

I. INTRODUCTION

  • 28.8% of the dominant corpus’s official test split has a training near-twin, while 95.5% of traceable test images share a patient with training.A second corpus leaks 22.3% of its test images byte-identically; removing leaked images changes balanced accuracy by -0.009 on average.Source: §I · p. 2
  • The authors argue that integrity must be measured directly rather than inferred from a stable leaderboard, alongside accuracy in reporting practice.They clarify that contamination is a dataset property inherited by users of the official splits, not evidence that an individual published number is fraudulent.Source: §I · p. 2
  • The framework separates contamination into duplicate, patient, and source-label leakage, measured with model-independent primitives.The proposed audit uses a pixel-correlation criterion, recovered patient identifiers, and file-header probes without anatomical content.Source: §I · pp. 1–2
  • The study jointly audits three public brain-tumor MRI corpora against a negative control and releases contaminated-file lists, patient mappings, and deduplicated splits.It evaluates nine architectures under three conditions, including the official test split, a deduplicated split, and a disjoint external corpus.Source: §I · pp. 1–2
  • Removing every identified leaked test image leaves balanced accuracy essentially unchanged, so score stability does not certify benchmark integrity.The controlled study spans nine architectures and three random seeds per configuration, for 52 training runs in total.Source: §I · p. 2
  • A standard perceptual hash falsely flags 93% of a clean control corpus, while a header-only baseline costing minutes would expose the source-label confound.The header probe uses file features containing no anatomical content.Source: §I · pp. 1–2

II. CONTAMINATION AUDIT FRAMEWORK

The framework treats benchmark integrity as independence between training and test data, and divides failures of that independence into three contamination layers. It pairs each layer with a model-independent measurement primitive.

II. CONTAMINATION AUDIT FRAMEWORK

  • A reported score is evidence about the task only when the official test partition is independent of training data.The framework measures each failure of independence and uses controlled re-evaluation to assess its score effect.Source: §II · p. 2
  • The duplicate layer includes test images with a pixel-level near-twin in training, including the same scan after resizing or re-encoding.It also covers scans shared across nominally distinct corpora.Source: §II · p. 2
  • The patient layer includes test images from patients who contribute another slice to training, even when the slices are distinct files.A duplicate detector does not flag these different slices.Source: §II · p. 2
  • The source layer is a corpus-level confound in which source class composition makes acquisition source predictive of labels before anatomy is considered.Unlike image-level duplicates and patient-level overlap, this confound survives deduplication because it arises from corpus construction.Source: §II · p. 2

A. Duplicate Criterion

Perceptual hashing is unreliable for these similar-appearing scans, so the audit uses normalized cross-correlation to identify duplicates. The threshold sweep and comparison with a chest-radiograph control check that the criterion isolates true matches.

A. Duplicate Criterion

  • A 64-bit difference hash flags 93% of the clean chest-radiograph control and 63% of C1’s test split, making its C1 flags untrustworthy.The false positives arise because scans share similar anatomy and pose.Source: A. Duplicate Criterion · pp. 2–3
  • The audit uses brightness- and contrast-invariant normalized cross-correlation, robust to JPEG re-encoding, and declares duplicates at τ = 0.98.Counts remain stable across the upper threshold range: 460 C1 test images are flagged at 0.98 and 432 at 0.995.Source: A. Duplicate Criterion · p. 3, Figure 1 · p. 3
  • C1 has 432 test images matching training images at ρ ≥ 0.995, with visually identical examples; its correlation distribution has a near-one mode absent from the control.The two distributions have nearly identical medians, 0.919 for C1 and 0.900 for the control.Source: A. Duplicate Criterion · p. 3, Figure 2 · p. 3, Figure 3 · p. 3

B. Patient Identity Recovery

The audit recovers patient identities by matching Kaggle images to figshare originals, probes source-label confounding with anatomy-free file features, and evaluates models under official, deduplicated, and external conditions. The corpora include a chest-radiograph negative control.

B. Patient Identity Recovery

  • Patient leakage is estimated by matching traceable Kaggle images to figshare originals, inheriting patient keys, and checking whether each test patient also appears in training.Because SARTAJ and Br35H lack identifiers, the patient-layer rate is conditional on traceable images.Source: B. Patient Identity Recovery · p. 3

C. Header-Only Probe

  • The header-only probe uses file metadata and JPEG features without anatomy; its balanced accuracy measures source-label confounding, not a channel directly available to the evaluated networks.The networks receive resized decoded pixels, so the probe is an upper bound on non-anatomical provenance signal.Source: C. Header-Only Probe · p. 4

D. Re-Evaluation Protocol

  • Models are scored on the official split, that split after duplicate-layer images are removed, and a disjoint external corpus cleared of images contained in training.Deduplication sensitivity is the change in balanced accuracy between official and deduplicated evaluation; calibration is also tracked.Source: D. Re-Evaluation Protocol · p. 4

III. CORPORA

  • The audit covers C1, C2, and C3 alongside a pediatric chest-radiograph negative control, which has zero byte-level and pixel-level duplicates.C1 combines figshare, SARTAJ, and Br35H; C3 is a small binary tumor/notumor corpus.Source: §III · p. 4

IV. EXPERIMENT SETUP

The experiment compares nine architectures trained on C1 under a fixed protocol, then evaluates them across official, deduplicated, and external conditions. Runs and per-case outputs are stored for reproducibility.

IV. EXPERIMENT SETUP

  • Nine architectures span hand-designed CNNs, established convolutional backbones, and modern ConvNeXt and vision-transformer models.The set is selected to cover backbones used in the literature and test whether conclusions vary by architecture.Source: §IV · p. 4
  • All models train on C1 with a stratified 15% validation holdout, fixed augmentation and optimization schedules, and checkpoint selection by validation balanced accuracy.The schedule is held fixed across architectures rather than searched separately.Source: §IV · p. 4
  • Each trained model is evaluated on C1’s official split, the split without 460 duplicate-layer images, and C2 after removing images contained in C1.Three independent seeds yield 52 training runs, with stored metric records and logits enabling recomputation of reported results.Source: §IV · pp. 4–5
  • The audit configures duplicate detection at τ = 0.98 and uses a gradient-boosted tree for the anatomy-free header probe.Attribution analysis measures the share of absolute attribution mass outside the brain mask.Source: §IV · p. 4

V. EVALUATION DIMENSIONS

The evaluation measures contamination, source-label confounding, score sensitivity to deduplication, external generalization and calibration, and mechanism attribution. Within-corpus and cross-corpus results show substantial overlap in the audited benchmarks.

V. EVALUATION DIMENSIONS

  • The evaluation spans contamination prevalence, source-label confounding, deduplication sensitivity, external generalization and calibration, and mechanism attribution across the reported dimensions.The dimensions include duplicate and patient leakage, header-only probe accuracy, calibration under each condition, and attribution comparisons for leaked versus clean images.Source: §VI · p. 5, §V · p. 5

A. Duplicate Leakage Within and Across Corpora

  • 28.8% of C1 test images have a near-twin in training, while 22.3% of C2 test images are byte-identical to training images; the chest-radiograph control is clean at 0.0%.The within-corpus criterion flags test images with a training match at normalized cross-correlation ≥ 0.98.Source: A. Duplicate Leakage Within and Across… · p. 5, Table I · p. 5
  • C1 and C2’s official train/test partitions contain leaked images, so models evaluated there are scored substantially on images already seen during training.The passage characterizes this as a property of the released partition rather than something modeling care can undo.Source: A. Duplicate Leakage Within and Across… · p. 5
  • 82.9% of C2 is present in C1, and 81.8% of that overlap is byte-identical; 48.6% of C3 is contained in C1.After contaminated images are removed, C2 retains 559 usable images and C3 retains 130 for genuinely external evaluations.Source: A. Duplicate Leakage Within and Across… · p. 5, Table II · p. 5

B. Patient-Level Leakage

Patient-level leakage remains invisible to duplicate detection: the figshare release contains many slices per patient, and most traceable test images share a patient with training. These measurements apply only to images whose provenance can be recovered.

B. Patient-Level Leakage

  • 3,064 figshare slices come from only 233 patients, or 13.2 slices per patient, making patient-disjointness essentially impossible under random image-level splits.Every published split on this benchmark is image-level.Source: B. Patient-Level Leakage · p. 5
  • 95.5% of C1’s traceable test images and 100.0% of C2’s traceable test images have another slice from the same patient in training.For C1, 201 test images (12.6%) could be traced to a figshare original; the estimate is conditional on traceability.Source: B. Patient-Level Leakage · p. 5, Table III · p. 6
  • The patient-leakage estimates cover only the traceable subset; SARTAJ and Br35H provide no identifiers for measuring whole-corpus leakage.C1’s traceable images represent 12.6% of its official test split.Source: B. Patient-Level Leakage · p. 5
  • 0 figshare patient-identifier matches occur for notumor, which originates from Br35H and serves as a check against indiscriminate matching.Table IV reports this per-class matcher check.Source: B. Patient-Level Leakage · p. 6, Table IV · p. 6

C. The Source–Label Confound

The audit finds a strong source–label confound and shows that deduplication leaves measured performance essentially unchanged. External evaluations and a chest-radiograph control further qualify what the benchmark scores establish.

C. The Source–Label Confound

D. Score Sensitivity to Decontamination

  • 0.044 in-domain calibration error versus 0.048 externally accompanies the near-unchanged deduplicated scores.The paper offers possible explanations for the null deduplication effect, including that leaked images may be no easier and that shortcuts may make memorization unnecessary.Source: D. Score Sensitivity to Decontamination · p. 6
  • A stable score after deduplication does not establish benchmark integrity because patient-level leakage and the source–label confound remain in place.Deduplication removes leaked test images but does not repair those other contamination layers.Source: D. Score Sensitivity to Decontamination · p. 6
  • ∆ = -0.009 mean deduplication sensitivity; ConvNeXt-T scores 0.950 officially and 0.950 after deduplication.For every architecture above 0.9, the change is under 0.003; transfer to the disjoint external condition costs 0.030 on average.Source: D. Score Sensitivity to Decontamination · p. 6, Table VI · p. 7, Figure 6 · p. 7
  • 0.745 within-corpus balanced accuracy for C3-trained binary detectors falls to 0.673 on C1’s test split and 0.565 on cleaned C2.The hand-designed CNN falls to 0.294, below chance; C3 contains only 253 images.Source: D. Score Sensitivity to Decontamination · pp. 6–7, Table VII · p. 7

E. What the Benchmark Cannot Distinguish

Header-only features approach neural-network performance on the binary task, so the benchmark cannot distinguish tumor recognition from source cues. Attribution analyses find similar leaked-versus-clean patterns, while a proposed zero-shot control fails to identify memorization.

E. What the Benchmark Cannot Distinguish

  • 0.959 balanced accuracy from file metadata alone is near the best network’s 0.989 and at parity with fine-tuned ResNet-50 (0.967) and ResNet-18 (0.965).The metadata contains no anatomy; the benchmark cannot separate learning tumor appearance from learning source-directory cues.Source: E. What the Benchmark Cannot Distinguish · pp. 7–8
  • The strongest model’s class recalls change little with deduplication, while the external corpus costs most in tumor subtypes rather than notumor.The architecture ordering is not preserved across the official, deduplicated, and external evaluations.Source: Figure 7 · p. 8

F. Mechanism Attribution, and a Control That Did Not Work

  • 29.1% (±19.4) of leaked-image attribution mass and 28.1% (±14.9) of clean-image attribution mass fall outside the brain; mean confidence is 0.923 and 0.908, respectively.The authors interpret the similar attribution patterns, alongside stable deduplication results, as evidence against memorization driving performance on leaked images.Source: F. Mechanism Attribution, and a… · p. 8, Figure 8 · p. 8
  • BiomedCLIP finds leaked images easier by 0.035, while CLIP finds them harder by 0.101, leaving the difference-in-differences estimand unidentified.The paper reports this failed control rather than choosing between its contradictory results.Source: F. Mechanism Attribution, and a… · p. 8, Table IX · p. 8

VII. DISCUSSION AND RECOMMENDATIONS

The discussion prioritizes practical integrity checks, especially a header-only baseline, and cautions that stable scores or deduplication alone cannot establish benchmark soundness. Its conclusions are bounded by conservative duplicate detection, incomplete patient traceability, and an audit of one benchmark family.

Recommendations

  • The authors give highest priority to reporting a header-only baseline because deduplication alone is not diagnostic.They also recommend duplicate checks with threshold sweeps, provenance checks, balanced accuracy and calibration with confidence intervals, and exact file lists.Source: §VII · pp. 8–9

Ethics

  • The study analyzes only publicly released, de-identified imaging and uses already distributed opaque patient keys that do not identify individuals.No new human-subject data were collected, and no institutional review was required.Source: §VII · p. 9

Reproducibility

  • Every table, figure, interval, and statistical test can be recomputed from stored outputs without a GPU or retraining.The analysis uses one JSON metric record per run and one compressed raw-logit array per run and evaluation set.Source: §VII · p. 9

Limitations

  • The duplicate criterion detects the same scan when re-encoded, but not different slices from the same patient.Patient-level estimates cover only 12.6% of test images traceable to figshare; whole-corpus leakage cannot be bounded for SARTAJ and Br35H, so reported rates are lower bounds.Source: §VII · p. 9
  • The architecture comparison uses one fixed training schedule rather than per-architecture search, and the reported rates concern one benchmark family.The authors frame the framework as general but caution against treating these rates as estimates for public medical-imaging datasets broadly.Source: §VII · p. 9

VIII. RELATED WORK

Prior work documents leakage, unstable evaluation, and acquisition shortcuts in medical imaging; this paper applies an audit to a widely used brain-tumor MRI benchmark family. The conclusion is that contamination can coexist with stable scores, so reported accuracy on these corpora alone does not establish tumor recognition.

VIII. RELATED WORK

  • Earlier benchmark studies reported duplicate test images and accuracy drops on fresh test sets; here, contamination is much larger but removal does not move the score.The related-work comparison includes CIFAR-100 duplicates and fresh ImageNet and CIFAR evaluations, [24].Source: §VIII · p. 9
  • Medical-imaging reviews have identified leakage, small test sets, dataset problems, and protocol-sensitive rankings as recurring evaluation concerns –.This work applies medical algorithmic audit practice to one benchmark family and supplies model-independent measures for its contamination layers and a negative control.Source: §VIII · p. 9
  • Prior imaging studies show classifiers can use hospital-specific acquisition signatures, while this paper quantifies a no-anatomy header-only probe and reports attribution evidence for a shortcut –.The authors distinguish this shortcut concern from memorizing specific scans.Source: §VIII · p. 9
  • The paper situates its audit within a literature built on figshare and derived Kaggle mirrors, without re-evaluating individual published classifiers,,.The released contaminated-file lists and deduplicated splits are intended to let that literature remeasure itself.Source: §VIII · pp. 9–10

IX. CONCLUSION

  • Removing identified leaked images changes balanced accuracy by -0.009, while transfer to a disjoint corpus costs 0.030 and calibration barely moves.The authors conclude that a stable leaderboard cannot certify benchmark integrity, because a header-only baseline performs at parity with fine-tuned ResNet backbones.Source: §IX · p. 10
  • The authors recommend measuring integrity directly and not treating accuracy on these corpora alone as evidence of tumor recognition.They propose duplicate audits, patient-disjointness statements, header-only baselines, and release contaminated-file lists, recovered identifiers, and deduplicated splits.Source: §IX · p. 10

APPENDIX A REPRODUCIBILITY

The appendices describe recomputation from stored outputs, release audit artifacts, and record analyses that did not support a conclusion.

APPENDIX A REPRODUCIBILITY

  • Every table, figure, confidence interval, and statistical test can be recomputed from stored outputs without a GPU or retraining.The analysis reads one JSON metric record per run and one compressed raw-logit array per run and evaluation set.Source: Appendix A · p. 11

APPENDIX B ARTIFACTS RELEASED

  • Released artifacts include contaminated-file lists, recovered patient mappings, deduplicated evaluation splits, and a GPU-free header-only probe.The file lists use a normalized cross-correlation criterion of 0.98; the patient mapping links Kaggle paths to figshare cjdata.PID.Source: Appendix B · p. 11

APPENDIX C NEGATIVE RESULTS RECORDED

  • The appendix records three analyses as inconclusive or misleading: a difference-hash detector, zero-shot vision-language comparison, and untransposed figshare arrays.The hash detector flagged 93% of a control set with zero measured duplicates; the two vision-language controls disagreed in sign, while untransposed arrays yielded zero patient matches.Source: Appendix C · p. 11

Related papers

Loading 2610.00421v1…