PaperlayerView exact revision on arXiv

Source-linked AI summary

On Hyperparameter Tuning on the Test Set

Matteo Fregonara, Tom Viering, Jan van Gemert

arXiv:2610.05902v1cs.LGcs.AIcs.CV

TL;DR

Textbook guidance cautions against tuning hyperparameters on test data, yet test-set tuning occurs in practice and its consequences merit measurement. The paper systematically compares test- and validation-tuning across five benchmarks. Overfitting is significant, but rankings largely persist, and the authors call for a nuanced view and open reporting.

  • Problem

    Test-set hyperparameter tuning occurs in practice, but its overfitting and consequences for model selection require systematic empirical evaluation.

    Source: §1 · p. 1

  • Method

    The authors compare repeated validation-set and test-set hyperparameter tuning across MNIST-1D, CIFAR-10, and three GLUE tasks using paired runs.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Test-tuning is significant across five benchmarks but generally stays within natural experimental variability, and rankings remain identical or change only by near-tie adjacent inversions.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The authors call for a nuanced view of test-set tuning and encourage researchers to report it openly.

    Source: §1 · p. 2

    Each estimate covers one search, not cumulative adaptive selection across groups reusing a benchmark over time.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

"Don't tune hyperparameters on the test set" is often stated in machine learning textbooks. Violating it is considered a cardinal sin that produces misleadingly optimistic results, corrupts benchmark integrity, and thus can even be interpreted as scientific fraud. Yet evidence suggests that test set hyperparameter tuning does occur in practice, making it all the more important to understand its actual consequences. So how bad is it, really? In this work we question this dogma and put it to an empirical test. We systematically study the magnitude of the performance inflation caused by tuning the hyperparameters on the test set for MNIST-1D, CIFAR-10, and three tasks from the GLUE benchmark. Our experiments show that while the effect is real and significant, it is frequently small relative to other sources of noise. In many cases, we find that tuning on the test set recovers exactly the same model as when tuning on the validation set. Most importantly, we find that the rankings of models remain essentially preserved after tuning on the test set and therefore that consistent test-set tuning may not invalidate benchmarks or model selection. Our results call for a more nuanced view of tuning hyperparameters on the test set, stimulating researchers to openly report test tuning.

1 INTRODUCTION

The paper examines whether tuning hyperparameters on test data causes meaningful overfitting, given that such tuning occurs in practice despite textbook prohibitions. It finds that overfitting is real, but rankings usually remain stable when models are tuned on the same data.

1 INTRODUCTION

  • Textbooks prohibit test-set tuning, which can be viewed as academic fraud because hyperparameters determine model training behavior like other parameters.The paper notes that test-set initialization tuning can produce excellent accuracy (Picard, 2021).Source: §1 · p. 1
  • Test-set tuning is used in computer vision and natural language processing, motivating investigation of its actual consequences despite the tension between theory and practice.The authors note that hyperparameter searches usually evaluate only a handful of configurations.Source: §1 · p. 1
  • The study systematically compares repeated validation-set and test-set tuning across MNIST-1D, CIFAR-10, and three GLUE tasks using multiple model families.The experiments train 36,000 model variants and examine overfitting ease, severity, and effects on model selection.Source: §1 · p. 1
  • Test-set tuning significantly overfits on every evaluated benchmark, but its accuracy gain is mild relative to other randomness and model rankings barely change.The authors did not observe a case where test-tuning failed to select the best model.Source: §1 · p. 2
  • The authors argue that test-tuning is relatively benign when all compared models are tuned on the same data, and encourage openly reporting it.They call for a nuanced view rather than hiding test-tuning.Source: §1 · p. 2

2 RELATED WORK

Prior work links repeated adaptive queries and hyperparameter optimization to overfitting, while empirical studies of reused benchmarks offer a more mixed picture. This paper situates its study amid that tension and the importance of benchmark variability.

2 RELATED WORK

  • Repeated adaptive queries may overfit a hidden test set, but CIFAR-10 and ImageNet replication studies attribute much observed accuracy drops to distribution shift while finding rankings preserved (Recht et al., 2018; 2019).Evidence at larger scale points in the same direction.Source: §2 · p. 2
  • Models trained on the same benchmark often make similar predictions, potentially reducing adaptive exploitation of test-set noise; prior work had not isolated direct test-set tuning on a widely used benchmark.The authors follow related work by giving the tuning procedure direct access to the test set (Schröder et al., 2025; Schneider et al., 2025; Makarova et al., 2021).Source: §2 · p. 2
  • Benchmark development often reuses labeled validation or development data for tuning and reporting, including in cited vision implementations and GLUE model selection procedures.Examples include PyTorch Contributors (2024), Wightman (2019), Devlin et al. (2019), and Liu et al. (2019).Source: §2 · p. 2
  • Hyperparameter tuning can substantially improve performance, but its effectiveness creates room for overfitting, including under adaptive search such as Bayesian optimization.Reported performance gains may partly reflect optimization thoroughness rather than algorithmic innovation (Brigato et al., 2021).Source: §2 · p. 3
  • Benchmark scores vary with sampling, initialization, training dynamics, and optimization, so repeated-run probabilities of outperforming can quantify uncertainty beyond single-run accuracy (Bouthillier et al., 2021).Large seed sweeps also find accuracy outliers on CIFAR-10 and ImageNet (Picard, 2021).Source: §2 · p. 3

3 METHODOLOGY

The methodology measures the test-accuracy advantage of selecting hyperparameters with test-set feedback rather than validation feedback. Paired runs and repeated sources of randomness are used to isolate and assess that difference.

3 METHODOLOGY

  • Adaptive overfitting is the accuracy advantage from selecting hyperparameters using test-set feedback rather than an independent validation set.The definition follows Dwork et al. (2015).Source: §3 · p. 3
  • The study randomly samples T configurations and compares validation-tuning with test-tuning, pairing runs so the evaluation dataset is the only selection-rule variation.Each configuration is trained on Dtrain and scored on Dval and Dtest; the selected configuration maximizes accuracy on the respective split.Source: §3 · pp. 3–4
  • The adaptive overfitting gap measures excess test accuracy attributable to direct test-set knowledge; it is nonnegative in paired runs and zero when both rules select the same or equally scoring configurations.The study also estimates test-tuning win probability, counting ties as 0.5, and calls overfitting significant when the 95% bootstrap interval lower bound exceeds 0.5.Source: §3 · p. 3
  • Each run uses an independent bootstrap split, and 30 paired runs vary data splits, hyperparameter samples, and initialization seeds to represent pipeline variance.The paired-run count is the minimum for the cited Noether-bound paired-comparison guarantee (Bouthillier et al., 2021).Source: §3 · p. 4

4 EXPERIMENTS

Across five benchmarks, the experiments test overfitting sensitivity and severity, then assess whether it changes model selection. Test-tuning is often statistically detectable, but its effect is generally comparable to run-to-run variability and rankings are largely preserved.

4 EXPERIMENTS

  • The experiments use MNIST-1D, CIFAR-10, and RTE, MRPC, and CoLA, evaluating eight architectures per benchmark with 30 paired runs and a 30-configuration search budget.For GLUE, train and validation sets are pooled because official test labels are withheld; MRPC is shown in the main text, with RTE and CoLA in Appendix A.Source: §4 · p. 4
  • As a sanity check, models trained and tuned directly on their test folds nearly memorize them, reaching 100% accuracy on both image benchmarks and 98.9%–100% on GLUE tasks.The GLUE test folds are smaller.Source: §4 · p. 5

4.1 EXP 1: OVERFITTING SENSITIVITY AND OVERFITTING SEVERITY

  • All models reach significant adaptive overfitting at about T=3 configurations on MNIST-1D and MRPC and T=13 on CIFAR-10; RTE and CoLA behave like MRPC.Win probability is not strictly increasing with T because a larger budget also benefits validation-tuning.Source: §4.1 · p. 5, Figure 1 · p. 5
  • The two tuning rules select the same configuration in many runs: the zero-gap share is 41.2% on MNIST-1D, 48.3% on CIFAR-10, and 12.9% on MRPC.The authors report that identical configuration selection, rather than accidental score ties, overwhelmingly explains zero gaps across benchmarks.Source: §4.1 · p. 5, Table 1 · p. 6
  • On 20 of 24 architecture–benchmark pairs, the mean overfitting gap is below run-to-run standard deviation; the image-benchmark gap is roughly half that deviation, while MRPC reaches 1.13 times it.Four MRPC architectures sit marginally above the deviation.Source: §4.1 · p. 6, Figure 2 · p. 6
  • With T=100 and larger fixed MNIST-1D splits, 16 of 30 runs still had zero gap, nearly unchanged from 17 of 30 in the main setup.The win probability was 0.73 at T=100, with its confidence-interval lower bound above 0.5.Source: §4.1 · p. 6
  • On independent CIFAR-10.1, validation-tuned and test-tuned configurations score within run-to-run noise of each other, while tuning directly on CIFAR-10.1 beats both by roughly 0.4%.The selected CIFAR-10 configurations were re-evaluated on a 2,000-image replication drawn from the same source distribution.Source: §4.1 · p. 6

4.2 EXP 2: HOW MUCH DOES OVERFITTING INFLUENCE MODEL SELECTION?

  • Test-tuning preserves architecture rankings across benchmarks: rankings are identical on MNIST-1D and CIFAR-10, with only a near-tie adjacent swap on MRPC, and the top-ranked model remains unchanged.RTE and CoLA show the same pattern; the near-tie MRPC architectures differ by 0.30% against 1.34% run-to-run standard deviation.Source: §4.2 · p. 7, Figure 3 · p. 7

5 DISCUSSION AND LIMITATIONS

Across five benchmarks, test-tuning produces small gains relative to experimental variability while leaving the top-ranked model unchanged and rankings nearly preserved. The findings are bounded by the study’s search procedure, budget, domains, and per-search design.

5 DISCUSSION AND LIMITATIONS

  • Zero-gap runs—41.2% on MNIST-1D, 48.3% on CIFAR-10, and 12.9%–25.8% on GLUE—reflect configurations winning on both splits; small, noisy GLUE splits weaken agreement.A larger-set MNIST-1D rerun with 100 trials kept the zero-gap fraction near 53%.Source: §5 · pp. 7–8
  • Across five benchmarks, test-tuning yields a consistent improvement within natural variability; rankings are identical or change by one adjacent inversion, and top-ranked models remain unchanged.The table reports model rankings alongside adaptive-overfitting differences and run-to-run standard deviations.Source: §5 · pp. 7–8, Table 2 · p. 8

5.1 LIMITATIONS

  • Random search evaluates configurations independently; Bayesian optimization, Hyperband, and manual iterative tuning reuse earlier results and could produce larger gaps.Source: §5.1 · p. 9
  • Each run uses T = 30 samples, and larger compute budgets remain unexplored despite a follow-up MNIST-1D run with T = 100 showing no meaningful difference.Batch size was fixed rather than tuned.Source: §5.1 · p. 9
  • The study excludes larger GLUE tasks, other modalities, and regression, where adaptive overfitting may be more severe.Source: §5.1 · p. 9
  • Each estimate covers one search, not cumulative benchmark reuse across groups, architecture choices, or publication filtering.Increasing T adds queries from the same procedure but does not capture community-level adaptation.Source: §5.1 · p. 9
  • With K = 30 paired runs, confidence intervals are 0.10–0.17 wide; weakest-architecture results are not robust individually, though the overall pattern persists.Source: §5.1 · p. 9
  • Evaluation covers common deep-learning architectures in image and text domains, not other machine-learning algorithms.Source: §5.1 · p. 9

6 CONCLUSION

The conclusion reports significant but small adaptive overfitting across the evaluated benchmarks, with model rankings largely preserved. It also describes the work’s disclosure, ethical, and reproducibility practices.

6 CONCLUSION

  • Test-tuning produced significant adaptive overfitting on all 40 architecture–benchmark combinations, but its effect was small relative to natural experimental variation.Validation and test tuning selected the same configuration in 41.2% of MNIST-1D runs, 48.3% of CIFAR-10 runs, and 12.9%–25.8% of GLUE runs.Source: §6 · p. 9
  • Test-tuning acted as a near-uniform accuracy shift: rankings were identical on MNIST-1D, CIFAR-10, and RTE, and the top-ranked model stayed unchanged everywhere.MRPC and CoLA each had one adjacent pair exchange places, separated by far less than run-to-run noise.Source: §6 · p. 9

AI USE STATEMENT

  • The authors report using generative AI for coding support and writing clarity, while manually verifying code and checking edits against the original intent.They state that AI was not used to generate research ideas, design experiments, produce results, or create data.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The work studies evaluation methodology on public datasets and releases code and analysis scripts intended to support independent verification.The authors anticipate no direct negative societal consequences and say the findings may help researchers and reviewers assess results in light of selection protocols.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors state that code, configurations, and analysis scripts are available and that all experiment randomness derives reproducibly from one master seed.The statement points to appendices for experimental settings, confidence intervals, and supporting analyses.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL GLUE DATASETS: RTE AND COLA

On RTE and CoLA, adaptive overfitting was significant, but its severity was generally within run-to-run variability. Test-tuning largely preserved model rankings and the top-ranked model on both datasets.

Overfitting sensitivity

  • Both datasets reached statistically significant adaptive overfitting at around T=3, and all 16 architecture–dataset pairs showed significant adaptive overfitting.The sensitivity threshold was similar to MNIST-1D and MRPC and below CIFAR-10’s T=13.Source: A ADDITIONAL GLUE DATASETS: RTE AND COLA · pp. 15–16

Runs with no gap

Overfitting severity

Model selection

  • On both RTE and CoLA, rankings were almost unchanged after test-tuning, and the top-ranked model stayed the same.RTE rankings were identical; CoLA had one adjacent inversion between xlnet-base and distilbert, whose validation-tuned means differed by 0.14% against 0.88% run-to-run standard deviation.Source: Table 4 · p. 15, A ADDITIONAL GLUE DATASETS: RTE AND COLA · p. 15

B IMPLEMENTATION DETAILS

The implementation details describe training and evaluation settings for the image and GLUE experiments, while the search-space section specifies the tuned GLUE hyperparameters and their ranges.

B IMPLEMENTATION DETAILS

  • Image experiments used training and evaluation batch sizes of 128 and 512, respectively; GLUE used 32 and 128.The GLUE training batch size followed published fine-tuning recipes for the evaluated model families.Source: B IMPLEMENTATION DETAILS · p. 17
  • CIFAR-10 training applied random crops with 4-pixel padding and random horizontal flips, while evaluation applied only per-channel normalization.Training data were normalized to zero mean and unit variance per channel.Source: B IMPLEMENTATION DETAILS · p. 17
  • CIFAR-10 training steps were tuned from 75,000 to 141,000, corresponding to roughly 160–300 epochs at batch size 128.The authors chose this range to allow configurations to converge while keeping experiments manageable, since the goal was to measure adaptive overfitting rather than reproduce state-of-the-art accuracy.Source: B IMPLEMENTATION DETAILS · p. 17
  • The GLUE trials fine-tuned all model parameters from fresh pretrained checkpoints with a new classification head, while holding batch size at 32.Epochs, warmup ratio, learning rate, and weight decay were tuned; each trial’s seed covered head initialization and data ordering.Source: B IMPLEMENTATION DETAILS · p. 17

C HYPERPARAMETER SEARCH SPACES

  • The GLUE search space was shared across three tasks and eight architectures, with ranges bracketing published fine-tuning grids for the model families.The tuned parameters were learning rate, weight decay, warmup ratio, and epoch count.Source: Table 7 · p. 18

D ARCHITECTURES

The appendix describes the evaluated MNIST-1D networks, CIFAR-10 architectures, and GLUE checkpoints, then reports ranking-preservation measures for selected benchmarks.

D.1 MNIST-1D

  • For MNIST-1D, the evaluated MLP family consists of networks that all use ReLU activations and residual connections between hidden layers, as stated in the table description.Source: Table 8 · p. 19

D.2 CIFAR-10

  • The CIFAR-10 architecture set includes reference implementations without architectural modifications.It covers ResNets, Shake-Shake, VGG, MobileNetV2, and ResNet-9.Source: Table 9 · p. 19

D.3 GLUE

  • The same eight public Hugging Face transformer checkpoints were evaluated on all three GLUE tasks, without architectural modification.Each trial freshly loaded a checkpoint and attached a randomly initialized classification head.Source: Table 10 · p. 19

E RANKING PRESERVATION DETAILS

  • Ranking preservation was assessed by fitting test-tuned accuracy against validation-tuned accuracy on per-architecture means and computing Spearman and Kendall rank correlations.Confidence intervals used a 100,000-resample pairs bootstrap for the linear fit and a 20,000-resample cluster bootstrap for rank correlations.Source: E RANKING PRESERVATION DETAILS · pp. 19–20

E.1 MNIST-1D

  • The reported results give Spearman ρ = 1.000 with 95% bootstrap CIs [0.929, 1.000], [0.976, 1.000], and [0.905, 1.000], and Kendall τ = 1.000 with CIs [0.857, 1.000], [0.929, 1.000], and [0.786, 1.000].Source: §E.1 · p. 20, §E.2 · p. 20, §E.3 · p. 20

E.4 MRPC

Across MRPC and CoLA, test-tuned model rankings track validation-tuned rankings closely, while the upper-bound analysis shows that random search captures little of the available test-set headroom.

E.4 MRPC

  • Spearman ρ = 0.976 and Kendall τ = 0.929 for MRPC; its single adjacent rank swap is between mobilebert and electra-small, whose means differ by 0.30% versus 1.34% run-to-run SD.The 95% bootstrap CIs are [0.905, 1.000] for Spearman ρ and [0.786, 1.000] for Kendall τ.Source: §E.4 · p. 21

E.5 COLA

  • Spearman ρ = 0.976 and Kendall τ = 0.929 for CoLA; its single rank swap is between xlnet-base and distilbert, whose means differ by 0.14% versus 0.88% run-to-run SD.The 95% bootstrap CIs are [0.952, 1.000] for Spearman ρ and [0.857, 1.000] for Kendall τ.Source: §E.5 · p. 21

F UPPER-BOUND REFERENCE

G MLP-GIANTT=100: ADAPTIVE OVERFITTING UNDER INCREASED TUNING AND DATA BUDGET

The MLP-Giant rerun tests a larger tuning budget and larger MNIST-1D splits; a separate CIFAR-10.1 comparison finds little transfer of the bootstrap-test tuning advantage.

G MLP-GIANTT=100: ADAPTIVE OVERFITTING UNDER INCREASED TUNING AND DATA BUDGET

  • The mean ∆AO stays small relative to validation-tuned noise, while ˆP(∆AO > 0) remains significant across the full tuning-budget range.The rerun uses T = 100 trials per run and 10,000/10,000/10,000 MNIST-1D splits; the figure reports a 95% percentile-bootstrap CI.Source: Figure 8 · p. 23

H ADAPTIVE OVERFITTING TRANSFER TO CIFAR-10.1

  • On CIFAR-10.1, the pooled val-tuned versus test-tuned gap is −0.04%, while CIFAR-10.1-tuned is ahead by +0.43%.The pooled val-tuned and test-tuned columns are indistinguishable within run-to-run noise; the directly CIFAR-10.1-tuned configuration has a small but consistent margin.Source: Table 12 · p. 23

I DATA SPLITTING

The benchmark splits use paired, nonoverlapping validation and test sets drawn from the same source pools. The authors also examine whether the chosen hyperparameter ranges are supported by winner distributions.

I DATA SPLITTING

  • Each run samples its training set with replacement from a benchmark-specific pool, then splits the unpicked remainder in half for validation and test.The pools contain 4,000 MNIST-1D examples, CIFAR-10’s 60,000-example training-plus-test pool, and concatenated official train and validation splits for GLUE.Source: I DATA SPLITTING · p. 23
  • Validation and test sets never overlap with each other or training, remain fixed across T trials, and are redrawn for each of K = 30 paired runs.Typical per-run validation and test sizes are 736 each for MNIST-1D, 11,040 each for CIFAR-10, and 509, 750, and 1,763 each for RTE, MRPC, and CoLA, respectively.Source: I DATA SPLITTING · p. 24
  • The GLUE pools combine official train and validation splits because official test labels are withheld, making validation the community’s de facto test set.The MLP-Giant rerun instead draws three disjoint 10,000-example sets from a larger 70,000-example MNIST-1D set, without replacement.Source: I DATA SPLITTING · p. 24

J HYPERPARAMETER RANGE VALIDITY

  • For GLUE, ranges bracket published fine-tuning grids except ELECTRA-small’s 3 × 10−4 recipe, above the 10−4 learning-rate cap; winner distributions put lr near 3–5 × 10−5 and epochs at the upper bound of 10.Weight decay and warmup ratio are close to flat; the learning-rate cap is kept below the divergence region of base-size models.Source: J HYPERPARAMETER RANGE VALIDITY · p. 24, Figure 11 · p. 25, Figure 12 · p. 26
  • MNIST-1D winner distributions show no edge clustering: all three hyperparameters have modes well inside their search bounds.Figure 9 excludes the 30.8% of trials with lr decay = 0 from its lr decay panel.Source: Figure 9 · p. 24
  • CIFAR-10 max steps winners cluster at the 141,000 upper bound; extending the range would benefit both selection rules equally and is unlikely to change ∆AO.The other reported CIFAR-10 hyperparameter modes lie well inside their bounds.Source: Figure 10 · p. 25
Loading 2610.05902v1…