PaperlayerView exact revision on arXiv

Source-linked AI summary

A Generalisation Signal Need Not Be a Model-Selection Signal

Aditya Nagarsekar, M P Ashish Bhat, Aadi Nesarkar, Vrishti Godhwani, Rahul Yedida, Aditya Challa, Danda Sravan, Snehanshu Saha

arXiv:2609.39099v1cs.LGcs.AI

TL;DR

When deployment shifts change which model is best, validation-based selection can fail. This paper tests a forward-only activation proxy alongside Hessian measures for selecting trained models, finding that generalisation-gap signal does not reliably translate into deployment-best selection.

  • Problem

    Validation-based selection can fail when deployment shifts change the ordering of candidate models, leaving model selection unreliable despite accurate validation estimates.

    Source: §1 · p. 1

  • Method

    The study evaluates a cheap forward-only activation proxy motivated by a loss-Hessian norm bound, alongside common curvature measures, for selecting trained models under biological shifts.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Across biological shifts, geometry did not reliably rescue weak validation rankings: the proxy could help, fail, or harm selection even when it carried generalisation-gap signal.

    Source: §3 · p. 3

  • Takeaways & Limitations

    Training-set measures can carry generalisation information without reliably selecting the best deployment model, so they are best treated as auxiliary diagnostics and selections should be audited for collapse.

    Source: §5 · p. 4

    The negative results cover the tested low-score, rank-sum, and multi-objective/rank-sum selection rules; combining geometry with training loss or varying signal direction was not tested.

    Source: A Limitations and scope · p. 7

Abstract

from arXiv · show

Model selection in computational biology often relies on validation data drawn from the training regime, even when deployment lies outside it. When validation no longer preserves which model is best, a natural alternative is to rank candidates using properties of the trained network itself. We test this idea using a novel, forward-only proxy motivated by the norm of the Hessian, alongside common Hessian measures, across molecular property, protein fitness, and drug-response tasks. Contrary to our hypothesis, geometry does not become more useful as validation Spearman correlation deteriorates: augmenting validation helps some shifts but significantly harms others. More surprisingly, the proxy still correlates with generalisation gap on most tasks even when Hessian trace and top-eigenvalue relationships are weak or reversed, yet this signal does not reliably identify the deployment-best model. A curvature bound need not preserve cross-model rankings, and low geometric scores can even favour collapsed predictors. Thus, a generalisation signal need not be a model-selection signal.

1 Introduction and Background

Validation is useful under distribution shift only when it approximately preserves which candidate performs best at deployment. The paper asks whether trained-network geometry can help select among models when validation rankings fail, and finds that generalisation signals do not necessarily select the deployment-best model.

1 Introduction and Background

  • Validation-based selection fails when a shift changes which candidate is best, even if validation is estimated accurately.Deployment may be harder without disrupting candidate ordering; the critical issue is whether validation preserves that ordering.Source: §1 · p. 1
  • Biological evaluation is complicated by dependence, confounding, preprocessing leakage, and distributional structure.Shifts can reverse conclusions in protein-fitness benchmarks, while unseen compounds and chemistry metrics can change drug-response rankings.Source: §1 · p. 1
  • Evidence that training-only geometry tracks out-of-distribution generalisation is mixed, and this paper tests whether such signals can select models when validation becomes unreliable.The study distinguishes selection among trained models from correlation with generalisation, using a cheap forward-only activation proxy motivated by a Hessian Frobenius-norm bound.Source: §1 · p. 2

2 Problem and Methodology

The study evaluates whether a training-only signal can replace or augment validation when validation and deployment rankings diverge. Its primary signal is a layer-averaged activation proxy motivated by a loss-Hessian norm bound, evaluated alongside curvature measures across paired biological-shift experiments.

2 Problem and Methodology

  • The primary signal is a cheap activation-based proxy motivated by a bound on the loss-Hessian norm, tested for association with generalisation gap and as a selection aid.It is computed from post-activation matrices using dropout off and on a fixed subset of training examples; the layer-averaged form is the preregistered Proxy.Source: §2 · p. 2
  • The study tests whether curvature becomes more useful as validation rank transfer deteriorates, including whether the proxy transfers its generalisation-gap signal to model ranking.The comparison includes exact λmax(∇²L_train), calculated over weights and biases without the L2 term using Hessian-vector products.Source: §2 · p. 2
  • Experiments pair selectors on the same 48 ReLU-MLP configurations per replication across molecular, protein-fitness, and drug-response tasks.Each replication redraws the split, candidate pool, initialization, and batch ordering; candidates train for 100 epochs without validation-based early stopping, and paired tests use Holm-corrected Wilcoxon signed-rank statistics.Source: §2 · p. 2

3 Results

Validation rank transfer varies substantially across biological shifts, and geometric signals do not reliably repair weak rankings. Proxy augmentation and geometry-guided search can help in some conditions but also harm or fail to improve selection.

3 Results

  • ρ=0.44–0.77 on molecular tasks, −0.10 on Hydrophobic Core, and 0.19 on GDSC2; training-set geometry did not reliably rescue weak ranking transfer.The table reports shared-pool deployment performance under benchmark-native metrics after the degeneracy audit.Source: §3 · p. 3, Table 1 · p. 3
  • The proxy roughly matched validation on molecular tasks but did not improve as validation became less reliable.On Hydrophobic Core it underperformed validation at 0.303 versus 0.320 Spearman; under NDCG, both geometric selectors were nominally below blind selection at 0.904 and 0.896 versus 0.909.Source: §3 · p. 3
  • On Lipophilicity scaffold, proxy-guided search scored 0.705 versus 0.688 for TPE and 0.701 versus 0.685 for HEBO, but Hydrophobic Core HEBO worsened at 0.273 versus 0.311.The sequential-HPO comparison reports deployment Spearman and uses random search as its reference; GDSC2 also deteriorated.Source: §3 · p. 3, Table 2 · p. 4
  • Augmenting validation significantly improved three confirmatory conditions (pHolm=0.027–0.047) but worsened Hydrophobic Core (∆=+0.73, pHolm<0.001).After the degeneracy audit, only Lipophilicity mismatch retained a significant augmentation benefit (pHolm=0.038); the Hydrophobic Core harm remained significant (pHolm=0.002).Source: §3 · p. 3
  • Correlation with generalisation gap was not sufficient for reliable model selection.Across conditions, the proxy could help, fail, or harm selection, so a generalisation signal need not be a model-selection signal.Source: §3 · p. 3

4 Failure analysis

The proxy can carry information about generalisation gap without matching standard curvature summaries or reliably ranking deployment performance. Extrapolation shifts preferred model configurations, and low geometric scores can favor collapsed predictors.

4 Failure analysis

  • After controlling for architecture and learning rate, the proxy retained modest gap associations on molecular tasks (ρ≈0.16–0.35), but not on Hydrophobic Core (−0.30) or GDSC2 (0.01).λmax and trace were weak or negative; on Lipophilicity mismatch the proxy anti-correlated with trace (−0.61) while correlating with relative flatness (+0.72).Source: §4 · p. 4
  • On Hydrophobic Core, the deployment oracle used a 2.05× larger learning rate, 4× greater width, and less regularisation than the validation-selected model.The shift truncates the observed target range, while deployment requires extrapolation to higher-fitness variants.Source: §4 · p. 4
  • On Hydrophobic Core, greater curvature accompanied better extrapolation (ρ(tr(H), Ldep−Ltrain)≈−0.52), reversing the expected trend.Because deployment loss is training loss plus generalisation gap, even an accurate gap signal can misrank candidates whose training losses differ.Source: §4 · p. 4
  • On Amylase, the proxy’s 1.39 median-MSE improvement coincided with a median dead-unit fraction of 0.998 and near-zero prediction variance.The selected predictors matched training-mean MSE (1.145), and Spearman was undefined in 14 of 20 replications; collapse can drive the proxy toward zero.Source: §4 · p. 4

5 Conclusion

Training-set measures can track generalisation without reliably selecting the best deployment model. The paper recommends treating them as auxiliary diagnostics and limits its conclusions to the models, representations, selection rules, and geometric criteria studied.

5 Conclusion

  • Training-set measures can carry generalisation signal without reliably selecting the best deployment model.The proxy correlates positively with generalisation gap on most tasks, but geometric signals can help, fail, harm selection, or favour degenerate predictors.Source: §5 · p. 4
  • The results show no consistent selection gain, not that training-set measures carry no information about deployment.Source: §5 · p. 4
  • Until a measure beats validation at choosing the deployed model, treat it as an auxiliary diagnostic and audit selections for collapse.The paper also recommends reporting rank transfer and comparing with validation and simple baselines.Source: §5 · p. 4
  • The conclusions are limited to the models, representations, selection rules, and geometric criteria studied.Source: §5 · p. 4

Use of large language models

Large language models assisted with code development, literature search, and language editing; the authors reviewed and take responsibility for the manuscript.

Use of large language models

Reproducibility statement

The reproducibility protocol fixes six seeds per outer run and retains predictions for post-hoc diagnostics. Conclusions are bounded by the tested architectures, representations, selection rules, and supervised regression setting; other approaches remain untested.

Reproducibility statement

  • Each outer run derives six deterministic seeds for data splitting, candidate generation, initialization, batch ordering, proxy estimation, and HPO sampling.Candidates are saved atomically, and predictions are retained so post-hoc diagnostics reuse the original trained models rather than retraining after an outcome is observed.Source: Reproducibility statement · p. 5

A Limitations and scope

  • The conclusions concern tested MLP architectures, fixed molecular and sequence representations, and evaluated geometric criteria.They do not establish that all curvature- or representation-based selectors fail, or that the same behavior holds for pretrained biological foundation models or learned representations.Source: A Limitations and scope · p. 7
  • The negative results apply to three tested selection rules, and the study did not test combining geometric signals with training loss or varying signal direction by condition.Because deployment loss is training loss plus generalisation gap, a gap signal alone may misrank candidates with different training losses; a negative proxy-gap association can also misdirect lowest-score selection.Source: A Limitations and scope · p. 7
  • The experiments focus on supervised regression with candidates selected from controlled hyperparameter sweeps.Uncertainty-aware selection, ensembling, active learning, and adaptation after shift may behave differently.Source: A Limitations and scope · p. 7
  • A next question is whether shift-aware or representation-aware signals can identify deployment ordering when validation rank transfer collapses.Source: A Limitations and scope · p. 7

B Protocol, reproducibility, and deviations from the pre-registered plan

The section specifies the training and selection protocol, constructs deployment shifts, and derives the activation-based motivation for the proxy while noting limits on its cross-model ranking interpretation.

B Protocol, reproducibility, and deviations from the pre-registered plan

  • Training used fixed-epoch MLPs and a stratified 48-configuration search; post-training signals and rank-sum rules selected candidates within each replication.The protocol was frozen before confirmatory runs; the degeneracy audit was added post hoc, and several analyses were labelled exploratory.Source: B Protocol, reproducibility, and… · pp. 8–9
  • The confirmatory family and audit affect interpretation: the dead-unit threshold followed the Amylase result, while adding Lipophilicity scaffold changes Val+Proxy significance.The primary inference uses unfiltered pools; audited results are reported as diagnostics, and added analyses are not pooled with confirmatory tests.Source: B Protocol, reproducibility, and… · pp. 9–10

C Datasets and split construction

  • Molecular, protein, and drug-response tasks use distinct shift constructions; the mismatch condition keeps scaffold-disjoint test data while making validation in-distribution.Molecular tasks use Morgan fingerprints; FLIP2 uses one-hot sequences, and GDSC2 holds out whole compounds.Source: C Datasets and split construction · p. 10

D.1 Setup and notation

  • The derivation defines a mean-squared-error network with penultimate activations and a final affine layer, then forms mini-batch activation matrices.The mini-batch matrix contains the corresponding penultimate activations as rows.Source: §D.1 · pp. 10–11

D.2 Statement

  • The final-layer Hessian is an exact quadratic-loss Hessian that is constant in the final-layer weights and depends on the network through its activations.Source: §D.2 · p. 11

D.3 Proof

  • The proof partitions training examples into disjoint mini-batches and combines inequalities to establish the stated bound.Source: §D.3 · p. 12

D.4 Why a valid bound need not give a usable ranking

  • A pointwise Hessian-norm bound does not ensure cross-model ranking preservation because candidate architectures can change the activations.The paper notes that its Section 4 comparisons do not directly test the proposition’s final-layer curvature ordering.Source: §D.4 · p. 13

D.5 From the bound to the proxy used in experiments

  • The experimental proxy adds width normalisation and hidden-layer averaging beyond the proposition; width normalisation can change rankings across widths.The penultimate-layer and layer-averaged forms are strongly correlated, but that does not address width normalisation.Source: §D.5 · p. 13

E Complete shared-pool results

Replacing validation with the proxy was Holm-significant only for Amylase in the unfiltered confirmatory results; augmenting validation improved some conditions and harmed Hydrophobic Core. Sequential search improved deployment where validation rankings transferred, but not on Hydrophobic Core.

E Complete shared-pool results

  • Only Amylase showed a Holm-significant proxy-versus-validation result in the unfiltered confirmatory analysis (20/20 replications); augmentation improved three conditions but harmed Hydrophobic Core.After the post-hoc audit, only Lipophilicity mismatch (pHolm = 0.038) and Hydrophobic Core (0.002) remained significant; Caco2 scaffold and Amylase moved to 0.094.Source: E Complete shared-pool results · p. 13, Table 12 · p. 17
  • For Caco2 scaffold, Val+Proxy had 7 wins, 3 losses, and 10 ties, so its median paired difference was zero despite a significant signed-rank test.The signed-rank test discards ties, and a changed adjusted p-value need not indicate a changed estimated effect.Source: E Complete shared-pool results · p. 13
  • On audited pools, the #params baseline remained robust on Hydrophobic Core across MSE, MAE, and Spearman, while the shared-pool geometric win was fragile across metrics.Table 13 reports selectors that beat validation at Holm-adjusted α = 0.05 under each metric.Source: E Complete shared-pool results · p. 14, Table 13 · p. 17

F Sequential hyperparameter optimisation

  • Multi-objective arms deployed the observed trial with the lowest rank sum of validation MSE and the training-side signal, not necessarily a Pareto-optimal trial.This rule made sequential-search and shared-pool comparisons comparable, but the arms measure search together with that deploy rule.Source: F Sequential hyperparameter optimisation · p. 17
  • The Lipophilicity-scaffold improvement is a boundary case: it was descriptive, failed to reproduce on matched random and Caco2 scaffold splits, and the signal family was harmful on held-out compounds.The scaffold condition was pre-registered as descriptive rather than confirmatory, so it does not establish a general advantage.Source: F Sequential hyperparameter optimisation · p. 17
  • Pooled comparisons across search arms are descriptive because method-by-replication observations share splits, initialisation seeds, and data ordering.Inference was instead paired within replication and corrected within each condition.Source: F Sequential hyperparameter optimisation · pp. 17–18
  • Extending search from 8 to 48 trials improved deployment where validation rankings transferred, but not on Hydrophobic Core.These results concern multi-objective search combined with the study's rank-sum deploy rule; other uses of geometry in search were not tested.Source: F Sequential hyperparameter optimisation · p. 17, Figure 2 · p. 18, Table 14 · p. 18

G Degeneracy audit

Low geometric scores can select collapsed predictors, and the consequences depend on the condition. The audit also shows that selection outcomes can reflect hyperparameter alignment or capacity-like signals rather than curvature.

G Degeneracy audit

  • Excluding networks with more than half their units dead shrinks the Amylase difference to a non-significant −0.36 (pHolm = 0.108).Removing only constant predictors leaves the proxy selecting a nearly inactive network with 96% dead units and prediction spread near 10−5.Source: G Degeneracy audit · p. 19, Table 17 · p. 20
  • On Hydrophobic Core, no selector deploys a constant predictor, and a training-mean predictor’s MSE of 29.8 is worse than every selector in Table 10.Unlike Amylase, many networks are partially dead but none is constant; Figure 3 contrasts these condition-specific outcomes.Source: G Degeneracy audit · p. 19, Figure 3 · p. 19, Table 16 · p. 20
  • On Amylase, the proxy improves deployment MSE by about 1.39 and selects a constant predictor in 80% of replications.The selected model has median dead-unit fraction 0.998 and median deployment MSE 1.145, equal to the training-mean predictor; its Spearman correlation is undefined in 14 of 20 replications.Source: G Degeneracy audit · p. 18, Figure 3 · p. 19

H Hyperparameter mechanism

  • On Hydrophobic Core, selecting the smallest model beats validation under all three metrics because the picks align better on other hyperparameters, not because smaller widths extrapolate better.The deployment oracle prefers wider models; #params and validation choose the same width, while #params selects the first-drawn among the one to five smallest networks.Source: H Hyperparameter mechanism · pp. 19–20

I Geometry and curvature diagnostics

  • The proxy’s partial correlation with the generalisation gap is positive on molecular conditions and Amylase, negative on Hydrophobic Core, and near zero on GDSC2.The proxy–λmax marginal association is largely induced by learning rate; controlling for it collapses the relationship.Source: I Geometry and curvature diagnostics · p. 20
  • On the fixed-architecture subset (two hidden layers of width 256), top Hessian eigenvalue has essentially no relationship with deployment MSE on constructed mismatch (ρ = −0.05), but reverses under fitness extrapolation (ρ = −0.76), with sharper minima deploying better.The top eigenvalue is a median 0.12–0.14 of the trace, but measuring the remaining curvature does not help. The relative-flatness proxy behaves like a capacity or norm quantity rather than curvature; on Hydrophobic Core, its decomposition inverts.Source: I Geometry and curvature diagnostics · pp. 20–21, Figure 4 · p. 21

J Ablations and additional experiments

The additional experiments do not reveal a setting where geometry reliably rescues selection. Their outcomes vary by regime, and harder deployment alone does not necessarily change model rankings.

Fixed architecture

  • With fixed architecture, validation is worse than uniform random selection on Hydrophobic Core, partly because it selects models with more dead units than the oracle.Pinning networks to two hidden layers of width 256 does not generally rescue selection.Source: J Ablations and additional experiments · p. 22, Table 21 · p. 23

Validation-free selection

  • In validation-free selection, Train+Proxy has lower median deployment MSE than Train in all four conditions and than random in three, but these differences were not tested.In sequential search, all six signal-augmented arms beat training-MSE-only search on Lipophilicity mismatch, while none do on Hydrophobic Core and one is significantly worse.Source: J Ablations and additional experiments · p. 22, Table 21 · p. 23

Shift severity

Hard-example negative control

  • The hard-example split raised rank transfer to +0.83 and +0.71 on two replications, rather than manufacturing validation failure.Because the construction is deliberately circular, it supports only the negative conclusion that prediction difficulty and model-ranking shift are separate phenomena.Source: J Ablations and additional experiments · p. 23

K Benchmark sanity checks

The benchmark checks show that candidate pools contain models of meaningfully different deployment quality, while also documenting performance and training-budget limits. The paper does not claim convergence or leaderboard-level molecular performance.

TDC molecular tasks

  • On matched scaffold splits, validation-deployed models have median Caco2-Wang MAE 0.416 versus a leaderboard best near 0.256, and Lipophilicity 0.663 versus 0.456.The controlled comparison uses Morgan fingerprints, a single MLP, fixed training and a shared candidate pool rather than task-tuned representations or searches.Source: K Benchmark sanity checks · p. 23

Training budget

  • Candidates train for 100 fixed epochs without early stopping, and the authors do not claim convergence.Training MSE fell by a median 13% between epochs 50 and 100; comparisons establish competitive candidate pools, not convergence of every candidate.Source: K Benchmark sanity checks · p. 24

FLIP2

  • On Hydrophobic Core, the one-hot MLP exceeds published benchmark baselines, while Amylase supervised performance is poor, consistent with the benchmark report.The appendix analyzes the consequences of the Amylase split for selection.Source: K Benchmark sanity checks · p. 24

GDSC2

Related papers

Loading 2609.39099v1…