PaperlayerView exact revision on arXiv

Source-linked AI summary

Revalidation Beats Stateful Routing for Scientific Surrogates Under Distribution Shift

Harshil Lodhiya

arXiv:2609.29715v1cs.LGcs.AI

TL;DR

When noise, input support, or physical parameters change, it is unclear whether surrogate selection should use a stateful controller or fresh validation of each batch. RegimeShift-Surrogates tests these choices in a reproducible streaming benchmark. Across its confirmatory episodes, current-window validation outperforms the best fixed model and the tested pooled stateful selectors, with scope limited to the benchmark's synthetic tasks and information setting.

  • Problem

    Surrogates are often selected during development and reused as new data arrive, leaving open whether changing conditions call for stateful control or renewed validation.

    Source: §1 · p. 1

  • Method

    The paper evaluates fixed, current-validation, and stateful selection in a streaming benchmark with eight task families, four regime schedules, and a surrogate portfolio.

    Source: §1 · pp. 1–2

  • Results

    Across 320 confirmatory episodes, current-window validation substantially outperforms the best fixed model and every pooled stateful selector tested, though the advantage is not universal.

    Source: §1 · p. 2, §9 · pp. 8–9

  • Takeaways & Limitations

    For streams with a modest current validation set available for every candidate, fresh evidence is a demanding baseline that a more elaborate controller should have to beat.

    Source: §7.1 · p. 7

    The benchmark uses analytic functions and simulated ODE trajectories rather than laboratory measurements or a large public PDE dataset, and trains every candidate in every window.

    Source: §8 · p. 8

Abstract

from arXiv · show

Surrogate models are often chosen during development and then left in place as new measurements arrive. That practice becomes risky when noise, input support, or physical parameters change. We asked whether such changes call for a stateful adaptive controller, or whether it is enough to validate the candidate models again on each new batch. To study this question, we built RegimeShift-Surrogates, a reproducible streaming benchmark spanning eight analytic and dynamical tasks, four stationary or shifting regimes, ten held-out seeds, and eight classical, multilayer-perceptron, and Kolmogorov-Arnold network surrogates. The confirmatory run contains 30,720 model fits and 3,200 scored deployment windows. Choosing the model with the lowest validation loss in the current window yields mean log regret 0.091 against a per-window oracle; the best fixed model chosen in hindsight yields 0.192. The paired difference is -0.101 (hierarchical bootstrap 95% CI [-0.165, -0.040]; Holm-adjusted p = 0.0469), with revalidation ahead in 26 of 32 task-scenario combinations. None of the stateful alternatives, including exponential smoothing, dual-timescale adaptation, Page-Hinkley resets, or margin gating, improves the pooled result, and delayed bias correction makes it worse. Oracle choices also differ substantially by task: k-nearest neighbors dominate the damped oscillator, vanilla KAN is often selected for two-dimensional surfaces, and MLPs lead on the Runge and Van der Pol tasks. In this benchmark, fresh validation evidence is useful; carrying old evidence forward is often not.

1 Introduction

Scientific surrogates are often selected once and reused, even as noise, input support, or physical parameters change; the paper asks whether to retain that choice or reconsider it with each new batch. It introduces a controlled streaming benchmark to compare fresh validation, fixed selection, and stateful alternatives.

1 Introduction

  • The benchmark spans eight task families, four regime schedules, eight surrogate families, and 30,720 confirmatory fits.It is designed as a reproducible benchmark.Source: §1 · p. 2
  • The study compares fixed, current-validation, temporally smoothed, change-detected, and oracle selectors under a predeclared protocol.Source: §1 · p. 2
  • Current revalidation outperforms the best fixed model chosen in hindsight, while the tested stateful selectors usually retain evidence too long.The comparison is conducted under a predeclared protocol.Source: §1 · p. 2
  • Task-level specialization and timing results show why a single surrogate is not sufficient across the benchmark.Source: §1 · p. 2

2 Related Work

Prior work spans scientific surrogate families, online selection under non-stationarity, and routing systems. This paper situates sequential surrogate choice where every candidate's current validation loss is visible before deployment.

2.1 Scientific surrogates

  • Scientific surrogate comparisons span RBF and Gaussian-process baselines, ordinary and physically constrained neural models, and KANs; portfolio differences make ranking changes across tasks or regimes plausible.Source: §2.1 · p. 2

2.2 Online selection under non-stationarity

  • Non-stationary bandit work includes forgetting older rewards [Garivier and Moulines, 2011], contextual methods, regret-preserving base-learner selection, and break detection with restart; here every candidate's current validation loss is visible, making current-best validation the baseline.Page–Hinkley is described as a classical cumulative detector.Source: §2.2 · p. 2

2.3 Routing systems

  • Routing research covers heterogeneous-model choice, learned quality–cost cascades, feedback-control routing, and preference-trained routers; prior scientific-agent selection was one-shot, while this study follows choices over time and reports stateful controllers.Source: §2.3 · p. 2

3 Problem Formulation

The formulation treats surrogate selection as a decision made from current validation evidence before clean deployment loss is revealed. It evaluates decisions against a per-window oracle using episode-level mean log regret.

3 Problem Formulation

  • Each model receives the same noisy training set, and the selector observes validation loss for every model before choosing, while clean deployment loss is revealed only afterward.The clean outcomes used by oracle references are retrospective and not available to deployable selectors.Source: §3 · pp. 2–3, Figure 1 · p. 3
  • The per-window oracle is the reference for selection decisions, while clean evaluation data remain unavailable at decision time.Source: §3 · pp. 2–3
  • Mean log regret is the primary episode-level endpoint; zero regret means matching the oracle, and the logarithmic form is unchanged by multiplicative task-error rescaling.It can be read as a log error ratio.Source: §3 · p. 3

4 Benchmark Design

RegimeShift-Surrogates combines analytic targets and simulated dynamical systems with stationary and shifting streams. It compares a fixed portfolio of classical and neural surrogates under deployable and retrospective selection rules.

4.1 Tasks

  • Eight task families comprise six analytic targets and two dynamical systems: the damped oscillator and Van der Pol trajectories generated with fourth-order Runge–Kutta integration.Training, validation, and test parameter draws do not overlap within a window.Source: §4.1 · p. 3

4.2 Regime schedules

  • Episodes have twelve windows spanning stationary noise, abrupt noise with recovery, gradual complexity shifts, or recurring mixed regimes; oscillator parameter ranges also change without selector-visible regime labels.The schedules specify noise, input-support, frequency, bias, outlier, or parameter-range changes.Source: §4.2 · pp. 3–4

4.3 Surrogate portfolio

  • The fixed portfolio includes four classical candidates and four neural candidates, including two MLP sizes, vanilla KAN, and ER-KAN; one global configuration is used per model rather than task-specific searches.Source: §4.3 · p. 4

4.4 Selectors

  • Margin gating retains history unless the current winner exceeds an estimated uncertainty margin.Source: §4.4 · p. 4
  • Deployable selectors include static development choice, current-window validation, EWMA, dual EWMA, Page–Hinkley reset, and margin gating; best-fixed and per-window oracle rows are retrospective and cannot be deployed.The two EWMA estimates use fast and slow rates, while the reset method uses a cumulative detector.Source: §4.4 · p. 4

5 Experimental Protocol

The confirmatory protocol extended the benchmark to eight task families and ten held-out seeds, then compared selectors using paired task–scenario–seed episodes. Hierarchical resampling and task-level randomization supported the comparisons.

Confirmatory run

  • The confirmatory run covered 320 episodes, 30,720 model-window fits, and 3,200 scored windows using ten seeds not inspected during development.The development runs used six tasks, five seeds, and ten windows to debug the stream, fix selector parameters, and choose current validation as the primary reference.Source: §5 · p. 4

Statistical analysis

  • Comparisons were paired within task, scenario, and seed, with 10,000 hierarchical bootstrap resamples for confidence intervals.The two-sided randomization test enumerated all 28 sign assignments across eight task-level mean differences; Holm correction covered six prespecified non-oracle comparisons.Source: §5 · p. 4

Reported comparisons

  • Table 1 reports selector results over 320 episodes with hierarchical-bootstrap confidence intervals; lower regret and MSE are better.Table 2 reports paired episode-level differences of current validation minus each comparator, where negative values favor current validation.Source: Table 1 · p. 5, Table 2 · p. 5

6 Results

Across the confirmatory benchmark, current-window validation beat the best fixed model and was the strongest deployable selector, although its advantage varied by task and scenario. The oracle favored different model families across tasks, and delayed feedback-based correction worsened regret.

6.1 Overall selector performance

  • Mean log regret was 0.091 for current validation versus 0.192 for the best fixed model chosen in hindsight, a 52.7% reduction.Geometric-mean clean MSE was 0.004812 versus 0.005326, and current validation matched the per-window oracle in 72.3% of scored windows.Source: §6.1 · p. 5

6.2 Paired comparisons

  • The paired difference against the best fixed model was −0.101 (95% CI [−0.165, −0.040]; Holm-adjusted p = 0.0469).The comparison with Page–Hinkley reset remained unresolved: the corrected task-cluster randomization result was p = 0.0703.Source: §6.2 · p. 5

6.3 Where revalidation helps

  • Current validation beat the best fixed model in 26 of 32 task–scenario cells, with the largest gaps where the oracle model changed across windows.Exceptions were stationary, abrupt, and gradual sine; stationary and gradual step; and damped oscillator under abrupt noise, where a single model tended to remain strong and revalidation could react to noise.Source: §6.3 · pp. 5–6

6.1 Overall selector performance

  • Current validation was the strongest deployable selector; adaptive dual EWMA had regret 0.120, Page–Hinkley reset 0.134, and ordinary EWMA 0.155.None of these stateful alternatives improved on current validation.Source: §6.1 · p. 5, Figure 2 · p. 6

6.4 Model specialization

  • Oracle choices varied by task: k-NN led in 96% of damped-oscillator windows, vanilla KAN in 47–51% on three task families, and MLP in 60% of Runge and 49% of Van der Pol windows.Vanilla KAN had the lowest geometric-mean clean MSE across fits but won fewer than one quarter of individual windows.Source: §6.4 · p. 6

6.5 Cost and delayed feedback

  • Classical models fit several orders of magnitude faster than neural models, while vanilla KAN had the highest median fit time and lowest unconditional error.With limited training budgets, deciding which candidates to fit becomes an additional deployment decision.Source: §6.5 · p. 6
  • Mean log regret with delayed-feedback correction was 0.171, 0.157, and 0.153 at delays of 0, 2, and 4 windows, versus 0.091 for the uncorrected rule.The apparently better longer-delay result was not evidence that old feedback helps.Source: §6.5 · p. 6
  • Sparse feedback from the selected model was a poor basis for correcting rankings built from validation losses for the full portfolio.Longer delays simply permitted fewer unstable updates in these short episodes.Source: §6.5 · p. 7

7 Discussion

Fresh validation can outperform carrying prior rankings forward when current validation data are available for every candidate. Across the studied streams, task-level specialization supports routing, but tested stateful controllers offer no reliable pooled advantage.

7.1 Why the simple method wins

  • After a regime change, stored rankings can become a source of bias, and reacting late costs more than smoothing gains in gradual and recurring streams.Smoothing can reduce variance while rankings remain stable.Source: §7.1 · p. 7

7.2 Implications for scientific agents

  • Task-stratified oracle choices show specialization, supporting a routing layer; for these streams, the most reliable router has neither a learned routing model nor long-term state.Stored evidence needs an expiration rule that reflects how quickly the environment can change.Source: §7.2 · p. 7, Figure 4 · p. 8

7.3 Negative results as design constraints

  • Dual-timescale smoothing does not beat current validation, Page–Hinkley resets lack a reliable multiplicity-adjusted advantage, and delayed bias correction performs worse.The authors argue more complex controllers should be motivated by a missing source of information, not complexity alone.Source: §7.3 · p. 8

8 Limitations

The benchmark and its conclusions are bounded by synthetic tasks, prescribed shifts, and an evaluation that trains every candidate. The authors identify public datasets, measured shifts, and restricted candidate training as useful next tests.

8 Limitations

  • The benchmark uses analytic functions and simulated ODE trajectories, without laboratory measurements or a large public PDE dataset; its noise and regime changes are prescribed.The shifts are not estimated from an instrument.Source: §8 · p. 8
  • Because every candidate is trained in every window, the main result concerns selection after predictions are available, not which models to train under a limited budget.Source: §8 · p. 8
  • Globally fixed hyperparameters leave some model–task pairs better tuned than others, while the neural models are deliberately small for the data-scarce setting.Source: §8 · p. 8
  • Clean targets are available to the evaluator but not in deployment, and eight task families leave a coarse task-level randomization test.Source: §8 · p. 8
  • A next test would use public scientific datasets and measured shifts, restrict candidate training, and assess whether a contextual policy can identify unreliable revalidation cases.Source: §8 · p. 8

9 Conclusion

Across confirmatory episodes, current-window validation outperforms the best fixed model and pooled stateful selectors tested. The authors recommend starting with fresh evidence and adding memory only when it demonstrably helps.

9 Conclusion

  • Across 320 confirmatory episodes, current-window validation substantially outperforms the best fixed model and every pooled stateful selector tested.Source: §9 · pp. 8–9
  • When one model remains dominant, repeated validation can cause unnecessary switching and a fixed choice may be better.Source: §9 · p. 9
  • The practical lesson is to begin with fresh evidence and add memory only when it demonstrably helps.Source: §9 · p. 9

Data and Code Availability

The project repository provides code, frozen protocols, and confirmatory records, including materials to regenerate figures and tables from stored CSV files without rerunning training.

Data and Code Availability

A Reproducibility Details

Reproducibility controls standardize the benchmark streams and scoring, while development runs establish current validation as the primary reference and fix the alternative selectors’ rules.

A Reproducibility Details

  • Every task-scenario-seed partition is written atomically, and all models share identical window inputs while calibration windows are excluded from scoring.Unit tests and a separate audit check determinism, predictions, record counts, unique keys, losses, and the calibration boundary.Source: A Reproducibility Details · p. 9

B Development Findings

  • Current validation had mean log regret 0.083, versus 0.198 for the best fixed oracle and 0.099 for adaptive dual EWMA in development runs.Those runs informed the choice of current validation as the primary confirmatory reference; confirmatory seeds and two additional task families were withheld until the protocol was frozen.Source: B Development Findings · p. 9

C Selector Definitions

  • Figure 5 plots accuracy against cost for eight surrogate families, with both axes logarithmic.Source: Figure 5 · p. 10
  • Adaptive dual EWMA switches to its fast state when fast and slow states differ by more than 0.45, while Page–Hinkley resets stored states after detection.Page–Hinkley uses tolerance 0.03 and threshold 0.75; the selector values were fixed during development.Source: C Selector Definitions · pp. 9–10
Loading 2609.29715v1…