Source-linked AI summary
Evidence-Gated Research: Statistically Controlled Model Adoption in Adaptive Search
Yifan Guo
TL;DR
Adaptive model search is path dependent: adopting a challenger changes the reference for later proposals, while ordinary model comparison does not address the full moving-incumbent adoption problem. EGR freezes candidates and constructs persistent evidence for a declared all-environment target before applying online control. The paper reports persistent FDR 0.621 for development-only e-LOND in the closed-loop benchmark and finds evidence placement can reduce validation cost without changing Stagewise crossing decisions.
Problem
Adaptive search couples evaluation and adoption to later proposals, while ordinary fixed-family comparison does not address when a challenger may replace the incumbent across declared environments under sequential evidence and repeated testing.
Source: §1 · p. 1Method
EGR freezes each challenger before decision evidence is opened, builds predictably routed anytime-valid evidence across declared environments, composes a persistent candidate e-value, and passes it to an online controller.
Source: §1 · p. 2Results
Development-only e-LOND attains persistent FDR 0.621, while matched replay finds evidence placement reduces acquisition cost without changing Stagewise crossing decisions.
Source: §5.1 · p. 6, §6 · p. 9Takeaways & Limitations
The results identify state-changing model adoption as a distinct statistical control point in adaptive search, with cost reductions supported once the target and validity contract are specified.
Source: §7 · p. 9Guarantees are conditional on the declared utility, environments, margins, valid evidence unit, and candidate-freezing or restart protocol, and do not imply robustness to undeclared shifts.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Adaptive model search is path dependent: once a challenger is adopted, it becomes the reference from which later candidates are generated. A statistically unsupported replacement can therefore alter hypotheses that have not yet been proposed. We introduce Evidence-Gated Research (EGR), a statistical adoption layer for moving-incumbent search. EGR freezes each challenger before decision evidence is revealed, builds anytime-valid evidence across a predeclared set of environments, routes evidence predictably toward unresolved components, composes a persistent candidate e-value, and passes that e-value to an online controller. Under explicit conditional-validity and predictability conditions, the resulting procedure controls false discovery rate for the declared all-environment adoption target even though earlier adoptions change later challengers. In a 5,000-trajectory closed-loop benchmark, development-only e-LOND attains persistent FDR 0.621, whereas no persistent false-adoption path is observed for the audited EGR variants in that finite run. In matched replay over 600 challenger--incumbent pairs, Stagewise EGR preserves fixed-anytime alternative crossing decisions while using 56.1% less decision evidence at the representative threshold. A three-environment public-data study and a 40,000-sample controlled neural benchmark reproduce the evidence-efficiency pattern. These results identify model replacement as a distinct statistical control point in adaptive model development.
1 Introduction
Adaptive model development links evaluation, adoption, and future proposal generation, so an unsupported replacement changes the reference for later candidates. EGR separates adaptive proposals from statistically controlled adoption.
1 Introduction
- An unsupported replacement changes the incumbent from which later candidates are generated in closed-loop model development.The proposal–evaluate–replace loop appears in AutoML, hyperparameter optimization, and automated research systems.Source: §1 · p. 1
- Adaptive search raises four linked problems: target mismatch across environments, sequential evidence acquisition, repeated-test multiplicity, and inefficient fixed allocation.Development-selected gains may fail in another declared environment, while evidence may be stopped or redirected as it arrives.Source: §1 · p. 1
- EGR freezes each challenger before opening decision evidence and permits adoption only when a persistent candidate e-value clears a stream-level threshold.The comparison tests whether gains exceed predeclared practical margins in every declared environment, conditionally on the history that generated the frozen comparison.Source: §1 · p. 2
- EGR's contributions include conditions for persistent candidate e-values, online FDR or stricter FWER control, and evidence-acquisition rules that reduce wasted sampling.The experiments examine target mismatch, stream multiplicity, evidence-unit dependence, candidate-time leakage, controller choice, and acquisition cost.Source: §1 · p. 2
2 Related Work
Related work addresses adaptive evaluation, anytime inference, online multiplicity, and intersection–union testing. EGR focuses on their combination for state-changing, moving-incumbent adoption across environments.
2 Related Work
- Reusable holdouts study repeated adaptive access to evaluation data, while LTT and adaptive LTT address risk calibration and sequential hyperparameter selection (Zecchin et al., 2025).Work on temporal shift compares candidate models as environment performance changes (Han et al., 2024).Source: §2 · p. 2
- Automated-science systems extend adaptive search to empirical software optimization and multi-agent discovery, but do not supply the moving-incumbent, multi-environment adoption control studied here (Lu et al., 2026; Ayg¨un et al., 2026; Ghareeb et al., 2026).EGR targets a state-changing challenger–incumbent stream with a predeclared intersection target across environments.Source: §2 · pp. 2–3
- Anytime-valid inference and online e-value controllers address sequential monitoring and stream-level multiplicity; EGR constructs and adaptively acquires the candidate-level multi-environment evidence they require (Xu and Ramdas, 2024).Related controllers include e-GAI methods and online e-closure or compound-e methods (Xu et al., 2026).Source: §2 · p. 3
- Persistent benefit has an intersection–union structure because its alternative requires the environment-specific alternative to hold in every environment.EGR combines this target structure with anytime-valid components and a changing incumbent.Source: §2 · p. 3
3 Persistent Adoption in an Adaptive Search Loop
The paper formalizes adoption as a sequence of history-dependent comparisons between a challenger and the current incumbent. Persistent benefit is defined against predeclared environments and practical margins.
3 Persistent Adoption in an Adaptive Search Loop
- A predictable proposal policy selects each challenger against the current incumbent using the history available before that candidate's decision evidence.The comparison is fixed before that evidence is opened.Source: §3 · p. 3
- A challenger is persistently beneficial only when its paired utility gain exceeds the predeclared practical margin in every declared environment.The margin is fixed before candidate decision evidence is revealed.Source: §3 · p. 3
- The persistent target is local to the declared all-environment criterion and does not imply robustness to undeclared future shifts..Source: §3 · p. 3
4 Evidence-Gated Research
EGR builds persistent candidate evidence from conditionally valid component processes, routes evidence predictably, and supplies candidate e-values to an online controller. Its guarantees depend on the declared target, valid evidence units, and predictable allocation and stopping.
4.1 Anytime-valid components and persistent composition
- A persistent candidate e-value can combine stopped component values without requiring independence across environments, though a slowly accumulating component can make the combination conservative.Under the union null, at least one component is null and bounds the combined value.Source: §4.1 · p. 4
- Each component e-process remains valid under predictable routing when selected updates satisfy the joint-history conditional evidence assumption.The sequential unit must match the valid acquisition mechanism; with dependent raw rows, a block or cluster may be the valid unit, and a marginal mean condition is insufficient.Source: §4.2 · p. 4, §4.1 · p. 4
4.2 Predictable routing and evidence allocation
- Stagewise EGR advances through environments after a component crosses its threshold; EGR-ALLOC recycles saved budget, while balancing targets an unresolved environment with weakest current log evidence.The operational validity contract is local to the declared target and evidence unit.Source: §4.2 · p. 4, Table 2 · p. 5
- In the deterministic growth benchmark, greedy balancing reaches evidence level L within ⌈S(L + r_max)⌉ observations, but this is not a general stochastic optimality claim.The guarantee assumes observations go to a current minimizer of n_e r_e.Source: §4.2 · p. 5
4.3 Across-candidate control
- Under the stated assumptions, e-LOND applied to conditionally valid candidate e-values controls persistent-target FDR at level α despite earlier adoptions affecting future proposals.Using threshold 1/(αγ_t) instead yields persistent FWER at most α; post-evidence changes to a candidate require restarting its evidence process and downstream testing index.Source: §4.3 · p. 5
5 Experiments
The experiments examine target mismatch, repeated-adoption multiplicity, evidence validity, and whether predictable evidence placement can reduce validation cost without changing the target. Finite-run error estimates are descriptive; the formal guarantees are stated in Theorem 2.
5.1 Target mismatch and repeated adoption
- Persistent FDR was 0.621 for development-only e-LOND; relative to fixed audited IUT, EGR-ALLOC used 36% less decision evidence and raised persistent-alternative power from .188 to .251, with no observed persistent false-adoption path in 5,000 trajectories.The one-sided 95% binomial upper bound for 0/5000 observed paths is approximately 6 × 10^-4.Source: §5.1 · p. 6, Figure 2 · p. 6
5.2 Stream-level controller sensitivity
- At T = 500 with 5% alternatives, persistent-alternative power was .619 for e-LOND, .890 for e-LORD, .888 for e-SAFFRON, and .415 for the e-FWER rule, while empirical FDRs were .00015, .00055, .00055, and .00020, respectively.The candidate e-value streams were identical across controllers, isolating downstream controller choice.Source: §5.2 · p. 6, Table 3 · p. 7
5.3 Evidence validity, candidate freezing, and adaptive allocation
- At within-cluster correlation ρ = .95, treating dependent raw rows as fresh evidence units crossed the candidate null threshold with probability .728, versus .0127 when each cluster was one bounded evidence unit.The stress test indicates that marginal mean validity alone is insufficient when the sequential evidence unit is misspecified.Source: §5.3 · p. 7
- At q = 20 with 64 post-evidence variants, reusing decision evidence to select a challenger raised null-crossing from .052% to 3.802%; selecting on that evidence but restarting on fresh evidence restored it to .063%.The experiments attribute the increase to reusing candidate-shaping evidence, not to adaptivity itself.Source: §5.3 · p. 7
- Across |E| ∈ {2, 3, 5} and q ∈ {5, 10, 20, 50, 100}, weakest-component balancing used realized evidence and had a balancing/oracle mean-evidence ratio between .970 and 1.034.The oracle was supplied with simulator evidence rates.Source: §5.3 · p. 7
5.4 Multi-environment and larger neural searches
- Stagewise EGR reduced mean evidence by 67.6% in the three-environment digits extension and 62.9% in the 40k controlled benchmark; EGR-ALLOC improved power to .3349 and .2666, respectively, with no observed persistent false-adoption path in those audited runs.The public-data extension used three declared environments and an evaluation-only hidden environment; the 40,000-sample neural benchmark remained controlled and synthetic.Source: §5.4 · pp. 7–8, Table 4 · p. 8
5.5 Matched replay isolates acquisition efficiency
- At q = 20, Stagewise EGR preserved fixed-anytime IUT’s component crossing events and persistent-alternative power (.6477), while reducing mean evidence from 1600.0 to 702.4 (56.1%) across 600 frozen challenger–incumbent pairs.EGR-ALLOC reached power .6699 at 715.7 observations; its power increase over Stagewise was .0222 (95% CI .0159–.0290).Source: §5.5 · p. 8
6 Discussion and Limitations
The experiments distinguish target mismatch, invalid evidence handling, and stream multiplicity from evidence-placement efficiency. Once the target and validity conditions are fixed, Stagewise EGR reduces evidence use without changing crossing decisions, while the framework’s guarantees remain bounded by their stated scope.
6 Discussion and Limitations
- Target mismatch, repeated-candidate multiplicity, dependent evidence units, and candidate-time leakage can each undermine adoption validity in distinct ways.In the leakage stress test, q = 20 boundary-null crossing rises from .052% to 3.802% with leakage and reuse, while a fresh restart restores .063%.Source: §6 · p. 8
- At q = 20, Stagewise EGR uses 56.1% less evidence at identical power while preserving fixed-anytime IUT alternative crossing events.The matched replay covers 600 challenger–incumbent pairs.Source: Figure 4 · p. 9, §6 · p. 9
- Guarantees depend on the declared utility, environments, margins, valid evidence unit, and freezing/restart protocol, and do not imply robustness to undeclared shifts.Public-data studies use finite empirical pools; the 40k neural benchmark is controlled/synthetic, and runtime overhead in large training systems is uncharacterized.Source: §6 · p. 9
7 Conclusion
EGR treats state-changing adoption as a statistical control point, separating flexible proposal generation from adoption validity. The appendix establishes conditional evidence validity under the stated history and stopping conditions.
7 Conclusion
- EGR separates proposal flexibility from adoption validity by freezing challengers, routing multi-environment anytime evidence, composing persistent candidate e-values, and applying stream-level control.The conclusion reports that target mismatch, candidate-time leakage, or missing stream correction can drive unsupported adoptions, while predictable placement can reduce validation cost when validity conditions are fixed.Source: §7 · p. 9
A Proofs
- The proof uses a frozen pre-candidate history and joint within-candidate history, with challenger, incumbent, environment laws, utilities, and margins measurable before decision evidence.The persistent null is represented by an indicator so the set of true component nulls need not be treated as deterministic.Source: A Proofs · p. 11
A.1 Proof of Proposition 1
- Averaging nonnegative betting processes over a precommitted grid yields a component e-process, and optional stopping preserves conditional validity for almost-surely finite stopping times.A finite evidence cap is given as a simple condition ensuring almost-sure finiteness.Source: §A.1 · p. 11
A.2 Proof of Lemma 1
The appendix establishes a union-null candidate e-value without requiring independence across environments, shows that predictable routing preserves component validity, and bounds the allocation cost of weakest-component balancing.
A.2 Proof of Lemma 1
- Events measurable before decision sampling partition the persistent null and select the first true component null, yielding a conditional e-value for the union null.The argument requires neither cross-environment independence nor simultaneous component stopping.Source: §A.2 · p. 12
A.3 Proof of Theorem 1
- Because each routing decision is known before the next outcome, predictable interleaving preserves component supermartingale validity and yields a valid persistent candidate e-value after stopping.The argument evaluates each local environment process at its routed allocation count, then averages and applies the stopping argument.Source: §A.3 · p. 12
A.4 Proof of Proposition 2
- Greedy weakest-component balancing reaches progress level L with at most S(L + r_max) allocations, compared with the necessary lower bound SL.The bound follows from the final minimum progress and the maximum increment size.Source: §A.4 · p. 13
A.5 Proof of Theorem 2
The appendix combines candidate-level conditional validity with online control and documents conventions and diagnostics for controller choice, synthetic streams, leakage, and allocation. Its empirical controller comparison shows differing power across controllers on identical candidate-e-value streams.
A.5 Proof of Theorem 2
- Fixing the challenger, incumbent, target, and proposal randomness before opening decision evidence gives conditional candidate validity despite earlier adoptions changing later hypotheses.Applying e-LOND yields persistent-target FDR control at level alpha under its stated dependence conditions; a stricter threshold gives the FWER claim.Source: §A.5 · p. 13
B.1 Common conventions
- Empirical FDR is the across-trajectory mean FDP, persistent-alternative power uses the declared all-environment practical-margin target, and evidence counts paired decision updates but excludes scout data.The appendix also specifies the betting grid and Monte Carlo standard-error convention.Source: §B.1 · p. 14
B.2 Synthetic closed loop and long streams
- The synthetic closed-loop setup uses scout observations for proposal only and paired Bernoulli decision gains; the headline M = 16 setting uses 5,000 trajectories.The main grid uses M in {1, 4, 16, 64}; long-stream diagnostics vary T from 20 to 500.Source: §B.2 · p. 14
B.3 Controller and clustered-dependence diagnostics
- At T = 500 and 5% alternatives, e-LOND, e-LORD, e-SAFFRON, and e-FWER powers are .619, .890, .888, and .415, with empirical FDR .00015, .00055, .00055, and .00020, respectively.All controllers receive identical persistent candidate e-value streams.Source: §B.3 · p. 14
- The clustered-dependence diagnostic contrasts treating dependent rows as fresh sequential units with aggregating each cluster into one bounded evidence unit.Within-cluster correlation varies from 0 to .95 across settings.Source: §B.3 · p. 14
B.4 E33: candidate-freezing / candidate-time leakage stress test
- The leakage stress test compares frozen selection, post-evidence selection with reuse, and post-evidence selection followed by discarding leaked data and restarting with fresh evidence.It uses independent streams across variants to expose candidate-time selection bias, not to estimate deployment error.Source: §B.4 · p. 14
B.5 Adaptive-allocation diagnostics
- The allocation diagnostics vary the number of environments and candidate threshold, comparing realized weakest-component balancing with an oracle that knows simulator evidence rates.The tested environment counts include 2, 3, 5, and an additional 8-environment calibration.Source: §B.5 · p. 15
B.6 Public-data adaptive search: original two-environment studies
The experiments evaluate adaptive search across public-data and controlled neural settings, including three-environment tests and frozen-candidate replay. E30 and E32 report no observed false-adoption paths for the audited main methods, alongside reduced evidence use for Stagewise EGR.
B.6 Public-data adaptive search: original two-environment studies
- The digits search used 808 training, 269 search, 270 development, 270 audit, and 180 hidden examples, with standardization fit on training only.Its RBF-SVC family had 64 configurations; its neural family had 25 one-hidden-layer MLPs.Source: §B.6 · p. 15
B.7 E30: three-environment public-data extension
- E30 declares clean development, Gaussian-noise audit, and dropout-plus-noise audit environments, while reserving stronger corruption for hidden evaluation.Evidence is sampled with replacement from fixed empirical pools, so validity is relative to those empirical distributions.Source: §B.7 · p. 15
B.8 E31: proposal-policy robustness
- E31 varies only the predictable proposal policy—local, wide-local, or global—while holding the E30 model/evidence bank and adoption rules fixed.Each policy cell uses 400 trajectories.Source: §B.8 · p. 15
B.9 E32: larger controlled neural benchmark
- E32 tests the evidence-efficiency pattern on a reproducibly generated 40,000-sample, 50-feature, five-class classification problem with a 16-model MLP bank.Its declared environments are clean development, strong additive-noise audit, and dropout-plus-noise audit; stronger corruption is hidden evaluation only.Source: §B.9 · p. 15
B.10 Frozen-candidate replay
- Frozen replay evaluates all 600 ordered non-self pairs from the 25-model neural grid, reusing identical development and audit streams across four procedures.Fixed-anytime IUT and Stagewise EGR have identical alternative crossing events by construction and differ only in evidence usage.Source: §B.10 · p. 15
C.1 Original two-environment public-data results
- Table 5 covers the original public-data adaptive searches and identifies EGR methods in its final columns.The supplied table caption does not report outcome values.Source: Table 5 · p. 16
C.2 Three-environment uncertainty and balancing
- E30 Stagewise EGR used 388.8 evidence versus fixed IUT’s 1200, with zero observed any-false paths in 800 trajectories; E32 used 444.6 versus 1200, with zero in 500.The corresponding one-sided Wilson upper bounds were .00478 for E30 and .00762 for E32.Source: §C.2 · p. 16
- E30 EGR-Balance reached .3687 power with 1106.4 evidence and one observed false-adoption path, unlike the zero-path fixed IUT, Stagewise EGR, and EGR-ALLOC variants.It is presented as an allocation diagnostic rather than the main efficiency comparison.Source: §C.2 · p. 16
C.3 Proposal-policy robustness
The diagnostics distinguish proposal-policy validity from finite-run power, and show that controller choice and evidence reuse affect reported outcomes. The results also emphasize that evidence placement is not a guarantee of higher power for every proposal dynamic.
C.3 Proposal-policy robustness
- Every displayed E31 audited method had zero observed FDR in 400 trajectories per proposal policy, but adaptive evidence placement is not a finite-run power guarantee for every policy.The stated validity claim is conditional, as in Theorem 2.Source: Table 6 · p. 16, §C.3 · p. 16
C.4 Frozen-pair diagnostics
- Table 7 reports candidate-level crossing frequencies at q=20, not online-stream FDR.The caption does not provide the crossing-rate values.Source: Table 7 · p. 17
C.5 Controller and evidence-unit diagnostics
- At T=500 with 5% alternatives and identical persistent candidate e-values, e-LORD and e-SAFFRON attained powers .890 and .888, versus .619 for e-LOND and .415 for e-FWER.Empirical FDR values remained small in these runs; treating correlated raw rows as fresh evidence gave null crossing probability .728 at ρ=.95, versus .0127 after cluster aggregation.Source: §C.5 · p. 17
C.6 Candidate-freezing / leakage diagnostic
- At q=20 and L=64, persistent-null crossing rates were .052% for Frozen, 3.802% for Leak+reuse, and .063% for Leak+restart.With one available variant, frozen and reuse protocols coincide; allocation diagnostics at q=20 report balancing/oracle powers and mean evidence across 2, 3, 5, and 8 environments, but establish no stochastic optimality theorem.Source: §C.6 · p. 17