PaperlayerView exact revision on arXiv

Source-linked AI summary

Lifetime-valid decisions by continually evolving AI analysts

Xiaojing Sun, Yuhan Zeng, Xiao Wang

arXiv:2610.30562v1stat.MEcs.AIcs.LGstat.ML

TL;DR

AI analysts combine internal randomness with continual, potentially adaptive updates, challenging Type I error control designed for fixed procedures. The paper introduces continual validation, which tests frozen versions on simulated null data and proves a lifetime risk bound. Its examples show that this approach maintains error control while permitting adaptive updates and preserving power.

  • Problem

    AI analysts can change adaptively and use evaluation feedback, creating a challenge for Type I error control designed for fixed procedures.

    Source: §1 · pp. 1–2

  • Method

    Continual validation tests each frozen proposed version on newly simulated null data before deployment and bounds the lifetime risk of deploying an invalid version.

    Source: §1 · p. 2, §3 · p. 9

  • Results

    The paper proves that continual validation controls Type I error across adaptive updates and directly controls statistical-decision error unlike methods that check only performance improvement.

    Source: §1 · p. 2, §3 · p. 9

  • Takeaways & Limitations

    With suitable upper confidence bounds, the framework can extend to other statistical risks, including the false discovery rate of a screening agent.

    Source: §3 · p. 10

    The theoretical guarantee strictly applies to validated null scenarios and does not automatically bound unobserved data-generating patterns.

    Source: §3 · p. 9

Abstract

from arXiv · show

AI agents increasingly act as analysts whose decision can often be cast as hypothesis tests. Classical tests control the Type I error rate, the probability of rejecting a true null hypothesis, for fixed procedures. However, AI analysts introduce internal randomness and can evolve continuously over time. As AI analysts evolve, the risk of approving versions with Type I error rates above the target can accumulate, and adaptive use of evaluation feedback can lead to overfitting. We introduce continual validation, which evaluates each proposed AI analyst version before deployment using simulated data under the null hypothesis, with scenarios that can be randomized and expanded over time. We prove that, regardless of how updates are chosen or how many are proposed, the lifetime probability of ever deploying a version whose Type I error rate exceeds the allowed level remains bounded. This guarantees that every subsequent decision controls the Type I error rate at the target level across these scenarios. At the same time, updates with sufficiently low Type I error rates pass validation with high probability, allowing the analyst to continuously improve its performance without sacrificing rigor. In two examples involving evolving AI analysts for bioequivalence studies and product A/B tests, Type I error rates reached 0.098 and 0.183 without continual validation, compared with at most 0.045 and 0.044 with it, while retaining 82% and 96% of the power from optimally selected valid updates.

1 Introduction

AI analysts make decisions through randomized procedures that can change repeatedly, so validating a fixed procedure does not address the risks of adaptive updates. Continual validation tests each frozen update on newly simulated null data, bounding the lifetime risk of deploying an invalid version while allowing sufficiently safe updates to pass.

1 Introduction

  • Classical tests control Type I error for fixed procedures, while AI analysts increasingly perform analyses that end in hypothesis-test decisions.Examples include bioequivalence claims and product launches.Source: §1 · p. 1
  • AI analysts can change models, prompts, tools, workflows, and decision rules through a potentially unlimited sequence of updates, with adaptive feedback creating a risk of overfitting and inflated Type I error.A version's decisions may also vary on the same data because of internal model randomness.Source: §1 · pp. 1–2
  • Continual validation freezes each proposed version and tests it on newly simulated null data before deployment, bounding the lifetime risk of approving a version above α′ by δ.The method aims to preserve power at reasonable computation cost and allows sufficiently low-error updates to pass with high probability.Source: §1 · p. 2
  • The paper applies continual validation to evolving analysts for bioequivalence studies and product launches and compares it with four alternative deployment strategies.Figure 1 also describes the two example settings and their experiment setup.Source: §1 · p. 2, Figure 1 · p. 3

2 Results

Across the two evolving analyst examples, continual validation maintained low Type I error while retaining substantial power, unlike several comparison strategies. Its guarantee controls lifetime deployment risk across updates, while applying to null scenarios covered by validation and allowing sufficiently safe updates to pass.

2.1 Why Common Validation Strategies Can Fail to Control Type I Error for Evolving AI Analysts

  • Continual validation addresses three failures: combining separately valid tools can raise error, per-update testing can accumulate false approvals, and reuse of a fixed benchmark can invite overfitting.Combining two tests near 0.05 yielded 0.0587; repeated independent passes accumulate as 1−(1−η)^G, and fixed-benchmark overfitting can produce error rate one on new data.Source: §2.1 · p. 4

2.2 Continual Validation Maintains Type I Error Control at Modest Cost

  • Theorem 1 bounds by δ the chance of ever deploying a version whose Type I error exceeds α′ in any tested null scenario, regardless of update number or selection.For each future study following a covered null scenario, the resulting decision controls Type I error at α even after unlimited adaptive updates.Source: §2.2 · pp. 5–6
  • Updates with Type I error sufficiently below α′ can pass with high probability, and for a fixed number of scenarios the required simulations grow only logarithmically with the update number.When an update adds no rejections relative to the reference, the required sample count is at least log(4/δr)/γ for the stated margin γ.Source: §2.2 · p. 6

2.3 Two Evolving AI Analysts

  • The experiments used Claude Sonnet 5.5 to propose updates after reviewing performance reports and prior validation feedback, with deterministic tools executing the analyses.Each deployment method ran 20 times from the same initial version, and outcomes were estimated using independent simulations on validation and off-grid scenarios.Source: §2.3 · pp. 6–7

Example 1, an Agent That Analyzes Bioequivalence Studies; Example 2, an Analyst of Product Experiments

  • The bioequivalence example evaluates average bioequivalence in replicate crossover studies, while the product example tests whether a feature improves conversion in a 14-day randomized experiment.The bioequivalence analyst claims equivalence when its confidence interval lies within regulatory limits; a product launch is a Type I error when the feature does not improve conversion.Source: Example 1, an Agent That Analyzes… · p. 7, Example 2, an Analyst of Product… · p. 8

Example 1, an Agent That Analyzes Bioequivalence Studies

  • In bioequivalence studies, continual validation’s maximum Type I error rate was 0.045 and its mean final power was 82 percent of the oracle’s; no version exceeded α′ across 20 runs.No validation and the improvement test reached Type I error rates up to 0.098, while Learn then Test reached 52 percent of oracle power because it could not evaluate novel rules.Source: Example 1, an Agent That Analyzes… · pp. 7–8

Example 2, an Analyst of Product Experiments

  • In product experiments, continual validation attained 96 percent of oracle power without exceeding α′, while no validation and the improvement test reached Type I error rates up to 0.183 and 0.817.The final version had power 0.21 and true Type I error 0.043; the analyst proposed 62 versions, including new decision rules.Source: Example 2, an Analyst of Product… · pp. 8–9

3 Discussion

Continual validation is presented as a way to preserve Type I error control while AI analysts evolve, including through adaptive updates. Its guarantee is limited to validated null scenarios, while randomized scenarios and suitable confidence bounds broaden the framework’s potential scope.

3 Discussion

  • The lifetime probability of ever approving an invalid version remains bounded, so each deployed decision controls Type I error under the covered scenarios.Freezing each update before simulating its validation data helps prevent overfitting during adaptive selection.Source: §3 · p. 9
  • Continual validation differs from finite-menu approaches by allowing adaptive, previously unanticipated updates, including new decision rules, while permitting performance-based deployment criteria.Comparing each update with a fixed reference analysis reduces simulation costs and avoids accumulating differences across successive versions.Source: §3 · p. 9
  • The theoretical guarantee applies strictly to validated null scenarios, not automatically to unobserved data-generating patterns.Randomly sampled scenarios can extend the guarantee to a broader data population; the method also requires relevant null scenarios to be simulatable and the system evaluable at sufficient scale.Source: §3 · pp. 9–10
  • Figure 2 encodes deployed-version error rates, runs with any deployment above α, and mean power relative to an oracle across two evolving analysts.Figure 3 depicts bioequivalence results on sample sizes outside the original validation grid and marks rejected updates that would raise off-grid Type I error while leaving grid analyses unchanged.Source: Figure 2 · p. 10, Figure 3 · p. 11
  • The framework can extend beyond Type I error to other error measures, such as a screening agent’s false discovery rate, when valid upper confidence bounds are available.The paper presents it as a deployment framework for continual improvement and adaptation while maintaining safety guarantees.Source: §3 · pp. 10–11

4 Methods

The methods define the decisions, analyst versions, and Type I error being controlled, then validate frozen updates against simulated null scenarios. A reference analysis and deployment conditions support the lifetime-risk guarantee.

4.1 Decisions, Versions, and Type I Error

  • A version specifies everything that can change a decision, and its Type I error is the probability of rejection over study data and internal randomness under a null scenario.Deferrals do not count as rejections; the stated assumption is that real studies matching validation scenarios follow the simulated distribution and cannot be distinguished from simulated studies.Source: §4.1 · p. 11

4.2 Validation of an Update

  • Each update is tested on simulated null datasets and passes only when every scenario passes the validation bounds.The exact one-sided Clopper–Pearson bounds use confidence levels δr/4, δr/4, and δr/2; the intersection-union argument requires no division of δr among scenarios.Source: §4.2 · pp. 11–12

4.3 Reference Analysis and Lifetime Risk

  • The reference analysis is run at α0 below α′, with one-time validation for real-study or non-Gaussian scenarios and δ0 of the lifetime risk.The method simulates all m datasets in every scenario before deciding deployment because optional stopping would invalidate the exact bounds; it uses α = 0.05, δ = 0.005, δ0 = 0.001, and α0 = 0.030 or 0.040 in the two examples.Source: §4.3 · p. 12

4.4 Conditions for Deployment

  • Deployment requires the validator to run each version on simulated studies, with study-learning memory frozen and round seeds drawn after the version is frozen.Each study uses one binding run; full validation pins a hosted model to a fixed snapshot, while validation through allowed paths protects against silent model updates.Source: §4.4 · p. 12

A1 Setting and notation

This section establishes the theory’s order: define its objects, specify the controlled Type I error rate, and state lifetime validity.

A1 Setting and notation

A1.1 Studies and decisions

The section specifies an analyst’s study-level decisions and the scenarios that determine their data distributions. It defines null scenarios as states where rejection is wrong, including composite bioequivalence nulls.

A1.1 Studies and decisions

  • The analyst returns reject, retain, or defer; deferral sends the study to a human and never counts as rejection.The guarantees concern only rejections made by the analyst itself.Source: A1.1 Studies and decisions · p. 17
  • A scenario θ specifies the study’s true state and data-generating distribution, including protocol and design features such as sample size, sequence allocation, and dropout.For bioequivalence, it also fixes the true exposure ratio and variability parameters.Source: A1.1 Studies and decisions · p. 17
  • The null-scenario set Θ0 contains states where rejection is wrong; in bioequivalence, the null is composite and includes ratios on or outside limits determined by true variability.For within-reference variability at or below 30%, the limits are 80.00 and 125.00 percent; otherwise the passage gives widened EMA limits, capped at 69.84 and 143.19 percent.Source: A1.1 Studies and decisions · p. 17
  • Validation scenarios can be parametric or built from real studies by resampling subjects and moving the true effect to the null boundary or beyond.Source: A1.1 Studies and decisions · p. 17

A1.2 Versions

An analyst version is a fixed configuration that maps study data and its own randomness to a decision. The version’s memory and components stay fixed while it is in force; changes create new versions.

A1.2 Versions

  • A version fixes the model and its sampling settings, instructions, tools, analyses, thresholds, routing rules, and computing environment.Every change to any component, including a provider’s behind-the-scenes model update, creates a new version.Source: A1.2 Versions · pp. 17–18
  • Each version is a randomized test: its decision depends on study data and internal randomness drawn independently from a version-fixed distribution.Two runs of one version on one study have the same distribution of decisions.Source: A1.2 Versions · p. 18
  • A version carries no memory across studies, and each study receives one binding run; rerunning until rejection is a different procedure.An analyst whose context grows with each study becomes a new version after each study.Source: A1.2 Versions · p. 18
  • Faithful simulation requires real and simulated studies to share the data law, and the version must not distinguish them.Under this assumption, simulated rejection rates equal Type I error rates on real studies in that state.Source: A1.2 Versions · p. 18

A1.3 Type I error rate

Type I error is measured over both study data and analyst randomness, and validity is defined by the maximum error rate over a set of null scenarios. The paper uses α = 0.05 and lifetime risk δ = 0.005.

A1.3 Type I error rate

  • Type I error rate is the probability of rejection over study data and the analyst’s internal randomness.For a scenario set, size is the supremum of the rate over its null scenarios; validity at level a means that size is at most a.Source: A1.3 Type I error rate · p. 18
  • Power is the probability of rejection under a false null hypothesis, and it plays no part in the guarantees.Most updates try to raise power.Source: A1.3 Type I error rate · p. 18
  • The paper sets α = 0.05, δ = 0.005, and α′ = 0.0452 for validation.The validation level is α′ = (α − δ)/(1 − δ); the reference levels are 0.030 in Example 1 and 0.040 in Example 2.Source: A1.3 Type I error rate · p. 18

A1.4 Scenario sets

Validation uses finite scenario sets that begin with a fixed initial set and can expand as the analyst encounters new kinds of information.

A1.4 Scenario sets

  • The initial set S0 contains J null scenarios, varying expected decision factors such as variability, sample size, and dropout.It includes scenarios both at the null boundary and inside the null hypothesis.Source: A1.4 Scenario sets · p. 19
  • When an updated version reads information not varied in the current set, the validator adds scenarios that vary it; scenarios are never removed.Thus the set in force at each update contains S0.Source: A1.4 Scenario sets · p. 19
  • The validator may also randomly draw scenarios after a version is frozen and use them only for that validation round.These draws extend the guarantee between fixed scenarios, except for a set of small probability.Source: A1.4 Scenario sets · p. 19

A1.5 An analyst that keeps changing

The analyst receives a sequence of proposed updates, each offering a new version that may pass or fail validation. A passing version may be deployed for subsequent studies, while the validator maintains a reference analysis for startup decisions.

A1.5 An analyst that keeps changing

  • Updates may come from self-revision, engineers, power optimizers, or adversaries, and each attempt counts whether it passes or fails.A passing version may be deployed; a failed update changes nothing.Source: A1.5 An analyst that keeps changing · p. 19
  • The reference analysis has simultaneous Type I error bounds, except with probability at most δ0.If its bounds are at most α′ on S0, the analyst starts with it or a version that rejects only where it rejects; otherwise, it starts by deferring every study.Source: A1.5 An analyst that keeps changing · p. 19
  • Each arriving study is decided by the version in force when its data arrive.The version in force is denoted v(s).Source: A1.5 An analyst that keeps changing · p. 19

A1.6 Lifetime validity

Lifetime validity concerns every decision made by an evolving analyst, under whichever version is in force and across the history leading to that decision. Continual validation provides this guarantee on scenario sets where all versions in force are validated.

A1.6 Lifetime validity

  • Lifetime validity applies to every study decision when its true scenario lies in the specified null-scenario set.The probability includes the updates, validations, earlier studies, current study data, and analyst randomness.Source: A1.6 Lifetime validity · p. 19
  • Continual validation delivers lifetime validity on scenario sets where every version in force is validated.Assumption A1 carries the guarantee to real studies in those states.Source: A1.6 Lifetime validity · p. 19
  • The guarantee covers decisions after any number of changes, however those changes are chosen.It applies under whichever version is in force for each decision.Source: A1.6 Lifetime validity · p. 19

A2 Why common checks fail

Common checks can fail through combining valid tests, repeatedly validating versions at a fixed confidence level, or reusing benchmark simulations. Figure A1 also illustrates how paired reference comparison affects validation costs.

A2 Why common checks fail

  • Two bioequivalence tests each near 5% Type I error can exceed that level when the second is used whenever the first fails.The figure evaluates nine designs using 400,000 simulated studies each.Source: Figure A1 · p. 21
  • Fixed 95% and 99.5% checks accumulate lifetime risk, while continual validation keeps the probability below δ.The curves consider proposed versions just above α′.Source: Figure A1 · p. 21
  • Paired reference comparison lets a version adding no rejections pass with certainty, while one adding rejections at rate (α′ − α0)/2 passes with probability 0.9.Without reference comparison, a version with Type I error rate α0 passes with probability 0.9.Source: Figure A1 · p. 21
  • Three mechanisms underlie excess rejections: valid analyses need not compose, stored simulations can be exploited, and per-version checks can fail over a long sequence.The section states that each mechanism receives an exact result below.Source: A2 Why common checks fail · p. 21

A2.1 Valid analyses do not compose

Combining individually valid analyses can inflate Type I error because a version may select among them. The section gives a worst-case bound and shows that the bound can be attained.

A2.1 Valid analyses do not compose

  • A version choosing among K analyses, each valid at level α, has Type I error at most min(1, Kα) when it rejects only where at least one analysis rejects.The bound holds for every null scenario.Source: A2.1 Valid analyses do not compose · p. 21
  • The min(1, Kα) bound is attainable even when every constituent analysis has Type I error α in every null scenario.The theorem constructs a version whose rejection rate reaches the bound.Source: A2.1 Valid analyses do not compose · p. 21
  • 0.0587 Type I error occurs at the bioequivalence boundary when a distribution-free interval is used whenever two one-sided tests fail.The FDA reference-scaled rule for highly variable drugs reaches about 2.4 times its nominal level when each criterion is judged where the switch applies it.Source: A2.1 Valid analyses do not compose · p. 22

A2.2 Selection through repeated updates

Updates selected to improve power may add rejections and raise Type I error alongside power. Repeated evaluation also creates risks from tuning to stored simulations and from accumulating failures across fixed-confidence checks.

A2.2 Selection through repeated updates

  • A change that adds rejections has at least the original version’s power and Type I error in every null scenario.A lineage of K such changes can reach min(1, Kα) without checks; examples include optional stopping and launching when any of several metrics improves.Source: A2.2 Selection through repeated updates · p. 22
  • When added rejections raise power, power and Type I error rise together under those changes.The section notes that changes raising power fastest are often of this kind.Source: A2.2 Selection through repeated updates · p. 22
  • Reusing simulated studies lets a proposer tune versions to the evaluation data, while fixed-confidence checks accumulate pass risk across updates.The section distinguishes these as two failures of checking each version separately.Source: A2.2 Selection through repeated updates · p. 22

A2.3 A reused set of simulations can be defeated

A proposer who can inspect a stored simulation suite can construct an analyst that passes validation on that suite yet has a high Type I error rate elsewhere. Fresh simulations drawn after each version is fixed prevent reuse-based overfitting; randomized scenarios confine the effects to a small-probability set.

A2.3 A reused set of simulations can be defeated

  • A version can match a reference analysis on every stored study and pass validation, yet have Type I error rate 1 in every nonatomic scenario.More generally, its Type I error rate is at least 1 − Prθ{D ∈ D} in each scenario.Source: A2.3 A reused set of simulations can… · p. 22
  • A visible simulation suite is an overfittable benchmark; even threshold tuning can select a version with a low estimated but excessive true Type I error rate.The threshold is tuned against stored studies, retaining the most permissive version whose estimate stays below the validation level.Source: A2.3 A reused set of simulations can… · pp. 22–23
  • Drawing each round’s studies after the version is fixed and independently of the proposer’s prior information removes both effects for those simulated studies.For a fixed set of scenarios, random scenarios drawn after each freeze confine the effects to a set of small probability.Source: A2.3 A reused set of simulations can… · p. 23

A2.4 A fixed confidence level per version fails over a lifetime

A fixed confidence level for each version does not control the lifetime risk: approvals of invalid versions can accumulate over updates. Continual validation instead divides a prespecified risk budget among updates, bounding the overall risk regardless of how many arrive.

A2.4 A fixed confidence level per version fails over a lifetime

  • An update is invalid when its Type I error rate exceeds the validation level in at least one validation scenario.Source: A2.4 A fixed confidence level per… · p. 23
  • With fixed per-version confidence, the probability that some invalid version passes among G updates can reach 1 − (1 − η)^G and tends to one.When every update is invalid and passes with conditional probability exactly η, the probability exceeds one half after 14 updates at 95 percent confidence and after 139 at 99.5 percent.Source: A2.4 A fixed confidence level per… · p. 23
  • Continual validation allocates shares δr across updates with total at most δ, bounding the familywise error rate by δ however many updates arrive.The bound follows by a union bound over updates.Source: A2.4 A fixed confidence level per… · p. 23

A3 Validating one version

This section develops validation methods for frozen and continually evolving analyst versions, covering scenario-based tests, lifetime risk control, randomized scenarios, and conditions for inheriting validity. It assesses these methods across bioequivalence, product and online experiments, and survival-trial decisions, and documents related implementation and reproducibility checks.

A3.1 Exact bounds in one scenario

  • Exact Clopper–Pearson binomial bounds hold for every simulated-study count without asymptotic approximation, and the upper bound falls at or below the true rate with probability at most η.This property implies that a version whose Type I error rate equals the target cannot be validated.Source: A3.1 Exact bounds in one scenario · p. 24

A3.2 Every scenario must pass

  • For a fixed version, passing every scenario bounds the probability of passing while above target by η, without dividing η across scenarios.The maximum scenario bound is an upper confidence bound at level η; requiring simultaneous coverage instead would require dividing the level among scenarios.Source: A3.2 Every scenario must pass · p. 24

A3.3 A version at the target cannot be validated

  • A version whose Type I error rate is at least a passes a scenario with probability at most η, so a version exactly at target fails with probability at least 1−η regardless of m.The reference analysis therefore uses a level α0 below validation level α′, leaving a fixed margin γ = α′ − α0 for future versions.Source: A3.3 A version at the target cannot be… · p. 25

A3.4 Paired and direct bounds

  • The combined paired and direct bound is valid at level η; paired bounds focus on added rejections, while direct bounds suit versions that change many reference decisions.The validator can take the smaller bound in each scenario without knowing in advance which will be smaller.Source: A3.4 Paired and direct bounds · pp. 25–26

A3.5 What validation costs

  • The two validation approaches have different study costs: paired validation can need order 1/γ studies for versions adding few rejections, while validation from scratch needs order a/γ² and has a matching lower bound.The combined bound costs no more than the better bound in each scenario.Source: A3.5 What validation costs · pp. 26–27
  • Paired validation needs order 1/γ studies versus order a/γ² from scratch; at Example 2’s margin, first-update counts differed 19.8-fold, and half-margin additions needed 11,928 versus 114,862.When added and removed rejections rise equally from 0.002 to 0.012, the paired-bound advantage falls from 4.8-fold to 1.2-fold.Source: A3.5 What validation costs · p. 27, Table A2 · p. 28

A3.6 A reference analysis validated by proof

  • For bioequivalence null scenarios under the stated model and response-independent dropout, the reference two-one-sided-tests analysis controls Type I error at α0; at ratio 0.95, level 0.040 yields 2.3–4.4 percentage points fewer bioequivalence declarations than 0.05.At the equivalence boundary without dropout, its Type I error rate tends to α0 as within-subject variance tends to zero.Source: A3.6 A reference analysis validated by… · pp. 28–29
  • The proof-based bound carries over to versions whose rejections are contained in those of the reference analysis, but not to versions that change which observations enter the analysis.Versions that defer high-variability studies or add a veto-only check are examples of the first kind; changing the analysis set requires scenario validation.Source: A3.6 A reference analysis validated by… · p. 29

A4.1 Lifetime risk control

  • Across arbitrarily many adaptive, randomized, or adversarial updates, the probability any passing version exceeds α′ on its round’s scenarios is at most δ.The risk share is spent per attempted update, whether or not it passes.Source: A4.1 Lifetime risk control · pp. 30–31

A4.2 Every decision the analyst makes

  • Under the stated conditions, each decision has Type I error bounded by α, provided its true scenario is covered and the version is fixed before that study’s data arrive.With probability at least 1−δ, the long-run rejection frequency among covered studies has lim sup at most α′.Source: A4.2 Every decision the analyst makes · pp. 31–32

A4.3 Valid updates pass

  • Valid updates with error rates at most α′−γ pass with high probability when m_r ≥ 8α′ℓ_r/γ², where ℓ_r = max{log(2/δ_r), log(J_r/ζ)}.The update is tested across J_r scenarios; the required simulation count grows only logarithmically with the update index under the described share schedule.Source: A4.1 Lifetime risk control · p. 31, A4.3 Valid updates pass · p. 33

A4.4 The reference analysis

  • In the bioequivalence analyst, reference-analysis validation on real-study scenarios spends δ₀ = 0.001, leaving 0.004 for updates.Reference bounds require simultaneous coverage across scenarios; on Gaussian null scenarios, bounds can be obtained by proof without simulation.Source: A4.4 The reference analysis · p. 34

A4.5 Scenarios grow with the information a version reads

  • A version can pass on scenarios with κ = 0 while its Type I error tends to 1 as κ grows, so scenario sets must expand when versions use previously unvaried information.The validator fixes additions at the round’s freeze; declared inputs and enforced software declarations can make needed extensions auditable.Source: A4.5 Scenarios grow with the… · pp. 34–35

A4.6 Random scenarios extend the guarantee between fixed scenarios

  • A fixed, known scenario set can be overfit: a version may pass by copying the reference analysis at tested sample sizes yet reject at every other sample size.Newly drawn scenarios can defeat overfitting to a fixed scenario set, up to a set of small probability.Source: A4.6 Random scenarios extend the… · p. 35
  • With εΠ=0.01 and ω=0.005, 580 scenarios on update 1 and 1,962 on update 1,000 give probability at least 1−δ−ω that deployed updates meet the level on fixed scenarios and all but εΠ of Π.The random draws add no share of lifetime risk; their added failure probability is controlled by ω.Source: A4.6 Random scenarios extend the… · p. 36

A5.1 Allowed paths

  • When software mechanically confines decisions to fixed allowed paths, the version’s Type I error rate is bounded by the any-path rule, regardless of how the model selects paths.Keeping the paths and decision functions unchanged preserves the bound through model, prompt, temperature, or provider updates.Source: A5.1 Allowed paths · pp. 36–37

A5.2 Deferral

  • Deferring on any event, including one based on data or analyst randomness, cannot raise Type I error; human review that can add a rejection requires validation as a new procedure.A stricter deferral rule or a human who can stop a rejection needs no validation.Source: A5.2 Deferral · p. 37

A5.3 Containment in the rejection region of the reference analysis

  • If a version rejects only when the reference analysis rejects, its Type I error is no greater than the reference’s and is at most α0 on Gaussian null scenarios without simulation.This containment holds exactly in every scenario when established from the version’s definition.Source: A5.3 Containment in the rejection… · p. 37

A5.4 Choosing among validated analyses

  • Routing among analyses valid at level a preserves that level when routing information is independent of their decision statistics and internal randomness; independence from data alone is insufficient.Selecting the first of K independent random tests that rejects can raise rejection probability to 1−(1−α)K.Source: A5.4 Choosing among validated analyses · pp. 37–38

A5.5 What each kind of change owes

  • Changes in Table A3 rows 1–5 inherit validity from an already valid version and need no new simulation or lifetime-risk share; other changes undergo validation as updates.The inherited scenario set depends on the type of change.Source: A5.5 What each kind of change owes · p. 38, Table A3 · p. 39

A6 The procedure

  • Algorithm A1 freezes each update, draws random scenarios and then fresh simulated datasets, and deploys only if every scenario’s bound is at most α′.The number of datasets is fixed within a round because optional stopping would invalidate the exact binomial bounds; failed attempts still spend their assigned risk share.Source: A6 The procedure · p. 40

A7 Example 1, a bioequivalence analysis agent

A7.3 Reference analysis, settings and new decision rules

  • The 320 configurations included a starting version that deferred studies with observed within-reference coefficient of variation above 0.30 and files with data-quality flags.Updates could also add a Python decision rule that returns equivalence shown, equivalence not shown, or deferral; the rules could use analyses and data from the fixed interface.Source: A7.3 Reference analysis, settings and… · p. 43

A7.4 Scenarios, workload and simulation sizes

  • Validation used 360 base scenarios from ten real-study reference datasets and expanded to 936 scenarios when updates read variability or added decision rules.The added sets included Gaussian scenarios varying within-reference variability and scenarios with unequal sequence allocation and period effects; power was measured over 180 equally weighted workload cells of equivalent studies.Source: A7.4 Scenarios, workload and… · pp. 43–45

A7.5 Design shared by both examples

  • The experiment compared five deployment methods using the same developer, which proposed one update per round across twenty independent ten-round runs.Continual validation checked every update on new null datasets across scenarios in force and deployed it only when every combined upper bound was at most α′; the lifetime probability of deploying an update above α′ on a validated scenario was bounded by δ.Source: A7.5 Design shared by both examples · pp. 45–47

A7.6 Metrics and analysis plan

  • The prespecified main comparison used the share of runs ever deploying above 0.05 and final-round power relative to the oracle.Type I error metrics were estimated on a separate evaluation stream, with a second estimation stage for the highest-rate and threshold-adjacent scenarios; the paper also reported off-grid error rates and validation workload.Source: A7.6 Metrics and analysis plan · pp. 47–48

A7.7 Implementation requirements

  • Validation analyzed 500,000 simulated studies in each of up to 936 scenarios using a vectorized GPU reimplementation of the estimators.The implementation agreed with direct least-squares calculations to within 10^-9 on simulated studies and produced the same decisions as every menu configuration.Source: A7.7 Implementation requirements · p. 48

A7.8 Results

  • Continual validation proposed 112 distinct versions, with 32% of updates adding decision rules, while Learn then Test proposed 37 new rules and deployed none.In one continual-validation run, a rule that applied ABEL but deferred studies with observed variability from 0.28 to 0.40 passed with an upper bound of 0.0452; the final version had power 0.79 and a largest true Type I error rate of 0.043.Source: A7.8 Results · pp. 48–49
  • Every run of no validation and the improvement test deployed a version above 0.05; continual validation deployed none above α′, while the static benchmark exceeded 0.05 in 20% of runs and α′ in 30%.The static benchmark’s failures occurred on Gaussian scenarios at a coefficient of variation of 0.30, absent from its stored base set; continual validation added these scenarios when updates used observed variability or new decision rules.Source: A7.8 Results · p. 49, Table A4 · p. 49
  • 82% of oracle power was retained with 500,000 datasets per scenario, and continual validation deployed no version above α′ in any run.With 100,000 datasets per scenario it attained 74% of oracle power and refused 17 valid updates; deployed versions reached at most 0.058 on 690 off-grid scenarios. Off-grid exceedances above 0.05 arose from response-dependent dropout in datasets 22 and 28, a mechanism absent from this example’s validation scenarios.Source: A7.8 Results · p. 49, Table A4 · p. 49
  • With the same ten prewritten updates, continual validation refused three unsafe changes in Example 1 and refused daily looks without sequential testing and a Bayesian 0.95 threshold in Example 2.In Example 1 it deployed the partial-replicate rule and the scaled-limits rule above coefficient of variation 0.40; Learn then Test refused all three new decision rules.Source: A7.8 Results · p. 50, Figure A2 · p. 51

A7.9 Random scenarios cover the space between the grid values

  • 0.098 was the true Type I error rate reached by fixed-scenario updates off-grid, while random-scenario validation kept deployed versions at or below 0.042 and retained the same 0.78 power.Random scenarios refused all twelve off-grid updates across three runs, while both methods deployed the same five realistic updates and refused the level 0.035.Source: A7.9 Random scenarios cover the space… · p. 52, Figure A3 · p. 53, Table A6 · p. 53

A7.10 The agent reaches the decisions of the validated rule

  • 900 simulated studies matched the vectorized decision rule, and the full language-model agent reached that rule’s decision in all 14 tested cases.The tools refused the 88 partial-replicate cases where the reference analysis defers; in the other 812, interval bounds agreed within 3.2 × 10^-15 on the log scale. The full agent claimed bioequivalence in four alternative cases and none of nine null cases.Source: A7.10 The agent reaches the decisions… · p. 54, Table A7 · p. 54

A8.1 The hypothesis test

  • A/B-test Type I error is launching a feature that does not improve conversion; daily optional stopping, post-hoc metric or segment selection, and switching decision rules can inflate that risk.The null is that relative conversion lift is at most zero, and the alternative is positive lift.Source: A8.1 The hypothesis test · p. 55

A8.2 GrowthBook and the analyst

  • GrowthBook’s gbstats supplies frequentist and Bayesian analyses, while the analyst’s configuration fixes analyses and the launch rule that validation checks on simulated experiments.The example’s reference analysis is a one-sided frequentist t test at α0 = 0.040; the configuration menu has 384 settings combinations, and updates may also add a Python launch rule.Source: A8.2 GrowthBook and the analyst · p. 55, A8.3 Reference analysis, settings and… · pp. 55–56

A8.4 Scenarios, workload and simulation sizes

  • The product-experiment validation set contains 64 scenarios, with 116 additional off-grid scenarios and a 48-cell power workload spanning relative conversion lifts of 1%, 2%, and 3%.Scenarios vary sample size, conversion rate, metric and covariate correlations, and null type; updates using daily looks, segments, or new rules receive 32 added scenarios.Source: A8.4 Scenarios, workload and… · p. 57

A8.5 Implementation requirements

  • The vectorized validator matched gbstats’ lifts, interval bounds, p values, and chances to win within 10^-12, with matching launch decisions and decision days in 75 tested pairs.It also gave the same decision for every menu configuration on GPUs and processors.Source: A8.5 Implementation requirements · p. 58

A8.6 Results

  • 0.82 was the largest Type I error rate under the improvement test; continual validation deployed no version above α′, kept off-grid rates at or below 0.044, and reached 96% of oracle power.Continual validation deployed 170 of 200 updates; evaluation placed 28 of 30 refused updates above α′. In 12 of 20 runs, no validation produced a version above 0.05, while the improvement test did so in 19.Source: A8.6 Results · pp. 58–59, Table A8 · p. 58

A9.1 Approving a therapy from a survival trial

  • In the survival trial, shortcut Type I error rates were 0.0684, 0.0843, and 0.0554; validated adaptive rules had bounds 0.0411 and 0.0450, with powers 0.627 and 0.600.The passed union rule also covered diagnostic, coin, and adversarial routing among its allowed paths.Source: Table A9 · p. 60, A9.1 Approving a therapy from a… · pp. 60–61

A9.2 Launching a feature from an online experiment

  • For online experiments, shortcuts reached Type I error rates 0.142, 0.144, and 0.370, while validated O’Brien-Fleming and Bonferroni rules had bounds 0.0481 and 0.0482.O’Brien-Fleming power was 0.776, close to the reference analysis’s 0.785, and it stopped after 81% of the planned sample on average.Source: Table A10 · p. 61, A9.2 Launching a feature from an… · p. 61

A10.1 A closed loop with a language-model proposer

  • Had the same update sequence been deployed without validation, the evaluated rate of the deployed version would have peaked at 0.0658 and stayed above 0.05 for five consecutive versions, until the harmful sensitivity rule was replaced in round 8.Source: A10.1 A closed loop with a… · p. 63
  • Across 25 reruns, all 47 updates with evaluated Type I error above 0.05 failed validation; a greedy adversary’s four passing versions had bounds at most 0.0478 and final power 0.579.The proposer’s repeated trajectories had a median of five passing updates.Source: A10.1 A closed loop with a… · pp. 63–64

A10.2 Valid analyses do not compose

  • Selecting whichever of K exact level-α analyses rejects can attain min(1, Kα); a bioequivalence fallback combining individually valid analyses reached Type I error 0.0587.The two analyses were strongly dependent, which kept the inflation moderate.Source: Table A12 · p. 64, A10.2 Valid analyses do not compose · p. 64

A10.3 A regulatory switching rule

  • Under the stated null-boundary convention, the regulatory switching rule’s largest Type I error rates were 0.1036, 0.1141, and 0.1204 for 24, 36, and 48 subjects.Just below the switching value, data-dependent selection can apply wider scaled limits; the reported grid maxima are lower bounds on the maximum rate over the null boundary.Source: A10.3 A regulatory switching rule · p. 65

A10.4 The validity reserve

  • For two one-sided tests at α = 0.05, validation required a reserve of 0.0048: running the test at 0.0452 yielded a largest bound of 0.04933 and passed.The required reserve shrank roughly like m^-1/2 as simulated studies per scenario increased, with modest power cost.Source: A10.4 The validity reserve · p. 65

A10.5 Language models as analysts

  • Under the charter naming no primary analysis, the smallest, mid-tier, and largest models deferred on 35 of 200, 241 of 300, and 198 of 200 decisive studies, respectively.Deferral never raises Type I error; continual validation means a model swap preserving allowed paths needs no new validation.Source: A10.5 Language models as analysts · p. 66

A10.6 Checks of the analysis code

  • Code checks reproduced all reported two-period estimates and 90% limits across eight studies [23], plus replicate-design EMA Method A values of 46.96% vs 47.0% and 151.29% vs 151.3% [24, 12, 11].The two-period results had no discrepancies at the reported precision; the replicate results included the published point estimate and 90% interval.Source: A10.6 Checks of the analysis code · pp. 66–67, Table A14 · p. 67

A11 Reproducibility and software

  • Code, real-study null-scenario construction, validation results, and outputs are planned for deposit in a public repository upon publication.The repository will include outputs behind every reported number, and figures are generated by code from the same programs.Source: A11 Reproducibility and software · p. 67

Software

  • Rerunning section A10.2 with Python 3.9 and NumPy 2.0 reproduced its archived outputs exactly.Other runs used Python 3.12 and NumPy 2.5 for supplementary A9/A10 studies, and Python 3.11, NumPy 1.26, SciPy 1.13, and CuPy 13.6 for examples A7/A8.Source: Software · p. 67

Randomness

  • Every simulation uses recorded random seeds, and validation-round studies are drawn only after the version is frozen.Source: Randomness · p. 67

Data

  • Section A10.6 uses reference datasets for 8 two-period bioequivalence studies published by Schütz and colleagues, plus replicateBE datasets 1 and 26; validation never uses five section A7.4 datasets.The unused datasets are numbers 7, 8, 22, 25, and 28; they ship with the package and were simulated by its authors.Source: Data · p. 68

Language models

  • Both examples use Claude Sonnet 5.5 as proposer at temperature 1, and retained raw model responses let analyses rerun without model access.Supplementary studies used claude-sonnet-5, claude-haiku-4-5, and claude-opus-5 with fixed instructions.Source: Language models · p. 68

Related papers

Loading 2610.30562v1…