PaperlayerView exact revision on arXiv

Source-linked AI summary

Always-On Experimentation

Ricardo J. Sandoval, David Arbour, Avi Feller, Michael I. Jordan

arXiv:2609.38695v1stat.MEcs.AI

TL;DR

Experimentation methods often require treatments to be set in advance, despite platforms increasingly adding and removing treatments as they are generated. This paper develops sequential tests and an online FDR procedure for that setting; it establishes time-uniform error control and almost-sure log-optimality under stated conditions.

  • Problem

    Existing experimental designs typically require treatments to be specified before an experiment starts, while modern platforms need to add and remove treatments dynamically and preserve statistical validity.

    Source: §1 · pp. 1–2

  • Method

    The framework uses testing-by-betting supermartingales for treatment effects and adapts online FDR control to experiments with dynamically changing treatments.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    The procedure controls online FDR, and its test supermartingales are almost-sure log-optimal under the stated assumptions and fixed treatment schedule.

    Source: §3.2 · p. 6, §4 · p. 8

  • Takeaways & Limitations

    Treatments can be dynamically added or removed while the experiment runs, with statistical validity maintained through online FDR control.

    Source: §1 · p. 2, §3.2 · p. 6, §6 · p. 10

    The log-optimality analysis fixes the treatment schedule and number of sub-experiments, with only the number of units per sub-experiment growing.

    Source: §4 · p. 8

Abstract

from arXiv · show

Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conceived and deployed. As a result, modern experimentation platforms often run continuously, with treatments added as they are ready and removed when they underperform. We formalize this "Always-On" experimental setting, in which treatments can be dynamically generated, added to, and removed from a running experiment, and study the statistical problem of deciding whether to accept or reject each treatment while controlling for the false discovery rate. We develop sequential tests that achieve time-uniform Type-I error control under arbitrary stopping times and "predictable" treatment schedules. Our approach builds on the testing-by-betting framework: we construct test supermartingales for testing the average treatment effect of each treatment, and show that the construction of these test supermartingales is growth-rate optimal in an almost-sure sense.

1 Introduction

Always-On experimentation addresses the mismatch between dynamically generated treatments and methods that require treatments to be fixed before an experiment begins. It combines continuous treatment changes and monitoring with statistical validity, including per-treatment Type-I error and multiple-comparison control.

1 Introduction

  • Existing methods typically require all treatments to be known before an experiment starts and do not allow continuous monitoring to decide when to remove a treatment.The paper motivates this gap by noting that generative AI lowers the barrier to proposing treatments on the fly.Source: §1 · p. 1
  • Figure 1 reports that Always-On experimentation matches platform-trial power while using fewer units for discoveries.The figure also depicts four treatments added and removed at different points in one experiment.Source: Figure 1 · p. 1
  • Always-On experimentation permits treatments to be added or removed during a continuously running experiment while preserving statistical validity.The design aims to support data-driven treatment decisions while controlling Type-I error over the experiment and accounting for multiple comparisons.Source: §1 · pp. 1–2

2 Related Work

Related work covers designs that evaluate multiple interventions, online multiple-testing procedures, and continuously run online experiments. The paper distinguishes Always-On experimentation by allowing treatment changes at arbitrary times based on past data.

Platform trials

  • Platform trials allow interventions to enter or leave a continuous experiment, but typically prespecify interim-analysis timing or adaptation rules; Always-On permits changes at arbitrary times based on past data.Platform trials are used in biomedical research to evaluate interventions for a disease or condition [52].Source: §2 · p. 2

Online multiple testing

  • Always-On builds on online multiple testing and resembles asynchronous methods that allow hypotheses to remain unresolved while others are introduced [16].Treatments may stay active until enough evidence supports discovery or a decision is made to remove them; related work studies online FDR methods in platform trials [67].Source: §2 · p. 2

Online experimentation and multi-armed bandits

  • Prior online experimentation work studies sequential tests of predetermined hypotheses, adaptive sampling [31], or bandit allocation to identify strong arms while controlling online FDR [66].Other sequential-testing work asks whether any of several arms is non-null [51] [22] [5].Source: §2 · pp. 2–3

3 Always-On experimentation

The design divides a continuously running experiment into sub-experiments with changing active-treatment sets and sequentially arriving units. It tests treatment effects with anytime-valid test supermartingales and adapts online FDR control to handle multiple treatments.

3 Always-On experimentation

  • Each sub-experiment has a finite active-treatment set and a control, with treatment assignments drawn anew for its units; the number of sub-experiments need not be known in advance but is finite.Units may be different across sub-experiments or reused, and the design uses each unit's potential outcomes across treatment paths.Source: §3 · p. 3
  • Treatment additions depend on past data, while decisions to keep existing treatments for the next sub-experiment may use data observed so far.The schedule decisions cannot depend on future information; the paper assumes units' potential-outcome vectors are identically and independently distributed across units, while outcomes within a unit may be dependent.Source: §3 · pp. 3–4

3.1 Testing by betting in Always-On experimentation

  • For each treatment, the test asks whether its effect exceeds a threshold, pooling treatment and control observations from the sub-experiments in which that treatment was active.The treatment-effect test statistic forms a test supermartingale under the stated assumptions, yielding time-uniform Type-I error control via Ville’s inequality.Source: §3.1 · p. 5

3.2 Multiple testing in Always-On experimentation

  • Online FDR is the chosen error metric because the experiment may test an unknown number of treatments, and it controls the expected proportion of false discoveries.The procedure adapts the online FDR notion to this setting.Source: §3.2 · pp. 5–6
  • The online FDR procedure adapts asynchronous e-LOND [65] to assign confidence levels to treatments whose number is not known in advance.The setting permits each treatment to be tested while other treatments may still be active or newly introduced.Source: §3.2 · pp. 5–6
  • The sequential tests with those confidence levels control the false discovery rate at level α across sub-experiments, including when treatment additions depend on past observations and test evidence evolves over time.The guarantee holds under Assumptions 1 to 3 when the procedure uses the specified test supermartingales.Source: §3.2 · pp. 6–7

4 Constructing Powerful Tests for Always-On Experimentation

For units observed only once, the paper constructs treatment-specific test supermartingales from transformed Horvitz-Thompson estimators. With the treatment schedule fixed, the proposed construction is almost-sure log-optimal within the stated comparator class.

4 Constructing Powerful Tests for Always-On Experimentation

  • In this setting, units participate in only one sub-experiment and their potential outcomes lie in [0, 1].The average treatment effect is identifiable from treatment and control means under the stated assumptions.Source: §4 · p. 7
  • A modified Horvitz-Thompson estimator and a nonnegative transform produce test supermartingales under Assumptions 1–4.The estimator targets the average treatment effect; the transform makes the quantities used in the test nonnegative.Source: §4 · pp. 7–8
  • The growth-rate analysis fixes the number and schedule of sub-experiments while the minimum number of units per sub-experiment grows.The minimum sample size is n_t = min_s n_s.Source: §4 · p. 8
  • Portfolios are computed independently across treatment arms, and the stated strategy yields the theorem’s log-optimality guarantee.The section introduces log-optimality relative to the class of test supermartingales of the specified form.Source: §4 · p. 8
  • Under Assumptions 1–4, the proposed test supermartingale is almost-sure log-optimal for any alternative distribution, given a fixed treatment schedule.The guarantee is relative to the class of test supermartingales of the specified form.Source: §4 · pp. 8–9

5 Semi-Synthetic Experiments

The semi-synthetic experiments assemble Upworthy headline tests by calendar month and compare the proposed method with oracle, ablated, and uncorrected variants. The proposed method controls FDR below the nominal level across traffic levels while finding substantially more true effects than the reward ablation.

5 Semi-Synthetic Experiments

  • The experiments use the Upworthy Research Archive, assembling tests within each of 18 calendar months and using each treatment’s observed timestamp as its start time.The archive contains randomized headline experiments run between 2013 and 2015.Source: §5 · p. 9
  • FDR stayed below the nominal α = 0.05 level across traffic levels; uncorrected variants had an order-of-magnitude worse FDR control.Figure 2(a) reports FDR over window traffic with 10 concurrent treatments.Source: Figure 2 · p. 9, §5 · p. 9
  • Comparisons include an oracle with true treatment effects, a reward-term ablation, an uncorrected version, and early peeking without correction.Figure 2(c) compares empirical growth rates for the proposed process, a no-restart variant, and an oracle with numeraire portfolios.Source: §5 · p. 9, Figure 2 · p. 9
  • Roughly 1,900 true effects per replication across 18 months—12% more than the reward ablation and within 30% of the oracle.The group sequential baseline had lower FDR but a substantially more conservative discovery rate.Source: §5 · p. 9, Figure 2 · p. 9

6 Conclusion

The paper presents an Always-On framework for dynamically adding and removing treatments while controlling online false discovery rate. It develops portfolio-selection strategies for two observation settings and reports that semi-synthetic experiments closely follow theory.

6 Conclusion

  • The framework allows treatments to be dynamically added and removed while testing effectiveness and controlling the online false discovery rate.Source: §6 · p. 10
  • For units observed at most once and experiments run over the same units, portfolio-selection strategies yield growth-rate optimal test supermartingales under any treatment schedule.Source: §6 · p. 10
  • An algorithm controls online false discovery rate, and semi-synthetic experiments based on Upworthy provide evidence that the methodology closely follows theory empirically.Source: §6 · p. 10
  • Future work includes adaptively choosing propensity scores to maximize discoveries and designing treatment schedules that control type-II error while removing treatments.Source: §6 · p. 10

A Further Information on Always-On Experimentation

The appendix formalizes data collection for Always-On experiments and explains why portfolio regret must be considered separately for each treatment and sub-experiment.

A Further Information on Always-On Experimentation

A.1 Simulations to compare Always-On experimentation versus platform trials

The simulations compare Always-On experimentation with an adapted platform-trial design across treatment effects, unit caps, and interim-analysis schedules. They track true discoveries and participant use, but the supplied passages do not report numerical outcomes.

A.1 Simulations to compare Always-On experimentation versus platform trials

  • The simulations compare true discoveries under Always-On experimentation with an adapted platform-trial design using gs-LOND, an O’Brien–Fleming boundary, and the Farrington-Manning test.Source: §A.1 · p. 16
  • Each simulation introduces 60 treatments, half under the null and half under the alternative, and varies the alternative effect and maximum unit cap.Source: §A.1 · p. 16
  • Figures 3 and 4 use alternative effects of 0.25 and 0.55, respectively, while platform-trial interim analysis occurs at 50% of each treatment's unit cap.Source: Figure 3 · p. 16, Figure 4 · p. 16
  • Figure 5 compares one platform-trial interim analysis at 50% of the unit cap with analyses at 25%, 50%, and 75%, tracking discovery differences and relative participants needed.Source: §A.1 · pp. 16–17, Figure 5 · p. 17

A.2 Further information about the semi-synthetic experiment.

The semi-synthetic experiment limits concurrent treatments to ten and uses a queue to admit replacements. It constructs monthly controls from historical click-through rates and runs 200 replications per month.

A.2 Further information about the semi-synthetic experiment.

  • At most ten treatments run concurrently, and a queued treatment enters only after another treatment is removed.Source: §A.2 · p. 17
  • The monthly control click-through rate is the median of that month's tested controls, and historical relative lift scales the constructed always-on control to simulate treatment outcomes.Source: §A.2 · p. 17
  • A treatment is removed when statistically significant against control or when its observation budget is exceeded; each month uses 200 replications.Source: §A.2 · p. 17

B Always-On Experimentation for Testing Treatment Histories

This section develops tests of whether a treatment’s effect exceeds δ across relevant treatment histories in repeated-unit Always-On experiments, under stated carryover and Markov assumptions. It also defines oracle-access processes used in the proofs, including variants that account for strata.

B Always-On Experimentation for Testing Treatment Histories

  • With carryover lasting ℓ sub-experiments, the target effect compares treatment a with control conditional on the recent treatment-and-outcome history, under an ℓ-Markov outcome distribution.For the first sub-experiment, this reduces to the standard average treatment effect.Source: B Always-On Experimentation for… · pp. 18–19
  • The null requires ψP(a, h) ≤ δ for every relevant history, while the alternative requires ψP(a, h) > δ for at least one history.The set of relevant histories is only fully observed after the last sub-experiment in which treatment a was active.Source: B Always-On Experimentation for… · p. 19
  • Under Assumptions 1–3, 5, and 6, the transformed Horvitz–Thompson estimator yields a test P(a)-supermartingale for each treatment.The estimator is conditionally unbiased given the filtration and inclusion event for units assigned treatment a or control with history h.Source: B Always-On Experimentation for… · pp. 19–20
  • For a fixed treatment schedule, under Assumptions 1–3, 5, and 6, the constructed test supermartingale is QRCT-almost-sure log-optimal within W(a) for any Q ∈ Q(a), including repeated testing on the same units.It uses transformed Horvitz–Thompson estimators and thresholds (12) and (13), with portfolios selected using strategy (14).Source: B Always-On Experimentation for… · p. 20

C Sequential hypothesis testing by betting

  • Sequential tests threshold test supermartingales at 1/α to control type-I error at arbitrary stopping times, and predictable betting portfolios support growth-rate log-optimality.The betting process uses predictable λ_n values and sequential e-variables whose conditional expectations under the null are at most 1.Source: C Sequential hypothesis testing by… · pp. 21–22

C.1 Numeraire Portfolios

  • Numeraire portfolios maximize expected logarithmic growth, with a separate portfolio for each active treatment and, in the repeated-unit setting, each history.Source: §C.1 · pp. 22–23

D Proofs of the Main Results

  • The proofs use oracle processes with access to numeraire portfolios or best-in-hindsight portfolios for active treatment arms across sub-experiments.The processes are defined in this proof section for use in proving the paper’s main results.Source: D Proofs of the Main Results · p. 23
  • For Section B proofs, analogous oracle processes are defined for each sub-experiment and stratum, using numeraire or best-in-hindsight portfolios.The stratum-specific definitions apply for each active treatment arm and each stratum in the sub-experiment.Source: D Proofs of the Main Results · pp. 23–24

D.1 Proof of Lemma 3.1

The proof establishes the test supermartingale by checking nonnegativity, the supermartingale property, and a mean bound. Predictability lets the conditional-expectation argument handle treatments that remain active or are removed.

D.1 Proof of Lemma 3.1

  • Nonnegativity follows by construction; the proof therefore focuses on the supermartingale property and an upper bound of one on the mean.The proof considers the conditional expectation for the last unit entering the experiment and distinguishes whether the treatment remains active.Source: §D.1 · p. 24
  • When a treatment is inactive, its test statistic’s conditional expectation equals its previous value; when active, the proof bounds the conditional expectation by one.For the active case, it applies the law of total expectation and uses treatment-inclusion probabilities.Source: §D.1 · p. 24
  • Predictability of the portfolios and transformation supports the conditional-expectation bound, completing the supermartingale argument.The proof also treats the first sub-experiment and concludes the mean bound.Source: §D.1 · p. 25

D.2 Proof of Proposition 3.2

The proof handles online false discovery control using predictable null-status indicators and test-supermartingale bounds. It also establishes conditional unbiasedness of the Horvitz–Thompson estimator and uses it to obtain a test supermartingale in the single-observation setting.

D.2 Proof of Proposition 3.2

  • J_s(a) indicates whether treatment a is null and is measurable from information available before sub-experiment s.In the repeated-history setting, null status means the treatment effect is below the threshold across the histories observed up to s.Source: §D.2 · p. 25
  • The false-discovery bound accounts for removed treatments by freezing their test supermartingales and uses the accumulating discovery set to bound the discovery proportion.The proof iterates bounds and uses confidence-level allocations whose sum is at most one.Source: §D.2 · pp. 25–26

D.3 Proof of Lemma 4.1

  • Conditioning on the filtration and treatment-inclusion event, the Horvitz–Thompson estimator has expectation equal to the treatment effect.The proof uses the conditional assignment probability π_s(a)/(π_s(a)+π_s(0)) and conditional ignorability.Source: §D.3 · p. 27

D.4 Proof of Proposition 4.2

  • Proposition 4.2 is an immediate consequence of Lemmas 3.1 and 4.1 and the fact that gs(·) is a non-decreasing linear map for each s ∈ [t].Source: §D.4 · p. 27

D.5 Proof of Theorem 4.4

The proofs establish log-optimality by comparing the constructed process with an oracle process and bounding the remaining portfolio-regret term. They also derive the history-specific treatment-effect estimand and extend the proof strategy to repeated use of the same units.

D.5 Proof of Theorem 4.4

  • Theorem 4.4 is proved by showing almost-sure equivalence to an oracle process, then showing no competing process in the class has a larger limiting growth rate.The proof uses a portfolio-regret bound and concludes the log-optimality argument.Source: §D.5 · pp. 28–29

D.6 Proof of Lemma B.1

  • For units with history h, the conditional expected difference between outcomes under treatment a and control is the history-specific effect ψP(a, h).The proof uses independence and assignment assumptions to connect the estimator’s conditional expectation to this estimand.Source: §D.6 · p. 30

D.7 Proof of Theorem B.4

  • Theorem B.4 follows the same two-step strategy, establishing oracle equivalence and then log-optimality for portfolios indexed by treatment history.The oracle process has access to the numeraire portfolios for all histories.Source: §D.7 · pp. 30–31

E Auxiliary Results

The auxiliary results establish controls based on portfolio regret, bound expected exponential decay, and show that per-arm allocation grows as sample sizes increase.

E.1 Proof of Lemma D.1

  • The proof of Lemma D.1 bounds the target probability through upper and lower deviations, then controls the terms using a union bound, Markov’s inequality, and Lemma E.3.The argument also invokes the Universal Portfolio regret bound for each arm and sub-experiment.Source: §E.1 · pp. 32–33

E.2 Proof of Lemma D.2

  • The proof of Lemma D.2 likewise splits the target probability into upper and lower deviations and bounds terms using a union bound, Markov’s inequality, and Lemma E.3.Its remaining bound uses the Universal Portfolio regret bound for each sub-experiment.Source: §E.2 · pp. 33–34

E.3 Path-wise control of portfolio regret

  • As the sample size grows, normalized portfolio regret tends to zero almost surely, supporting path-wise control across sub-experiments.The argument uses finite histories, sublinear regret bounds, and almost-sure growth in the number of included units.Source: §E.3 · pp. 35–36

E.4 Upper bound on the expected exponential decay as a function of the number of units in a treatment arm.

  • The expected exponential-decay bound applies to stratified units and also holds without stratification, with the displayed rate bounded by exp{−2p_t(1−exp{−ε})}.The proof links the bound to a positive probability of assigning a unit to the treatment arm or control.Source: §E.4 · pp. 36–37

E.5 Almost sure infinite per-arm allocation

  • Under Assumption 2, the included-unit count for a treatment in a sub-experiment tends to infinity almost surely as that sub-experiment’s sample size grows.Positive probabilities of assignment to the treatment arm and control underpin the strong-law argument.Source: §E.5 · pp. 37–38
  • The auxiliary allocation result states that the treatment-arm count grows without bound as the overall sample size increases.This result is stated for the experiment-wide count in Lemma E.4.Source: §E.5 · pp. 37–38
Loading 2609.38695v1…