Offline Policy Evaluation as a decision support tool for designing Adaptive Experiments

1 Introduction

• Fixed A/B tests treat individuals identically, foregoing personalization based on characteristics that may moderate treatment effects.Contextual bandits instead adjust assignments as data accumulate, potentially improving participant outcomes and focusing exploration where uncertainty is high.Source: §1 · p. 2
• The paper proposes a two-level framework: use OPE on historical A/B test logs to rank candidate adaptive policies, then evaluate selected policies in a controlled warm simulator.The simulator uses reward probabilities known from the synthetic data-generating process, enabling ground-truth-anchored evaluation.Source: §1 · p. 3

2 Background

• Doubly robust OPE combines outcome modeling with inverse-propensity correction and remains consistent when either the outcome or propensity model is correctly specified.The paper favors DR for practical policy screening, while noting IPS can have high variance when propensities are small or the target differs substantially from the logging policy.Source: §2 · pp. 3–4
• Unlike offline policy learning, which searches a policy class for one optimized policy, this work compares a pre-specified portfolio of contextual-bandit strategies.The paper reframes uplift evaluation from ranking heterogeneous effects to policy value and regret.Source: §2 · p. 4
• Hillstrom, Criteo, and LaLonde/NSW provide benchmarks spanning modest heterogeneity, severe treatment imbalance and rare conversions, and known-ground-truth causal inference, respectively.The LaLonde/NSW data are interpreted through policy regret rather than estimator-bias matching.Source: §2 · p. 4
• The contribution combines fixed-test OPE with a controlled warm-start simulator to assess whether adaptive experimentation is worth deploying and which policy classes are promising.The framework evaluates offline-selected policies before live interaction, complementing work focused separately on estimator accuracy, synthetic benchmarks, or uplift ranking.Source: §2 · p. 5

3 Methodology

• Synthetic data use binary treatment and outcome under balanced logging, with arm-specific response probabilities and an oracle that selects the higher-reward arm for each context.Four representative heterogeneous-treatment-effect regimes are considered.Source: §3 · pp. 5–6

No context.

• With no context, response probabilities and the optimal treatment rule are constant, making this regime a check that policies identify the better arm rather than demonstrate personalization.The no-context case is described as a sanity check because there is no personalization problem to solve.Source: No context · p. 6

Categorical heterogeneity.

• The heterogeneous settings include categorical segments, a continuous linear effect with a monotone switching threshold, and an oscillatory effect with alternating treatment advantage.The two-covariate setting combines linear and oscillatory covariates to create a smooth two-dimensional decision surface.Source: Categorical heterogeneity · p. 6
• OPE fits nuisance components on data disjoint from policy evaluation, compares a fixed portfolio using DR with IPS and SNIPS robustness checks, and quantifies uncertainty with bootstrap intervals.Policies are initialized with offline-learned parameters and then receive rewards from the same response functions in the warm simulator; this is a controlled best-case transfer benchmark, not a proxy for model misspecification, covariate shift, or temporal drift.Source: Categorical heterogeneity · pp. 7, 9
• Figure 1 depicts ground-truth probability of effect by treatment and covariate value.The simulator records reward, regret relative to the oracle, and action accuracy across replications.Source: Figure 1 · p. 8, Categorical heterogeneity · p. 9

4 Results

• The results use two complementary stages: doubly robust OPE compares pre-specified adaptive and non-adaptive policies, followed by warm-start simulation when synthetic ground truth is available.The evaluation asks whether adaptivity helps over fixed allocation, whether OPE screening carries over online, and whether strong adaptive candidates differ enough to warrant further optimization.Source: §4 · p. 9

4.1 No Context

• Without context, the strongest adaptive policies identify the better arm, but a strong fixed policy is already sufficient and further optimization is unlikely to add practical value..Source: §4.1 · pp. 10–11

4.2 Categorical Heterogeneity

• With categorical heterogeneity, several adaptive policies achieve similarly strong OPE values and low regret, while fixed baselines remain clearly inferior.Relatively simple context-aware rules capture most of the available personalization gain.Source: §4.2 · p. 11

4.3 Continuous and Structured Heterogeneity

• Across linear, oscillatory, and two-covariate settings, adaptive policies outperform fixed baselines, recover oracle-implied structure, and remain close to the oracle in warm-start simulations.The learned structures include a monotone switching threshold, alternating treatment advantage, and probability mass in oracle-favored regions.Source: §4.3 · p. 11, Figure 2 · p. 12, Figure 3 · p. 13
• In the two-covariate setup, average and product combinations show no meaningful difference in policy ordering or qualitative fit, although overall mean reward changes.Source: Figure 5 · p. 12, Figure 6 · p. 13

4.4 Validating on Open Datasets

• Hillstrom shows weak evidence for added adaptivity, Criteo a strong signal, and LaLonde an overlap-limited regime; the value of adaptivity depends on heterogeneity and data support.On Hillstrom and LaLonde, top adaptive policies are not clearly separated from strong fixed alternatives; Criteo shows higher estimated value and lower regret for adaptive policies.Source: §4.4 · pp. 14–15

• Criteo shows consistent adaptive-policy gains over non-adaptive baselines, whereas Hillstrom and LaLonde show no clear separation from the strongest fixed alternatives.Source: Figure 7 · p. 16, §5 · p. 17
• Adaptive policies improve on fixed baselines and remain near the oracle when treatment effects are genuinely heterogeneous, but gains are negligible when heterogeneity is weak or absent.Source: §5 · p. 16
• OPE reliably distinguishes broad policy contrasts in the matched regime, but does not necessarily rank history-dependent adaptive learners finely.Source: §5 · p. 17
• DR is never clearly dominated across 38 synthetic scenario-policy comparisons, and its confidence interval overlaps the best estimator in all 38.Source: §5 · p. 17
• The warm-start simulator is a matched best-case benchmark, so the results do not establish robustness to offline-online mismatch.Source: §5 · pp. 16–17
• The evaluated policies are reward-oriented; oracle-policy plots diagnose recovery of context-dependent decisions, not best-arm-identification objectives.Source: §5 · p. 18

• Adaptive policies improve on fixed allocation when meaningful contextual heterogeneity is present, while offering little benefit when it is absent.Source: §6 · p. 18
• The study combines off-policy evaluation with controlled warm-start simulation to compare a pre-specified policy portfolio and inform deployment decisions.Source: §6 · p. 18
• The study evaluates a pre-specified portfolio rather than learning policies, and its matched best-case simulator does not establish robustness to misspecification, covariate shift, or temporal drift.Source: §6 · p. 18
• Because the setting is contextual bandits rather than full sequential reinforcement learning, conclusions concern one-step assignment rather than long-horizon control.Source: §6 · p. 19
• Future extensions include richer policy classes and action spaces, non-stationary or partially observable settings, operational constraints, and further logged datasets or controlled live experiments.Source: §6 · p. 19