Policy Learning with Observational Data

Susan Athey; Stefan Wager

Source-linked AI summary

Policy Learning with Observational Data

Susan Athey, Stefan Wager

arXiv:1702.02896v6Feb 9, 2017math.STcs.LGecon.EMstat.ML

TL;DR

The paper asks how to learn constrained treatment policies from observational data when treatment assignment may be complex or endogenous. It adapts doubly robust, semiparametrically efficient estimation to policy optimization and obtains rate-optimal regret guarantees across several treatment and identification settings. The results remain asymptotic and rely on point-identification assumptions.

Problem
The paper addresses the need to learn budget-, fairness-, and form-constrained treatment policies from observational data beyond randomized or known-assignment settings.
Method
The method maximizes a doubly robust value estimate over a prespecified policy class, using semiparametric scores that can support binary treatments, continuous-treatment nudges, and instrumental-variable settings.
Results
The resulting policies satisfy rate-optimal minimax-regret guarantees, with regret bounded on the order of VC(Π)/n under regularity conditions.
Takeaways & Limitations
The framework provides a general approach to best-in-class policy learning when treatment propensities are unknown or treatment assignment is endogenous.
Takeaways & Limitations
The results rely on point-identification through selection on observables or an instrument satisfying conditional homogeneity, and the regret bounds are asymptotic.

Abstract

from arXiv · show

In many areas, practitioners seek to use observational data to learn a treatment assignment policy that satisfies application-specific constraints, such as budget, fairness, simplicity, or other functional form constraints. For example, policies may be restricted to take the form of decision trees based on a limited set of easily observable individual characteristics. We propose a new approach to this problem motivated by the theory of semiparametrically efficient estimation. Our method can be used to optimize either binary treatments or infinitesimal nudges to continuous treatments, and can leverage observational data where causal effects are identified using a variety of strategies, including selection on observables and instrumental variables. Given a doubly robust estimator of the causal effect of assigning everyone to treatment, we develop an algorithm for choosing whom to treat, and establish strong guarantees for the asymptotic utilitarian regret of the resulting policy.

1 Introduction

The paper studies constrained treatment-policy learning from observational data and develops a semiparametric, doubly robust approach applicable across broader treatment and sampling settings. It establishes regret guarantees whose dependence on policy-class complexity and sample size is rate-optimal under regularity conditions.

Motivation: Practitioners need treatment-assignment policies that respect budget, implementation, functional-form, and fairness constraints.Examples include finite-depth decision trees, rapid lookup rules, and policies restricted to selected covariates.
Problem and scope: The paper addresses observational policy learning when treatment assignments may be unknown, endogenous, continuous, or identified using instruments.This extends beyond literature focused mainly on randomized trials or known treatment-assignment probabilities.
Contribution: The framework achieves optimal dependence on sample size and policy-class complexity under more general sampling designs than earlier methods.The paper treats unknown exogenous assignment probabilities and endogenous assignments requiring an instrument, as well as infinitesimal continuous-treatment nudges.
Approach: The proposed procedure maximizes a doubly robust value estimate over a prespecified policy class, using semiparametrically efficient scores for the intervention target.The framework covers binary-treatment effects, average derivatives for continuous treatments, and related estimands.
Guarantees: VC(Π)/n bounds the resulting policy regret with high probability under regularity conditions.The bound depends on the Vapnik–Chervonenkis dimension of Π and the sample size n.
Scope: The paper’s regret guarantees are asymptotic rather than finite-sample because they rely on asymptotic semiparametric-estimation results.This distinguishes the results from Kitagawa and Tetenov (2018), which provide finite-sample regret bounds in their setting.

2 From Efficient Policy Evaluation to Learning

The paper formulates constrained policy learning from observational data and uses doubly robust scores to evaluate and optimize policies across binary, continuous, and instrumental-variable settings. Under regularity conditions, the resulting policies achieve asymptotic regret guarantees, including when the policy class has structural limitations.

Problem formulation: The goal is to learn a policy π ∈ Π that maps individual features to binary treatment decisions while respecting constraints encoded by Π.The policy class can represent restrictions such as budget, functional form, or fairness.
Problem formulation: The framework covers binary treatments, infinitesimal interventions on continuous treatments, and endogenous treatments identified with an instrument.For continuous treatments, the policy makes a binary decision while observed treatment levels remain continuous.
Efficient policy evaluation: The method estimates causal effects using doubly robust scores constructed from nuisance components and uses those scores to evaluate policy value.The estimator is √n-consistent, asymptotically Gaussian, and semiparametrically efficient under stated conditions.
From evaluation to learning: The policy-learning objective plugs the same doubly robust scores into an empirical maximization over Π rather than merely averaging them to estimate one average treatment effect.This extends efficient estimation from evaluating a single average effect to optimizing across policies.
Guarantees and scope: Nontrivial learning rates for monotone decision rules require additional assumptions on the feature distribution, which the paper does not impose.Without such assumptions, monotone rules can match arbitrary rules along a feature-space curve.
Guarantees and scope: The main result gives asymptotic regret bounds for the learned policy under assumptions on nuisance-estimation rates, noise, overlap, and policy-class complexity.The result also applies uniformly to approximate maximizers when the optimization tolerance is ψn > 0.

3 Upper Bounds

The upper-bound analysis controls policy-learning error by combining empirical-process bounds with uniform coupling of feasible doubly robust objectives to oracle objectives. The resulting argument exploits policy-class complexity and score variance while retaining strong guarantees over approximate optimizers.

3.1 Rademacher Complexities and Oracle Regret Bounds: The analysis first studies an oracle objective based on true influence scores and relates its uniform deviations over Πn to the class’s Rademacher complexity.This separates empirical-maximization complexity from the estimation tools used to construct feasible objectives.
3.1 Rademacher Complexities and Oracle Regret Bounds: Rademacher complexity measures how well policies in Π can fit randomly generated labels, thereby quantifying the class’s ability to overfit random noise.The construction uses independent sign variables ξi = ±1.
3.1 Rademacher Complexities and Oracle Regret Bounds: The proof analyzes slices of Πn because doubly robust scores can evaluate low-regret policies more accurately than high-regret policies.This variance-sensitive slicing produces sharper bounds than a uniform treatment of the entire class.
3.1 Rademacher Complexities and Oracle Regret Bounds: The concentration bound is based on second moments of the scores rather than their supremum, avoiding the uniformly bounded-score requirement in earlier optimal-rate arguments.Earlier alternatives either depend on max{Γi}/√n or introduce an additional log(n) factor.
3.2 Uniform Coupling with the Doubly Robust Estimator: The feasible doubly robust objective is uniformly coupled to the oracle objective over all π ∈ Πn under cross-fitting and nuisance-estimation conditions.The coupling remains comparable to the bound for a single policy when VC(Πn) does not grow too quickly.
3.3 Combining the Bounds: The final argument applies the concentration and coupling bounds at different λ-slices to show that approximate maximizers have regret at the desired asymptotic rate.The tolerance ψn can be omitted from the leading-order result when it decays sufficiently fast.

4 Lower Bounds

The paper establishes minimax lower bounds for policy learning under asymptotically ambiguous treatment-effect sequences, showing that the upper bound is sharp up to a universal constant. The analysis focuses on the 1/√n treatment-effect regime, where policy learning is neither trivial nor impossible and doubly robust evaluation matters.

Lower-bound result: The minimax lower bound applies to binary unconfounded treatments and depends on the policy class through VC(Π).The paper presents this case for simplicity and states that analogous arguments extend to other settings.
Problem construction: The lower-bound construction uses smooth functions f(x) and e(x), variance bounded away from zero and infinity, and bounded treatment effects τ(x).These conditions define the asymptotically ambiguous problem sequences used in the analysis.
Lower-bound result: Theorem 5 shows that the lower bound is sharp up to a universal constant smaller than 200.This result holds for policy classes with finite VC dimension under the stated data-generating conditions.
Treatment-effect regimes: When treatment effects scale as 1/√n, policy learning is neither trivial nor impossible, making doubly robust policy evaluation valuable.Faster decay makes learning better-than-random policies effectively impossible, while larger effects make optimal treatment assignment increasingly obvious.
Comparison with prior bounds: The lower-bound comparison shows that inverse-propensity-weighting bounds can be arbitrarily looser than the adaptive lower bound on asymptotically ambiguous sequences.The ratio between the cited upper and lower bounds scales as M/(η√SP), although the inverse-propensity bound remains optimal under only bounded-outcome and overlap information.

5 Implementation and Experiments

The paper implements policy learning by estimating nuisance components, constructing cross-fitted doubly robust scores, and optimizing policies within constrained classes. Experiments apply finite-depth trees to observational and randomized settings, while simulations show improving regret with sample size and empirical evaluations expose both strong performance and estimation caveats.

Implementation: The three-step procedure estimates nuisance components, forms cross-fitted doubly robust scores, and optimizes a policy over a prespecified class.The nuisance estimators and policy-class optimizer can be chosen independently, subject to the conditions needed for the main theorem.
Implementation: The approach permits flexible nuisance estimators, including sieve-based and kernel methods, provided their mean-squared error converges sufficiently quickly.Theorem 1 requires the estimation error to satisfy a specified rate condition.
Implementation: The policy optimization is computationally challenging because it is nonconvex, but it is numerically equivalent to weighted classification and can be implemented with policytree.The formal guarantees apply only when the weighted-classification problem is solved exactly, not when using approximate alternatives.
GAIN application: The feasible doubly robust estimator gives an average treatment effect of 0.141 ± 0.026, compared with 0.146 ± 0.028 for the oracle estimator and 0.208 ± 0.028 for naive differences in means.The reported estimates suggest that controlling for available covariates changes the result relative to pooling county information.
Experiments: Depth-2 trees outperform depth-1 trees and are competitive with the unconstrained plug-in policy, while feasible evaluation preserves method ordering but is somewhat optimistic about policy quality.The simulation study finds regret improving with sample size and approaching best-in-class regret; sharp treatment-effect jumps produce a phase transition between n = 2,000 and n = 4,000.

6 Discussion

The paper adapts doubly robust treatment-effect estimation to policy learning under observational data and obtains rate-optimal minimax-regret guarantees over constrained policy classes. It identifies confidence sets, dynamic decision making, and non-point-identified settings as directions for further work.

Main contribution: Doubly robust average-treatment-effect estimators can evaluate policies, whose value-maximizing rule over a prespecified class achieves rate-optimal minimax-regret guarantees.The approach separates nuisance estimation from optimization of the doubly robust value function.
Open directions: Future work includes confidence sets that contain an optimal policy within a constrained class with high probability.The discussion gives depth-L decision trees as an example of such a class.
Open directions: The framework could be extended to dynamic decision problems involving sequences of decisions and time-varying covariates.Related work has studied doubly robust policy evaluation and observational stopping rules.
Scope boundary: The results rely on point-identification through selection on observables or an instrument satisfying conditional homogeneity.The paper notes that applications violating these assumptions require methods robust to identification failures.

A Characterizing the VC Dimension

This section characterizes policy-class complexity through distribution-independent Hamming covering numbers and relates that entropy to VC dimension. The resulting bound is used as the proof's complexity condition, with non-personalized rules treated as trivial.

Hamming entropy: The ε-Hamming covering number is the smallest number of policies needed to ε-cover a policy class on a finite set of points.The supremum version ranges over all finite discrete point sets.
Hamming entropy: Hamming entropy is geometric and does not depend on the distribution generating the covariates.This makes it a distribution-independent complexity measure.
VC dimension: A policy class has finite VC dimension exactly when its Hamming entropy satisfies an appropriate logarithmic covering-number bound.The section invokes the Pakes–Pollard characterization.
VC dimension: For VC classes, quantitative upper and lower relationships connect Hamming covering numbers to the class's VC dimension.The lower relationship follows because a VC-dimension-d class can shatter d points.
Proof condition: The proof works with the covering-number bound and assumes VC(Π) ≥ 2, while non-personalized rules with VC(Π) = 1 are treated as trivial.This assumption is used whenever the proof invokes its complexity condition.

B Additional Simulation Experiments

The additional simulations study policy learning for continuous treatment nudges using doubly robust scores and flexible nuisance estimators. Learned policy value improves with sample size, but derivative estimation remains difficult, especially for non-Gaussian treatments.

Setup: Continuous-treatment policies infinitesimally nudge dose Wi for selected samples, with policy value defined for these dose changes.The simulations assume exogenous Wi and learn over depth-2 decision trees.
Policy learning: The policy-learning objective maximizes cross-fit doubly robust scores adjusted by a treatment-cost parameter over Π.The scores enter the objective through (2π(Xi) − 1)(bΓi − C).
Nuisance estimation: The simulations use penalized series regression for conditional response derivatives and an adapted Lindsey method for the conditional treatment density.The response function uses third-order Hermite polynomials, while the density model discretizes Wi and fits penalized logistic regression with interactions.
Simulation designs: Gaussian and non-Gaussian treatment distributions require different basis choices, with quadratic expansions for the former and fifth-order natural splines for the latter.The Gaussian case is described as easier because its logistic regression problem is well specified with a quadratic expansion.
Results: The doubly robust average-derivative estimator converges with n and outperforms regression adjustment and weighting, while learned policy value also improves with n.The estimator remains bias-dominated, and standardized error is much larger than 1, especially in challenging non-Gaussian settings.
Interpretation: The simulations suggest that semiparametric efficiency asymptotics may emerge slowly in this difficult nonparametric problem.The paper suggests that more carefully tailored weighting-function estimators could improve performance.

C.1 Proof of Lemma 2

The proof establishes uniform control of doubly robust policy-value fluctuations by combining distribution-independent entropy bounds, a chaining construction, and concentration inequalities. Fine-scale approximation terms become asymptotically negligible, yielding the target complexity bound.

Chaining construction: The proof uses a Dudley-style chaining argument with increasingly precise approximating policy sets.The construction controls approximation accuracy, prevents branching, and bounds each level's cardinality through covering numbers.
Conclusion: The final entropy step bounds the distribution-dependent Dn-entropy by distribution-independent Hamming entropy controlled through the VC-type assumption.This completes the proof of the desired bound after replacing the initial threshold choice with a sharper one.
Complexity control: The approximation-set cardinalities are controlled by covering numbers in a random distance based on the observed covariates and doubly robust scores.This connects the chaining metric to the policy-class entropy used in the proof.
Concentration strategy: The argument first proves a weaker Rademacher-complexity bound using worst-case variance, then sharpens it using slice-adapted variance.Bernstein's inequality and chaining control the terms uniformly over the policy class.
Assumptions: The proof assumes sub-Gaussian doubly robust scores and controls their maximum on a high-probability event.These conditions support the concentration steps applied across approximation levels.
Negligible terms: The third group of chaining terms does not contribute to first-order Rademacher complexity, and the fourth group vanishes deterministically at the 1/√n scale.The wrap-up combines these negligibility results with the earlier bounds to recover the target result.

C.2 Proof of Corollary 3

The proof controls uniform deviations using truncation and concentration inequalities, then combines the resulting bounds to establish the claimed high-probability conclusion.

Concentration bound: Truncation makes the unbounded sub-Gaussian statistics suitable for applying Talagrand’s inequality.The truncation level is chosen so that the relevant terms satisfy |·| ≤ log(n).
Concentration bound: Uniform sub-Gaussianity is used to verify the remaining concentration bound.
Conclusion: The proof combines a polynomially decaying term with an exponentially decaying term to obtain the stated probability guarantee.
Conclusion: These bounds establish the proof’s second claim.

C.3 Proof of Lemma 4

The proof decomposes the estimation error into three terms, bounds each using cross-fitting, foldwise concentration, and overlap conditions, and then combines the bounds.

Error decomposition: The target difference is decomposed into three summands, D1(π), D2(π), and D3(π), which are bounded separately.
First term: Cross-fitting permits conditioning on out-of-fold nuisance estimates, making the relevant summands independent within folds.
First term: Instrument exogeneity and the exclusion restriction reduce the conditional variance of D1(π) to the sum of constituent variances.
First term: Finite, evenly sized folds satisfy n_k/n → 1/K, allowing the assumed risk bounds to control foldwise terms.
Remaining terms: Jensen’s inequality combines the foldwise bounds, while the weighting function’s mean-zero property and the overlap bound control subsequent terms.
Remaining terms: A deterministic Cauchy-Schwarz bound controls D3(π) uniformly over policies, completing the argument after the three bounds are combined.

C.4 Proof of Theorem 5

The proof constructs a lower-bound instance using shattered groups with unknown treatment effects, then relates policy regret to the difficulty of estimating each effect’s sign efficiently.

Construction: A class with VC dimension d supplies d non-overlapping groups that can realize every binary labeling.
Construction: Restricting to groupwise-constant treatment effects yields a valid smaller class for lower-bounding minimax policy-learning risk.
Optimal policy: If the coefficients were known, the optimal policy would treat exactly the groups with positive coefficients.
Efficient estimation: The minimax learner thresholds an efficient estimator of each coefficient at zero.
Efficient estimation: The semiparametric efficient variance for each coefficient is S_P/d.
Regret lower bound: The probability of incorrectly estimating a coefficient’s sign tends to Φ(−c_j√(d/S_P)), with the signal scaling as 1/√n.
Regret lower bound: Each sign error incurs expected utility loss 2|c_j|, producing asymptotic policy regret from coefficient-sign mistakes.