Source-linked AI summary
Policy Evaluation and Optimization with Continuous Treatments
Nathan Kallus, Angela Zhou
TL;DR
The paper addresses off-policy evaluation and learning for continuous treatments, where discrete rejection sampling almost surely rejects all observations. It extends IPW and DR with kernel-based treatment proximity, establishes consistency and bandwidth results, and reports convergent policy optimization with strong simulated and Warfarin-case-study performance.
Problem
Discrete-treatment off-policy methods rely on rejection sampling, which fails for continuous treatments because exact treatment matches occur with probability zero.
Method
The paper replaces exact-treatment rejection in IPW and DR with kernel weighting of nearby treatments and analyzes bandwidth choice, estimator consistency, and policy optimization.
Results
The estimator is consistent, policy optimization converges toward the best-in-class policy, and experiments show good performance on simulated data and Warfarin dosing data.
Takeaways & Limitations
Continuous-treatment policies can be evaluated and learned from observational data by leveraging local similarity between observed and target treatments.
Takeaways & Limitations
The approach assumes unconfoundedness, common support, and a known generalized propensity score, although the score can otherwise be imputed.
Abstract
from arXiv · showhide
We study the problem of policy evaluation and learning from batched contextual bandit data when treatments are continuous, going beyond previous work on discrete treatments. Previous work for discrete treatment/action spaces focuses on inverse probability weighting (IPW) and doubly robust (DR) methods that use a rejection sampling approach for evaluation and the equivalent weighted classification problem for learning. In the continuous setting, this reduction fails as we would almost surely reject all observations. To tackle the case of continuous treatments, we extend the IPW and DR approaches to the continuous setting using a kernel function that leverages treatment proximity to attenuate discrete rejection. Our policy estimator is consistent and we characterize the optimal bandwidth. The resulting continuous policy optimizer (CPO) approach using our estimator achieves convergent regret and approaches the best-in-class policy for learnable policy classes. We demonstrate that the estimator performs well and, in particular, outperforms a discretization-based benchmark. We further study the performance of our policy optimizer in a case study on personalized dosing based on a dataset of Warfarin patients, their covariates, and final therapeutic doses. Our learned policy outperforms benchmarks and nears the oracle-best linear policy.
1 Introduction
The paper addresses off-policy evaluation and optimization when treatments are continuous, extending personalized decision-making methods beyond discrete actions. It proposes using outcomes with treatments close to a target policy and analyzes the resulting estimator and optimization procedure.
- Off-policy evaluation and optimization assesses and improves personalized policies using observational data collected under historical policies.
- Continuous treatments arise in applications such as personalized medicine, dynamic pricing, and intervention intensity.
- Existing policy-evaluation literature primarily targets discrete action spaces, making continuous-treatment applications difficult to handle faithfully through discretization.
- The proposed framework uses data points whose observed treatments are close to the target policy and analyzes estimator bias, variance, mean-squared error, and policy consistency.
- Experiments evaluate the approach on simulated data and a clinical dataset of Warfarin patients.
2 Policy Evaluation Methodology
The methodology extends inverse-propensity weighting from discrete to continuous treatments by replacing exact-treatment rejection with kernel-weighted proximity. It defines the policy-evaluation problem, states causal and support assumptions, and addresses normalization, bandwidth selection, and doubly robust extensions.
- The generalized propensity score is the conditional treatment density fT|X(ti | xi), extending discrete propensity scores to continuous treatments.
- The policy value is the expected outcome induced by a deterministic treatment policy under potential outcomes.
- The analysis assumes unconfoundedness and common support, requiring treatment assignment to depend only on covariates and positive density at policy treatments.
- Discrete IPW uses rejection and importance sampling, but exact-treatment rejection fails for continuous treatments because P[τ(xi) = ti] = 0.
- The proposed estimator replaces exact rejection with a kernel that weights observations according to treatment proximity, extending IPW to continuous treatments.
- Bandwidth controls kernel smoothing and can be selected to minimize mean-squared error, while self-normalization can reduce variance without sacrificing consistency.
3 Off-Policy Evaluation Analysis
The analysis characterizes the estimator’s bias, variance, mean-squared error, and consistency under technical regularity and support assumptions. Kernel smoothing creates a bias–variance trade-off, while treatment sparsity can increase variance.
- The analysis assumes existing, differentiable conditional outcome and treatment densities, bounded outcomes, common support, and unconfoundedness.
- Kernel-density bias is O(h^2) and depends on outcome-density curvature at the policy treatment.
- Variance increases when the generalized propensity score is small, corresponding to regions where the policy treatment is rarely observed.
- The mean-squared-error analysis combines bias and variance and characterizes an optimal bandwidth.
- Under Assumptions 1–5 and 1/(nh) → 0, the proposed policy-value estimator is consistent for Vτ.
- Under the same condition, the self-normalized off-policy evaluation estimator is also consistent for Vτ.
4 Continuous Policy Optimization
The paper analyzes empirical policy optimization for continuous treatments, including its generalization error and convergence under bandwidth and complexity conditions. It also discusses variance regularization for overly expressive policy classes.
- Policy optimization: Accurate off-policy evaluation underpins policy optimization, which selects policies by minimizing estimated expected outcomes.The analysis studies how the empirically optimal policy performs out of sample.
- Policy optimization: The empirical policy optimizer minimizes the proposed policy-value estimator over a constrained policy class, such as bounded linear policies.This optimization can be viewed as weighted empirical risk minimization over the constrained class.
- Generalization analysis: Theorem 5 bounds generalization error using the kernel, propensity-weight, outcome, bandwidth, and Rademacher-complexity terms.The bound is derived from concentration, the relation between policy value and empirical estimates, and a Rademacher comparison lemma.
- Consistency of policy optimization: The policy optimizer’s regret converges to zero when h=o(1), h=ω(R_n(T)), and h=ω(1/√n), achieving best-in-class performance under the stated assumptions.The result applies to the empirical minimizer relative to the best policy in the class.
- Practical considerations: Variance regularization mitigates policies that assign treatments far outside the observed treatment range and otherwise exploit near-zero kernel weights.Without such control, overly expressive policy classes can trivially minimize the loss.
5 Experiments
Experiments show that continuous off-policy evaluation converges in synthetic settings, supports consistent policy optimization, and performs well in higher-dimensional simulations and a Warfarin dosing study. The approach generally outperforms discretization-based or naive benchmarks and approaches strong policy baselines.
- Validation on Synthetic Data: Continuous policy evaluation converges to the truth, whereas discretization is systematically biased, inconsistent, and higher-variance.The comparison uses 50 replications for each n from 10 to 300 in randomized and confounded settings.
- Validation on Synthetic Data: Policy evaluations are tight near the optimum, optimization identifies the true β, and out-of-sample error converges to zero as n increases.The simulations evaluate β over [0, 1.3] with n=300 and use 20 replications for out-of-sample error.
- Policy Optimization Simulation: In a 10-dimensional quadratic-outcome simulation, continuous off-policy evaluation improves mean risk over competing methods and nears the best full-information treatment assignment.The best full-information assignment is outside the linear policy class; the linear policy better personalizes treatment than the best constant policy.
- Warfarin case study: In the Warfarin study, Continuous OPE achieves a mean loss of 10.2 and is competitive with the best-in-class linear policy.It improves on the direct method and reduces median and mean dose differences relative to the mean-dose benchmark.
- Warfarin case study: Discrete off-policy optimization struggles on Warfarin because propensity scores are mostly zero or one under tested dosage discretizations.This makes the choice of an appropriate discretization difficult for the real dataset.
6 Conclusion
The paper develops inverse-propensity-weighted policy evaluation and learning for continuous treatments by replacing discrete rejection sampling with kernel-based local treatment information. Its theory establishes convergence, and experiments demonstrate efficacy on simulated and Warfarin data.
- The estimator extends inverse-propensity weighting from discrete actions to continuous treatments using a kernel to incorporate information from similar treatments.This replaces rejection sampling with treatment-proximity weighting.
- The policy optimization generalization bound implies that the empirically optimal policy converges to the policy minimizing expected loss.
- The approach is evaluated on simulated data and a real-world dataset of Warfarin dosages for patients.
7 Appendix
The appendix analyzes the estimator’s bias, variance, mean-squared error, consistency, and optimization structure. It characterizes bandwidth choices, convergence rates, assumptions, and alternative kernel formulations for optimization.
- Bias and variance: The estimator’s bias arises from a second-order Taylor expansion of the conditional outcome density around the policy treatment.For symmetric kernels, odd-order moments integrate to zero, while the second-order term determines the bias.
- Bias and variance: Bias vanishes asymptotically when h^2 tends to zero under bounded outcomes and bounded second derivatives of the conditional density.
- Bias and variance: The multivariate analysis uses product kernels with coordinate-specific bandwidths and a multidimensional change of variables.
- Mean squared error: The asymptotically optimal bandwidth has order O(n^-1/5), but its constants generally depend on the unknown true density.
- Consistency: At the optimal bandwidth, the estimator converges in probability at rate n^-2/5, described as standard for kernel density estimation.
- Consistency: Uniform consistency requires density regularity and moment conditions, with almost-sure convergence under the stated assumptions.The appendix also gives a strong uniform consistency rate involving n^2/5 and a logarithmic factor.
- Policy optimization: The optimization can be nonsmooth and nonconvex with the Epanechnikov kernel, while a triangular kernel permits a difference-of-convex formulation over a convex policy space.