PaperlayerView exact revision on arXiv

Source-linked AI summary

Confounding-Robust Policy Improvement

Nathan Kallus, Angela Zhou

arXiv:1805.08593v3cs.LGstat.ML

TL;DR

Observational policy learning can fail when unobserved confounding invalidates unconfoundedness, potentially producing harmful personalized decisions. The paper minimizes worst-case regret against a baseline over propensity-weight uncertainty sets, with guarantees and efficient optimization. Synthetic and clinical assessments show safety relative to baseline and improvement when supported by the data.

  • Problem

    Unobserved confounding can invalidate observational policy learning and cause personalized decisions to perform worse than current standards of care.

  • Method

    The framework minimizes worst-case estimated regret against a baseline over uncertainty sets for propensity weights, using robust optimization and efficient weight-based computation.

  • Results

    The approach provides finite-sample safety and population minimax-regret guarantees, while synthetic and clinical assessments demonstrate personalized improvement without worsening the baseline when the uncertainty set is well specified.

  • Takeaways & Limitations

    Confounding-robust policy improvement can support personalized treatment decisions by prioritizing well-evidenced improvement over potentially harmful gains from nominal observational analyses.

  • Takeaways & Limitations

    The guarantees depend on a well-specified uncertainty set, and the framework's preliminaries assume iid observational data with observed treatment outcomes linked to potential outcomes.

Abstract

from arXiv · show

We study the problem of learning personalized decision policies from observational data while accounting for possible unobserved confounding. Previous approaches, which assume unconfoundedness, i.e., that no unobserved confounders affect both the treatment assignment as well as outcome, can lead to policies that introduce harm rather than benefit when some unobserved confounding is present, as is generally the case with observational data. Instead, since policy value and regret may not be point-identifiable, we study a method that minimizes the worst-case estimated regret of a candidate policy against a baseline policy over an uncertainty set for propensity weights that controls the extent of unobserved confounding. We prove generalization guarantees that ensure our policy will be safe when applied in practice and will in fact obtain the best-possible uniform control on the range of all possible population regrets that agree with the possible extent of confounding. We develop efficient algorithmic solutions to compute this confounding-robust policy. Finally, we assess and compare our methods on synthetic and semi-synthetic data. In particular, we consider a case study on personalizing hormone replacement therapy based on observational data, where we validate our results on a randomized experiment. We demonstrate that hidden confounding can hinder existing policy learning approaches and lead to unwarranted harm, while our robust approach guarantees safety and focuses on well-evidenced improvement, a necessity for making personalized treatment policies learned from observational data reliable in practice.

1 Introduction

Observational data enables personalized policy learning when randomized experiments are scarce, but unobserved confounding can invalidate unconfoundedness-based methods and produce harmful policies. The paper develops confounding-robust policy improvement that seeks policies no worse than a baseline while improving when supported by the data.

  • Randomized experiments may be scarce, costly, or unethical, motivating personalized policy learning from observational data such as EMRs.
  • Unconfoundedness is controversial and may fail in observational healthcare data because important treatment-driving clinical information can remain unobserved.
  • The WHI observational study suggested a 40-50% reduction in CHD incidence, whereas its clinical trial found dangerously elevated CHD incidence in the HRT arm.
  • When unconfoundedness fails, existing policy-learning methods can exploit nonexistent individual effects, intervene unnecessarily, and cause net harm.
  • The proposed framework optimizes worst-case improvement over a baseline using uncertainty sets around nominal inverse propensity weights.
  • The framework supports data-driven confounding uncertainty sets and proves uniform convergence over restricted policy classes and possible confounded distributions.

2 Problem Statement and Preliminaries

The paper formulates policy learning with multiple discrete treatments using observational tuples, potential outcomes, and restricted randomized policies. Because nominal propensities may not equal true propensities under unobserved confounding, policy value is defined through potential outcomes and policy-induced treatment assignment.

  • The data consist of iid covariate, treatment, and outcome tuples with treatments drawn from m discrete options and observed outcomes equal to assigned-treatment potential outcomes.
  • Nominal propensities condition on observed covariates, whereas true propensities also condition on potential outcomes and are not estimable from observed data.
  • Unconfoundedness is the equality between nominal and true propensities; the framework does not assume this equality.
  • A policy maps covariates to treatment probabilities, and its value is the expected outcome under an independent policy-induced treatment variable.
  • The policy class can be restricted to structures such as deterministic linear policies or bounded-depth decision trees.

3 Related Work

Prior policy-learning methods commonly rely on unconfoundedness and inverse-propensity or outcome-model estimators, while sensitivity-analysis work typically evaluates causal conclusions. This paper instead applies data-driven minimax regret to personalized policy learning under bounded unobserved confounding.

  • Observational policy-learning methods include regression-based, reweighting-based, and doubly robust strategies, generally under unconfoundedness.
  • Under unconfoundedness, inverse-propensity weighting can estimate policy value by changing measure from historical treatment assignment to a new policy.
  • Propensity division can create extreme weights and high-variance estimates, motivating probability clipping or balancing weights.
  • With unobserved confounding, neither policy value nor the optimal policy is generally identifiable using these standard approaches.
  • Earlier minimax-regret approaches can be highly conservative, whereas this paper uses observed selection information and a data-driven uncertainty set around inverse propensity weights.
  • Causal sensitivity analysis commonly evaluates robustness of treatment-effect inferences, while this work targets personalized policy learning over an infinite policy family.

4 Robust policy evaluation and improvement

The framework evaluates candidate policies by worst-case regret against a baseline over uncertainty sets for true inverse propensities. It provides sharp population regret bounds, supports alternative uncertainty structures, and targets minimax-optimal policies under plausible confounding.

  • The method minimizes a regret bound against a specified baseline by maximizing a reweighting-based regret estimate over an uncertainty set around nominal propensities.
  • The baseline can represent the current standard of care or an existing guideline-based personalized policy that the learned policy should not worsen.
  • Weight-normalized estimators remain consistent under the stated oracle conditions and make the baseline material to the robust bound, unlike the unbounded regret estimate.
  • Unknown true inverse propensities are replaced by worst-case values consistent with observed data and restrictions on unobserved confounding.
  • The marginal sensitivity model bounds pointwise deviations from nominal propensities using Γ, with Γ = 1 corresponding to unconfoundedness and Γ = ∞ imposing no restriction.
  • The population worst-case regret is the largest possible true regret compatible with the observable distribution and sensitivity restrictions, and the resulting interval is sharp.
  • The minimax-optimal policy provides best-possible uniform control over all compatible population regrets.
  • The framework also accommodates total-variation and budgeted uncertainty sets, including settings where confounding is concentrated among exceptions rather than every unit.

5 Analysis, improvement guarantees, and minimax optimality

The framework provides finite-sample safety and minimax-regret guarantees for policies learned under bounded unobserved confounding. These guarantees depend on manageable policy-class complexity and sensitivity models, and extend to estimated nominal propensities with an error controlled by propensity estimation.

  • Assumptions: The guarantees require bounded outcomes, strong overlap, finite VC-major dimension, and a policy class that is flexible but not unrestricted.The policy class includes linear, logistic, and bounded-depth tree policies under the stated complexity condition.
  • Improvement guarantee: The learned policy’s true population regret is bounded by its worst-case empirical regret plus finite-sample terms that vanish as n grows.The bound holds even with unobserved confounding when the sensitivity model is appropriately specified.
  • Improvement guarantee: O(n^-1/2) finite-sample terms have sub-Gaussian tails regardless of unobserved confounding.When the baseline policy belongs to the candidate class, the empirical objective is nonpositive, ensuring no worse performance up to vanishing terms.
  • Minimax optimality: The policy achieves uniform minimax regret control over all distributions compatible with the sensitivity model, with finite-sample bounds and asymptotic minimax optimality.This controls the entire range of possible population regrets rather than relying on point identification.
  • Sensitivity-model complexity: Uniform convergence over all policies and sensitivity-model weights can fail because the uncertainty set WΓ is too large, requiring complexity control for the sensitivity model.The limitation is explicit for uncertainty sets isomorphic to all bounded functions.
  • Estimated propensities: Estimated nominal propensities add an error controlled by propensity-estimation accuracy, which generally vanishes with more data.The resulting policy remains near-optimal relative to the objective computed using true nominal propensities.

6 Algorithms for Optimizing Robust Policies

The robust policy problem is solved by decomposing it into an inner worst-case regret problem and an outer policy optimization problem. The resulting procedures support differentiable policies, decision trees, and broader policy classes compatible with conic or integer programming.

  • Parametric policies: For differentiable parametrized policies, a subgradient method alternates between finding worst-case weights and updating policy parameters.Logistic policies are a principal example, while multiple restarts and averaged parameters are used in the procedure.
  • Structured policies: Decision-tree and other structured policy classes can be optimized using mixed-integer, conic, or linear formulations when their policy constraints admit compatible representations.The framework covers sparse linear policies and fixed-depth decision trees under such formulations.
  • Worst-case regret: A Charnes-Cooper transformation converts the linear-fractional inner objective into an equivalent linear program with normalized weights.The dual formulation supports an efficient line-search procedure for identifying pessimal weights.
  • Worst-case regret: The inner worst-case regret problem can be solved efficiently by sorting the data and ternary-searching a unimodal function.This generalizes prior results to arbitrary pointwise bounds on the weights.
  • Parametric policies: The objective is convex in the reward vector because it is a maximum over linear functions, enabling subgradient updates even when it is not convex in policy parameters.The subgradient is obtained from the argmax set of the inner weight problem.
  • Practical optimization: Algorithmic refinements empirically stabilize optimization when computing robust policies across multiple plausible values of Γ.This is especially relevant when a decision-maker evaluates a range of confounding-sensitivity levels.

7 Empirical Results

The experiments compare confounding-robust policies with methods assuming unconfoundedness on synthetic and WHI-based data. Across settings, the robust approach avoids harm and can improve over baseline when the treatment-effect environment supports it, although its performance depends on the scalarization and confounding regime.

  • Experiments: The study evaluates confounding robustness using a controlled synthetic experiment and a semi-synthetic WHI case study.The WHI analysis uses actual clinical data with semi-synthetic outcomes and treatment-effect bonuses.
  • Synthetic experiments: The binary-treatment simulation compares nominal IPW and GRF policies with robust logistic policies using unbudgeted or budgeted uncertainty sets.The policies are evaluated against a never-treat baseline across values of the confounding parameter Γ.
  • Synthetic experiments: The synthetic evaluation reports out-of-sample regret relative to the baseline and attributes worse IPW and GRF performance to violated unconfoundedness rather than algorithmic defects.Data are generated with an unobserved confounder affecting treatment assignment and outcomes, and true policy values are computed from known counterfactuals.
  • Synthetic experiments: The three-treatment experiment extends the policy class to multinomial logistic probabilities and includes a treatment arm with heterogeneous effects and greater confounding.The additional parametrization makes optimization noisier, so further refinement is left for future work.
  • WHI case study: For λ = −0.93 in the WHI case study, the robust policy finds benefit for some sensitivity parameters while IPW and RF linear policies incur harm relative to all-control.The WHI outcomes combine cardiovascular health with clinical benefits of hormone replacement therapy through a scalarization parameter λ.
  • WHI case study: Across λ, robust policies avoid harm when treatment is nearly uniformly bad, may underperform frequent-treatment policies when treatment is nearly uniformly good, and achieve safety plus improvement in moderate heterogeneous regimes.The moderate-regime result holds for moderate values of Γ, while very negative λ can make robust defaulting to baseline worse than confounded IPW.

8 Practical Considerations in Calibrating Uncertainty Sets

The paper treats Γ as a practical sensitivity parameter that must be calibrated to plausible unobserved confounding. It recommends benchmarking Γ against observed selection predictors and using calibration plots to expose the safety–conservatism trade-off.

  • Benchmarking: A practical calibration strategy benchmarks omitted-confounder strength against the propensity informativeness of observed covariates.The paper suggests examining the odds-ratio change caused by omitting each observed covariate and incorporating domain knowledge about plausible unobserved predictors.
  • Benchmarking: In the WHI case study, most induced odds ratios are bounded by Γ around 0.8 to 1.2, apart from highly treatment-predictive variables such as age.This benchmark informs which confounding levels should be considered when assessing policy safety.
  • Choosing Γ: Choosing Γ too high provides uniform regret control over more confounding but can be conservative, whereas choosing it too low weakens control over larger confounding.The trade-off is between robustness to a wider uncertainty range and performance when actual confounding is smaller.
  • Calibration plots: A calibration plot trains policies at Γ_k and evaluates their estimated worst-case regret under Γ_k′, revealing how regret changes when actual confounding exceeds the training assumption.The procedure fixes a sequence of Γ values, trains one robust policy for each, evaluates every policy across the sequence, and plots the resulting regrets.
  • Calibration plots: In the WHI example, a policy trained at Γ = 1.05 can incur large regret when evaluated at confounding levels around 1.1 to 1.2.The result illustrates why a policy calibrated to too little confounding may lose safety under modestly larger confounding.
  • Calibration plots: The calibration analysis helps decision-makers choose a policy balancing safety and performance according to application requirements.The recommended choice depends on the acceptable range of confounding and the cost of conservatism.

9 Conclusion

The paper develops confounding-robust policy improvement for personalized interventions learned from observational data with unobserved confounding. It minimizes worst-case regret against a baseline and provides safety guarantees alongside efficient optimization and empirical assessments.

  • The framework optimizes personalized policies against a baseline while accounting for possible unobserved confounding in observational data.It generalizes inverse-propensity-weighted estimators using uncertainty sets centered at nominal weights.
  • The method guarantees, under a well-specified uncertainty set, that the learned policy does no worse than the standard of care and may improve upon it.The guarantee is finite-sample and supports safe implementation when the data support improvement.
  • Efficient optimization and uniform convergence analyses establish population-level minimax-optimal regret for the robust policy.The approach uses subgradient descent on robust risk and provides uniform convergence bounds.
  • Synthetic and clinical-data assessments show that the method can recommend personalized treatment while maintaining performance guarantees relative to baseline preferences.The framework also enables assessment of how plausible confounding levels affect robust policy performance.

A Proofs for optimization structure

The proofs characterize the structure of optimal uncertainty-set weights and use that structure to reduce optimization and establish stability properties. In particular, optimal weights are organized monotonically by a coefficient index, enabling finite candidate searches in some settings and supporting uniform-convergence analysis.

  • The optimal threshold lies between adjacent order statistics, so candidate solutions can be found by checking finitely many threshold values.The analysis identifies λ between ordered coefficients and restricts attention to values λ(k).
  • For discrete outcomes or mixtures with atoms, lexicographic ordering by outcome and weight-width preserves the required unimodality structure.This extends the ordering argument beyond continuously distributed outcomes, where ties occur with probability zero.
  • Complementary-slackness analysis shows that optimal weights can be represented through thresholded coefficient indices and nondecreasing functions.For the displayed structure, weights switch according to whether coefficients lie above or below a threshold λ.
  • The monotone-function composition class remains a VC-major class when generated from a VC-major class.This stability result supports entropy and uniform-convergence arguments for the relevant weight-function classes.
  • The proofs therefore show that optimization can be restricted to worst-case weights with additional structure without changing the relevant robust analysis.

B.2 Proof of Theorem 6

The proof of Theorem 6 derives uniform convergence for robust regret by decomposing deviations into several empirical-process terms and bounding them uniformly over policies and admissible weights. Combining these bounds yields the stated high-probability result.

  • The regret analysis begins with a decomposition that separates the terms needed to control minimax regret through uniform convergence.The decomposition is combined with subadditivity and bounds over the policy and uncertainty-set classes.
  • Weight normalization provides deterministic boundedness, while treatment-wise partitions and union bounds control deviations across treatment groups.The argument applies concentration inequalities to the relevant treatment-specific empirical quantities.
  • Uniform convergence is established over policies and uncertainty-set weights using symmetrization, contraction, chaining, and tail inequalities.The proof reduces product-class deviations to bounds for the individual policy and weight classes.
  • The resulting high-probability bound is obtained by combining the bounds for the decomposed terms and collecting constants.
  • The proof of the general theorem analyzes uniform convergence around true propensity weights assumed to lie in the uncertainty set.Well-specification ensures that the true inverse-propensity weights are covered by the admissible set.

B.4 Proof of Theorem 8

The proof of Theorem 8 analyzes budgeted uncertainty through partial dualization and reparameterization. It establishes that optimal rescaled weights are nondecreasing in a coefficient index, reducing the search space to a structured class useful for uniform-convergence analysis.

  • Partial dualization of the budget constraint and strong duality reduce the optimization to a problem over rescaled weights and scalar parameters.The transformation introduces a normalized variable u in [0, 1] together with scaling and dual parameters.
  • The proof interprets the reduced problem as a fractional-knapsack-like optimization sorted by the multipliers r.
  • The optimal rescaled weight function u*(ψ, η) is nondecreasing in the coefficient index r for all ψ and η.This is the central structural result of Lemma 6.
  • The monotonicity result permits an equivalent search over scalar parameters and the intersection of the feasible rescaled-weight set with the monotone-function class.
  • The structural characterization is sufficient for uniform-convergence analysis but is weaker than the separate result that provides an algorithmic solution.

B.5 Proof of Theorem 9

The proof establishes uniform control of the robust policy objective by combining duality, boundedness, and concentration arguments over the uncertainty set. These steps connect the sample and population minimax-regret problems.

  • Restricted uncertainty sets: The nondecreasing optimal-solution characterization shows that restricted budgeted weights recover the optimal regret.The result follows by optimizing over the corresponding nondecreasing sample-budgeted uncertainty set.
  • Dual reformulation: Strong duality and Charnes–Cooper transformations reparameterize the uncertainty-set optimization through primal and dual weight variables.The proof analyzes treatment partitions separately and uses equivalent formulations over restricted, budgeted weight classes.
  • Boundedness: Bounded feasible weights and strong overlap restrict the relevant primal and dual variables to compact ranges.These bounds support subsequent perturbation and uniform-convergence arguments.
  • Uniform convergence: Lipschitz contraction and tail inequalities control uniform deviations when optimizing over the restricted weight class.The argument applies a union bound across treatments to obtain a high-probability bound.
  • Theorem conclusion: Applying the deviation bound to sample- and population-optimal policies yields the theorem through a union bound and the triangle inequality.The proof compares the corresponding sample and population minimax-regret objectives.

C.1 Subgradient Approach Refinements

The section refines optimization for nested confounding parameters and develops interpretable decision-tree policies through robust mixed-integer optimization. A recursive partitioning heuristic supplies warm starts but has a narrower applicability than the full formulation.

  • Subgradient refinements: Warm starts from nearby Γ values and checks against earlier policies stabilize optimization over nested uncertainty sets.The refinements are intended for a feasible range of plausible Γ values considered in practice.
  • Robust decision trees: Optimal confounding-robust trees combine the dual robust-risk formulation with a mixed-integer representation of fixed-depth decision trees.The formulation jointly selects covariate splits, leaf assignments, and treatment probabilities.
  • Tree formulation: Tree constraints encode leaf membership, ancestor split consistency, minimum leaf size, and treatment-assignment probabilities.Auxiliary product variables and big-M constraints connect assignments to leaf-level policy probabilities.
  • Recursive partitioning: Recursive partitioning greedily refines treatment assignments using covariate splits and robust-risk changes, then provides a warm start for the mixed-integer program.The heuristic initializes with a constant treatment and recursively evaluates candidate univariate splits.
  • Scope and limitation: The recursive warm-start approach applies only to the unbudgeted uncertainty set because uncertainty budgets couple risk across tree levels.Optimal assignments at one level can also depend on assignments at other nodes at the same level.

D WHI Case Study details

The WHI case study examines policy regret under different treatment-effect scalarizations and compares the distribution of odds ratios induced by dropped covariates. Most induced odds-ratio mass lies near Γ = 1, with wider distributions for some covariates.

  • Policy regret: Table 1 reports policy regret for WHI under different λ scalarizations.The table is organized around how the treatment-effect scalarization changes the regret comparison.
  • Odds-ratio distributions: Most odds-ratio probability mass lies within Γ ∈[0.8, 1.2], while a few covariates produce wider informativity distributions.Figure 7 uses odds ratio on the x-axis, density on the y-axis, and a common y-scale of 0 to 10 across subplots.
Loading 1805.08593v3…