Confounding-Robust Policy Improvement

• Unobserved confounding can make existing policy-learning methods exploit nonexistent individual effects and recommend interventions that cause net harm in sensitive applications.Unconfoundedness is unverifiable and may fail even with richer electronic medical records or randomized trials affected by compliance, censoring, or site selection bias.Source: §1 · p. 2
• The WHI comparison shows the danger: observational data suggested a 40-50% CHD reduction from HRT, whereas the clinical-trial HRT arm stopped early for dangerously elevated CHD incidence.The conflicting evidence reduced HRT prescribing by 80% and motivates the paper’s later confounded-data case study.Source: §1 · pp. 2–3
• The proposed framework optimizes worst-case improvement over a baseline policy using weighted outcomes with propensity weights drawn from an uncertainty set around nominal inverse propensity weights.The baseline is material: it can represent treat-all, treat-none, or the current standard of care, and the policy must improve regardless of the direction of confounding.Source: §1 · pp. 3–4
• The theory establishes uniform convergence over restricted-complexity policies and confounded data-generating distributions, yielding asymptotic minimax-regret optimality and near-baseline safety guarantees.The uncertainty sets use pointwise propensity-odds-ratio bounds or a global budget-of-uncertainty parameter; the objective value can validate improvement over baseline up to vanishing complexity terms.Source: §1 · p. 4
• Efficient algorithms solve robust policy optimization for parameterized policies and decision trees, while synthetic experiments and a WHI observational-trial case study assess uncertainty effects and show confounded learning can cause harm.The WHI design permits learning from observational data with unobserved confounding and evaluating out-of-sample performance on an unconfounded clinical trial.Source: §1 · p. 4

Setup

• The data are iid tuples of covariates, one of m ≥2 discrete treatments, and real-valued outcomes, with observed outcomes equal to corresponding potential outcomes and lower outcomes preferred.Potential outcomes are defined for every treatment level, and empirical expectations are sample averages.Source: §2 · p. 5

Propensities and confounding

• Because unconfoundedness is not assumed, the nominal propensity conditioned on covariates can differ from the true propensity conditioned additionally on potential outcomes, which cannot be estimated from data.Nominal propensities may be estimated using probabilistic classifiers, whereas true propensities account for confounding and are not observable.Source: §2 · p. 5

Policies and value

• A policy maps covariates to a probability distribution over treatments, with an independent policy-generated treatment variable defining its value as the expected outcome under that assignment.The policy value is V(π) = E[Y(Zπ)].Source: §2 · p. 5

Policy classes

• Learning is restricted to a policy class Π, including deterministic linear policies and bounded-depth decision trees that can assign probability vectors at each leaf.These examples illustrate both deterministic treatment selection and randomized policies within the restricted class.Source: §2 · p. 5

Causal inference for personalization

• Under unconfoundedness, regression, reweighting, and doubly robust methods learn personalized policies, but unobserved confounding makes outcome effects and policy values unidentifiable.Regression-based methods can fail under misspecification, while reweighting methods may suffer extreme weights and high variance; without unconfoundedness, these approaches lack guarantees and may cause harm.Source: §3 · pp. 6–7

Minimax regret

• Unlike prior minimax-regret approaches based on arbitrary-confounding bounds or asymptotic experimental designs, this work uses a data-driven uncertainty set around inverse propensity weights.The uncertainty set incorporates observable selection information while assessing reasonable violations of unconfoundedness.Source: §3 · p. 7

Policy improvement

• Safe policy-improvement methods in reinforcement learning use uncertainty sets or trust regions, but do not address confounding in observational causal policy learning.This robust or minimax perspective motivates the method’s name and framing.Source: §3 · p. 7

Sensitivity analysis

• Causal sensitivity analysis studies robustness to unconfoundedness assumptions, whereas this work learns personalized policies across an infinite family of candidate policies without requiring auxiliary data or structural assumptions.The method borrows Tan’s marginal sensitivity model, which bounds the strength of unobserved confounding in treatment selection, and applies it to personalized treatment decisions.Source: §3 · pp. 7–8

Sampling and estimator bounds

• Related sampling work derives bounds for Hájek estimators under weight restrictions or parametric and shape constraints, while this paper applies sensitivity analysis to robust personalized policy learning with additional convergence challenges.The minimax objective is a Hájek estimator for policy value.Source: §3 · p. 8

4 Robust policy evaluation and improvement

• The framework minimizes a worst-case regret bound relative to baseline π0, ensuring no worse performance and possible improvement under confounding.The baseline may represent current standard of care, usually assigning control, or an established personalized guideline.Source: §4 · pp. 8–9

4.1 Confounding-robust policy learning by optimizing minimax regret

• The learning method ranges over inverse propensity weights consistent with observed data and a marginal sensitivity model, with Γ = 1 representing unconfoundedness and Γ = ∞ imposing no restriction.It uses normalized Hájek-style regret estimation and recommends plugging in estimated nominal propensities when those are unknown.Source: §4.1 · pp. 9–10

4.2 The population minimax-optimal policy

• The population worst-case regret is the tight largest possible true regret over distributions compatible with observed data and the marginal sensitivity model.The smallest- and largest-possible regrets form a sharp interval equal to the closure of all possible regrets, and the corresponding minimax policy achieves the best possible uniform control.Source: §4.2 · p. 11

4.3 Extension: Budgeted uncertainty sets to address “local” confounding

• Budgeted uncertainty sets extend the framework beyond pointwise odds-ratio bounds by targeting localized confounding exceptions and producing a less conservative policy criterion.The additional set uses average deviation budgets Λt, which can be calibrated as a fraction ρ < 1 of the deviation allowed by WΓ; subsequent sections establish similar minimax-optimality and tractability guarantees.Source: §4.3 · pp. 11–12

5 Analysis, improvement guarantees, and minimax optimality

• The guarantees require searching over a flexible but restricted policy class, with bounded outcomes, strong overlap, and finite VC-major complexity.The framework covers linear, logistic, and bounded-depth tree policies.Source: §5 · pp. 12–13

5.1 Improvement guarantee

• The learned policy’s population regret is bounded by its worst-case empirical regret plus O(n^-1/2) terms, ensuring no harm relative to π0 when π0 ∈Π.The guarantee holds under well-specified uncertainty weights, bounded outcomes and true propensities, strong overlap, and finite policy-class VC-major dimension; sufficiently negative objective values certify strict improvement.Source: §5 · p. 12, §5.1 · pp. 13–14

5.2 Minimax optimality

• The policy achieves asymptotically minimax-optimal uniform regret by restricting worst-case weights to a structured, bounded-complexity subclass rather than the full sensitivity set.The full WΓ is too flexible for uniform convergence, while optimization identifies monotone thresholding solutions with finite VC-major dimension, enabling finite-sample regret bounds.Source: §5.2 · pp. 14–15

5.3 Estimated propensities

• Using estimated nominal propensities adds an error controlled by propensity-estimation accuracy, which vanishes with more data and preserves the preceding guarantees with an additive adjustment.The estimated-propensity policy is near-optimal relative to the true-propensity objective, and the improvement guarantee only requires validity of the marginal sensitivity model.Source: §5.3 · pp. 16–17

6.1 Dual Formulation of Worst-Case Regret

• The worst-case regret problem is transformed into a linear program and solved efficiently by sorting rewards and ternary-searching a unimodal function.The transformation uses Charnes–Cooper rescaling, while the resulting solution generalizes Aronow and Lee (2012) to arbitrary pointwise weight bounds.Source: §6.1 · pp. 18–19

6.2 Optimizing Parametric and Differentiable Policies

• For differentiable parametric policies, a projected subgradient method alternates between finding pessimal weights and updating policy parameters.The method exploits convexity of the inner maximum in reward vectors, while the outer objective need not be convex in policy parameters.Source: §6.2 · p. 20
• Algorithmic refinements empirically stabilize optimization when computing robust policies across multiple plausible values of Γ.The procedure averages parameters across initializations and uses the best-performing restart before applying these refinements.Source: §6.2 · pp. 20–21

6.3 Optimizing Over Other Policy Classes

• The framework extends to policy classes whose constraints are compatible with conic or integer programming, including sparse linear policies and fixed-depth decision trees.For polyhedral uncertainty sets, the conic formulation becomes linear, and existing integer-optimization formulations can encode the policy-class constraints.Source: §6.3 · p. 21

7 Empirical Results

• The empirical study combines a controlled synthetic experiment with a WHI observational-study and clinical-trial case study to assess whether confounding robustness prevents harmful intervention while preserving personalized improvement.The WHI experiments report sizable baseline improvements for several scalarizations of blood-pressure reductions and known clinical benefits.Source: §7 · p. 21

7.1.2 Multiple Treatments

• The multiple-treatment simulation has three treatments, including one highly variable, more-confounded arm, uses 50 replications with n = 200, and evaluates on known-counterfactual test data.Additional parametrization leads to a noisier optimization process for Algorithm 1, and further refinement is left for future investigation.Source: §7.1.2 · p. 23, Figure 2 · p. 23

7.2.1 WHI Case Study Background

• The WHI analysis revisits hormone replacement therapy because unaccounted-for confounding historically produced harmful intervention policies, testing robust learning across possible treatment-benefit scalarizations rather than prescribing a single clinical policy.The semi-synthetic outcomes combine cardiovascular outcomes with treatment-effect bonuses reflecting known ancillary benefits.Source: §7.2 · p. 24, §7.2.1 · pp. 24–25

7.2.2 WHI Case Study Evaluation Setup and Outcome Measures

• WHI policies are learned from observational data while varying λ ∈ [−0.5, −1.5] and Γ, then evaluated on randomized trial data using an out-of-sample Horvitz-Thompson regret estimator against the all-control baseline.The complete-case samples contain n = 13594 trial participants and n = 48458 observational participants, with treatment randomized at probability 1/2 in the trial.Source: §7.2.2 · p. 25

7.2.3 WHI Case Study Policy Learning Results

• In the WHI case study, for λ = −0.93, robust policies find benefit at some Γ values, whereas IPW and RF lin. incur harm relative to the all-control baseline.Policies are trained on confounded observational data and evaluated on held-out randomized clinical-trial data.Source: §7.2.3 · p. 26, Figure 3 · p. 26
• Across WHI scalarizations, robust policies avoid harm from unconfoundedness-based learning in moderate regimes and achieve improvement for moderate Γ, while conservatively defaulting to control when treatment is mostly harmful.When treatment is mostly beneficial, less-treatment robust policies can underperform IPW; when robust specification is poor, confounded IPW may outperform defaulting to baseline.Source: §7.2.3 · p. 26, Figure 4 · p. 27

8.1 Comparison to effect of observed covariates on treatment selection

• A practical Γ benchmark compares omitted-covariate effects on treatment-selection odds, using domain knowledge to assess whether unobserved confounders could be similarly informative.This adapts traditional sensitivity analysis from testing inference robustness to protecting policy learning against harmful confounding.Source: §8.1 · p. 28
• In the WHI benchmark, variables other than highly predictive age induced odds ratios bounded around Γ=0.8 to 1.2, motivating safety checks across that range.The recommended range applies when omitted confounders are believed less informative than age.Source: §8.1 · p. 28

8.2 Calibration plots

• In WHI, Γ=1.14 offered safe regret control for confounding up to Γ′=1.2 with only minimal inefficiency when confounding was smaller.Calibration plots compare policies trained under Γ across possible true-confounding levels Γ′, revealing the safety–performance trade-off.Source: Figure 6 · p. 29, §8.2 · p. 29
• Calibration plots show that higher Γ provides uniform regret control over broader confounding ranges but can be conservative, while lower Γ risks positive regret under stronger confounding.In the WHI example, Γ=1.05 incurred large regret for Γ′ between 1.1 and 1.2.Source: §8.2 · pp. 28–29

9 Conclusion

• The framework optimizes personalized policies against unobserved confounding by minimizing worst-case regret relative to a baseline policy.It generalizes IPW-based estimators and constructs uncertainty sets around nominal IPW weights, calibrated using causal sensitivity-analysis methods.Source: §9 · p. 30
• The framework addresses the risk that standard observational-data methods become corrupted by hidden confounding and cause harm relative to current standards of care.This risk is especially consequential in sensitive domains including medicine, public policy, and civics.Source: §9 · p. 30
• A future extension is a confounding-robust variant based on the doubly robust estimator of policy value.This is identified as a future line of investigation rather than a result established in the paper.Source: §9 · p. 30
• When the uncertainty set is well-specified, the method is guaranteed not to underperform the standard of care and may improve it when supported by the data.The guarantee is finite-sample and checkable, enabling safe implementation under the assumed confounding range.Source: §9 · p. 30
• The analysis establishes uniform convergence bounds showing that the learned policy achieves population-level minimax-optimal regret.This complements the finite-sample safety guarantee with a population-level optimality result.Source: §9 · p. 30
• The optimization structure of weight-normalized policy-value estimators enables efficient policy learning through subgradient descent on the robust risk.This makes the confounding-robust policy computationally tractable.Source: §9 · p. 30
• Synthetic and clinical-data assessments demonstrate that confounding-robust policy improvement can recommend personalized treatments while maintaining strong performance guarantees relative to baseline preferences.The approach also lets analysts assess how different plausible confounding levels affect robust personalized-policy performance.Source: §9 · p. 30

A Proofs for optimization structure

• For discrete outcomes, lexicographic sorting by outcome and weight-range width preserves the optimization argument’s unimodality structure.The tie-breaking order sorts first by Y and then by b(i)−a(i), extending the continuous-outcome characterization to mixtures with atoms.Source: A Proofs for optimization structure · p. 35
• A linear change of variables and Charnes–Cooper transformation reduce the fractional program to a linear program whose dual yields thresholded weights.The dual variables satisfy positive-part expressions around λ, and complementary slackness implies zero weights below the threshold and unit weights above it.Source: A Proofs for optimization structure · pp. 35–36
• The optimal weights are characterized by an order-statistic threshold, yielding a nondecreasing function of outcome-weighted policy advantage that attains the upper bound.Complementary slackness implies threshold behavior, with λ lying between adjacent sorted sample values and optimal weights recovered from the corresponding order-statistic candidate.Source: A Proofs for optimization structure · pp. 34, 36

B.1 Uniform convergence: tail inequalities

• The uniform-convergence analysis uses packing and covering numbers, VC-hull entropy bounds, Orlicz norms, symmetrization, and chaining to control empirical processes.The framework applies to the policy class Π and reparameterized optimal weight classes, with subgaussian tails obtained from suitable Orlicz choices.Source: §B.1 · pp. 37–38
• Composing VC-major classes with nondecreasing functions preserves the relevant major-class property, enabling structured worst-case weights in the uniform-convergence analysis.This stability result is used to show that optimizing over worst-case weights with additional monotone structure is sufficient for Theorem 5.Source: §B.1 · p. 38
• The analysis derives uniform convergence bounds separately for policies, fixed-policy weight functions, and jointly optimized policy–weight compositions.The joint composition remains monotone under the stated assumption, although its VC-major dimension is not explicit from the stability result.Source: §B.1 · pp. 39–41
• The weight-function bounds rely on embedding monotone classes in a symmetric convex hull and applying VC-hull entropy estimates before a standard chaining argument.This establishes the entropy control needed for maximal inequalities over u(y(π(t|x)−π0(t|x))) and its treatment-specific restrictions.Source: §B.1 · pp. 40–41

B.2 Proof of Theorem 6

• Theorem 6 reduces minimax-regret analysis to uniform convergence through a regret-decomposition lemma.The proof uses the decomposition to separate the policy and uncertainty-set components before bounding them.Source: §B.2 · pp. 41–42
• Theorem 6 assembles high-probability bounds for the three decomposed terms by applying tail inequalities, subadditivity, normalization, and a union bound.The resulting bound is obtained with probability at least 1 − p, after collecting constants depending on v.Source: §B.2 · pp. 42, 44
• Theorem 6 controls the decomposed product-class term using symmetrization, contraction, Orlicz norms, and empirical-process inequalities over policies and weight functions.The argument reparameterizes weights by u(π), decomposes products into sums of squares, and applies contraction and tail bounds to the resulting function classes.Source: §B.2 · p. 43

B.3 Proof of Theorem 2

• Theorem 2 analyzes uniform convergence for the true inverse propensity weights W ∗, assumed to lie in the uncertainty set.Its proof begins with an analogous regret decomposition and uses Lemma 2 together with Hoeffding inequalities for sample expectations.Source: §B.3 · pp. 44–45
• For Theorem 2, the proof bounds regret deviations uniformly over policies using chaining and a bounded envelope, then controls the remaining terms with treatment-group concentration inequalities.The analysis conditions on sufficiently representative treatment-group counts and bounds empirical denominators and expectations separately.Source: §B.3 · p. 45
• Theorem 2 combines the component bounds by a union bound and concludes using well-specification, which places each true weight component W ∗ in its corresponding uncertainty set.The final statement folds all v-dependent constants into a single quantity.Source: §B.3 · pp. 45–46

B.4 Proof of Theorem 8

• Theorem 8’s proof uses partial Lagrangian duality and strong duality to reduce the budgeted uncertainty-set problem to optimization over nondecreasing functions on [0, 1].The reduction fixes optimal η* and ψ*, then uses the resulting structural restriction to control the optimization class’s complexity.Source: §B.4 · p. 46
• Because the restricted class has controlled complexity, the same stability arguments as Theorem 6 provide the same regret guarantees and establish uniform convergence for policy-indexed coefficients.For the estimator, r_i depends on the policy π through a restricted-complexity class, and the VC-major property of u(r) suffices to apply Lemma 1.Source: §B.4 · pp. 46, 49
• For every ψ, η ≥ 0, an optimal rescaled weight function u*(ψ, η) is nondecreasing in the coefficient index r.The proof interprets the subproblem as a fractional-knapsack-like allocation, where sorting by r supports the monotone parametrization.Source: §B.4 · p. 48
• The monotone structure remains valid under the transformation back to the original primal formulation, so optimization is equivalently restricted to scalars ψ, η and u ∈ S(ψ) ∩ U.The linear-fractional and linearized formulations are equivalent, and scaling by ψ preserves monotonicity in r.Source: §B.4 · p. 49

B.5 Proof of Theorem 9

• Theorem 9 recovers optimal regret by optimizing over restricted budgeted weight functions and combines uniform convergence with minimax regret bounds.The proof uses dual sensitivity analysis, Lipschitz contraction, and a union bound over treatments and relevant policies.Source: §B.5 · pp. 50, 52
• Theorem 9 reparameterizes monotone uncertainty weights through a Charnes–Cooper transformation while preserving monotonicity and bounding dual variables.The argument controls deviations between sample and population budget constraints uniformly over the restricted weight class.Source: §B.5 · pp. 50–51

C Optimization Algorithm Details

• Across the proofs, strong duality and minimax arguments connect primal uncertainty-set programs with compact dual formulations suitable for analysis.The supplied passages establish these equivalences for budgeted weights, propensity perturbations, and treatment-specific convex uncertainty sets.Source: §B.5 · p. 50, §B.6 · p. 53, §B.7 · p. 54

B.6 Proof of Theorem 10.

• Theorem 10 bounds optimization error caused by replacing oracle propensity-based weight bounds with perturbed estimated-propensity bounds.The proof works within treatment partitions, uses strong LP duality and compact restrictions, and applies the resulting bound under a product uncertainty set.Source: §B.6 · pp. 52, 54
• For Theorem 10, bounded outcomes, strong overlap, strict feasibility, and compact feasible regions ensure bounded primal and dual optimization variables.The dual variables are restricted using quantile and inverse-probability bounds before perturbation analysis.Source: §B.6 · p. 53

B.7 Proof of Theorem 12

• Theorem 12 converts optimization over a convex set into a conic-hull and dual-cone formulation using Charnes–Cooper transformation and strong duality.The resulting characterization is applied separately to each treatment level.Source: §B.7 · p. 54

C.1 Subgradient Approach Refinements

• For nested Γ values, the subgradient method warm-starts optimization at Γ_k+1 using Γ_k and checks whether policies optimized at earlier Γ_i achieve lower objectives.If a prior policy has a lower objective under the current comparison, the procedure reverts to that previous policy.Source: §C.1 · p. 55
• These subgradient refinements empirically stabilize optimization across the nested Γ sequence anticipated for practical use.The procedure leverages the nested property of the corresponding uncertainty sets and supports additional checks across plausible Γ values.Source: §C.1 · p. 55

C.2 Optimal Confounding-Robust Trees

• OCRT globally optimizes interpretable axis-aligned decision-tree policies with constant treatment probabilities at leaves by combining the dual robust-risk formulation with mixed-integer optimization.The formulation jointly determines covariate partitions and treatment probabilities, with a specialized warm start.Source: §C.2 · p. 55
• The OCRT formulation encodes leaf assignments, ancestor split consistency, leaf-size requirements, split activation, and nonempty-leaf conditions through mixed-integer constraints.Its objective uses the dual formulation, while constraints enforce policy assignment products and consistency between tree structure, splits, and assigned observations.Source: §C.2 · pp. 56–57

C.3 Recursive Partitioning: MIP Warm Start

• The recursive-partitioning heuristic initializes a uniform treatment policy and greedily refines limited-depth trees using univariate splits that minimize minimax risk, simultaneously selecting partitions and within-leaf treatment assignments.Candidate thresholds are evaluated through incremental policy changes while preserving assignments from leaves above each node.Source: §C.3 · pp. 57–58
• Specialized B-tree updates and cumulative sums make recursive-partitioning risk updates efficient, with O(log(N)) updates for sorted multiplier lists.The implementation maintains and updates sorted multipliers and reuses precomputed cumulative sums when recomputing optimization solutions.Source: §C.3 · p. 57
• The recursive-partitioning warm start applies only to the unbudgeted uncertainty set UΓ_n because an uncertainty budget would couple risk across tree levels.The resulting partitioning procedure is used to provide a warm start for the OCRT mixed-integer program.Source: §C.3 · p. 57, §C.2 · p. 57
• Unlike impurity-based tree approaches, the recursive heuristic evaluates objective changes from policy-assignment changes while determining the split location and direction.In general, the optimal assignment can also depend on policies assigned to other nodes at the same tree level.Source: §C.3 · p. 58

D WHI Case Study details

• Policy regret for WHI is evaluated under different λ scalarizations.Source: Table 1 · p. 59
• The case study compares policy regret across λ-based scalarization choices.Source: Table 1 · p. 59
• Figure 7 compares densities of odds ratios induced by training propensities with one dropped covariate per line.Source: Figure 7 · p. 60

Calibration for WHI

• The odds-ratio comparison uses odds ratio on the x-axis and density on the y-axis for each subplot.Source: Figure 7 · p. 60
• All Figure 7 subplots use the fixed y-scale y ∈[0, 10].Source: Figure 7 · p. 60
• Most probability mass falls within Γ ∈[0.8, 1.2].Source: Figure 7 · p. 60
• A few covariates instead produce wider distributions of informativity.Source: Figure 7 · p. 60