PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Risk Scores Robust to Unobserved Confounders

Ryan Edmonds, Yingxiao Ye, Sina Aghaei, Andrés Gómez, Çağıl Koçyiğit, Phebe Vayanos

arXiv:2609.27144v1cs.LG

TL;DR

The paper asks how to learn risk scores robust to unobserved confounding in resource allocation. It proposes a robust framework that models ground-truth propensity weights and improves calibration over traditional benchmarks and the state of the art.

  • Problem

    The paper asks how to learn risk scores robust to unobserved confounding in high-stakes resource allocation.

    Source: §1 · pp. 2–3

  • Method

    The paper introduces a framework for robust risk-score learning in the presence of unobserved confounding by modeling ground-truth propensity weights.

    Source: §5 · p. 15

  • Results

    Calibration improves by up to 29.2% over traditional benchmarks and up to 11.1% over the state of the art.

    Source: §5 · p. 16

  • Takeaways & Limitations

    The method outperforms baselines across all metrics, with the best choice of Γ generally between e^γ and e^2γ.

    Source: §5 · p. 16

    Scaling to very large datasets is constrained by quadratic growth in computational requirements.

    Source: §5 · p. 16

Abstract

from arXiv · show

We consider the problem of learning risk scores to prioritize individuals for scarce resources or interventions, from historical observational data affected by unobserved confounding. Decisions about who receives scarce resources are often guided by risk scores based on recorded characteristics, such as responses to a survey. These risk scores are increasingly being learned directly from observational data: historical records of individuals' characteristics, allocation decisions, and outcomes. Standard methods such as inverse propensity weighting (IPW), which corrects for the bias introduced by the historical allocation policy, can be used to learn accurate risk scores if the historical decision process is fully explained by the recorded characteristics. In practice, however, historical decisions often depend on unrecorded information, causing learned risk scores to systematically under-prioritize exactly the individuals whose unrecorded circumstances drove past prioritization. We propose a method for learning risk scores that are robust to this kind of unobserved confounding, building on IPW. Since propensity weights cannot be reliably estimated under unobserved confounding, we instead treat them as belonging to an uncertainty set determined by the observable data and domain-informed estimates of the degree of confounding, combining sensitivity analysis from causal inference with Wasserstein distributionally robust optimization. The resulting robust risk score learning problem admits a sample-based approximation that we reformulate as an exponential cone program compatible with off-the-shelf solvers. We demonstrate the effectiveness of our approach on semi-synthetic data derived from datasets in the UCI Machine Learning Repository. Our method improves calibration by up to 29.2% over traditional benchmarks and up to 11.1% over the state of the art, without compromising other metrics.

1. Introduction

The introduction addresses how unrecorded information in historical allocations can cause risk scores learned from observational data to mis-prioritize people, and proposes learning scores robust to plausible confounding by treating propensity scores as uncertain. It frames scores as inputs to human decisions rather than direct treatment policies, distinguishing the method through Wasserstein-based consistency constraints and propensity-bound uncertainty sets with a causally interpretable Γ.

1. Introduction

  • Inverse propensity weighting requires unconfoundedness, but unrecorded factors shaping historical allocations can confound propensity scores and undermine risk scores learned from those data.In homelessness prevention, caseworkers may use in-person observations, such as social connections or perceived resilience, that are absent from recorded data.Source: §1 · p. 3
  • The framework maximizes worst-case expected log-likelihood over uncertain propensity scores, using marginal sensitivity and Wasserstein constraints to bound confounding and match observable data distributions.The uncertainty set can also incorporate assumptions about historical decision-makers’ use of richer information.Source: §1 · p. 3
  • The sample-based approximation uses observational data, can encode rationality assumptions, and is reformulated as an exponential cone program solvable with off-the-shelf tools.Rationality can reflect the assumption that decision-makers used richer information to make better decisions on average.Source: §1 · pp. 3–4

1.1. Related Work

  • The work extends optimization-based risk-score learning to observational settings by optimizing against a worst-case confounder realization within a causal uncertainty set.This setting observes outcomes only under the treatment individuals received.Source: §1.1 · p. 4
  • Unlike sensitivity analyses targeting causal estimands, the framework optimizes predictive risk scores, accommodates outcome-effect intervals, and encodes decision-maker rationality as [0,∞) without another sensitivity parameter.The interval constrains the difference between true and observed treatment effects; the one-sided choice represents decision-makers using richer information to make better decisions on average.Source: §1.1 · p. 5
  • The method enforces distributional consistency with a Wasserstein bound on the unknown distribution, which can be relaxed when compared distributions have different supports.At the population level, the bound is equivalent to per-covariate consistency.Source: §1.1 · p. 5
  • The consistency bound is independent of the optimized objective, unlike earlier constructions tied to the specific causal estimand being bounded.Source: §1.1 · p. 5
  • The uncertainty set uses propensity score bounds from the MSM rather than direct bounds on observed-data distributions, yielding a causally interpretable robustness parameter Γ.This makes the robustness parameter something practitioners can reason about directly.Source: §1.1 · p. 6
  • Risk scores estimate vulnerability to inform human decisions rather than prescribe treatment, and the method adds consistency constraints to avoid distributions incompatible with observed data.The paper’s example shows that the uncertainty set of Kallus and Zhou includes distributions incompatible with the observed data.Source: §1.1 · p. 6

2. Problem Description

The section formulates learning a linear risk score for untreated adverse-outcome risk under unobserved-confounding uncertainty, and illustrates the role of a covariate-distribution condition and how the bounds can accommodate alternative historical decision logics.

Problem Description

  • The target is β for s(x)=β⊤x, estimating the probability of an adverse untreated outcome, with Y0=−1 denoting adverse outcome.Individuals have observed covariates X, unobserved confounders U, and potential outcomes across treatments including no treatment.Source: §2 · p. 7
  • The causal setup assumes consistency, positivity, and conditional exchangeability given both observed covariates and unobserved confounders.U is defined broadly enough to include missing confounders needed for conditional exchangeability.Source: §2 · p. 7
  • Because U is unobserved, P(T=0|X,U) is not identifiable, so the method learns risk scores robustly over plausible propensities in an adversarial uncertainty set.The reweighted untreated population can be estimated from observed data, but the unknown propensity used for weighting requires robust treatment.Source: §2 · pp. 8–9
  • Proposition 1 states that when Z = P(T = 0 | X,U), conditions (C1) and (C2) are satisfied, ensuring consistency with the unobservable ground truth distribution.Source: §2 · p. 10
  • Plausible propensities obey an odds-ratio bound Γ relative to X-only odds and historical rationality, requiring the confounder-weighted untreated-outcome expectation not to be worse on average.Γ quantifies potential hidden bias; Γ=1 corresponds to no unobserved confounding, while larger values allow greater effects.Source: §2 · p. 10

2. Problem Description

  • The bounds can be adapted to reflect alternative historical decision logics.Source: §2 · p. 10
  • When Condition (C2) fails, reweighting the untreated group may not recover even the overall covariate distribution.The example says this defeats the purpose of reweighting the untreated group to recover the distribution across all treatment groups.Source: §2 · p. 11

3. Sample-Based Approximation

The sample-based approximation handles unknown propensity weights as uncertain and reformulates the resulting problem as an exponential cone program that off-the-shelf solvers can solve.

3. Sample-Based Approximation

  • The sample-based formulation replaces unknown population quantities with quantities based on the observed sample and its empirical distribution.The empirical distribution is induced by the sample observations.Source: §3 · p. 11
  • Unknown true propensity weights are treated as uncertain, while estimated propensities define corresponding estimated weights.The sample-based approximation focuses on observations assigned to no treatment.Source: §3 · p. 11
  • The population-level uncertainty set encodes the stated conditions using the empirical distribution, a confounding bound Γ, and a Wasserstein radius ε.The Wasserstein radius may be nonzero because treated and untreated groups can have different covariate supports, even without confounding.Source: §3 · p. 12
  • The reformulation yields an exponential cone program that off-the-shelf solvers can use to compute robust risk scores.It applies when the inner infimum is feasible and estimated weights are at least 1.Source: §3 · p. 12
  • The reformulation has a number of constraints that grows quadratically in sample size, which can create computational issues beyond the experiment-sized datasets.Delayed constraint generation can improve computation time.Source: §3 · p. 12

4. Experiments

Experiments use UCI classification datasets to create observational settings with systematically introduced unobserved confounding. Across tested settings, the robust method performs best among non-full-information benchmarks for a range of Γ values and improves calibration against both DR and the robust baseline.

4. Experiments

  • The experiments select 15 UCI classification datasets where logistic relations explain the data decently but imperfectly.Selection used cross-validated log loss between 0.35 and log2.Source: §4 · pp. 13–14
  • Historical treatments are generated across γ settings to introduce unobserved confounding while retaining access to ground truth and a confounding sample.The resulting setup is described as a semi-synthetic observational dataset.Source: §4 · pp. 13–14
  • Comparisons include Full Information, Direct Method, IPW, Doubly Robust, and a robust baseline without Wasserstein or historical rationality constraints.The robust baseline retains constraints C1 and C3 and uses an odds-ratio uncertainty set.Source: §4 · p. 14
  • Evaluation measures log-likelihood improvement, expected calibration error, and ranking agreement with test-data-optimal risk scores.The ranking measure is Spearman’s rank correlation.Source: §4 · pp. 14–15
  • As γ increases, methods other than Full Information decline in performance, while Full Information remains unchanged because it accesses counterfactuals.Full Information can ignore confounding in treatment assignment because it has access to counterfactual outcomes.Source: §4 · p. 15
  • For Γ between e^γ and e^2γ, the robust method outperforms non-full-information benchmarks in all metrics.The reported range is where the robust method generally has the best performance.Source: §4 · p. 15
  • Calibration error falls by up to 29.2% relative to DR and 11.1% relative to the robust baseline.The paper attributes the latter improvement to distributional consistency and historical rationality constraints.Source: §4 · p. 15

5. Conclusion

The paper presents a robust framework for learning risk scores under unobserved confounding by optimizing against plausible propensity weights. Experiments report improved calibration and performance across metrics, while the framework's constraint growth presents a scaling challenge.

5. Conclusion

  • The framework treats ground-truth propensity weights as uncertain and maximizes worst-case performance over plausible confounding.The resulting max-min problem has an exact exponential cone reformulation that can be solved with off-the-shelf solvers.Source: §5 · p. 15
  • 29.2% calibration improvement over traditional benchmarks and 11.1% over the state of the art are reported for the robust method.The conclusion also reports that it outperforms other methods as confounding increases and performs well across metrics.Source: §5 · p. 16, Table 1 · p. 16
  • The framework's uncertainty-set choice depends on estimating confounding severity using domain knowledge.The conclusion identifies homelessness services and public health as domains where practitioners may have intuition about missing information and its influence.Source: §5 · p. 16
  • Constraint growth is quadratic in dataset size, which may create computational issues for very large datasets.Additional integrality or sparsity constraints may require further computational strategies.Source: §5 · p. 16
  • The framework allows objectives beyond log-likelihood, pointing to robust learning of quantities beyond risk scores.The choice of B is modular and can accommodate additional integrality or sparsity constraints.Source: §5 · p. 16

EC.1. Additional Definitions

This section defines the 1-Wasserstein distance between distributions through joint distributions with the specified marginals.

EC.1. Additional Definitions

  • The 1-Wasserstein distance between Q1 and Q2 is defined using joint distributions whose marginals are Q1 and Q2.The set of such joint distributions is denoted M(Q1,Q2).Source: EC.1.1. 1-Wasserstein distance… · p. 20

EC.2. Proofs

The proofs establish the stated propositions and lemma, then reformulate the sample-based robust optimization problem using optimal transport and duality.

EC.2.1. Proof of Lemma 1

  • The proof of Lemma 1 derives equalities using parts (iii) and (i) of Assumption 1, followed by standard conditional-expectation properties.Source: EC.2.1. Proof of Lemma 1 · p. 20

EC.2.2. Proof of Proposition 1

EC.2.3. Proof of Proposition 2

  • The proof of Proposition 2 expresses the Wasserstein distance between discrete distributions as an optimal transport problem, with ζij representing transported probability mass.A constraint ensures feasible weights rescale the untreated population to the full-population scale.Source: EC.2.3. Proof of Proposition 2 · pp. 21–22
  • Strong duality converts the inner infimum into a supremum that combines with the outer supremum, yielding an exponential cone problem.Source: EC.2.3. Proof of Proposition 2 · p. 22

EC.3. Experimental Details

The experimental appendix lists compute resources and UCI datasets, and relates the uncertainty-set parameter Γ to confounding severity γ under the treatment-generation model.

EC.3.1. Compute Resources

EC.3.2. Datasets

  • The experiments use datasets from the UCI Machine Learning Repository, including Abalone, Adult, Connect-4, Covertype, and Wine Quality.Source: EC.3.2. Datasets · p. 23

EC.3.3. Approximately relating Γ to γ

  • Under the experimental treatment-generation model, e^-2γ and e^2γ are described as the lowest and highest reasonable values for Γ in the uncertainty set.The stated bounds follow from Jensen's inequality and 0 < π0 < 1.Source: EC.3.3. Approximately relating Γ to γ · p. 23
Loading 2609.27144v1…