Source-linked AI summary
Conformal Policy Selection: When to Deploy AI Decisions for Targeted Intervention
Junyu Cao, Ruijiang Gao, Maytal Saar-Tsechansky, Mingzhang Yin
TL;DR
Responsible AI-policy deployment requires identifying who may benefit, not only whether average outcomes improve. Conformal Policy Selection tests whether each unit’s outcome under the target policy exceeds a chosen threshold, using weighted conformal p-values and selection-and-pruning. The paper establishes finite-sample FDR control and a doubly robust asymptotic guarantee, and illustrates the safety gate in a semi-synthetic study.
Problem
Existing policy-evaluation and safe-learning guarantees are population-level, while deployment also requires identifying individuals who may be harmed or benefit.
Source: §1 · p. 1, §2.1 · p. 3Method
Results
The procedure has finite-sample FDR control, and its doubly robust guarantee preserves asymptotic FDR control when either the outcome score or weights are correctly specified.
Source: §4.3 · p. 7, §5.1.3 · p. 11, §6 · p. 14Takeaways & Limitations
The selected-set FDR bound provides a deployment safety gate, and q sets an interpretable budget for the expected false-discovery proportion.
Source: §5.2.3 · p. 13, §6 · p. 14In the semi-synthetic study, fitted values are treated as true potential outcomes, and unobserved counterfactual outcomes preclude conclusions about actual treatment benefits.
Source: §5.2.1 · p. 13
Abstract
from arXiv · showhide
Artificial-intelligence (AI) policies are increasingly used to make consequential decisions in medicine, finance, education, and public services, yet deploying them responsibly requires knowing not only whether a policy helps on average, but for whom. Existing tools for off-policy evaluation and safe policy learning provide population-level guarantees, but some individuals may experience adverse treatment effects. We recast targeted deployment as a multiple-testing problem: for each candidate unit, test whether the outcome under the AI policy would exceed a user-specified threshold, and control the false discovery rate (FDR) among the selected set. Our method, Conformal Policy Selection, combines weighted conformal $p$-values that account for the distributional shift from behavior to target policy with a two-step selection-and-pruning procedure. We prove finite-sample FDR control and establish a doubly robust asymptotic guarantee, where FDR remains controlled if either the outcome model or the propensity model is correctly specified. Synthetic experiments verify these guarantees across four misspecification regimes, and a semi-synthetic study built on real electronic-health-record covariates illustrates how the same FDR control can act as a deployment safety gate, bounding the harmful treatments that a globally trained policy would otherwise administer to a high-risk subpopulation where it misgeneralizes. The framework offers a principled method for safe AI deployment in high-stakes applications.
1 Introduction
Policy deployment must address not only whether a policy improves outcomes on average, but also which individuals should receive it. Conformal Policy Selection frames that decision as multiple testing and combines policy-weighted conformal p-values with selection and pruning to control false discoveries.
Existing approaches
- Off-policy evaluation and safe policy learning address average improvement, but do not resolve which individuals should receive a policy.CATE-based decision rules also do not directly address the multiple-testing structure or FDR under joint outcome- and propensity-model uncertainty.Source: §1 · pp. 1–2
Motivation
- Average policy gains can conceal identifiable subgroups with worse outcomes, motivating selective rather than universal deployment.The decision problem becomes testing whether each selected unit exceeds a deployment-relevant threshold while controlling incorrect selections.Source: §1 · p. 2
Existing approaches
- Predictive-model rankings lack rigorous subgroup-selection guarantees, and standard calibration can fail under behavior-to-target policy distribution shift.Marginal conformal intervals likewise lack conditional coverage guarantees, and thresholding them naively does not control selection FDR.Source: §1 · p. 2
Our contribution
- The method returns units predicted to exceed a user-specified outcome threshold under a future policy while controlling the expected false-discovery fraction at a chosen level.It uses weighted conformal p-values tailored to the shift from the behavior policy to the target policy, followed by selection and pruning.Source: §1 · p. 2
2 Related Work
Existing safe policy learning generally certifies population-level performance rather than identifying individuals for intervention. Conformal selection and weighted conformal inference provide a basis for FDR-controlled selection, which this work adapts to policy deployment under behavior-to-target shift.
2.1 Safe Policy Learning
- Safe policy-learning approaches include high-confidence policy-value bounds, PAC-style minimum-performance certification, user-specified behavioral constraints, and safety guarantees for sequential settings.Other related work improves policy-value estimation but does not directly enforce safety constraints.Source: §2.1 · pp. 2–3
- Existing safe policy-learning guarantees bound population-level performance but do not specify which individuals should receive the new policy.This work instead controls FDR among selected individuals, bounding the fraction harmed by incorrect assignment with finite-sample guarantees.Source: §2.1 · p. 3
2.2 Conformal Inference and Selection
- Conformal prediction offers marginal coverage, but that alone does not ensure coverage within a selected subgroup.Naively thresholding marginally valid intervals does not control the FDR of the resulting selection.Source: §2.2 · p. 3
- Prior conformal selection combines conformal p-values with Benjamini–Hochberg for FDR-controlled subset selection, while weighted conformal selection extends it to covariate shift.This work adapts the framework to treatment-covariate shift through the policy ratio and establishes a doubly robust asymptotic FDR guarantee.Source: §2.2 · p. 3
3 Preliminaries: Policy Evaluation
The setup distinguishes observational data generated under a behavior policy from test units assigned under a target policy. Policy evaluation estimates the target policy’s average reward, while the deployment motivation is to select individuals who may benefit rather than apply the policy universally.
3.1 Problem Setup
- The setup uses observed covariate-treatment-outcome tuples, with outcomes represented as potential outcomes under the assigned treatment and historical treatment assignment generated by a behavior policy.Test units are assigned according to a target policy, and their corresponding outcomes are potential outcomes under that assigned treatment.Source: §3.1 · p. 4
3.2 Policy Evaluation and Deployment
- Policy evaluation commonly estimates a future policy’s reward using data collected under a different behavior policy.Direct-method, inverse-propensity-weighted, and doubly robust estimators target unbiased or consistent estimates under suitable model assumptions.Source: §3.2 · p. 4
- The policy-evaluation estimand is the average potential outcome under the target policy for covariates drawn from the target population.Because not all potential outcomes under that policy are observed, estimators address missing outcomes using outcome modeling, propensity weighting, or both.Source: §3.2 · p. 4
- Universal deployment may yield average reward gains while harming some individuals or subpopulations, so selective service is motivated by identifying those likely to benefit.The goal is to serve only individuals for whom the new policy is likely to improve outcomes.Source: §3.2 · p. 4
4 Policy Selection with FDR Control
The section formulates targeted deployment as selecting units whose outcomes under a future policy exceed specified thresholds while controlling the expected fraction of false selections. It develops weighted conformal selection and gives finite-sample and asymptotic FDR guarantees under different nuisance-model conditions.
4 Policy Selection with FDR Control
- The target is to select as many units as possible whose future-policy outcomes exceed their thresholds while controlling the expected fraction of false discoveries.A false discovery is a selected unit whose outcome falls at or below its threshold.Source: §4 · pp. 4–5
4.1 Weighted Conformal Selection
- Weighted conformal selection tests whether each unit’s outcome under the target policy exceeds its threshold, accounting for differences from the behavior policy through distributional weights.When covariate marginals match, the weight reduces to the target-to-behavior policy ratio.Source: §4.1 · p. 5, §4.1.1 · p. 6
4.1.1 Conformal p-values
- Conformal p-values use a monotone nonconformity score to measure how well an outcome conforms to a prediction, with weighted calibration addressing policy-induced distribution shift.For a point prediction, one score is the outcome minus its predicted value.Source: §4.1.1 · pp. 5–6
4.2 Conformal Policy Selection
- Because outcomes under the future policy are unobserved and its treatment assignment differs from the behavior policy, the procedure constructs a selection set from weighted conformal p-values.The method uses calibration data generated under the behavior policy.Source: §4.2 · p. 6, §4.2.1 · p. 6
4.2.1 Construct the Selection Set
- The selection procedure applies BH-based thresholds to auxiliary p-values, then uses a uniform random pruning rule to address inflated selection from correlated threshold sizes.The correlation can make K_j accidentally too large, so the p-value threshold alone may not guarantee FDR control.Source: §4.2.1 · p. 6
4.3 Non-asymptotic FDR Guarantee When Weight Is Known
- FDR is at most q for every n and m when the density ratio is known and the stated conditions hold.The guarantee is finite-sample and requires no additional sample-size or distributional-regularity assumptions.Source: §4.3 · p. 7
4.4 Asymptotic Doubly Robust Guarantee When the Weight Model Is Misspecified
- With a correctly specified outcome model and bounded working weights, the procedure retains asymptotic FDR control even when the weight model is misspecified.The guarantee is asymptotic rather than finite-sample; bounded weights can be enforced by clipping.Source: §4.4 · p. 8
4.5 Asymptotic Guarantee: When Weights and Estimators Are Asymptotically Correct
- When both nuisance functions are estimated from independent training data and are uniformly consistent, the plug-in procedure has asymptotic FDR at most q.This result requires consistency of both the propensity and regression estimators.Source: §4.5 · p. 9
5 Numerical Experiments
Across synthetic and semi-synthetic experiments, conformal selection controls FDR under the stated model conditions and can gate deployment in a high-risk EHR subgroup. The experiments examine correct and misspecified models, with the EHR findings limited to a semi-synthetic ground truth.
5 Numerical Experiments
- The synthetic comparison evaluated Weighted BH and the pruned Weighted Conformal Selection procedure across nominal levels q, reporting empirical FDR and power over replications.The finite-sample guarantee applies to Weighted Conformal Selection; Weighted BH is reported as a practical baseline whose FDR is assessed empirically.Source: §5 · pp. 9–10
5.1.1 Data Generating Process
- The synthetic setup generated covariates and outcomes under specified behavior and AI policies, then drew independent calibration and test samples of n = 500 and m = 100.The AI policy was trained on an independent sample of 200 observations, and the selection threshold was c = 0.Source: §5.1.1 · p. 10
5.1.2 FDR Control with Correctly Specified Models
- With correctly specified scores and weights, empirical FDR stayed below the nominal level across q, while power rose to approximately 0.80 at q = 0.95.This experiment used 400 replications with n = 500 calibration and m = 100 test observations.Source: §5.1.2 · p. 10, Figure 1 · p. 11
5.1.3 Doubly Robust FDR Control
- FDR stayed controlled when either scores or weights were correctly specified, but with both misspecified it reached about 0.30 at q = 0.05.The misspecified outcome model reduced power without breaking validity when weights were correct; control failed when weights were also wrong.Source: §5.1.3 · p. 11, Figure 2 · p. 12
5.2.1 Dataset and setting
- The EHR analysis used 5,993 patient-years from patients aged at least 60 with BMI at least 30; 8.3% were treated.The dataset came from a longitudinal Israeli health-insurer cohort, and the outcome measured avoidance of diabetes onset within three years.Source: §5.2.1 · p. 13
- Because antidiabetics were prescribed reactively and counterfactual outcomes are unobserved, the EHR results do not establish their actual benefits.The paper treats fitted values as potential outcomes for the semi-synthetic experiment, so conclusions concern that setup.Source: §5.2.1 · p. 13
5.2.2 Conformal selection as a deployment gate
- The EHR gate deploys the AI policy only to selected units, while other units retain the status quo; its FDR is evaluated using semi-synthetic counterfactual outcomes.A causal forest supplied conditional treatment effects and a random forest supplied the control surface, allowing exact evaluation against the constructed ground truth.Source: §5.2.2 · p. 13
5.2.3 Results and implication
- At q ≤0.85, no harmful intended treatments were deployed, while the gate certified 93% of the EHR batch at q = 0.85.Across replications, empirical FDR stayed below the nominal level; it reached 0.046 when selection first began at q = 0.75.Source: §5.2.3 · p. 13, Figure 3 · p. 14
6 Discussion
The discussion frames targeted deployment as selecting units by their realized outcomes under the deployed policy, with finite-sample and doubly robust asymptotic FDR guarantees. The accompanying proofs develop weighted conformal validity and the selection-bound argument.
6 Discussion
- The method controls finite-sample FDR among selected units and has an asymptotic guarantee when either the outcome model or weight model is correctly specified.The discussion contrasts acting only on selected units with universal deployment.Source: §6 · p. 14
- The selection target is whether a unit’s outcome under policy π clears threshold c, not the unit’s individual causal effect.The causal effect itself can be targeted by redefining the nonconformity score.Source: §6 · p. 14
A.2 Lemmas
- Conditional on the unordered calibration-and-test multiset, the test tuple is weighted by the shift weight, yielding weighted exchangeability.The oracle p-value is uniform, while the observable oracle p-value is super-uniform.Source: §A.2 · pp. 18–19
- The finite-sample argument reduces FDR to a per-unit bound, then uses oracle substitution and conditional independence in the proof of Theorem 1.The proof compares observable and oracle p-values using score monotonicity and the BH step-up rule.Source: §A.2 · p. 19, §A.3 · pp. 20–21
A.4 Proof of Theorem 2
- For the asymptotic argument, residual-score independence and a bounded weight ratio support convergence to a procedure whose FDR remains at most q.The proof sums the per-unit bound and applies bounded convergence.Source: §A.4 · pp. 22–23
A.5 Proof of Theorem 3
Theorem 3 establishes asymptotic FDR control for Algorithm 1 with estimated nuisance functions. Its proof shows that the estimated selection set converges to the ideal set, then applies the finite-sample result under the limiting procedure’s conditions.
A.5 Proof of Theorem 3
- The estimated scores, p-values, rejection counts, and selection set converge almost surely to their ideal counterparts, apart from exact ties.The proof notes that randomized tie-breaking resolves exact ties if needed.Source: §A.5 · p. 23
- The limiting procedure meets Theorem 1’s conditions when the score is monotone in outcome, weights satisfy the shift condition, and sample splitting preserves independence.The proof uses the residual score and inverse propensity weights.Source: §A.5 · p. 23
- As the training sample grows, the false discovery rate satisfies FDR ≲ q.The proof obtains this from convergence of the estimated selection set to its ideal counterpart.Source: §A.5 · p. 23
B.1 Data
The experiments use synthetic data and real-world datasets, including LaLonde’s job-training benchmark. The LaLonde setup fits outcome and propensity models on stratified splits, evaluates a treatment policy, and selects units against a $4,000 earnings threshold.
B.1 Data
- The study includes synthetic Gaussian-covariate experiments, an EHR dataset of 89,999 patients, and LaLonde’s 614-person job-training benchmark.The EHR analysis restricts to fully observed follow-up windows and a high-risk subgroup, yielding 5,993 patient-years.Source: §B.1 · p. 24
B.2 Models and procedure
- For each dataset, the procedure fits an outcome model, a deployment policy, and a propensity model, then compares weighted conformal selection with weighted BH.The experiments sweep nominal FDR levels from 0.05 to 0.95 over independent splits.Source: §B.2 · p. 24
C Real-World Data: LaLonde Job Training
- LaLonde’s benchmark contains 614 individuals, with 185 treated and 429 controls; its outcome is post-program earnings in 1978.Covariates include demographics, degree status, and prior earnings.Source: C Real-World Data: LaLonde Job Training · p. 24
C.0.1 Setup
- The LaLonde setup uses 100 stratified train-calibration-test splits, a random-forest outcome model, and a logistic propensity model clipped to [0.05, 0.95].The AI policy treats when estimated CATE is positive; the selection threshold is $4,000 in annual earnings.Source: §C.0.1 · p. 24, Figure 4 · p. 25
C.0.2 Results
On LaLonde, empirical FDR stayed at or below q through q = 0.65 before approaching the baseline negative rate, while power rose as q increased. At small q, selected individuals also had higher mean earnings than non-selected individuals.
C.0.2 Results
- At q = 0.15, FDR was ≈0.025 and power was 0.012; at q = 0.65, FDR was ≈0.125 and power was 0.34.FDR stayed at or below nominal q through q = 0.65 and then plateaued near 23%; power approached 1 as q → 0.85.Source: §C.0.2 · p. 25
- At q = 0.15, selected individuals averaged $20,780 in earnings versus $8,244 for non-selected individuals, a difference of $12,535.The fraction clearing the threshold was 93.1% among selected versus 76.7% among non-selected.Source: §C.0.2 · p. 25
- When q is near 0.9, the procedure selects almost everyone, and selected-group mean earnings decline toward the overall average.At small q, selected-group mean earnings are substantially larger than non-selected-group mean earnings.Source: §C.0.2 · p. 25
D Experimental Details of the Diabetes Experiment
The diabetes experiment uses a high-risk EHR subgroup to examine estimated treatment effects, construct semi-synthetic potential outcomes, and apply the AI policy with weighted conformal selection.
D Experimental Details of the Diabetes Experiment
- In the diabetes experiment, analysis focuses on the clinically high-risk EHR subgroup (age ≥60, BMI ≥30), comprising 5,993 patient-years, 305 features, 8.3% treated, and a 36.5% three-year onset rate.Source: D Experimental Details of the Diabetes… · p. 25
D.1 Estimating Treatment Effect
- Among patients aged ≥60 with BMI ≥30, all four heterogeneous-effect estimators find negative treatment effects for essentially all patients; treated patients fare worse even in balanced, high-propensity propensity deciles.Avoidance means no type II diabetes onset within three years.Source: Table 2 · p. 26, §D.1 · p. 26, Table 3 · p. 27
D.2 Semi-synthetic ground truth
- The semi-synthetic ground truth fits a causal forest for τ(x) and a random forest for the untreated response, then constructs treated outcomes as µ1(x) = clip(µ0(x) + τ(x), 0.001, 0.999).Potential outcomes are generated with Gaussian noise of σ = 0.10, allowing outcomes under any policy to be computed.Source: §D.2 · pp. 26–27
D.3 AI policy and selection procedure
- The AI policy treats patients above the 90th training percentile of predicted onset risk, and weighted Benjamini–Hochberg selection is applied to weighted conformal p-values across q ∈ [0.05, 0.95].The study uses 20 stratified train/calibration/test splits and propensity weights clipped to [0.05, 0.95].Source: §D.3 · p. 27
D.4 Safety metric
The safety metric counts selected patients whom the AI policy would treat as harmful deployments; the figure presents FDR, power, and deployment counts across q.
D.4 Safety metric
- A harmful deployment is a selected unit whom the AI policy would treat; all other units retain the status quo.The figure plots empirical FDR against its nominal level, selection power, and counts of beneficial treated units and harmful deployments as q varies.Source: §D.4 · p. 27, Figure 6 · p. 28
- Harmful deployments are zero for all q ≤ 0.85 and reach only about two units at q = 0.95, despite roughly 44 policy treatments per batch.Because treatment is harmful throughout this subgroup, each such deployment degrades the realized outcome; FDR control bounds their count by q|S|.Source: §D.4 · p. 27
E Diabetes Experiment with Heterogeneous Treatment Effects
A second semi-synthetic study introduces heterogeneous treatment effects on the real EHR covariates, with effects beneficial for some patients and harmful for others.
E Diabetes Experiment with Heterogeneous Treatment Effects
- This study is designed to test whether the gate discriminates when treatment effects vary, unlike the all-harm subgroup where the ideal selected set is empty.The second study injects effects that benefit some patients and harm others on the same real EHR covariates.Source: E Diabetes Experiment with… · p. 28
- With heterogeneous effects, realized FDR never exceeds target q, and selected treatments include far more beneficial units than harmful deployments.The AI policy treats about 49% of the cohort; the study runs 20 repetitions.Source: E Diabetes Experiment with… · p. 28