Winner's Curse Drives False Promises in Data-Driven Decisions: A Case Study in Refugee Matching

1. Introduction

• Model-free inverse probability weighting avoids estimating counterfactual outcomes but can have very high variance with large action spaces or high-dimensional covariates, and is invalid when treatment assignment is nonrandom.In the random-assignment setting, it evaluates decisions only when they align with historical decisions.Source: §1 · p. 2
• When optimization and evaluation use the same estimated counterfactual model, optimization exploits estimation errors and evaluation leaves them uncorrected, producing optimistic estimates even when true gains are absent.The paper defines valid evaluation as providing a high-probability confidence interval containing the true policy improvement.Source: §1 · pp. 2–3
• Among 55 recent Management Science papers reporting data-driven policy improvement, all but two used model-based evaluation, and many gave no justification.The justifications that were offered typically concerned model accuracy, stability, and calibration; random treatment assignment; a well-specified model family; or sample splitting.Source: §1 · p. 3

1.1. Our Contributions

• Theoretical examples show that misspecification or finite-sample regularization bias can produce arbitrarily large, unsubstantiated policy improvements, even when the cited justifications hold.With regularization, bias that is small and stable on the historical data distribution can become significant on the shifted distribution induced by a different policy.Source: §1.1 · p. 4
• Regularization allows the winner’s curse under broader conditions, with systematic positive bias possible even when the estimated model is highly accurate and stable.Prior work on misspecification shows bias but not that it is systematically positive or can arise with a highly accurate, stable model.Source: §1.1 · p. 4
• In a simulated refugee-matching setting where no policy can outperform random assignment, model-based evaluation reports spurious employment gains around 60%, and bootstrapping still yields substantial, stable improvements under a null ground truth.The reported gains are on par with estimates in the literature.Source: §1.1 · p. 5

1.2. Related Literature

• Model-free policy evaluation provides asymptotically unbiased estimates but can have high variance in sparse, large action spaces; doubly robust estimation improves efficiency, yet finite-sample variance remains problematic.Source: §1.2 · p. 5
• Against these limitations, many studies use model-based evaluation, which is subject to the winner’s curse; the authors rebut common justifications for dismissing that risk.Source: §1.2 · p. 5

2. A Survey of the Literature

• A two-stage screen using language-model classification followed by manual review reduced 875 search results to 55 estimate-then-optimize papers.The automated screen identified 111 potentially fitting papers before full-text review.Source: §2 · p. 6
• The survey of Estimate-Then-Optimize papers in Management Science yielded 55 papers; 53 (96%) relied on model-based evaluation of their proposed policy’s performance.Source: §2 · p. 6, Table 1 · p. 7
• 35 of 53 papers (66%) evaluated optimized policies with the same estimated model used to generate them, letting optimization exploit residual errors.The remaining papers generally re-estimated parameters in the same model structure, which the authors say does not eliminate bias.Source: §2 · pp. 7–8, Table 2 · p. 7
• Accuracy, stability, calibration, and sample splitting are insufficient to rule out spurious gains from the winner’s curse.The survey also reports that confidence intervals were uncommon and did not account for the optimization process.Source: §2 · p. 8
• The winner’s curse is not confined to black-box machine learning: 35 papers (66%) used simple parametric models, 13 (25%) structural models, and five (9%) black-box models.The survey describes the issue as fundamental to estimate-then-optimize rather than specific to black-box methods.Source: §2 · p. 8

3. Misspecification Causes the Winner’s Curse

• Misspecification drives the winner’s curse in the theoretical construction even when accuracy, stability, calibration, random assignment, and sample splitting hold.The paper also says the justifications, including a well-specified model family, are insufficient to rule out the curse in theory.Source: §3 · p. 8, §3.3 · p. 11

3.1. Theoretical Analysis

• The model-selected treatment has predicted performance comparable to the true optimum, yet its realized outcome is zero, so the optimistic bias can be arbitrarily large.This occurs despite arbitrarily low mean-squared error on the historical data distribution.Source: §3.1 · p. 10

3.2. Contextualizing Our Result

• The construction produces winner’s-curse bias even with an accurate, stable, calibrated model, random treatment assignment, and sample splitting.In the infinite-data setup, independently trained splits yield identical models, so sample splitting does not change the estimated improvement.Source: §3.2 · pp. 10–11

3.3. Connection to Practice

• Although the example’s abrupt outcome drop relies on a one-dimensional, no-covariate setup, richer feature spaces let optimizers exploit locally upward-biased predictions.Even small average bias can coexist with exploitable overprediction on particular covariate-treatment pairs.Source: §3.3 · pp. 11–12

4. Regularization Causes the Winner’s Curse

• The winner’s curse can persist in well-specified models because finite-sample regularization introduces bias, which can be stable across data subsamples.Although the curse disappears asymptotically under the stated conditions, finite-sample regularization used to avoid overfitting can produce bias that is repeated across subsamples.Source: §4 · p. 12

4.1. Theoretical Analysis

• In a well-specified linear model, ridge regularization alone suffices for the winner’s curse when rare treatment assignments make treatment-interaction effects hard to identify.When treatment is rare, the covariate and treatment-interaction features are strongly correlated; regularization can yield a stable estimate with good historical-distribution accuracy while misrepresenting their separate effects, producing optimism under a policy that targets using them.Source: §4.1 · pp. 12–14

4.2. Contextualizing Our Result

• The example permits arbitrarily large spurious policy-improvement estimates even when the paper’s four stated justifications hold, including sample splitting.Those justifications include accurate and stable models, random treatment assignment, and a well-specified model family.Source: §4.2 · p. 15

4.3. Connection to Practice

• Regularization creates stable but biased counterfactual estimates in low-probability regions, where low historical-data MSE lets the optimizer exploit prediction errors and generate the winner’s curse.The example’s rare region is t = 0.Source: §4.3 · p. 15
• In practice, such low-probability regions can involve rare covariates as well as rare treatments, and the example is limited to one rare treatment in a low-dimensional setting.The passage notes that practical settings usually have many more features, making low historical variation along some features more likely and regularization more necessary.Source: §4.3 · pp. 15–16
• An optimizer can exploit biases caused by regularization to achieve spurious improvements.Source: §4.3 · p. 16
• Random forests and gradient-boosted machines may exhibit the winner’s curse, a possibility illustrated by the simulation study in Section 5.Random forests average trees, regularizing estimates toward zero when uncertain; gradient-boosted machines use shallow base trees and inherit their biases.Source: §4.3 · p. 16

5. Case Study on Refugee Matching

• Despite accurate, calibrated, stable models, random historical assignments, and sometimes well-specified models, model-based methods falsely report improved employment because of the winner’s curse.The simulation is designed so no assignment policy can improve expected employment.Source: §5 · p. 16

5.1. Background

• The task is to assign refugees to feasible locations to improve short-term employment, using 90-day employment as a proxy and constraints that create assignment variation.Source: §5.1 · pp. 16–17

5.2. Optimizing Refugee Matches

• Because alternative-location outcomes are unknown, estimated outcome models feed an integer program that assigns refugees to maximize predicted employment subject to location capacity.Source: §5.2 · pp. 17–18

5.3. Evaluating Refugee Matches

• Reusing the optimization model for evaluation invites winner’s curse; IPW is unbiased under unconfoundedness and known random propensities but can have high variance with 43 locations.Source: §5.3 · pp. 18–19

5.4. Synthetic Environment

• The US-calibrated synthetic environment uses random historical assignments and outcomes that depend on refugee and location features but not their interaction, so no policy can improve expected employment.Source: §5.4 · p. 19
• The synthetic causal model makes expected employment equal across feasible matchings when locations are at capacity, so no matching can improve the expected rate.It allows assignments to rearrange employment probabilities across refugees, but excludes covariate-refugee interactions that could enable better matching.Source: §5.4 · p. 20

5.5. Simulation Details

• The refugee-matching method is tested with LASSO-constrained logit, Honest Random Forests, and gradient-boosted classification trees, training on 33,000 refugees, evaluating on 1,000, and repeating 250 times.Source: §5.5 · pp. 20–21

5.6. Simulation Results

• In simulations, all three prediction-model classes exhibit large winner’s-curse bias, while IPW estimation remains unbiased regardless of which prediction model optimized the policy.Source: §5.6 · pp. 21–22
• Bootstrapped LASSO evaluations perpetuate winner’s-curse bias, while stable LASSO bootstraps can show more bias than the original model.The authors attribute this to misspecification causing bootstrapped LASSO models to reproduce the original model’s predictions.Source: §5.6 · pp. 22–23, Figure 3 · p. 23
• Bootstrapping reduces but does not eliminate GBM bias; Honest Random Forests eliminate most bias in this simulation, but alternate setups show sizable inaccuracies.Those alternate inaccuracies indicate that bootstrapping cannot generate trustworthy confidence intervals.Source: §5.6 · pp. 22–23
• Bootstrapping suffers from correlation between the training dataset and its bootstraps, and although this bias is expected to decrease as training-set power increases, its size relationship to dataset size is unclear.This uncertainty makes bootstrapping difficult to use for evaluation.Source: §5.6 · p. 23
• Without improved policy evaluation methods, confidently assessing complex data-driven assignment policies requires large testing datasets to counteract IPW estimation variance, which is often economically infeasible.Source: §5.6 · p. 23

6. Discussion and Broader Implications

• Model-based methods produced large, stable, entirely spurious estimated gains in the refugee-matching case study.The discussion argues that the winner’s curse is a systematic failure mode, not merely a theoretical curiosity or one confined to obviously misspecified models.Source: §6 · p. 23
• The authors caution that these simulations do not establish Honest Random Forest as the best model, since other setups retain large biases.This is a scope caveat about the simulation evidence, not a claim that the model avoids bias in general.Source: §6 · p. 23
• The winner’s curse is most acute in medium-data settings: models appear accurate and calibrated, but data are insufficient for low-variance model-free evaluation.Conventional diagnostics can create false confidence even as policy optimization searches for and exploits residual model errors, especially in rare regions.Source: §6 · p. 24
• In the refugee-matching case study, roughly 30,000 observations supported plausible predictive models, yet assignment-space sparsity made model-free validation too noisy.As a result, unbiased checks could not robustly validate the optimizer’s purported gains.Source: §6 · p. 24
• The authors argue against using model-based imputation for policy evaluation, even when estimated models appear strong by conventional metrics.The discussion describes a bias–variance trade-off: model-free methods can have high variance, while model-based estimates are systematically optimistic.Source: §6 · p. 24

EC.1. Proofs for Section 4

• The appendix develops general results on ridge-regression accuracy and stability.These results provide the general theory used in the proof of Proposition 3.Source: EC.1. Proofs for Section 4 · p. 27
• Proposition 3’s proof relies on the general theory establishing accuracy and stability.Source: EC.1. Proofs for Section 4 · p. 27
• The proof that the winner’s curse occurs is specific to the problem construction established in Section 4.Source: EC.1. Proofs for Section 4 · p. 27

EC.1.1. General Problem Formulation

• The decision objective is linear in features, and each parameter vector induces a policy that selects the treatment maximizing that objective.Source: EC.1.1. General Problem Formulation · p. 27
• Historical covariates and treatments follow a product distribution formed from their respective covariate and treatment distributions.Source: EC.1.1. General Problem Formulation · p. 27
• The feature map is assumed measurable, bounded in expected maximum norm, and subgaussian; β* denotes the true objective parameters.Source: EC.1.1. General Problem Formulation · p. 27
• The formulation evaluates parameter estimates by global prediction accuracy and optimism bias.Excess MSE is defined relative to the true model, while optimism bias compares estimated-model policy improvement with improvement under β*.Source: EC.1.1. General Problem Formulation · p. 27
• The parameter model is estimated from a historical dataset with i.i.d. Gaussian noise, using ridge regression as a proxy for stable regularized methods.Source: EC.1.1. General Problem Formulation · p. 27

EC.1.2. Results for the General Case

• If two evaluation parameters are within ε in Euclidean norm, their estimated policy improvements differ by at most εϕmax.Source: EC.1.2. Results for the General Case · p. 30
• The accuracy bound separates a regularization-dependent bias term from a sample-size-dependent variance term.The bias term shrinks as λ approaches zero, while the variance term shrinks as n grows.Source: EC.1.2. Results for the General Case · p. 30
• With high probability, the estimated ridge parameter concentrates around a deterministic quantity, implying pairwise closeness of estimates.The stated bound requires λ/∆ to be sufficiently large when both λ and ∆ scale as 1/√n.Source: EC.1.2. Results for the General Case · p. 31
• The optimism bias under the estimated parameter concentrates around the bias under its deterministic approximation.Source: EC.1.2. Results for the General Case · p. 32

EC.1.3. Proof of Proposition 3

• The proof applies its high-probability events jointly to four specified ridge-parameter estimates.Source: EC.1.3. Proof of Proposition 3 · p. 33
• A triangle-inequality step bounds the distance between two estimated parameter vectors using the preceding bounds.Source: EC.1.3. Proof of Proposition 3 · p. 33
• In the construction, optimism bias equals estimated policy improvement because the true-parameter policy improvement is zero.Source: EC.1.3. Proof of Proposition 3 · p. 34

• Parameters, sample sizes, and marginal distributions were calibrated wherever possible using the US-site data reported in Bansak et al. (2018).The section provides implementation details for the simulation environment.Source: EC.2. Details of Refugee Matching Case… · p. 34

EC.2.1. Additional Details on Bansak et al. (2018) and Ahani et al. (2021)

• Bansak et al. (2018) predicted employment for refugee–location pairs using covariates, fitting separate location models and selecting gradient-boosted trees by out-of-sample classification accuracy.Predictions were aggregated into family-level scores and optimized subject to capacity and practical constraints.Source: EC.2.1. Additional Details on Bansak… · pp. 34–35
• Backtests trained on earlier arrivals and tested on later arrivals with no placement restrictions reported 40–75% improvements in predicted employment relative to existing procedures.Source: EC.2.1. Additional Details on Bansak… · p. 35
• Ahani et al. (2021) developed Annie MOORE for flexible “free cases,” using predicted employment probabilities and a mixed-integer program to maximize allocations under operational constraints.LASSO was selected as the primary prediction model for its combination of discrimination and interpretability.Source: EC.2.1. Additional Details on Bansak… · p. 35
• 20–40% improvements in predicted employment were suggested by Bansak et al. (2018) backtests using the same LASSO model to impute counterfactual outcomes.Bansak et al. trained machine-learning models on U.S. and Swiss data.Source: EC.2.1. Additional Details on Bansak… · pp. 34–35
• Bootstrap estimates of Bansak et al. (2018)’s gains were tight and positive, which the authors interpreted as evidence of robustness.They repeatedly resampled training data, re-estimated LASSO, and evaluated the same optimized allocation.Source: EC.2.1. Additional Details on Bansak… · p. 35

EC.2.2. Covariate Marginals and Location Assignment Probabilities

• Covariate marginals followed Bansak et al. (2018), with independent assignment probabilities reported for the covariates and location probabilities sorted across refugee samples.For binary covariates, only one option is reported.Source: EC.2.2. Covariate Marginals and… · p. 36, Table EC.1 · p. 36, Figure EC.1 · p. 36

EC.2.3. Employment Rates by Location

• Employment-rate distributions across locations were tuned to resemble the empirical distribution reported in Bansak et al. (2018).Source: EC.2.3. Employment Rates by Location · p. 37, Figure EC.2 · p. 37

EC.2.4. Prediction Model Calibration and ROC Curves

• Calibration curves indicate that each prediction model is well-calibrated and fairly accurate; calibration and ROC curves are reported for the test dataset.Source: EC.2.4. Prediction Model Calibration… · p. 37, Figure EC.3 · p. 37, Figure EC.4 · p. 38