Source-linked AI summary
A Unified Perspective on Conformal Prediction and Wasserstein Distributionally Robust Optimization for Uncertainty Quantification
Kehan Long, Yiqi Zhao, Pol Mestres, Lars Lindemann, Nikolay Atanasov, Jorge Cortés
TL;DR
Finite-data uncertainty quantification must support reliable decisions despite limited samples and distribution shift. This paper unifies conformal prediction and Wasserstein DRO as data-dependent quantile estimators, showing that they correct empirical quantiles through different coordinates and exhibit distinct certification and tail-behavior trade-offs.
Problem
Finite calibration data do not fully characterize the score-distribution tail, making reliable calibration-conditional coverage difficult under distribution shift.
Method
The paper formulates CP and Wasserstein DRO as data-dependent quantile estimators, comparing level-space and value-space corrections with calibration-conditional guarantees.
Results
CP and DRO attain the two-level guarantee across ImageNet, MMLU, and nuScenes, with CP producing the smallest prediction sets among methods that meet it.
Takeaways & Limitations
CP is preferable without distribution shift for smaller sets, while DRO is useful for ambiguity-set coverage and can suit continuous scores better than bounded scores.
Takeaways & Limitations
Certified DRO radii depend on unknown distributional properties, may saturate bounded scores, and shift-aware estimators require an input shift budget.
Abstract
from arXiv · showhide
Uncertainty quantification from finite data is central to machine learning, optimization, and automation systems, where decisions must remain reliable under limited samples and test-time distribution shift. Conformal prediction (CP) and distributionally robust optimization (DRO) offer two complementary approaches: CP constructs data-dependent prediction sets with distribution-free finite-sample validity under exchangeability, while DRO optimizes worst-case performance over an ambiguity set around an empirical distribution. We develop a unified probabilistic perspective on CP and DRO by viewing both as ways to turn finite calibration data into a data-dependent quantile estimator that a test score falls below with high probability. From this perspective, CP and DRO correct the empirical quantile along two coordinates of the same family of estimators: CP inflates the quantile level, whereas DRO shifts the quantile value through an ambiguity radius. Both methods provide the same calibration-conditional guarantee for the true distribution, requiring the target coverage to hold with high probability over the calibration sample. Their constructions differ, however: CP uses a closed-form, distribution-free level correction, while DRO uses a value-space correction whose certified radius depends on properties of the unknown distribution and additionally guarantees coverage uniformly over the ambiguity set. This distinction emerges in the tails of the score distribution. Because CP relies on sparse upper-tail order statistics of the calibration samples, its level inflation barely moves the estimator when those samples are dense near the target quantile but overshoots when they are sparse, whereas a well-chosen DRO radius corrects in value space and may avoid this overshoot.
A Unified Perspective
The section foregrounds uncertainty quantification, distribution shift, conformal prediction, and distributionally robust optimization.
- Uncertainty quantification is a central topic.
- Distribution shift is identified as a key concern.
- Conformal prediction and distributionally robust optimization are the named methodological approaches.
1. Introduction
The introduction frames finite-sample uncertainty quantification as a challenge for data-driven systems and presents CP and DRO as complementary approaches. It unifies them as quantile-estimation methods while comparing their corrections, guarantees, and behavior under distribution shift.
- Finite-sample uncertainty quantification is challenging in machine learning, robotics, and decision-making.
- Calibration-conditional validity requires target coverage to hold for the realized calibration set with high probability over calibration samples.A two-level guarantee combines test-score coverage with confidence over the random calibration set.
- The paper develops a unified probabilistic view of CP and Wasserstein DRO as finite-sample data-dependent quantile-estimation methods.
- CP inflates the empirical quantile level, whereas DRO adds a value-space ambiguity radius.Both methods provide calibration-conditional guarantees for the true distribution, while DRO also certifies coverage uniformly over the realized ambiguity set.
- The study compares coverage, conservativeness, two-level guarantees, and robustness under distribution shift across three representative tasks.The tasks are image classification, multiple-choice question answering, and autonomous-driving trajectory prediction.
- The paper aims to provide theoretical basis and practical guidance for selecting uncertainty-quantification methods from finite data.
2. Related Work
Related work covers conformal prediction, distributionally robust optimization, their applications, and prior efforts to connect them. The paper distinguishes its unified probabilistic framing and systematic comparison from earlier shift-robust combinations.
- Conformal prediction: Conformal prediction provides finite-sample coverage guarantees under exchangeability without parametric assumptions.Related work discusses marginal, training-conditional, and restricted conditional validity, along with extensions beyond exchangeability.
- Conformal prediction: CP has been applied to model calibration, clinical imaging, control, autonomous systems, robotics, planning, and human-robot interaction.
- Distributionally robust optimization: DRO optimizes against worst-case distributions in an ambiguity set around an empirical distribution, supporting robustness to estimation error and distribution shift.Wasserstein ambiguity sets are emphasized for combining computational tractability with finite-sample guarantees.
- Distributionally robust optimization: DRO has been used in robust learning, control, autonomous systems, and robotics through ambiguity sets and worst-case risk formulations.
- Connections between CP and DRO: Prior work connects CP with distributionally robust coverage under f-divergence and Lévy–Prokhorov ambiguity sets.
- Connections between CP and DRO: The paper distinguishes its goal of unified probabilistic framing, finite-sample quantile analysis, and systematic trade-off comparison from prior work.
3. Problem Formulation and Preliminaries
The formulation treats uncertainty quantification as estimating a threshold that a future test score falls below, using calibration scores from an unknown distribution. It defines calibration-conditional coverage and introduces CP and DRO as estimator constructions, including distribution-shift settings.
- The framework models calibration scores and a test score through probability distributions on a measurable subset of the real line.
- The empirical p-quantile is the ⌈np⌉-th smallest of n values, while an unknown test distribution would ideally determine the target quantile directly.
- The test-distribution quantile problem seeks the smallest threshold satisfying the target risk constraint given calibration scores, a discrepancy, and a known shift budget.
- Calibration-conditional quantile estimation requires the threshold to cover an independent future test score after calibration samples are realized.
- Calibration-test distribution shift can arise across environments, sensing conditions, populations, or tasks, motivating a shift-robust formulation.
- CP and DRO are presented as methods for constructing the data-dependent estimators.
4. CP and DRO Without Distribution Shift
Without distribution shift, CP and DRO both estimate a target quantile with calibration-conditional coverage, but CP corrects the probability level while DRO corrects the value through an ambiguity radius. Their finite-sample behavior differs: CP can be vacuous or overshoot in sparse tails, whereas DRO can remain finite but requires distributional assumptions or tuning for certification.
- Calibration-conditional conformal prediction: Finite-sample, distribution-free calibration-conditional guarantees cannot hold for every calibration set, so CP targets coverage for all but a small fraction of calibration samples.This produces a two-level guarantee: target test coverage holds with high probability over calibration samples.
- Radius certification: DRO’s certified radius depends on unknown distributional properties and is often replaced in practice by validation or tuning, which forfeits the high-probability guarantee for the true distribution.For a fixed radius, DRO still guarantees coverage uniformly over its ambiguity ball.
- Unified comparison: CP inflates the empirical quantile level, whereas DRO shifts the quantile value through a Wasserstein ambiguity radius.Both address the gap between the empirical and true distributions, but along probability and score-value coordinates respectively.
- Tail behavior: Tail sparsity governs conservativeness: CP barely moves near dense light-tail samples but can overshoot deeply in heavy tails, while a tuned DRO radius adds a value-space margin.The DRO correction depends on K, β, and ambiguity-set construction rather than realized sample spacing.
- Asymptotic properties: Both CP and DRO are strongly consistent under their stated regularity conditions, with the same first-order asymptotic variance but different deterministic shifts.CP’s shift depends on the density at the target quantile, whereas DRO’s shift is inherited from the ambiguity-set radius.
5. CP and DRO With Distribution Shift
Under distribution shift, CP and DRO extend calibration-based thresholds to test distributions using Wasserstein and Lévy–Prokhorov ambiguity models. Both retain calibration-conditional guarantees, while their corrections differ in how they account for shift and ambiguity.
- Wasserstein shift: The W∞ model assumes that the test distribution lies within a radius-η ball around the calibration distribution.The radius controls the amount of calibration-test shift allowed and may be specified from prior knowledge or validation data.
- Wasserstein shift: Under a W∞ shift model, CP adds the shift radius η directly to its calibration threshold.This transfers the calibration guarantee to the test distribution at the cost of additional conservativeness.
- Wasserstein shift: Under the same model, DRO combines the finite-sample radius r_K(β) with η as two additive corrections.The first term addresses empirical-to-calibration uncertainty, while η addresses calibration-to-test shift.
- Lévy–Prokhorov shift: The Lévy–Prokhorov model permits both bounded value-space movement and arbitrary displacement of a fraction ρ of the mass.Its ambiguity set reduces to a TV ball when η=0 and to a W∞ ball when ρ=0.
- Lévy–Prokhorov shift: Under LP shift, a quantile above K/(K+1) returns ∞, so the guarantee is trivially valid but non-useful.A finite threshold therefore requires the non-vacuity condition described in Remark 4.5.
- Lévy–Prokhorov shift: The paper extends LP-robust conformal prediction from marginal coverage to calibration-conditional coverage by combining index inflation with the LP shift relation.The extension is stated under an LP-ball assumption and includes a corresponding DRO counterpart.
6. Application Studies in Vision, Language, and Autonomous Driving
Across vision, language, and trajectory prediction, calibration-conditional CP reliably meets the two-level guarantee with compact prediction sets, while Split CP controls only marginal coverage and DRO can trade robustness for vacuity or larger sets. Under distribution shift, Theorem 5.6 is especially robust, but increasing ambiguity budgets enlarges sets and fixed budgets eventually fail as severity grows.
- Cross-application comparison: Calibration-conditional CP satisfies the stronger two-level guarantee in essentially all trials while keeping prediction sets small across the evaluated applications.On ImageNet, mean set sizes are 2.4–2.7 of 1000 classes; in language QA they are approximately 2.0 of 4 options; trajectory radii are 26–27 m.
- Distribution shift: Under distribution shift, Split CP degrades sharply, whereas Theorem 5.6 maintains the two-level guarantee at severity 1–2 across all three corruptions.In a representative motion-blur cell, Theorem 5.6 achieves rate 1.000 with median set size 6.8 and no vacuous trials.
- Distribution shift: For Theorem 5.6, increasing Wasserstein radius η raises two-level reliability but grows prediction sets steeply, while level parameter ρ supplies robustness more efficiently when η is small.The additive value correction can push thresholds toward the all-class set; the preferred operating point keeps η small and uses ρ.
- Distribution shift: As distribution-shift severity increases, all methods degrade; fixed ambiguity budgets cannot absorb sufficiently large shifts, while larger budgets enlarge prediction sets.Budget selection therefore trades efficiency for robustness, and the score domain affects the cost: continuous trajectory scores avoid discrete-label vacuity.
7. Conclusions
The paper unifies CP and Wasserstein DRO as quantile-threshold procedures while exposing different corrections, certification properties, and finite-sample trade-offs. Experiments show calibration-conditional methods meet the two-level guarantee, with CP yielding smaller prediction sets and DRO facing bounded-score limitations.
- CP raises the empirical quantile level, whereas DRO shifts the quantile value through an ambiguity radius.
- CP uses a closed-form distribution-free correction, while DRO’s certified radius depends on a typically unknown density lower bound and certifies coverage uniformly over the ambiguity set.
- In the scalar case, both estimators have the same first-order asymptotic variance and differ only by a deterministic offset.With certified radius, DRO’s offset is provably larger and depends on a uniform density lower bound, whereas CP’s depends on density at the target quantile.
- Calibration-conditional CP and DRO attain the two-level guarantee that standard split-CP fails to meet, consistently across ImageNet, MMLU, and nuScenes.
- Among methods meeting the guarantee, CP produces the smallest prediction sets, while DRO’s additive correction can make prediction sets vacuous for bounded scores.The same distinction applies to shift-aware estimators under both Wasserstein and Lévy–Prokhorov shift models.
- Open problems include estimating adaptive DRO radii and shift budgets from data, accounting for estimation error, and extending two-level guarantees to decisions and online calibration.