A Unified Perspective on Conformal Prediction and Wasserstein Distributionally Robust Optimization for Uncertainty Quantification

• CP constructs prediction sets with finite-sample distribution-free coverage under exchangeability, whereas DRO optimizes worst-case performance over empirical-centered ambiguity sets.Source: §1 · p. 1
• Calibration-conditional validity requires target test coverage for the observed calibration set, with that requirement holding for all but a small fraction of possible sets.This is a two-level guarantee: target coverage for the test score and confidence over the random calibration set.Source: §1 · p. 1
• The comparative study examines theoretical formulations, statistical properties, coverage guarantees, conservativeness, and numerical performance to guide method selection from finite data.Source: §1 · pp. 1–2
• CP and Wasserstein DRO can be unified as finite-sample, data-dependent quantile estimators with calibration-conditional coverage guarantees.Source: §1 · p. 1
• Without test-time shift, CP inflates the empirical quantile level, while DRO adds a value-space ambiguity radius; both give the same calibration-conditional guarantee for the true distribution, and DRO also certifies coverage over its ambiguity set.The score density near the target quantile affects finite-sample conservativeness.Source: §1 · pp. 1–2
• Under test-time shift, the paper compares Wasserstein- and Lévy–Prokhorov-based shift models and derives CP and DRO estimators with calibration-conditional two-level guarantees.The methods use value-space and quantile-level corrections to account for different forms of shift.Source: §1 · p. 2
• Experiments cover image classification, multiple-choice question answering, and autonomous-driving trajectory prediction, comparing coverage, conservativeness, two-level guarantees, and robustness under shift.Source: §1 · p. 2

Conformal prediction

• CP provides finite-sample coverage under exchangeability, but exact distribution-free guarantees conditional on arbitrary test inputs are generally unattainable and can yield conservative sets [91, 35].Source: Conformal prediction · p. 2
• CP supports calibrated uncertainty across machine learning, control, and robotics, including adaptive prediction sets, safety certificates, planning, and risk-calibrated human-robot interaction [51] [24].Source: Conformal prediction · pp. 2–3

Distributionally robust optimization

• DRO optimizes worst-case performance over an ambiguity set containing the true distribution, a stronger assumption than CP’s exchangeability; Wasserstein sets offer tractability and finite-sample guarantees.Source: Distributionally robust optimization · p. 3
• DRO applications span robust learning, control, energy systems, and robotics, including Wasserstein learning guarantees, safe control, power-system chance constraints, and planning under uncertainty [23].Source: Distributionally robust optimization · p. 3

Unified perspective

• Related CP–DRO approaches use ambiguity sets for worst-case coverage or center a DRO set on a fitted predictive distribution and calibrate its radius using split CP [39].Source: Unified perspective · p. 3
• Other distribution-free routes include tolerance limits and scenario optimization, with conformal calibration linked to scenario programs and risk allocation [16].Source: Unified perspective · pp. 3–4
• Unlike prior CP–DRO work focused on shift-robust coverage, this paper unifies the methods through finite-sample quantile estimation and compares their statistical behavior and practical trade-offs.Source: Unified perspective · pp. 3–4

Problem Formulation and Preliminaries

• The empirical p-quantile of n values is the ⌈np⌉-th smallest, while estimating the test quantile from calibration data requires a known bound on distribution discrepancy when P may differ from Ptest.When Ptest is known, the target is its q_(1−δ) quantile.Source: §3 · p. 4
• Given a risk level, calibration scores from P, and a known discrepancy bound between P and Ptest, the formulation seeks the smallest threshold for test-distribution quantile estimation.P denotes the calibration-score distribution and Ptest the test-score distribution; if Ptest were known, the target would be its (1 − δ)-quantile.Source: §3 · p. 4
• When P = Ptest, the threshold depends on calibration data and must cover an independent future score with probability at least 1 − δ, with high probability over calibration samples.The section calls this a calibration-conditional coverage guarantee.Source: §3 · p. 4
• Because calibration and test scores may come from different environments or populations, the shift-robust formulation allows Ptest to differ from P subject to a discrepancy budget.It uses K i.i.d. calibration samples from P and an independent test score from Ptest.Source: §3 · p. 4

4.1. Calibration-Conditional Conformal Prediction

• Exact calibration-conditional guarantees of the form (3) cannot be achieved in a finite-sample, distribution-free setting; instead presents a conformal variant providing calibration-conditional coverage guarantees.These guarantees imply marginal coverage of at least (1 − δ)(1 − β), but marginal coverage may coexist with failure to attain coverage 1 − δ for a non-negligible fraction of calibration samples.Source: §4.1 · pp. 4–5
• Hoeffding, exact-binomial, and Bernstein CP corrections bound the calibration tail, with the exact-binomial and Bernstein constructions requiring K_min = Θ(ln(1/β)/δ) rather than Hoeffding’s Θ(ln(1/β)/δ^2).The exact-binomial construction has a smaller constant and becomes non-vacuous at a smaller calibration size; otherwise, CP may return the threshold ∞.Source: §4.1 · pp. 5–6

4.2. Distributionally Robust Optimization

• Certified Wasserstein radii depend on problem-specific constants that are often unknown, so practitioners may tune the radius and lose the formal guarantee.The certified value-space shift uses a uniform density lower bound and can be more conservative, while DRO remains finite for every calibration size.Source: §4.1 · p. 6, §4.2 · pp. 7–8
• Wasserstein DRO yields coverage at least 1 − δ for every distribution in its ambiguity ball and every calibration realization, implying marginal coverage of at least (1 − δ)(1 − β).Its radius can be certified under assumptions including bounded one-dimensional support and a positive density lower bound; the robust guarantee holds once the ball is fixed.Source: §4.2 · pp. 6–7
• CP raises the quantile level, while DRO shifts its value; sparse heavy-tail calibration scores can make CP overshoot, whereas a tuned DRO radius may avoid it.CP’s upper-tail order statistics cluster near the target quantile for light-tailed scores but are sparse for heavy-tailed scores; tuned DRO can be more accurate at small calibration sizes, though bounded discrete scores can also yield vacuous DRO sets.Source: Figure 1 · p. 7, §4.2 · p. 8

4.3. Statistical Properties of CP and DRO Estimators

• Under continuity and strict increase of the score CDF near the target quantile, the Hoeffding-based CP quantile estimator converges almost surely to that quantile as K grows.The supplied passage also states an asymptotic distribution result under differentiability and positive density, but its formula is not present in the excerpt.Source: §4.3 · p. 8
• CP and DRO are strongly consistent, with K^-1/2 estimation errors sharing the same random sampling fluctuation but differing in their deterministic upward correction.CP’s shift depends on the density at the target quantile, whereas DRO’s comes from the Wasserstein radius; with the radius of Lemma 4.6, DRO’s correction is never smaller.Source: §4.3 · p. 9

4.4. Choosing the Wasserstein Radius

• A fixed Wasserstein radius guarantees coverage uniformly over its ambiguity set, but heuristic radius selection does not guarantee high-probability coverage for the true distribution.Finite-sample radii based on concentration or inference can depend on unknown distributional constants and be overly conservative; CP’s effective value-space correction depends on local density, with no distribution-free conversion in general.Source: §4.4 · p. 10

4.5. Illustrative Examples

• With radius r_K = r_1/√K, DRO converges to the true quantile, but larger r_1 yields more conservative small-sample estimates and the Normal requires the largest radius.The Normal’s density at the target quantile is smallest among the three illustrated distributions; with small r_1, samples may miss the upper tail and the DRO estimate can initially fall below the truth.Source: §4.5 · pp. 10, 12, Figure 3 · p. 11
• At K=1000, CP overshoots by 4–22% (Exp(1): 22%; Normal: 21%; Truncated Normal: 12%), while DRO stays within 3–10% at guarantee-meeting radii.At K=1, DRO provides finite estimates for all five distributions while CP is vacuous. The table also reports CP overshoots of 4% for Uniform and 6% for Beta at K=1000.Source: §4.5 · p. 10, Table 3 · p. 12
• In the five-distribution examples, only Uniform and Truncated Normal meet Lemma 4.6’s bounded-density condition; heuristic radii for the others show practical utility, not a finite-sample guarantee.The simplified radius r_K = r_1/√K meets the two-level target for r_1 ≥ 1 on Uniform and Beta, but requires r_1 = 4 for Normal; convergence alone does not establish the guarantee.Source: Figure 3 · p. 11, §4.5 · p. 12

5. CP and DRO With Distribution Shift

• The section models calibration–test discrepancy with W∞ or LP shift, and η may come from prior knowledge, validation data, or a robustness choice.The LP model additionally permits a fraction of mass to move arbitrarily.Source: §5 · p. 12

5.1. Wasserstein Shift Model

• Under W∞ shift, CP adds η to its calibration-conditional threshold, while DRO adds η to r_K(β), separating shift from finite-sample uncertainty.The DRO correction r_K(β) accounts for empirical-to-calibration uncertainty; η accounts for calibration-to-test shift.Source: §5.1 · pp. 12–13

5.2. Lévy–Prokhorov Shift Model

• Under LP shift, CP combines the usual index inflation with ρ; the result strengthens prior marginal coverage to a calibration-conditional guarantee.The LP model allows local value perturbations and arbitrary displacement of a fraction ρ of the mass; the LP ball includes TV and W∞ balls as special cases.Source: §5.2 · p. 13
• Under LP shift, DRO combines quantile-level correction ρ with value correction r_K(β) + η, with its conditional guarantee requiring bounded support and density bounded below.The theorem assumes support on [a, b], density infimum at least m > 0, and 0 ≤ ρ < δ.Source: §5.2 · p. 14

6.1. Image Classification

• On ImageNet, split CP’s two-level rate was 0.547 versus 1.000 and 0.996 for Lemmas 4.2 and 4.4, whose mean sets contained 2.4–2.7 classes.DRO at r=0.03 reached a 0.965 rate but produced vacuous 1,000-class sets in 0.060 of trials; the additive threshold can exceed 1 on bounded scores.Source: Figure 5 · p. 16, §6.1 · p. 16

6.2. Image Classification Under Distribution Shift

• Under ImageNet-C shift, Theorem 5.6 maintained the two-level guarantee at severities 1–2 across all corruptions, while split CP’s Gaussian-noise coverage fell from 0.87 to 0.50.For motion blur at severity 1, Theorem 5.6 met the guarantee in all trials with a median set size of 6.8 and no vacuous sets; Theorem 5.7 produced vacuous sets in 34% of trials.Source: Figure 7 · p. 17, Figure 8 · p. 17, §6.2 · p. 17

6.3. Language Model Question Answering

• On MMLU, split CP’s two-level rate was 0.557; Lemmas 4.2 and 4.4 met it in essentially all trials with ≈2.0 options, while Lemma 4.3 reached 0.891 with 1.84.DRO reached 0.955 at r_K=0.003 but returned the full four-option set in 0.160 of trials; the calibration-conditional methods were never vacuous.Source: §6.3 · pp. 18–19, Figure 11 · p. 19, Table 7 · p. 19

6.4. Autonomous Driving Trajectory Prediction

• On nuScenes, Lemmas 4.2 and 4.4 met the two-level guarantee with 26–27 m radii, while DRO reached it at r≥3 m without vacuity; maneuver shift reduced split-CP coverage to 0.669.The maneuver shift roughly doubled the 90th-percentile error, from 20.2 to 29.7 m; geographic shift barely changed it, from 23.9 to 23.7 m.Source: §6.4 · pp. 19–20, Figure 13 · p. 20, Table 8 · p. 20
• Under nuScenes maneuver shift, split CP under-covered (0.669), Theorem 5.5’s two-level rate was 0.657, while Theorem 5.6 attained both guarantees (0.969, 1.000).The correction enlarges the safety ball from 20 to 30–37 m; Theorem 5.7 also attains both guarantees with a tighter ball.Source: Table 9 · p. 20, §6.4 · p. 20

6.5. Choosing an Estimator

• Across the three applications, CP estimators rank identically by set size and two-level rate; split CP gives the smallest sets when only marginal coverage is required, but its two-level rate is below 0.9.Split CP does not control coverage for the particular calibration set drawn.Source: §6.5 · p. 21
• Among calibration-conditional corrections, Lemma 4.3 gives the smallest sets, while Lemmas 4.2 and 4.4 provide a wider two-level margin at the cost of larger sets.Lemma 4.3’s rate tracks 1−β and can fall on either side; the explicit corrections of Lemmas 4.2 and 4.4 can be allocated across constraints or time steps, and Lemma 4.4 is preferable once δ falls below roughly 0.1.Source: §6.5 · p. 21
• Without shift, CP is preferable for smaller sets; DRO is preferable below CP’s non-vacuity thresholds or when coverage must hold over an ambiguity set.On nuScenes, DRO matches calibration-conditional sets (26.9 against 26.8 m; two-level rate 1.000), while on bounded MMLU its correction returns the full label set in 0.160 of trials.Source: §6.5 · p. 21
• Under shift, estimator choice depends on budget assumptions, LP guarantee strength, and whether the score is bounded or unbounded.Every shift-aware estimator requires a budget (η for W∞, (η, ρ) for LP), and none is assumption-free; for purely value-space shifts W∞ is tighter (31.1 against 36.7 m). Under LP, Theorem 5.5 gives marginal coverage only (0.905; two-level 0.657), while Theorems 5.6 and 5.7 give both; ρ inflates bounded-score sets more slowly than η, but on unbounded distance scores the budgets are interchangeable.Source: §6.4 · p. 20, §6.5 · p. 21

Conclusions

• CP raises the empirical quantile level, whereas DRO shifts its value through an ambiguity radius, yielding distinct coverage guarantees.CP’s correction is closed-form and distribution-free; DRO’s certified radius depends on an unknown density lower bound but guarantees coverage uniformly over the ambiguity set.Source: §7 · p. 21
• In the scalar case, CP and DRO have the same first-order asymptotic variance and differ only by a deterministic offset.With a certified radius, DRO’s offset is provably larger and depends on a uniform density lower bound, while CP’s depends on density at the target quantile.Source: §7 · p. 21
• Across ImageNet, MMLU, and nuScenes, calibration-conditional CP and DRO meet the two-level guarantee that standard split-CP fails to meet.Among methods meeting it, CP yields the smallest prediction sets; DRO’s additive correction suits continuous scores but can make sets vacuous for bounded scores. The same distinction holds for shift-aware estimators under both shift models.Source: §7 · p. 21
• A key open problem is selecting a certified DRO radius without relying on unknown distribution properties or saturating bounded scores.Data-driven or adaptive radii may help, but require more than the K calibration scores, such as a score-distribution model or repeated problem instances.Source: §7 · p. 21
• Estimating the shift budget from data and accounting for estimation error in the guarantee remain open problems for shift-aware estimators.Every shift-aware estimator requires a shift budget as input, so none is assumption-free.Source: §7 · p. 21
• Further directions include extending the two-level coverage guarantee to decision-making and maintaining coverage through online calibration under non-stationary drift.Applications include chance-constrained optimization, model predictive control, and planning; online thresholds would need continual updating for long-horizon autonomous systems.Source: §7 · p. 21