Source-linked AI summary
Conformal Prediction via Transported Beta Laws
Thiago R. Ramos, Helton Graziadei, Luben M. C. Cabezas
TL;DR
Split conformal’s marginal coverage guarantee averages over random calibration samples, so it does not describe the distribution of coverage after one sample is realized. This paper compares that realized-coverage law with its i.i.d. beta reference using Wasserstein distance; the framework yields coverage and bad-calibration bounds, and its approximation tracks empirical distance in dependent-process simulations.
Problem
Marginal coverage averages over calibration samples, leaving the law of realized calibration-conditional coverage and its departure from the beta reference as the central question.
Source: §1 · pp. 2–3Method
The framework compares realized-coverage laws with the i.i.d. beta benchmark using Wasserstein distance, treating test-side shift as coverage-scale transport and calibration dependence as a change in the order-statistic law.
Source: §1 · p. 3, §5 · p. 20Results
Simulations on dependent uniform processes show that the Berry–Esseen approximation tracks empirical Wasserstein distance closely, even at moderate sample sizes.
Source: §5 · p. 20Takeaways & Limitations
The beta law remains a finite-sample reference beyond the i.i.d. setting, with Wasserstein distance giving direct control of marginal coverage gaps and bad-calibration probabilities.
Source: §5 · p. 20The Markov penalty for bad-calibration probabilities can be conservative, especially on the W1 scale.
Source: §3.1 · p. 10
Abstract
from arXiv · showhide
Split conformal prediction provides finite-sample marginal coverage under exchangeability, but this guarantee averages over the random calibration sample. We study instead the law of the calibration-conditional coverage induced by a realized conformal threshold. In the continuous i.i.d. setting this law is exactly $Beta(k,n+1-k)$, so the usual marginal guarantee corresponds to its mean. We take this beta law as a finite-sample reference object and quantify departures from it using Wasserstein distances on $[0,1]$. The framework yields direct bounds on marginal coverage gaps and on bad-calibration probabilities, and separates different sources of non-i.i.d. behavior according to how they deform the beta reference: test-side shift acts through a transport map on the coverage scale, while calibration dependence changes the order-statistic law itself. We instantiate the framework in scale-shift, clustered, and stationary mixing settings, where the induced deformations can be characterized explicitly or through Berry-Esseen approximations. Simulations on dependent processes confirm that the first-order approximation tracks the empirical Wasserstein distance even at moderate sample sizes.
1. Introduction
Split conformal’s marginal guarantee averages over calibration samples, leaving the distribution of coverage after a particular calibration sample uncharacterized. This paper studies that distribution against its i.i.d. beta reference and uses Wasserstein distance to relate departures to coverage degradation.
1. Introduction
- The usual guarantee controls mean coverage, but not the sampling distribution of calibration-conditional coverage; under continuous i.i.d. scores, that realized coverage has a beta law.For n = 30 and γ = 0.9, Beta(28, 3) assigns about 0.15 probability to coverage below 0.85 and about 0.04 below 0.80.Source: §1 · p. 2
- Under distribution shift, dependence, or non-identical score distributions, realized coverage may depart from beta, motivating comparison of its law with the beta reference.The calibration-conditional coverage is the inner probability given the calibration sample; marginal coverage averages it over calibration samples.Source: §1 · pp. 2–3
- Wasserstein distance between the realized-coverage law and the beta benchmark directly bounds marginal coverage gaps and bad-calibration probabilities.The framework treats test-side shift as transport on the coverage scale and calibration dependence as a change to the order-statistic law.Source: §1 · p. 3
1.1. Related work
- Prior optimal-transport approaches compare score laws, transported weights, or ambiguity sets, whereas this paper compares realized-coverage laws on [0, 1] after calibration.The cited approaches include Xu et al. (2025), Correia and Louizos (2025), and Aolaritei et al. (2025).Source: §1.1 · p. 3
- Other work studies non-exchangeability through weighted quantiles or coverage penalties, and asymptotic joint coverage across test samples; this paper instead studies finite-sample fixed-quantile coverage laws.The cited non-exchangeability approaches include Barber et al. (2023) and Oliveira et al. (2024); the asymptotic comparisons include Gazin et al. (2024) and Gazin (2024).Source: §1.1 · pp. 3–4
2. Background
The background establishes split conformal’s finite-sample marginal guarantee and refines it by describing the random coverage induced by a realized calibration sample. It then reviews Wasserstein tools used to compare such coverage laws.
2.1. Conformal prediction
- Under exchangeability and no ties, the split conformal threshold at kγ = ⌈(n + 1)γ⌉ gives marginal coverage γ ≤ P(Sn+1 ≤ q̂n,γ) < γ + 1/(n + 1).The threshold is the calibration-score order statistic, with the n + 1 index providing the finite-sample correction.Source: §2.1 · p. 4
- For continuous i.i.d. scores, realized calibration-conditional coverage has the beta law associated with the chosen order statistic, whose mean recovers the classical marginal guarantee.The beta lower tail gives the i.i.d. reference probability of bad calibration; later bounds compare non-beta coverage laws with that reference tail.Source: §2.1 · pp. 5–6
2.2. Optimal Transport and Wasserstein Distance
- The background also supplies coupling, Wasserstein-order monotonicity, and monotone rearrangement results for constructing and relating transport bounds.The monotone rearrangement gives an explicit optimal map when a distorted coverage law can be transported back to its beta reference.Source: §2.2 · pp. 6–7
- On [0, 1], Wasserstein comparisons of coverage laws bound differences in their means through W1, since the identity map is 1-Lipschitz.The one-dimensional formula applies when the coverage law is described through an order statistic or its counting process.Source: §2.2 · pp. 6–7
3. A Transportation Framework for Conformal Prediction
The framework treats the continuous i.i.d. beta law as the reference distribution for realized calibration-conditional coverage and measures departures from it on [0, 1].
3. A Transportation Framework for Conformal Prediction
- The radius ρ measures departure from the beta reference; under point-mass contamination, transport cost grows linearly in π and increases as c moves from the beta bulk.Figure 2 contours show which contamination pairs (π, c) fit a given Wasserstein budget.Source: §3 · pp. 8–9, Figure 2 · p. 9
3.1. Coverage Guarantees
- Wasserstein radius ρ bounds the marginal coverage gap by ρ plus 1/(n+1), while bad-calibration tail bounds add a transport penalty to the beta lower tail.The Markov penalty can be conservative, especially on the W1 scale.Source: §3.1 · p. 10
3.2. Decoupled Test Scores and Transported Beta Laws
- With decoupled test scores, test-side shift acts through h = F_eT ∘ F_T^-1; the distance for dependent uniform calibrations equals an integrated empirical-count versus binomial-tail difference.The counting formula does not require independence of the uniform variables.Source: §3.2 · p. 11
3.3. Berry–Esseen Transport for Dependent Calibration
- A Berry–Esseen approximation for dependent uniform order statistics, with asymptotic scale τγ, yields a quantile-to-beta Wasserstein bound.The result applies when the stated approximation assumptions hold.Source: §3.3 · p. 12
4. Applications
The applications distinguish test-side distribution shift, which transports the beta reference, from calibration dependence, which changes the order-statistic law. Examples characterize these effects through an explicit scale-shift map, effective cluster size, and mixing-based approximations.
4.1. Exact Transported Beta Laws under Distribution Shift
- Under distribution shift, the calibration order statistic retains its beta law, while the calibration-to-test map transports that law; the map is identity when test and calibration distributions match.For half-normal scores, r > 1 shifts realized coverage left and produces undercoverage; r < 1 shifts it right and produces overcoverage.Source: §4.1 · p. 13, §4.1.1 · p. 13
4.1.1. Example: half-normal scale shift
- A 10% increase in test-score scale (r = 1.1) gives an approximate coverage loss of 0.032, or roughly three percentage points, at γ = 0.9.In the half-normal model, the transport distance equals the marginal coverage loss, and its Wasserstein cost is larger for undercoverage than for an equal-sized overcoverage departure.Source: §4.1.1 · pp. 14–15
4.2. Clustered Calibration and Effective Sample Size
- With perfect within-cluster dependence, n calibration scores behave for realized coverage more like b independent units, even though each score remains marginally uniform.The example creates ties, but the counting identity and Wasserstein comparison do not require their absence.Source: §4.2 · pp. 15–16
4.3. Stationary Mixing Processes
- For stationary mixing scores, the bound separates test-calibration decoupling error from calibration-order-statistic deviation; a Berry–Esseen approximation gives an O(n^-1/2) remainder under the stated assumptions.The low-coverage probability is controlled by the i.i.d. beta lower tail plus the decoupling and approximation errors.Source: §4.3 · pp. 17–18
4.3.1. Example: AR(1) process
- In the AR(1) example, the test-calibration dependence contribution decays geometrically with horizon ℓ, at a rate bounded by a term of order |a|^ℓ.As ℓ grows, the coverage law approaches the calibration-order-statistic law, which need not equal the beta reference when calibration scores remain dependent.Source: §4.3.1 · p. 19
- The AR(1) simulations confirm coverage gap ≤ W1 ≤ Berry–Esseen bound across configurations, while the bound is conservative for small n, large a, and short horizons.The bad-calibration bound can be loose or exceed one when W1 is comparable to or larger than η, and tightens as ℓ increases.Source: §4.3.1 · p. 20
5. Conclusion
The conclusion presents the realized-coverage law as a finite-sample object, using the i.i.d. beta law as a reference beyond the i.i.d. setting. The examples show how Wasserstein comparisons capture shift and dependence, while simulations find that the Berry–Esseen approximation tracks empirical distances at moderate sample sizes.
5. Conclusion
- In the continuous i.i.d. setting, realized calibration-conditional coverage has law Beta(k, n + 1 − k), whose mean gives the classical split conformal coverage level.Its lower tail quantifies bad-calibration events.Source: §5 · p. 20
- Wasserstein distance between the realized-coverage law and the beta benchmark directly controls marginal coverage gaps and bad-calibration probabilities.Test-side shift transports the beta law through a coverage map, whereas calibration dependence changes the order-statistic law.Source: §5 · p. 20
- Simulations on dependent uniform processes show that the Berry–Esseen approximation tracks empirical Wasserstein distance closely, even at moderate sample sizes.The framework is presented as a diagnostic and comparison tool, not as a new conformal algorithm.Source: §5 · p. 20
A.1. Proof of Proposition 2
The proof conditions on the calibration sample to express coverage at the realized threshold, then identifies the sampling law of that quantity through a uniform order statistic.
A.1. Proof of Proposition 2
- Conditional on the calibration sample, the order statistic is fixed, so coverage is the score distribution function evaluated at that threshold.This is the conditional coverage quantity whose law is studied as the calibration sample varies.Source: §A.1 · p. 23
- The remaining distributional step is to identify the kth order statistic among n i.i.d. uniform variables.Source: §A.1 · p. 23
- Applying the probability integral transform gives i.i.d. uniform variables, with their ordering preserved by the score distribution function.Source: §A.1 · p. 23
A.2. Proof of Theorem 11
The proof compares the mean of the beta reference with the expectation of realized coverage to obtain the marginal coverage gap, alongside a finite-sample discretization bound.
A.2. Proof of Theorem 11
- The beta reference has mean k/(n + 1), which is compared with expected realized coverage to express the marginal coverage gap.Source: §A.2 · p. 23
- The finite-sample rounding correction is at most 1/(n + 1).The final absolute-error bound follows from the beta-neighborhood bound and this discretization bound.Source: §A.2 · p. 24
A.3. Proof of Theorem 12
The appendix establishes the bad-calibration bound and supporting results by combining probability inequalities, order-statistic arguments, and transport properties. It also develops the half-normal transport calculation and a Berry–Esseen comparison for the beta reference.
A.3. Proof of Theorem 12
- Theorem 12’s proof combines Markov’s inequality with the optimality of Tmon to establish the claimed bound.Source: §A.3 · p. 24
A.4. Proof of Proposition 13
- Applying the probability integral transform preserves calibration-score ordering and gives uniform marginals without requiring independence.In the i.i.d. uniform reference case, the order statistic has law Beta(k, n + 1 − k).Source: §A.4 · p. 25
A.5. Proof of Proposition 14
- The event U(k) ≤ t is equivalent to at least k calibration variables falling at or below t, yielding the order-statistic and Wasserstein identities.In the i.i.d. reference case, the count is Bin(n, t) and the order statistic has law βn,k.Source: §A.5 · p. 25
A.6. Details for the Half-Normal Scale-Shift Example
- For the half-normal scale-shift example, hr is increasing and hr − id has constant sign, so the monotone coupling is optimal for W1.The beta reference mean is E[Bn,k] = k/(n + 1).Source: §A.6 · p. 27
A.7. Proof of Proposition 15
- The i.i.d. beta order statistic has a Berry–Esseen approximation, and its centering difference is absorbed into the O(n^-1/2) remainder.Source: §A.7 · p. 28
Appendix B. Example additional results
The appendix estimates Wasserstein distances, coverage gaps, and transported laws by Monte Carlo across both worked examples. It reports generally low simulation standard errors and explains a wider band near zero in Figure 7.
Appendix B. Example additional results
- Each configuration uses 50,000 Monte Carlo simulations, drawing directly from the underlying process to estimate W1, the coverage gap, and transported laws.Source: Appendix B · p. 29
- Shaded bands show ±2 standard errors across simulations, which are generally very low.Source: Appendix B · p. 29
- Figure 7’s wider band near zero reflects its logarithmic scale; the absolute error remains stable as the coverage gap approaches zero.Source: Appendix B · p. 29
B.1. Half-normal illustrations and results
The half-normal scale-shift example transports the beta reference law and makes the resulting Wasserstein distance visible through CDF and coverage-gap comparisons. The deformation is stable across calibration sizes and asymmetric around r = 1.
B.1. Half-normal illustrations and results
- Figure 5 shows the full CDF deformation over a wider range of r and n; shaded W1 areas are visible for each pair, and the deformation is stable across calibration sizes.The right panel traces W1 and the coverage gap jointly as functions of r, verifying exact equality from both sides of r = 1 and across all three values of n.Source: §B.1 · p. 29
- The right panel verifies the equality between W1 and the coverage gap from both sides of r = 1 and across three calibration sizes.Source: §B.1 · p. 29
- Across r ∈ [0.3, 3.0], W1 equals the absolute marginal coverage gap, with larger transport cost for undercoverage than overcoverage at equal |r − 1|.Source: §B.1 · p. 30
B.2. AR(1) illustrations and results
In the AR(1) example, the coverage gap, Wasserstein distance, and asymptotic bound form a chain whose tightness depends on sample size and dependence. Increasing the horizon reduces the test–calibration contribution, while the bad-calibration bound can be loose at small horizons.
B.2. AR(1) illustrations and results
- For a > 0, W1 and its bound decay geometrically with ℓ before leveling at a Berry–Esseen floor; for a = 0, W1 is negligible.The floor depends on n and the long-run variance τ2.Source: §B.2 · p. 30, Figure 6 · p. 31
- The coverage gap ≤ W1 ≤ asymptotic bound chain holds in the asymptotic regime, though the bound is conservative for small n, large a, and small ℓ.Source: §B.2 · pp. 30, 32
- The bad-calibration bound can exceed one when large W1 makes the Markov penalty dominate, and tightens as ℓ grows and W1 decays.Source: §B.2 · p. 33, Figure 8 · p. 33