PaperlayerView exact revision on arXiv

Source-linked AI summary

CONTRA: Conformal Prediction Region via Normalizing Flow Transformation

Zhenhan Fang, Aixin Tan, Jian Huang

arXiv:2605.08561v1stat.MLcs.LG

TL;DR

Existing density-based regions lack inherent coverage guarantees, and multi-dimensional conformal prediction often relies on restrictive one-dimensional scores. CONTRA maps calibrated high-density latent regions through normalizing flows, while ResCONTRA extends the approach to residuals from other predictors. The methods are reported to achieve reliable coverage and compact, flexible regions, although practical conditional-coverage bounds remain unresolved.

  • Problem

    Multi-dimensional prediction regions need reliable coverage guarantees while avoiding the inflated or irregular shapes produced by existing approaches.

  • Method

    CONTRA calibrates distances in a normalizing-flow latent space and maps a calibrated high-density region into the output space; ResCONTRA applies the idea to residuals from other predictors.

  • Results

    CONTRA and ResCONTRA achieve desired coverage with high accuracy and produce compact regions with flexible shapes and smooth boundaries.

  • Takeaways & Limitations

    The methods provide compact, interpretable multi-dimensional prediction regions for conditional density estimation within the supported settings.

  • Takeaways & Limitations

    Conditional coverage depends on deviation from an ideal x-independent latent distribution, and useful bounds remain an ongoing theoretical and empirical challenge.

Abstract

from arXiv · show

Density estimation and reliable prediction regions for outputs are crucial in supervised and unsupervised learning. While conformal prediction effectively generates coverage-guaranteed regions, it struggles with multi-dimensional outputs due to reliance on one-dimensional nonconformity scores. To address this, we introduce CONTRA: CONformal prediction region via normalizing flow TRAnsformation. CONTRA utilizes the latent spaces of normalizing flows to define nonconformity scores based on distances from the center. This allows for the mapping of high-density regions in latent space to sharp prediction regions in the output space, surpassing traditional hyperrectangular or elliptical conformal regions. Further, for scenarios where other predictive models are favored over flow-based models, we extend CONTRA to enhance any such model with a reliable prediction region by training a simple normalizing flow on the residuals. We demonstrate that both CONTRA and its extension maintain guaranteed coverage probability and outperform existing methods in generating accurate prediction regions across various datasets. We conclude that CONTRA is an effective tool for (conditional) density estimation, addressing the under-explored challenge of delivering multi-dimensional prediction regions.

1 INTRODUCTION

Existing density-based regions lack inherent coverage guarantees, while multi-dimensional conformal methods often produce inflated boxes, ellipsoids, or irregular regions. CONTRA and ResCONTRA use normalizing-flow transformations to produce compact, flexible, and interpretable regions while retaining coverage guarantees.

  • Density-based prediction regions may target probabilities such as 90%, but their actual coverage depends on model assumptions and lacks inherent guarantees.
  • Multi-dimensional conformal methods commonly restrict regions to boxes or ellipsoids, which can inflate around multimodal or unequal-tailed distributions.
  • PCP permits flexible shapes but can produce irregular boundaries and disconnected regions that hinder interpretability.
  • In the NYC Taxi example, CONTRA regions are more connected than PCP and ST-DQR regions while maintaining theoretical rigor.
  • CONTRA maps high-density regions from a normalizing-flow latent space into the output space and calibrates them using latent representations of calibration data.
  • ResCONTRA trains a simple normalizing flow on residuals to add reliable prediction regions to other predictive models.
  • CONTRA and ResCONTRA achieve desired coverage with high accuracy while producing smaller, smoother, and more interpretable regions than restricted-shape methods.

2 CONDITIONAL NORMALIZING FLOW

Conditional normalizing flows model output distributions by transforming a simple latent variable into the conditional output space through a differentiable bijection. Complex transformations are composed from simpler layers whose Jacobian determinants remain tractable.

  • Conditional density estimation seeks an estimate of the conditional density pY|X(·|x) for each input x.
  • A conditional normalizing flow transforms a q-dimensional output into a simple latent random vector, such as a q-dimensional Gaussian.
  • The transformation y = tθ(z, x) is a differentiable bijection from latent space to the conditional output support, with an inverse transformation.
  • Complex bijections are constructed compositionally by chaining simpler transformations.
  • The paper uses realNVP affine transformations with coupling layers to make Jacobian determinant evaluation computationally efficient.
  • The base distribution is chosen independently of x so latent representations across inputs can be compared and pooled for calibration.

3 CONTRA: CONFORMAL REGION VIA NORMALIZING FLOW TRANSFORMATION

CONTRA calibrates distances of latent representations from the center of a normalizing-flow base distribution, then maps the resulting conformal ball back to the output space. ResCONTRA applies the same idea to residuals from an arbitrary point predictor, while practical conditional-coverage guarantees remain an open challenge.

  • CONTRA construction: A naive latent high-density region can have inaccurate coverage when the learned flow poorly approximates the conditional distribution, especially for underrepresented inputs.
  • CONTRA construction: Split conformal prediction separates training and calibration data so latent nonconformity scores can be calibrated without refitting for each new input.
  • CONTRA construction: CONTRA represents each calibration output in latent space and uses the empirical quantile of Euclidean norms to determine the conformal radius.
  • Coverage: The calibrated latent ball contains at least (1 −α)100% of calibration points, and exchangeability yields P(ˆzn+1 ∈ˆE) ≥ 1 −α.
  • Coverage: The output prediction region is the conditional-flow image of the calibrated latent ball, with its coverage guarantee stated in Proposition 2.
  • Region display: The region can be displayed by mapping samples from the latent ball or only its boundary into output space.
  • Geometry: CONTRA favors smooth boundaries and connected regions, properties associated with more robust and interpretable inferences.
  • ResCONTRA: ResCONTRA splits data into three sets, fits a user-chosen point predictor, models residuals with a normalizing flow, and calibrates the residual-based region.

4 RELATED WORK

Related work covers conformal methods that target multi-dimensional outputs directly and one-dimensional approaches upgraded to multi-target settings. CONTRA differs by using a bijective latent representation and directly calibrating a Gaussian high-density region, while MCQR provides a box-shaped alternative with marginal coverage guarantees.

  • Multi-dimensional conformal approaches: RCP uses a global covariance matrix for ellipsoidal regions, while NLE adds local covariance matrices to adapt regions to x.
  • Multi-dimensional conformal approaches: PCP generates prediction regions as unions of K balls, yielding flexible but often irregular and disconnected shapes sensitive to K and α.These properties can make the regions difficult to interpret.
  • Multi-dimensional conformal approaches: ST-DQR learns a lower-dimensional latent representation and applies directional quantile regression to form convex latent probability regions before mapping samples back to output space.
  • Comparison with CONTRA: Unlike ST-DQR, CONTRA uses a bijection that can closely match a Gaussian reference, allowing direct calibration of a Gaussian high-density region rather than union balls.ST-DQR and PCP introduce additional steps involving latent probability regions or calibrated balls around generated samples.
  • One-dimensional approaches upgraded to multi-dimensional outputs: MCQR extends conformalized quantile regression to multi-dimensional outputs, retaining a box-shaped region and satisfying a marginal coverage guarantee for any weight vector.Its construction trains lower and upper quantile estimators for each output dimension.

5 EXPERIMENTS

Experiments on synthetic and real datasets show that CONTRA and ResCONTRA achieve nominal coverage while producing compact, flexible, and smooth prediction regions. The methods generally outperform competing approaches in region volume and shape.

  • Experimental setup: Experiments used four synthetic and six real datasets, with RealNVP-based conditional normalizing flows trained using 6–10 coupling layers.The study also examined underfitting, overfitting, and data-size effects.
  • Synthetic data: All conformal methods achieved the nominal 0.9 coverage level in both synthetic-data setups.Results were averaged over 20 replications with standard errors.
  • Synthetic data: CONTRA and PCP generated the two smallest synthetic prediction regions, while ResCONTRA produced comparable results and other methods produced significantly larger regions.The comparison used methods sharing the same conditional normalizing flow where applicable.
  • Synthetic data: CONTRA, ResCONTRA, and PCP aligned well with true highest-density regions, whereas elliptical and rectangular methods included extensive low-density areas.PCP approximates flexible regions with a union of K circles, increasing computational cost and boundary complexity as K grows.
  • Real data: Across real datasets, coverage was generally near 0.9, while energy produced lower rates ranging from 0.84 to 0.88.The lower empirical coverage was attributed to finite training and calibration sample sizes relative to marginal guarantees.
  • Real data: In multidimensional real-data experiments, CONTRA-type methods produced more compact regions than competitors, with CONTRA usually attaining the smallest volume and smoother shapes.The experiments included a four-dimensional output task, described as challenging for traditional approaches.

6 SUMMARY AND DISCUSSION

The paper presents CONTRA for reliable multidimensional conditional density estimation and introduces extensions for users who prefer other predictive models. It reports compact, flexible, and smooth regions, while identifying training complexity and continuous-outcome support as limitations.

  • CONTRA: CONTRA is a multidimensional conformal prediction method designed to provide reliable conditional density estimation.Its reported regions are compact, flexible, and have smooth boundaries.
  • Extensions: ResCONTRA extends user-selected point predictors with valid prediction regions, while MCQR extends CQR to compact box-shaped conformal regions.These methods address users who prefer predictive models other than flow-based models.
  • Method: CONTRA uses latent-space distance from the origin as a multidimensional nonconformity score after transforming outputs through a learned bijection.The approach relies on choosing a projection that yields desirable prediction regions while retaining marginal coverage under exchangeability.
  • Limitations: A key challenge is that CONTRA requires learning a q-dimensional output-to-latent bijection, which becomes harder with complex distributions, higher dimension, and limited data.The paper suggests inspecting latent deviations from a standard multivariate Gaussian as a possible adjustment strategy.
  • Limitations: The current Gaussian latent base restricts CONTRA to continuous outcomes, leaving discrete and mixed-type multidimensional outcomes outside its present scope.The paper identifies conformal methods for these outcome types as an important direction for future work.

A PROOF

The proof establishes coverage for a conformal region built from any conditional normalizing flow under exchangeability. It also uses the flow’s bijective structure to support the region construction and boundary argument.

  • Coverage result: Under exchangeability, the conformal ball based on any conditional normalizing flow satisfies the stated marginal coverage result.The proposition applies to any CNF model t̂θ.
  • Coverage result: If calibration and new latent representations are almost surely distinct, the coverage probability is bounded above by 1 − α + 1/(n₂+1).This is the finite-sample upper bound stated in the proposition.
  • Proof mechanism: The proof uses that a CNF-trained transformation is a differentiable bijection.This property connects output-space regions with their latent representations.
  • Boundary argument: For a homomorphism, transforming a set transforms its boundary: t(∂E) = ∂t(E).The result is applied to the latent and output spaces used in the CNF construction.

A.3 GUARANTEED COVERAGE PROOF OF MCQR

The MCQR proof establishes coverage by using exchangeability of calibration and future non-conformity scores, quantile inflation, and expectation over the training data.

  • The MCQR prediction region is defined through the algorithm's construction and is evaluated under exchangeable observations.
  • Exchangeability of the calibration and future non-conformity scores enables the coverage argument through Lemma 1.The proof applies the inflation-of-quantiles result to exchangeable scores.
  • When calibration residuals are almost surely distinct, the conditional probability is bounded by 1 − α + 1/(n^2 + 1).
  • Taking expectation over the training data yields guaranteed marginal coverage.

B SYNTHETIC DATA STRUCTURE

The synthetic studies use several non-Gaussian outcome-error structures, while the normalizing-flow construction transforms partitioned output coordinates through coupling layers and invertible functions.

  • Synthetic models: The simulations include mixture-Gaussian, spiral, moon-shaped, and ring-shaped error distributions.These settings provide varied two-dimensional structures for evaluating prediction regions.
  • Synthetic models: The synthetic covariates are Gaussian, with X ∼ N(µ, I2) and µ = [−2.0, −1.5]T.
  • Flow transformation: A coupling layer partitions y into two subspaces and applies a bijection to one subspace conditioned on the other and x.
  • Flow transformation: The conditional transformation uses scale and shift functions, with a Jacobian determinant expressed through the exponential scale terms.
  • Flow transformation: The transformation is composed of multiple coupling layers whose subset roles can be switched to avoid consecutive layers reducing to the identity.

D ADDITIONAL SIMULATION STUDIES

Additional simulations compare coverage and volume for two-dimensional 90% conformal regions and examine how normalizing-flow fit affects CONTRA regions.

  • Synthetic comparisons: Table 4 reports average coverage and volume for 2-dimensional 90% conformal regions across two synthetic datasets and 20 random splits.Standard errors are shown in parentheses, and the smallest volume is boldfaced.
  • Flow fit: Using an NF that is too complex or too simple can make conformal regions overly sensitive or excessively large.
  • Flow fit: Figure 4 compares CONTRA regions and latent z under underfitting, overfitting, and regular-fitting NF models for NYC taxi drop-off prediction.The models use 2, 16, and 6 coupling layers, respectively, with corresponding training times of 4, 205, and 17 seconds.
  • Flow fit: Calibration latent variables can be checked against a standard Gaussian for bias, overdispersion, and underdispersion when tuning the flow model.Classical metrics can supplement the visual check.

F IMPACT OF DATA SIZE ON CONTRA AND ITS MAIN COMPETITOR

The data-size study compares CONTRA with ST-DQR based on a diffusion model across increasing sample sizes, finding that regions shrink as uncertainty falls and then stabilize.

  • The study compares CONTRA and ST-DQR based on a diffusion model at sample sizes 300, 1200, and 4800.Both methods use a 75%-25% training-calibration split.
  • As sample size increases from very small to moderate, prediction regions become smaller as uncertainty decreases.
  • At larger sample sizes, the prediction regions stabilize at approximately the same size, reflecting persistent uncertainty for new subjects.

G A COMPARISON BETWEEN CONTRA AND RESCONTRA

The comparison indicates that CONTRA is preferable when the conditional distribution is easier for a normalizing flow to learn, whereas ResCONTRA can work better when complex relationships leave simple residuals. Users can compare both methods and select the prediction region that best suits their needs.

  • Method choice: When the relationship y|x is highly complex but residuals are relatively simple, ResCONTRA tends to provide better prediction regions.This advantage is attributed to combining point-estimation tools with a simpler normalizing flow for residuals.
  • Method choice: When a normalizing flow effectively learns y|x, one-step CONTRA generally outperforms two-step ResCONTRA because it uses a larger training set.
  • Practical guidance: Users need not choose CONTRA or ResCONTRA in advance; they can try both and use the region that better suits their needs.
  • Empirical comparison: In the complex-example comparison, ResCONTRA methods using specialized point estimators and relatively simple residual flows produced small latent radii and small conformal regions.The comparison examines latent calibration-set values against a bivariate standard-normal reference and reports the radius r of the smallest circle containing 90% of the calibration points.
Loading 2605.08561v1…