Distribution Calibration for Regression

1 Introduction

• Regression models can achieve strong performance on metrics such as mean squared error while still providing poorly quantified uncertainty about target values.The paper extends probability-calibration concepts from classification to regression.Source: §1 · p. 1
• Post-hoc regression calibration remains largely unexplored, although it could calibrate outputs from existing specialized models such as pretrained deep neural networks.The cited prior approach applies isotonic regression to match predicted CDFs with empirical frequencies, improving quantile calibration.Source: §1 · p. 1
• Distribution calibration targets each prediction’s uncertainty, addressing a limitation of quantile calibration, which is defined uniformly over the input space.A quantile-calibrated regressor does not guarantee that cases with the same predicted mean and standard deviation follow a Gaussian distribution with those moments.Source: §1 · p. 1
• The paper introduces distribution calibration, which naturally leads to calibrated quantiles, and proposes a multi-output GP approach with a Beta link for post-hoc density calibration.The approach models distributions over calibration parameters; the authors also provide stochastic variational inference with inducing pseudo-points for scalability.Source: §1 · p. 2

2 Background and Definitions

• Post-hoc calibration learns a transformation of a pretrained probabilistic model’s output distribution to better align predicted and empirical distributions.The paper uses X for input features and Y for the corresponding target value.Source: §2 · p. 2

2.1 Calibration in Classification

• Classification calibration requires predicted class probabilities to agree with observed class frequencies among instances receiving the same probability vector.Alternative definitions may instead require accuracy among instances with the same confidence level to match that confidence.Source: §2.1 · p. 3
• Isotonic calibration fits a non-decreasing, piecewise-constant map, while Beta calibration provides a parametric map with flexible shapes including the identity.Isotonic calibration can overfit smaller datasets; Beta calibration is defined on [0, 1].Source: §2.1 · p. 3

2.2 Quantile-Calibrated Regression

• Quantile calibration requires each predicted τ-quantile to have marginal coverage τ, and isotonic regression can map quantile levels to empirical frequencies to improve calibration.The mapping is learned from quantile levels and the observed proportions of targets falling below corresponding predictions.Source: §2.2 · p. 3
• Because quantile calibration is marginal, it does not ensure that predictions with the same moments have target distributions with those moments.The paper therefore proposes a stronger regression-calibration definition that more closely mirrors classification calibration.Source: §2.2 · pp. 3–4

3 Distribution-Calibrated Regression

• Distribution calibration requires the conditional target distribution to agree with the model’s predicted distribution, with the definition restricted to absolutely continuous distributions over real-valued targets.The condition is imposed for every model prediction and target value.Source: §3 · p. 4
• Predictions sharing a mean µ and variance σ² correspond to targets with the same mean and variance on average.Source: §3 · p. 4
• Extracting any τ-quantile from a distribution-calibrated model yields a quantile-calibrated regressor.The theorem assumes the model’s output distributions are absolutely continuous.Source: §3 · p. 4
• Quantile calibration alone does not guarantee distribution calibration because marginal coverage gives no conditional information for a particular model prediction.Source: §3 · p. 4
• Distribution calibration minimizes expected negative log likelihood when the calibration-loss KL divergence is zero, with refinement loss fixed after training.The paper identifies negative log likelihood as its metric for quantifying the benefits of distribution calibration.Source: §3 · p. 4

4.1 Beta link function for regression

• The Beta link transforms regression-output CDFs and acts as a density ratio; its identity setting leaves the density unchanged, while valid parameters preserve a normalized distribution.The identity setting is a = 1, b = 1, c = 0; nonnegative a and b give a nonnegative ratio.Source: §4.1 · pp. 4–5

4.2 The GP-BETA Model

• A multi-output GP predicts Beta calibration parameters from each regressor’s mean and variance, borrowing information across nearby outputs when individual targets are sparse.The three jointly distributed latent functions correspond to parameters a, b, and c; the kernel acts on regression-output distributions.Source: §4.2 · p. 5

4.3 Scalable Inference

• Sparse variational inference with inducing points and Monte Carlo gradients addresses the nonlinear link’s intractable integration during training.The method approximates a Gaussian posterior over inducing points and estimates the expected log likelihood with Monte Carlo samples.Source: §4.3 · p. 6
• Training cost includes an O(m^3) KL-bound computation, while prediction costs O(m^2 + l) per instance; kernel-matrix operations remain challenging as data grows.Here m is the number of inducing points and l is the number of Monte Carlo samples; both are user-selectable.Source: §4.3 · pp. 6–7

Synthetic data

• On the synthetic example, isotonic calibration barely changes the CDF and produces nonsmooth PDFs, while GP-BETA recovers conditional CDF variation and the true density’s bimodal shape.GP-BETA estimates different calibration maps for different model outputs; isotonic calibration uses the same map for each output.Source: §5 · p. 7

Real world datasets

• The real-world evaluation uses NLL, MSE, and pinball loss across six UCI datasets, with GPR limited to 1,000 training points for computational reasons.The paper averages pinball loss over quantiles from 0.05 to 0.95 in increments of 0.05.Source: §5 · pp. 7–8
• GP-BETA is clearly superior on the synthetic example and most real-world examples, and almost always has the best NLL regardless of inducing-point count.Table 1 reports predictive performance across base models; the prose gives the overall comparison but no exact table values.Source: Table 1 · p. 8, §5 · p. 8
• GP-BETA does not improve some results on dataset 3, while isotonic calibration’s unusually low NLL on dataset 4 may reflect its target distribution’s roughly half-zero values.The authors suggest an isotonic step near zero may yield a very high likelihood on dataset 4.Source: §5 · p. 8

• Distribution calibration improves uncertainty quantification on the target variable while also providing quantile-level calibration.Source: §6 · p. 8
• GP-BETA combines multi-output Gaussian Processes with Beta calibration and distribution regression as a post-hoc approach for regression.Its sparse variational inference scheme allows scaling to large datasets and shows strong empirical performance.Source: §6 · p. 8
• Although distribution calibration applies to any conditional density estimator, the paper focuses on regression and identifies non-parametric calibration maps and other density-estimation forms as future directions.Source: §6 · p. 8