Source-linked AI summary
Accurate Uncertainties for Deep Learning Using Calibrated Regression
Volodymyr Kuleshov, Nathan Fenner, Stefano Ermon
TL;DR
Bayesian uncertainty estimates can fail to match real outcome frequencies, motivating a general calibration method for regression. The paper applies a Platt-scaling-inspired post-processing procedure to Bayesian and probabilistic models, finding consistently well-calibrated estimates that are useful in forecasting and reinforcement learning.
Problem
Bayesian credible intervals can be inaccurate because model misspecification and approximate inference may prevent them from matching true outcome frequencies.
Method
The paper trains an auxiliary recalibrator on a pretrained regression model’s outputs or features to produce calibrated probabilistic forecasts.
Results
The method consistently produces well-calibrated confidence estimates across Bayesian linear regression and feedforward and recurrent Bayesian neural networks, with uses in forecasting and model-based reinforcement learning.
Takeaways & Limitations
Calibrated uncertainty estimates correspond to empirical frequencies and support reliable, interpretable machine learning applications, including time series forecasting and reinforcement learning.
Takeaways & Limitations
Recalibration can worsen quantile estimates with insufficient recalibration data or a shifted data distribution, and may reduce forecast sharpness when original intervals are too tight.
Abstract
from arXiv · showhide
Methods for reasoning under uncertainty are a key building block of accurate and reliable machine learning systems. Bayesian methods provide a general framework to quantify uncertainty. However, because of model misspecification and the use of approximate inference, Bayesian uncertainty estimates are often inaccurate -- for example, a 90% credible interval may not contain the true outcome 90% of the time. Here, we propose a simple procedure for calibrating any regression algorithm; when applied to Bayesian and probabilistic models, it is guaranteed to produce calibrated uncertainty estimates given enough data. Our procedure is inspired by Platt scaling and extends previous work on classification. We evaluate this approach on Bayesian linear regression, feedforward, and recurrent neural networks, and find that it consistently outputs well-calibrated credible intervals while improving performance on time series forecasting and model-based reinforcement learning tasks.
1. Introduction
Reliable uncertainty estimates matter for machine learning, but Bayesian credible intervals can be miscalibrated because models may fail to represent the true data distribution. The paper extends post-processing calibration to regression and applies it to Bayesian deep learning, with reported benefits for forecasting and reinforcement learning.
- Uncertainty assessment is important for accurate, reliable, and interpretable machine learning, including supply chain planning, medical diagnosis, and autonomous driving.
- Bayesian uncertainty estimates can be miscalibrated: a 90% posterior credible interval may not contain the true outcome 90% of the time.The paper attributes this problem to model bias and insufficient expressiveness.
- Existing deep-neural-network uncertainty methods may require modifying the model and may not always produce perfectly calibrated forecasts.Weather-forecasting calibration techniques are described as specialized and difficult to generalize beyond climate applications.
- The paper introduces a simple procedure for recalibrating any regression algorithm, extending Platt scaling from classification to regression.Applied to Bayesian and probabilistic deep learning models, it is stated to produce calibrated credible intervals given sufficient i.i.d. data.
- Across Bayesian linear regression and feedforward and recurrent Bayesian neural networks, the method consistently produces well-calibrated confidence estimates useful for time series forecasting and model-based reinforcement learning.
2. Calibrated Classification
Calibrated classification requires predicted probabilities to match observed frequencies, while useful forecasts also need sharpness. Recalibration estimates empirical conditional probabilities from a pretrained model’s outputs or features and evaluates calibration with binned prediction-frequency curves.
- Calibration means that events assigned probability p occur approximately p of the time; formally, P(Y = 1 | H(X) = p) = p for all p ∈ [0, 1].
- Calibration vs. Sharpness: Calibration alone is insufficient because a forecaster that always predicts E[Y] can be calibrated without being useful; sharpness requires predictions near zero or one.
- Recalibration trains an auxiliary model R on a pretrained forecaster H so that the composition R ◦ H is calibrated.
- Choosing R(p) = P(Y = 1 | H(X) = p) frames recalibration as estimating the conditional probability associated with each forecast value.Platt scaling approximates this conditional probability with a sigmoid, while other methods use isotonic regression or kernel density estimation.
- Projections and Features: Non-probabilistic classifiers can be recalibrated by mapping low-dimensional features, such as an SVM margin, to empirical class probabilities.The recalibrator estimates P(Y = 1 | H(X) = φ) in feature space.
- Diagnostic Tools: Calibration curves compare predicted and observed average probabilities within prediction intervals, with perfect calibration represented by a straight line.Sharpness can also be assessed from the distribution of predictions: sharp forecasts cluster near 0 and 1.
3. Calibrated Regression
Calibrated regression matches empirical and predicted confidence behavior, while useful forecasts must also be sharp. The proposed post-processing recalibrates arbitrary regression models using an auxiliary mapping and diagnostic tools.
- Calibration: Regression calibration requires outcomes to fall within predicted confidence intervals at their stated frequencies as data grows.The empirical and predicted CDFs should match asymptotically, extending the notion to general confidence intervals.
- Calibration and Sharpness: Calibration alone is insufficient: useful forecasts must produce tight confidence intervals that account for the input-dependent uncertainty.A forecast based only on the marginal distribution can be calibrated without using features, while intervals with uniform width are less useful for locating uncertain inputs.
- Training Calibrated Regression Models: The recalibration scheme trains an auxiliary model R on forecast-derived data, then transforms each pre-trained forecast into a calibrated forecast R ◦ F_t.The method can use isotonic regression, whose monotonicity and non-parametric form support learning the target distribution with enough i.i.d. data.
- Training Calibrated Regression Models: The approach is simple, applies to any regression model, and produces calibrated forecasts given enough i.i.d. data, including for Bayesian deep-learning models.Unlike existing methods described here, it does not require modifying the forecaster.
- Training Calibrated Regression Models: A 95% forecast quantile is adjusted to 80% when only 80 of 100 observed outcomes fall below it.The recalibrator estimates the empirical probability of outcomes lying below each predicted quantile.
- Features for Recalibration: Non-probabilistic regressors can be recalibrated using features such as distance from a point prediction, but heteroscedastic features can improve forecast sharpness.Distance-based features yield calibrated forecasts with uniform-width intervals; modeling µ(x) and σ(x) captures uncertainty as a function of x.
- Diagnostic Tools: Calibration plots compare the true frequency of outcomes in confidence intervals with the predicted fraction, while calibration error provides a numerical score.The diagnostic set should be distinct from the calibration and training sets for best results.
4. Experiments
Experiments evaluate post-hoc recalibration across Bayesian regression, neural networks, depth estimation, forecasting, and model-based reinforcement learning. The method consistently improves calibration while preserving accuracy and supporting better forecasting and planning outcomes.
- Experimental setup: The study evaluates Bayesian ridge regression, feedforward and recurrent Bayesian neural networks, depth estimation, time series forecasting, and model-based reinforcement learning.UCI experiments use eight datasets, while forecasting uses grocery sales and reinforcement learning uses inventory management.
- Experimental setup: Recalibration uses a post-processing isotonic regression model applied after fitting the base predictor.The authors fit the base model and recalibrator on the same training set and report no significant overfitting.
- UCI experiments: Recalibrated models achieve significantly lower calibration error than the original model while retaining similar accuracy to stronger baselines.Concrete dropout and deep ensembles often improve calibration, but recalibrated models improve it further without sacrificing comparable accuracy.
- Time series forecasting: A recalibrated recurrent forecast produces confidence intervals containing about 90% of true grocery-sales values, unlike the uncalibrated intervals.Test mean absolute percent errors range from 17.3-21.8% across four stores.
- Forecasting and reinforcement learning: The recalibration technique is the only compared method reported to achieve almost perfectly calibrated forecasts, and calibrated transition models yield higher reinforcement-learning reward.The reinforcement-learning experiment uses dynamic programming for 14-day inventory planning.
5. Discussion
The discussion frames recalibration as a hybrid frequentist-Bayesian adjustment that matches empirical frequencies without requiring fully probabilistic uncertainty models. It preserves point-estimate accuracy, but may reduce sharpness or fail under insufficient recalibration data or distribution shift.
- Calibrated Bayesian forecasts: The approach adjusts Bayesian models to match true empirical frequencies, combining frequentist and Bayesian ideas rather than integrating over all possible models.The discussion relates this strategy to model criticism techniques such as posterior predictive checking.
- Calibrated Bayesian forecasts: Fully probabilistic models are not necessary for confidence estimates; simple features such as distance from a point forecast can support recalibration.More complex models may provide richer uncertainty signals that can be recalibrated into sharper forecasts.
- Probabilistic forecasting: The paper distinguishes its empirical-frequency calibration from Gneiting et al.'s calibration relative to latent generative distributions.It also reports that marginal calibration was too weak for the paper's purposes.
- Probabilistic forecasting: Given a pre-trained model, the method prioritizes calibration rather than maximizing sharpness subject to calibration.This differs from the sharpness-oriented framing described in prior probabilistic forecasting work.
- Limitations: Recalibration preserves point-estimate accuracy, but insufficient data or distribution shift can worsen recalibrated quantiles and underestimating uncertainty can reduce forecast sharpness.The discussion identifies these conditions as the principal boundaries on the method's behavior.
6. Previous Work
Previous work developed calibration concepts and methods mainly for classification and probabilistic forecasting. This paper builds on that literature by addressing the generation of calibrated forecasts for continuous variables.
- Calibrated classification: Platt scaling and isotonic regression are established recalibration methods for binary classification, with extensions to multiclass, structured prediction, and neural-network settings.These methods recalibrate outputs from pre-trained classifiers.
- Probabilistic forecasting: Calibration research originated in statistics and includes proper-loss analyses that decompose forecast quality into calibration and sharpness.This literature also includes Bayesian treatments of calibration.
- Probabilistic forecasting: Probabilistic forecasting work, especially in meteorology, developed specialized calibration systems and definitions for continuous variables, but did not explore techniques for generating calibrated forecasts.The paper focuses on that generation problem.
7. Conclusion
The paper formalizes calibration for continuous variables and proposes a simple recalibration method that produces well-calibrated probabilistic forecasts. It demonstrates practical usefulness across time series forecasting, reinforcement learning, and reliable machine learning systems.
- The paper formalizes calibration for continuous variables and connects it closely to calibrated classification.
- The proposed recalibration technique generates calibrated probabilistic forecasts given enough i.i.d. data.
- The paper introduces visualizations and metrics for evaluating calibration and sharpness.
- The method consistently produces well-calibrated uncertainty estimates when applied to Bayesian neural networks.
- The authors apply the method to time series forecasting and reinforcement learning, framing calibration as useful for reliable, interpretable, and interactive machine learning systems.