PaperlayerView exact revision on arXiv

Source-linked AI summary

Classifier Calibration: A survey on how to assess and improve predicted class probabilities

Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez, Meelis Kull, Peter Flach

arXiv:2112.10327v2cs.LGstat.ML

TL;DR

Classifier calibration has become difficult to navigate as its literature, methods, metrics, visualisations, and multiclass subtleties have expanded. This survey organises the main concepts and techniques while highlighting how reliability diagrams and calibration measures support method selection and evaluation, alongside important limitations and open problems.

  • Problem

    The calibration literature is sufficiently rich that obtaining and maintaining a clear overview, especially amid multiclass subtleties and differing evaluation practices, is difficult.

  • Method

    The survey combines introductory material with up-to-date technical coverage of calibration concepts, evaluation metrics, visualisation approaches, post-hoc methods, and advanced topics.

  • Results

    The survey presents a broad account of calibration practice, including reliability diagrams for diagnosing miscalibration and a range of post-hoc techniques suited to different miscalibration patterns.

  • Takeaways & Limitations

    Reliability diagrams can help select an appropriate calibration method, while ECE measures can assess whether calibration is needed, subject to binning choices.

  • Takeaways & Limitations

    Post-hoc calibration does not improve robustness against out-of-distribution inputs, and no standard method determines the binning scheme or number of bins for evaluation.

Abstract

from arXiv · show

This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.

1 Introduction and motivation

Classifier calibration assesses whether predicted probability vectors accurately represent observed class frequencies and associated uncertainty. This survey unifies the expanding binary and multiclass literature, including methods, metrics, visualisations, and practical tools.

  • A K-class classifier is calibrated when instances receiving probability vector s have class frequencies approximately distributed as s.This property quantifies predictive uncertainty and confidence.
  • Calibration supports cost-sensitive classification, human decision making, and autonomous systems by making uncertainty estimates usable for decisions.
  • Post-hoc calibration learns a map from hold-out validation data to transform predictions from a previously trained model.Examples include Platt scaling, isotonic calibration, and binning methods.
  • Multiclass calibration is more involved because one-vs-rest calibrators produce unnormalised vectors that may not be multiclass-calibrated after normalisation.Native multiclass extensions of Platt scaling have also been introduced, particularly for neural networks.
  • The survey provides an introductory and up-to-date overview of calibration concepts, methods, proper scoring rules, visualisations, evaluation metrics, and binary and multiclass post-hoc techniques.It was motivated by a literature that had become difficult to overview or maintain systematically.
  • The survey distinguishes full calibration from weaker confidence calibration and supplies PyCalib implementations for metrics, calibration methods, and combined classifier-calibrator pipelines.Its unified perspective addresses variation in methods, metrics, and probability visualisation.

2 A brief introduction to classifier calibration

Classifier calibration means that predicted probabilities match observed frequencies over sufficiently large sets of instances, making uncertainty estimates useful for decisions under varying costs and class proportions. Reliability diagrams and binning reveal calibration patterns, while post-hoc calibration maps can correct miscalibrated scores; multiclass calibration requires additional care.

  • Calibration concept: Calibration requires that predicted probabilities match empirical outcome frequencies across a sufficiently large and diverse set of forecasts.A single forecast cannot establish calibration; repeated forecasts with comparable scores are needed.
  • Visual assessment: Reliability diagrams compare predicted probabilities with observed proportions, with deviations from the diagonal indicating under- or over-estimation.They can display forecast distributions alongside bin-level empirical frequencies or plot empirical probability directly against average prediction.
  • Binning: Binning makes calibration measurable for continuous scores, but bin width trades off information loss against sufficiently precise estimates of event frequencies.In the weather example, merging ten bins into five preserves more information than using two, while higher forecast probabilities remain about 10% too low.
  • Why calibration matters: Well-calibrated probabilities support standard decision rules and allow thresholds to be adjusted for changed misclassification costs or class priors.Post-calibration can avoid relearning a decision rule whenever class skew changes.
  • Calibration methods: Post-hoc methods learn a calibration map from hold-out data, while logistic calibration can transform uncalibrated scores into calibrated scores.For multiclass problems, classwise reliability diagrams and binning are practical tools, but calibration definitions must be relaxed for finite data.

3 From proper scoring rules to calibration

Proper scoring rules evaluate probabilistic predictions against observed outcomes, while their decompositions separate calibration quality from refinement. The survey connects these principles to empirical evaluation and calibration methods.

  • A scoring rule measures how well a predicted probability vector matches the true class vector, with lower values indicating better predictions.
  • Brier score and log-loss are popular proper scoring rules, and both are strictly proper because truthful probabilities minimize expected loss.
  • Log-loss is widely used for training and calibration methods, while isotonic calibration is a notable exception that optimizes Brier score.
  • Proper scoring rules decompose into calibration loss, measuring the gap between predictions and conditional class frequencies, and refinement loss, measuring residual predictive resolution.
  • Empirical decompositions apply to representative test data under the assumption that the test distribution represents the population.
  • A calibrator can potentially reduce calibration loss to zero, thereby decreasing proper-scoring-rule loss, whereas refinement loss depends on ties between instances from different classes.

4 Evaluation and visualisation of classifier calibration

Calibration evaluation combines reliability diagrams, binning, and numerical error measures to inspect how predicted probabilities correspond to observed frequencies. These tools are informative but sensitive to bin choices and should be interpreted alongside proper scoring rules.

  • 4.1 Visualising calibration: Reliability diagrams compare predicted scores with observed proportions of positives across score ranges, usually after discretizing the continuous score space into bins.
  • 4.1 Visualising calibration: Markers, sample-size indicators, histograms, confidence intervals, and alternative graphical layouts provide different ways to inspect calibration information.
  • 4.1 Visualising calibration: Width binning uses equal-width score intervals, whereas frequency binning assigns approximately equal numbers of samples to each bin.
  • 4.1 Visualising calibration: Using bin centres instead of average bin scores can make an uncalibrated model appear calibrated or a calibrated model appear uncalibrated, with discrepancies reaching half the bin width.
  • 4.1 Visualising calibration: Multiclass reliability can be visualized through confidence diagrams or separate one-vs-rest diagrams for each class.
  • 4.2 Evaluation metrics: Binary-ECE is the bin-size-weighted average calibration gap, while maximum calibration error targets the worst bin-wise discrepancy; in the example, binary-ECE = 0.1873.
  • 4.4 Confidence calibration error: ECE and MCE values can vary substantially with the number of bins and may be minimized by predicting the overall class distribution, so proper scoring rules should also be used.

5 Calibration methods

This section systematically reviews post-hoc calibration methods through calibration maps, covering their formulations, learning objectives, predictive functions, empirical behavior, advantages, disadvantages, and implementation guidance.

  • Scope: The review focuses on post-hoc calibration methods that adjust probability outputs separately from initial model training.It presents calibration maps and examines their mathematical formulations, learning objectives, and predictive functions.
  • Evaluation and guidance: Each method is empirically analysed on test cases, with discussion of practical advantages, disadvantages, overfitting, and computational effort.
  • Related work: Related work includes training-time calibration, calibration for other predictive tasks, and task-specific approaches in computer vision and natural language processing.

5.1 Preliminaries

Calibration maps transform uncalibrated classifier outputs into probability vectors using calibration data and an objective function, with training schemes addressing overfitting risks.

  • Calibration maps: A calibration map transforms outputs from an original classifier space S_Y into the probability-vector space P_Y.P_Y contains nonnegative K-dimensional vectors whose entries sum to 1; S_Y may contain probabilities, real-valued vectors, or pre-softmax neural outputs.
  • Learning setup: Calibration maps are learned from paired uncalibrated outputs and labels in a calibration set by optimizing an objective function, typically built from proper scoring rules.
  • Training schemes: Small training and calibration sets can cause overfitting, motivating multifold dataset splits during model and calibrator fitting.
  • Illustration: The test cases use a synthetic dataset whose right-hand visualization gives the Bayes-optimal posterior probability for class 1.

5.2 Two-Class Test Cases

The two-class test cases use a known synthetic distribution and one-dimensional features to visualize calibration behavior, contrasting under-confidence, over-confidence, and an MLP classifier.

  • Experimental design: The experiments use directly sampled instances from a known distribution with one univariate feature, enabling fitted functions and reliability diagrams to be inspected as x-y plots.
  • Synthetic dataset: The synthetic dataset has equally likely classes, with each class generated from a mixture of two normal feature distributions.
  • Reliability diagrams: The generative assumptions allow smooth reliability diagrams to approximate the true reliability behavior using many random samples and small bins.
  • Classifier cases: Under-confidence produces probabilities concentrated away from the extremes, whereas over-confidence yields a shallower reliability curve with sharp transitions near 0 and 1.
  • Classifier cases: The MLP exhibits under-confident regions near s ∈[0,0.1] and s ∈[0.65,1], and is mostly over-confident elsewhere.

5.3 Calibration for Binary Classification

Binary calibration methods map scalar probabilities from [0,1] to [0,1], spanning non-parametric binning and isotonic regression alongside logistic and Beta calibration.

  • Overview: Four widely applied binary methods are reviewed: empirical binning, isotonic regression, logistic calibration, and Beta calibration.
  • Empirical binning: Empirical binning assigns calibrated values to probability intervals using empirical label frequencies and can directly optimize a variant of ECE.
  • Empirical binning: With 10 equal-width bins, binning produces reliability diagrams close to the diagonal but restricts outputs to 10 values, increasing grouping loss.
  • Isotonic regression: Isotonic regression learns non-decreasing segments with varying-width bins and can interpolate between edge values, while minimizing training-set Brier score.
  • Isotonic regression: Isotonic regression can be sub-optimal for non-monotonic reliability diagrams, costly on large datasets, and prone to extreme outputs that may harm log-loss.
  • Platt scaling: Platt scaling is straightforward and fast but restricts calibration to a sigmoid and requires transformations for probabilistic inputs bounded in [0,1].
  • Beta calibration: Beta calibration supports a richer family of maps, including inverse sigmoids and the identity map, but may be inadequate for severely uncalibrated models needing complex maps.

5.4 Calibration for Multi-class Classification

Multiclass calibration methods extend binary approaches with increasingly flexible parameterizations, trading calibration-map richness against overfitting risk. The MLP examples show that richer methods can correct class-specific reliability patterns more effectively than shared or restricted maps.

  • Temperature scaling: Temperature scaling improves some probability regions for classes 1 and 2 but worsens others because it applies the same map despite class-specific reliability diagrams.
  • Temperature scaling: Temperature scaling uses one shared temperature, which can prevent overfitting on small datasets but may be sub-optimal when its calibration-map family cannot represent the reliability diagram.
  • Vector scaling: Vector scaling assigns class-specific temperatures and intercepts, balancing the restrictive temperature model against the richer matrix model and their respective overfitting risks.
  • Matrix scaling: Matrix scaling allows fully linear class-wise logit adjustments, substantially improving class 3 in the MLP example, but its broader map family may overfit.
  • Dirichlet calibration: Dirichlet calibration applies logistic regression to log-probabilities, supports the identity map, and has results close to matrix scaling while retaining small-data overfitting risk.

5.5 Extended methods for post-hoc classifier calibration

Extended post-hoc methods broaden calibration beyond a single global map or standard binning. They use local regions, continuous-map averages, debiasing, and shape restrictions to improve flexibility or calibration efficiency.

  • Local calibration ensembles learn a decision tree over the original feature space and estimate a separate calibrator within each resulting region.
  • Combining scaling with binning uses averages from a pre-fit continuous calibration map and can improve calibration with smaller calibration sets.
  • A debiased estimator can provide better calibration error in sample complexity, while shape-restricted polynomial regression produces flexible non-decreasing univariate maps.

5.6 Methods that can improve calibration during training time

Training-time approaches incorporate calibration directly into model learning or improve it through efficient ensembles. These methods target calibrated models before post-hoc correction or improve calibration under distributional change.

  • Training-time calibration can add a differentiable kernel-mean-embedding regulariser that approximates calibration error to the learning objective.
  • Efficient ensembles train multiple deep networks with lower computational and memory costs and achieve better calibration on out-of-distribution test samples.
  • Modified Dirichlet link functions have also been proposed to improve calibration for Bayesian nonparametric Gaussian-process classifiers.

5.7 Methods for probability calibration for other predictive settings

Calibration extends beyond class probabilities to regression, structured outputs, and conformal prediction. The literature distinguishes global, local, threshold-based, and set-coverage notions across these settings.

  • Regression: For regression, quantile calibration requires predicted quantiles to match the target’s marginal distribution, with both training-time and post-hoc methods available.
  • Regression: Stronger regression notions require calibration of local conditional distributions or groups satisfying threshold conditions rather than only marginal quantiles.
  • Structured outputs: Structured-output calibration includes reducing sequential outputs to binary problems and adapting calibration ideas to domains such as natural language processing and time-series modelling.
  • Conformal prediction: Conformal classification predicts label sets designed to include the true label with probability 1 − α, providing a coverage-based alternative to label-probability calibration.

5.8 Methods designed for other research fields and applications

Calibration concepts are being extended beyond standard classification into related statistical, machine-learning, and AI applications. The survey briefly introduces these connections while leaving comprehensive analysis for future work.

  • Recent work connects calibration to other topics in statistics, machine learning, and artificial intelligence.
  • Calibration procedures have been used to improve posterior distributions for Bayesian non-parametric ensemble models.
  • Semantic segmentation has been studied as an application of calibration improvement.
  • Calibrated classifiers can estimate model performance on out-of-distribution test sets.

6 Hypothesis tests for calibration

The survey reviews goodness-of-fit and resampling-based hypothesis tests for assessing whether predicted probabilities are calibrated. These methods rely on binning or calibration measures and require attention to sample size and finite-sample effects.

  • Finite datasets can show deviations from perfect calibration through sampling effects, motivating tests of whether observed evidence indicates genuine uncalibration.
  • Goodness-of-fit tests compare observed and expected class counts after grouping instances, but sparse groups can make Pearson statistics unreliable.
  • The multiclass Hosmer–Lemeshow extension bins instances using complement probabilities, then compares observed and expected counts across classes.
  • C = 25.3 and p = 0.0003 in the toy example, so the multiclass Hosmer–Lemeshow test rejects calibration at α = 0.05.
  • The Hosmer–Lemeshow test is not recommended for very small samples because experiments found satisfactory power only above 400 instances.
  • A resampling-based test can evaluate calibration according to arbitrary measures such as ECE, log-loss, or Brier score.
  • ≈0.016, so the toy classifier is rejected as calibrated according to ECEclasswise while scores remain fixed and labels are resampled.

7 Concluding remarks

The survey synthesizes classifier-calibration principles, methods, evaluation tools, and open problems. It recommends a workflow combining proper training losses, diagnostic evaluation, hypothesis testing, and post-hoc correction, while emphasizing scope limits around distribution shift.

  • The survey provides a detailed overview of calibration principles, definitions, evaluation, related approaches, and topics beyond standard classification.
  • Calibration research has a history predating machine learning, and navigating its many options requires concepts and tools beyond confidence calibration, temperature scaling, and ECE.
  • Decision boundaries are insufficient for calibration when class priors or misclassification costs may change after training; broader calibration analysis is then needed.
  • A recommended workflow uses proper scoring rules during training, reliability diagrams or ECE for evaluation, hypothesis tests, and post-hoc calibrators when needed.
  • PyCalib provides calibration measurement tools, visualisations, and methods and pipelines for training calibrated classifiers.
  • Open problems include choosing binning schemes and bin counts, robustness to out-of-distribution inputs, and representing second-order uncertainty.
Loading 2112.10327v2…