PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, Peter Flach

arXiv:1910.12656v1cs.LGstat.ML

TL;DR

Multiclass classifiers often produce overconfident probabilities, while general-purpose native multiclass calibration methods are limited. This paper proposes Dirichlet calibration and reports best or tied-best performance across 21 datasets × 11 models on all 8 evaluation measures.

  • Problem

    General-purpose native multiclass calibration methods are limited, despite the importance of calibrated probabilities for cost-sensitive classification, human decision making, and autonomous systems.

    Source: §1 · pp. 1–2

  • Method

    Dirichlet calibration is a general-purpose, natively multiclass method derived from Dirichlet distributions and applicable to any probabilistic classifier.

    Source: §3 · p. 4, §5 · p. 9

  • Results

    Across 21 datasets × 11 models, Dirichlet calibration achieved best or tied-best performance on all 8 evaluation measures; ODIR-regularised methods outperformed temperature scaling on many deep neural networks.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The learned Dirichlet calibration parameters provide insights into biases in the uncalibrated model.

    Source: §5 · p. 9

    Reliability diagrams and ECE measures can indicate near-perfect calibration even when predictions ignore the individual instance and reflect only the overall class distribution.

    Source: §2 · p. 4

Abstract

from arXiv · show

Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.

1 Introduction

The introduction motivates post-hoc calibration as a way to improve overconfident classifiers and presents Dirichlet calibration as a general-purpose method for multiclass probabilities. It also highlights distinctions in how multiclass calibration is defined and evaluated.

1 Introduction

  • A classifier is calibrated when instances assigned probability vector p follow class distribution p; this matters for cost-sensitive decisions, human decision making, and autonomous systems.Post-hoc methods learn a calibration map from hold-out validation data to improve predictions that are often overconfident due to overfitting.Source: §1 · p. 1
  • Binary calibration methods include Platt scaling, binning, isotonic calibration, and beta calibration, with Bayesian and near-isotonic extensions, [23], and.The introduction also traces calibration evaluation to the Brier score, reliability diagrams, and proper scoring rules.Source: §1 · pp. 1–2
  • Multiclass calibration has used one-vs-rest binary tasks, whose normalized outputs are not guaranteed calibrated, and neural-network methods including matrix, vector, and temperature scaling [9].Other approaches alter training or inference, including MMCE, SWAG [19], and a regression-to-Gaussian-Process method [20].Source: §1 · p. 2
  • Dirichlet calibration is a general-purpose, natively multiclass post-hoc method for calibrating probabilistic classifiers, and the paper flags overlooked subtleties in multiclass definitions and evaluation.The introduction says Section 2 clarifies these issues, Section 3 presents the method, and Section 4 reports experiments.Source: §1 · p. 2
  • The introduction distinguishes confidence calibration, which concerns only the predicted probability of the most likely class, from broader multiclass calibration.It also notes variation in evaluation metrics and visualizations, motivating the paper’s clarification of these issues in Section 2.Source: §1 · p. 2

2 Evaluation of calibration and temperature scaling

The section defines multiclass calibration and shows how reliability diagrams and ECE measures evaluate it in practice. It illustrates that temperature scaling can improve confidence calibration while leaving classwise biases, and explains why calibration metrics should be considered alongside proper losses.

2 Evaluation of calibration and temperature scaling

  • A classifier is multiclass-calibrated when, conditional on any prediction vector q, each class’s observed frequency equals its predicted probability q_i.Formally, P(Y = i | p̂(X) = q) = q_i for each class i.Source: §2 · p. 2
  • Classwise-ECE is 0.1857 for temperature scaling versus 0.1795 for Dirichlet calibration; the test rejects temperature scaling (p < 0.0001) but Dirichlet calibration’s result depends on significance level (p = 0.016) [25].Figure 1d shows class 2 closer to calibrated after Dirichlet calibration.Source: Figure 1 · p. 3, §2 · p. 4
  • On CIFAR-10, temperature scaling brings the wide ResNet closer to confidence calibration, with confidence-ECE low enough that the test does not reject calibration at α = 0.01 (p-value 0.017) [9] [25].Temperature scaling learns one t > 0 on a hold-out validation set and rescales logits z to z/t before softmax; reliability diagrams show accuracy below confidence in most bins before calibration.Source: Figure 1 · p. 3, §2 · p. 3
  • Although nearly confidence-calibrated, temperature scaling remains far from classwise-calibrated: it overestimates class 2 probability, underestimates class 4 probability, and its single tuneable parameter cannot act differently across classes.Source: §2 · p. 3
  • Reliability diagrams and ECE can favor predictions of the overall class distribution regardless of instance, so evaluation should also use proper losses such as Brier score or log-loss, which reflect calibration and refinement.Post-hoc calibration learns a map on a hold-out validation set to estimate the classifier’s canonical calibration function.Source: §2 · p. 4

3 Dirichlet calibration

Dirichlet calibration models predicted probability vectors with class-conditional Dirichlet distributions and uses Bayes’ rule to define a multiclass calibration map. Equivalent generative, linear, and canonical parametrisations support fitting, neural-network implementation, and interpretation.

3 Dirichlet calibration

  • The method models predicted probability vectors with a different Dirichlet distribution for each class, then combines class likelihoods with class priors to obtain calibrated probabilities by Bayes’ rule.The family was designed to express class- or class-pair biases, include the identity map, admit synthetic canonical examples, and have somewhat interpretable parameters.Source: §3 · p. 4
  • Theorem 1 establishes that the generative, linear, and canonical parametrisations contain exactly the same calibration maps.The linear form uses log-transformed probabilities followed by a fully connected layer and softmax; the canonical form uniquely represents each map.Source: §3 · pp. 4–5

Implementation details

  • Implementation can use neural-network layers, multinomial logistic regression, or Newton–Raphson on the L2-regularised objective, with predicted probabilities clipped above zero when needed.The logistic-regression formulation uses k log-transformed probability features; Newton–Raphson can converge faster when the number of target classes is small enough for a tractable Hessian.Source: §3 · pp. 5–6

Interpretability

  • Canonical parameters offer interpretable probes: the simplex centre maps to c, while each of k facet-centre points is governed by one column of A.Increasing a_ij raises calibrated class i probability and lowers the others, with an effect dependent on uncalibrated class j probability; in the SVHN example, a_3,9 = 0.63 was associated with 26 fewer class-2-for-8 confusions.Source: §3 · pp. 5–6, Figure 2 · p. 5

Relationship to other families

  • For two classes, Dirichlet calibration coincides with beta calibration, and temperature scaling is a member of the Dirichlet family.Unlike matrix scaling [9], Dirichlet calibration applies its fully connected softmax layer to log-transformed probabilities rather than logits; the passage notes matrix scaling’s informational advantage on deep neural networks.Source: §3 · p. 6

Fitting and ODIR regularisation

  • Fitting minimises validation log-loss with L2 regularisation for non-neural models or ODIR regularisation for deep neural networks.ODIR regularises off-diagonal weights and intercepts separately while leaving diagonal weights free to follow class biases; its hyperparameters are tuned by internal cross-validation.Source: §3 · p. 6

4 Experiments

Across non-neural classifiers, Dirichlet_L2 ranked among the best calibrators and led on several measures. In deep networks, Dir-ODIR was the top-ranked general-purpose method, while logit-based methods and off-diagonal class effects improved results in some cases.

4.1 Calibration of non-neural models

  • Dirichlet_L2 was best on log-loss, p-cw-ECE, and accuracy, and was considered calibrated in more than 60% of p-cw-ECE tests.It ranked among the best calibrators on every measure. The comparison covered 21 UCI datasets and 11 classifiers, using nested cross-validation to evaluate six calibration methods.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 Calibration of deep neural networks

  • Zeroing MS-ODIR’s off-diagonal coefficients changed performance little in 6/14 cases but significantly worsened it in the other 8/14.In those eight cases, MS-ODIR beat vector scaling on log-loss, and reruns on two additional train-test splits confirmed the advantage, supporting the importance of pairwise class effects in those cases.Source: §4.2 · pp. 8–9
  • MS-ODIR outperformed Dir-ODIR in 8/14 cw-ECE and 11/14 log-loss cases, though average relative reductions were under 1%.Those reductions were compared with over 30% from the uncalibrated model. MS-ODIR ranked best on log-loss; vector scaling ranked best on cw-ECE, but its average cw-ECE was 2% lower than MS-ODIR’s at the expense of increased log-loss.Source: §4.2 · p. 8, Table 3 · p. 9, Table 4 · p. 9
  • Dir-ODIR had the best average rank among general-purpose deep-net calibrators, without statistical significance; its advantage over temperature scaling varied by dataset and model.Both Dirichlet variants outperformed temperature scaling in most CIFAR-10 cases. On CIFAR-100, Dir-L2 performed poorly, while Dir-ODIR improved cw-ECE over temperature scaling at a minor log-loss cost; experiments used CIFAR-10, CIFAR-100, and SVHN across 11 networks.Source: §4.2 · p. 8, Table 3 · p. 9, Table 4 · p. 9

5 Conclusion

The paper introduces Dirichlet calibration as a general-purpose multiclass method and reports best or tied-best performance across 21 datasets × 11 models on all 8 evaluation measures. It also presents classwise-ECE and ODIR regularization, and identifies dataset shift as a setting where Dirichlet calibration can remain useful.

5 Conclusion

  • Dirichlet calibration extends beta calibration to multiclass problems and can be implemented as a neural-network layer or multinomial logistic regression on log-transformed class probabilities.Although derived from Dirichlet-distributed likelihoods, it does not assume class-conditional probability vectors are Dirichlet-distributed; its parameters can reveal model biases.Source: §5 · p. 9
  • Across 21 datasets × 11 models, Dirichlet calibration achieved best or tied-best performance on all 8 evaluation measures; ODIR-regularized methods outperformed temperature scaling on many deep neural networks.ODIR regularization was introduced for matrix scaling and Dirichlet calibration.Source: §5 · p. 9
  • Classwise-ECE evaluates calibration across all classes, unlike confidence-ECE, which evaluates only the most likely predicted class.Source: §5 · p. 9
  • On many deep nets, Dirichlet calibration learns a map close to the temperature-scaling family, but can reveal dataset shifts and recalibrate predictions for a new context.The authors identify which architectures and training methods yield canonical calibration functions within the temperature-scaling family as an open theoretical question.Source: §5 · p. 9
  • Dirichlet-derived calibration maps suggest exploring other exponential-family distributions for different score types and mixtures of distributions within each class.Source: §5 · p. 9

Supplementary material

The supplementary material lists sections on classifier comparisons and deep neural networks.

E.4 Comparison of classifiers

E.5 Deep neural networks

B Proofs · C Dirichlet calibration

The proofs establish that Dirichlet calibration’s generative, linear, and canonical parameterizations define the same calibration maps, and that temperature scaling is a special case. The following section introduces plots for comparing the representational power of calibration methods.

B Proofs

  • The generative, linear, and canonical Dirichlet calibration families contain exactly the same calibration maps.The proof establishes the equivalence by relating the parameterizations in both directions.Source: B Proofs · p. 13
  • The generative parameterization can be rewritten as a linear-softmax calibration map through log-density algebra.The proof uses softmax to normalize positive vectors and defines linear parameters from the generative parameters.Source: B Proofs · p. 13
  • The linear parameterization can be expressed canonically using softmax invariance to constant-vector shifts.The proof defines canonical parameters and uses the fact that adding a constant to every softmax input leaves the output unchanged.Source: B Proofs · pp. 13–14
  • The canonical parameterization can be mapped back to the generative form by defining its concentration parameters and class weights.The proof specifies these parameters using the canonical matrix, vector, and the multivariate beta function.Source: B Proofs · p. 14
  • Temperature scaling, defined as softmax of logits divided by temperature, is a special case of the Dirichlet calibration map family.The proposition states that this holds for any temperature.Source: B Proofs · p. 14

C Dirichlet calibration

  • The section presents reliability-diagram examples and other plots to help compare Dirichlet calibration’s representational power with other calibration methods.Source: C Dirichlet calibration · p. 15

C.1 Reliability diagram examples · D Experimental setup

Reliability-diagram examples show that aggregate calibration can conceal class-specific errors, while Dirichlet L2 and One-vs.Rest isotonic regression obtain the lowest ECE in the reported examples. The experimental setup implements Dirichlet calibration with Newton–Raphson, multinomial logistic loss, and L2 regularisation.

C.1 Reliability diagram examples

  • For balance-scale, the MLP’s aggregate reliability can mask classwise errors: class 1 is underconfident, while classes 2 and 3 are overconfident at mid-range and underconfident at higher probabilities.The overall diagram’s small gaps between predicted means and true proportions can conceal those contrasting classwise patterns.Source: §C.1 · pp. 15–16
  • On balance-scale and car, Dirichlet L2 and One-vs.Rest isotonic regression obtain the lowest ECE; frequency binning and temperature scaling can worsen calibration.The reported comparisons are for an MLP on balance-scale and AdaBoost SAMME on car.Source: Table 1 · p. 15, §C.1 · p. 15, Table 2 · p. 15
  • On balance-scale, temperature scaling reduces overconfidence for classes 2 and 3 at higher probabilities but worsens it over [0.2, 0.6], despite reducing overall ECE.The comparison concerns the MLP reliability diagrams and the confidence-ECE and classwise-ECE results.Source: §C.1 · pp. 15–16, Table 1 · p. 15
  • On car, uncalibrated AdaBoost SAMME is underconfident for classes 1–3; Dirichlet L2 produces the largest probability spread and narrows mean gaps, while temperature scaling hurts overall performance.Temperature scaling improves ECE for class 1 but harms the other classes overall.Source: §C.1 · pp. 15, 17, Table 2 · p. 15
  • For good and very good classes, uncalibrated outputs and temperature scaling retain low variance, while isotonic and Dirichlet L2 show higher variance; for the acceptable class, only Dirichlet L2 gives the highest mean probability to the correct class.The authors note that the variance observation is not justified by quantitative analysis in this example.Source: §C.1 · pp. 15, 17

D Experimental setup

  • Dirichlet calibration is implemented with standard Newton–Raphson, multinomial logistic loss, and L2 regularisation; logistic-regression implementations can also be applied to log-transformed predicted probabilities.The setup covers experiments with non-neural classifiers and datasets.Source: D Experimental setup · p. 17

D.1 Datasets and performance estimation

The section describes the datasets and a nested cross-validation procedure for estimating performance across dataset, classifier, and calibrator combinations, with results summarized across seven metrics.

D.1 Datasets and performance estimation

  • Table 3 presents each dataset’s brief description and number of samples, features, and classes.Source: §D.1 · p. 17
  • Each dataset used 5 repetitions of 5-fold cross-validation to create 25 test partitions, with an inner 3-fold procedure for classifier and calibrator training.For each test partition, the three inner-fold calibrators predicted its samples, and their predictions were averaged.Source: §D.1 · p. 17
  • Performance was measured with accuracy, Brier score, log-loss, maximum calibration error, confidence-ECE, classwise-ECE, and the ECE metrics’ p test statistic.The 25 resulting measures were averaged.Source: §D.1 · p. 18

D.2 Full example of statistical analysis · E Results

The sections show how calibration methods are ranked and statistically compared using a naive Bayes/log-loss example, then outline the broader final-results analyses across classifiers and metrics.

D.2 Full example of statistical analysis

  • For naive Bayes on log-loss, Dirichlet calibration with L2 regularisation is significantly better than the other methods; the Friedman statistic is 73.8 (p-value 6.71e−14).The comparison is based on the ranking table and its critical-difference diagram.Source: §D.2 · pp. 18–19, Figure 5 · p. 19
  • The example estimates log-loss by averaging 5-times 5-fold cross-validation losses of inner 3-fold aggregated predictions, then ranks calibrators per dataset, averaging ranks for ties.The resulting rank table is used to compute the Friedman test statistic.Source: §D.2 · pp. 18–19
  • The critical-difference diagram uses a Bonferroni-Dunn one-tailed test to calculate the minimum ranking distance for the naive Bayes/log-loss comparison.The diagram presents the average ranks from the ranking table.Source: §D.2 · p. 18, Figure 5 · p. 19
  • The same ranking and testing process is applied to each of the 11 classifiers for every metric.Table 6 presents the final average results across classifiers; the naive Bayes row has rounded average rankings from Figure 5a.Source: §D.2 · p. 18
  • Table 6 reports final average results for all classifiers, with the naive Bayes row matching the rounded average rankings from Figure 5a.Figure 5a shows the critical-difference diagram for log-loss rankings.Source: §D.2 · p. 18, Figure 5 · p. 19

E Results

  • The results section presents metric-specific ranking tables and critical-difference diagrams, selected hyperparameters for three calibrators, calibration comparisons for 11 classifiers, and additional deep-neural-network results.The named calibrators are Dirichlet calibration with L2 regularisation, Frequency binning, and Width binning.Source: E Results · p. 20

E.1 Final ranking tables for all metrics

This section presents final rankings across accuracy, log-loss, Brier score, MCE, confidence-ECE, and classwise-ECE; lower rankings indicate better metric values.

E.1 Final ranking tables for all metrics

E.2 Final critical difference diagrams for every metric … E.5 Deep neural networks

Across the final comparisons, Dirichlet L2 ranks best on accuracy, log-loss, and p-cw-ECE, while classifier rankings vary by model. The appendices also report selected calibration hyperparameters and deep-network reliability findings.

E.2 Final critical difference diagrams for every metric

  • The final method rankings are obtained by grouping each dataset-classifier combination, ranking the calibration methods, and applying Friedman and one-tailed Bonferroni-Dunn tests for critical differences.The group count is n = #datasets × #classifiers.Source: §E.2 · p. 20
  • Dirichlet L2 is best with statistical significance on accuracy, log-loss, and p-cw-ECE, and is among the best methods on other metrics without significant differences within that group.For log-loss, the reported significant ordering is Dirichlet L2, OvR Beta, OvR width binning, then the remaining calibrators.Source: §E.2 · p. 21, Figure 6 · p. 21

E.3 Best calibrator hyperparameters

  • Dirichlet L2’s selected regularisation value is often λ = 1e−3 or lower, including the minimum 1e−7, while OvR frequency and width binning favor 10 and 5 bins, respectively.The passage notes that selection of the minimum regularisation value did not seem to hurt the overall results.Source: §E.3 · pp. 21–22, Figure 8 · p. 22

E.4 Comparison of classifiers

  • The classifier comparison excludes four datasets because at least one classifier could not complete the experiment, leaving 17 datasets; accompanying figures show p-conf-ECE and p-cw-ECE pass proportions.The removed datasets are shuttle, yeast, mfeat-karhunen, and libras-movement.Source: §E.4 · p. 22, Figure 10 · p. 23
  • Across eight metrics, MLP and linear-kernel SVC are always in higher-ranking groups and never the lowest, while random forest and radial-basis SVC appear in both best and worst groups.Random forest is also in the worst group on four measures; radial-basis SVC is in the best group six times and the worst group three times.Source: §E.4 · p. 22, Figure 9 · p. 23

E.5 Deep neural networks

  • The deep-network appendix presents critical-difference diagrams and calibration-method tables for error rate, log-loss, Brier score, MCE, confidence-ECE, classwise-ECE, p-conf-ECE, and p-cw-ECE.The supplied passages identify the scope of these comparisons but do not give their score values.Source: §E.5 · p. 22, Figure 11 · p. 24, Table 13 · p. 24
  • MS-ODIR and vector scaling are compared on log-loss across three dataset-split replications, with MS-ODIR-zero also evaluated by zeroing MS-ODIR’s off-diagonal entries.The best-scoring model in each replication is bolded in the table.Source: §E.5 · p. 22, Table 21 · p. 27
  • On CIFAR-10’s c10_resnet_wide32 model, temperature scaling systematically underestimates class 4 probabilities; the section also shows classwise reliability after Dirichlet calibration.The reliability diagrams compare class 4 after temperature scaling and Dirichlet calibration.Source: §E.5 · p. 24, Figure 12 · p. 25

Related papers

Loading 1910.12656v1…