PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, Peter Flach

arXiv:1910.12656v1cs.LGstat.ML

TL;DR

Multiclass classifiers often produce overconfident probabilities, while existing calibration approaches are limited across model classes or calibrate classes indirectly. The paper introduces Dirichlet calibration and finds best or tied-best performance across 21 datasets, 11 models, and eight evaluation measures.

  • Problem

    Multiclass classifiers often produce overconfident probabilities, motivating calibration methods that evaluate and improve probabilities across all classes.

  • Method

    Dirichlet calibration is a general-purpose native multiclass method derived from Dirichlet distributions and implementable as multinomial logistic regression on log-transformed probabilities.

  • Results

    Dirichlet calibration achieved best or tied-best performance across 21 datasets, 11 models, and all 8 evaluation measures.

  • Takeaways & Limitations

    Dirichlet calibration provides a broadly applicable multiclass calibration method whose fitted parameters also offer insights into model biases.

  • Takeaways & Limitations

    Reliability diagrams and ECE measures alone can reward predictions of the overall class distribution, so proper losses should also be considered.

Abstract

from arXiv · show

Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.

1 Introduction

The introduction defines calibration as agreement between predicted probability vectors and empirical class distributions, emphasizing its importance for decision-making. It motivates Dirichlet calibration as a general-purpose native multiclass method for probabilistic classifiers.

  • Calibration motivation: A classifier is well-calibrated when instances assigned probability vector p have an approximately matching class distribution.Calibration matters for cost-sensitive classification, human decision making, and autonomous systems.
  • Calibration motivation: Most machine-learning algorithms produce overconfident models because of overfitting unless dedicated calibration procedures are applied.The passage gives Laplace smoothing in decision trees as an example of such a procedure.
  • Related work: Binary calibration methods include logistic or Platt scaling, binning, isotonic calibration, and beta calibration.These methods have been extensively studied for binary classifiers.
  • Related work: Multiclass calibration has used k one-vs-rest binary tasks, but normalized outputs from those models are not guaranteed to be calibrated.Recent native multiclass methods include matrix scaling, vector scaling, and temperature scaling, with a focus on neural networks.
  • Contribution: The paper proposes Dirichlet calibration, a general-purpose natively multiclass method applicable to any probabilistic classifier.The introduction also highlights subtleties in multiclass calibration, including the weaker notion of confidence calibration.

2 Evaluation of calibration and temperature scaling

This section defines multiclass, classwise, and confidence calibration, then shows that temperature scaling can improve confidence calibration while leaving classwise biases. It therefore motivates evaluating calibration with multiple measures, including proper losses, rather than reliability diagrams or ECE alone.

  • Calibration definitions: Multiclass calibration requires the empirical class proportions among instances sharing prediction vector q to equal q.Formally, P(Y = i | ˆp(X) = q) = q_i for each class i.
  • Temperature scaling: Temperature scaling learns one t > 0 and predicts probabilities as σ(z/t), decreasing confidence when t > 1 and increasing it when t < 1.The parameter is learned on a hold-out validation set.
  • Temperature scaling: Temperature scaling can become nearly confidence-calibrated yet remain far from classwise-calibrated, systematically overestimating class 2 and underestimating class 4 probabilities.For class 2, predicted probabilities are smaller than observed frequencies in every equal-width bin shown.
  • Comparative evaluation: 0.1857 versus 0.1795: temperature scaling and Dirichlet calibration achieve these overall classwise-ECE values, respectively.The reliability diagram also shows the model becoming closer to class-2 calibration after Dirichlet calibration.
  • Evaluation measures: Reliability diagrams and ECE can reward predicting the overall class distribution for every instance, so evaluation should also include error rate and proper losses such as Brier score.These measures alone cannot establish useful instance-dependent probabilities.

3 Dirichlet calibration

Dirichlet calibration models class-conditional prediction vectors with Dirichlet distributions and yields equivalent generative, linear, and canonical calibration maps. Its linear form supports neural-network implementation, while the canonical form provides unique and partially interpretable parameters.

  • Dirichlet calibration map family: The method assumes the uncalibrated prediction vectors for each class follow separate Dirichlet distributions, combining class-conditional likelihoods with class priors through Bayes’ rule.This produces the generative calibration map by normalising prior-weighted Dirichlet densities.
  • Parametrisations: The generative, linear, and canonical parametrisations define exactly the same family of calibration maps.The canonical parametrisation is unique, unlike the other two forms.
  • Implementation: The linear parametrisation is implemented as component-wise logarithms of predicted probabilities, followed by a fully connected layer and softmax activation.It can therefore be added directly as layers in a neural network.
  • Interpretability: The canonical parametrisation links fitted linear parameters to a unique matrix A and vector c, with parameters describing changes in calibrated probabilities and decision boundaries.Larger off-diagonal coefficients can produce larger changes in the confusion matrix.
  • Relationship to other families: For two classes, Dirichlet calibration coincides with beta calibration, and temperature scaling is a special case within the Dirichlet family.Unlike matrix scaling, Dirichlet calibration applies its fully connected softmax layer to log-transformed probabilities rather than logits.
  • Fitting and ODIR regularisation: Fitting uses regularised multinomial logistic regression, with L2 regularisation generally sufficient for non-neural models and ODIR proposed for deep neural networks.Hyperparameters are tunable by internal cross-validation on validation data.

4 Experiments

Experiments across 21 UCI datasets and 11 classifiers found Dirichlet_L2 consistently strong, ranking best on log-loss, p-cw-ECE, and accuracy. Deep-network experiments showed ODIR can improve calibration over temperature scaling, while logit-based matrix scaling offers only small additional gains over softmax-based calibration.

  • Non-neural classifiers: The non-neural evaluation covered 231 dataset–classifier settings using nested cross-validation and eight measures, including accuracy, log-loss, Brier score, MCE, and ECE variants.The 21 UCI datasets were paired with 11 classifiers.
  • Non-neural classifiers: Dirichlet_L2 was among the best calibrators for every measure and ranked best on log-loss, p-cw-ECE, and accuracy.It was also in the group of best calibrators for the other measures.
  • Deep neural networks: Across deep networks, both Dirichlet variants outperformed temperature scaling in most CIFAR-10 cases, while Dir-ODIR surpassed TempS in CIFAR-100 cw-ECE at a minor log-loss cost.Dir-L2 performed poorly on CIFAR-100, highlighting the effect of ODIR regularisation.
  • Deep neural networks: MS-ODIR outperformed Dir-ODIR in 8 of 14 cw-ECE cases and 11 of 14 log-loss cases, but its average relative reductions were less than 1%.The corresponding average relative reduction from the uncalibrated model exceeded 30%.
  • Deep neural networks: MS-ODIR’s off-diagonal coefficients were meaningful: zeroing them produced significantly worse performance in 8 of 14 cases, where MS-ODIR also beat VecS in log-loss.In the other 6 of 14 cases, MS-ODIR-zero and MS-ODIR performed almost identically because regularisation had effectively zeroed those coefficients.

5 Conclusion

The paper introduces Dirichlet calibration as a general-purpose multiclass method that extends beta calibration, is easy to implement, and yields interpretable model-bias parameters. Across broad comparisons, it performs best or ties for best on all eight evaluation measures, while motivating extensions based on other exponential-family distributions.

  • 5 Conclusion: Dirichlet calibration extends beta calibration to multiclass settings and can be implemented as a neural-network layer or multinomial logistic regression on log-transformed probabilities.Its parameters provide insights into model biases.
  • 5 Conclusion: Across 21 datasets and 11 models, Dirichlet calibration achieved best or tied-best performance on all 8 evaluation measures.The measures include confidence-ECE, classwise-ECE, log-loss, and Brier score.
  • 5 Conclusion: On many deep nets, Dirichlet calibration learns a map close to temperature scaling, yet remains useful after dataset shift through an interpretable calibration map.The conclusion identifies a theoretical question about when canonical calibration functions belong to the temperature-scaling family.
  • 5 Conclusion: Dirichlet-based calibration maps motivate using other exponential-family distributions for score-specific maps and studying mixtures of distributions within classes.This extends the proposed distribution-derived approach beyond Dirichlet calibration.

Supplementary material

The supplementary material includes sections comparing classifiers and examining deep neural networks, both listed on page 11. The paper was published at NeurIPS 2019 in Vancouver, Canada.

  • Supplementary material: The supplementary material lists “E.4 Comparison of classifiers” on page 11.
  • Supplementary material: It also lists “E.5 Deep neural networks” on page 11.
  • Supplementary material: The paper appeared at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019) in Vancouver, Canada.

B Proofs · C Dirichlet calibration

The proofs establish that generative, linear, and canonical Dirichlet calibration parametrisations define exactly the same calibration maps. They also show temperature scaling is a special case of this family, while the next section illustrates its representational power through reliability diagrams and related plots.

  • B Proofs: Theorem 1 shows that the generative, linear, and canonical Dirichlet parametrisations contain exactly the same calibration maps.
  • B Proofs: Positive vectors can be renormalised with softmax applied to their element-wise logarithms, enabling the generative parametrisation to be written in linear form.
  • B Proofs: The linear and canonical forms are equivalent because affine constant shifts vanish under softmax, including the relationship ln σ(x) = x + const.
  • B Proofs: The canonical parametrisation can be converted back to the generative form by setting α_ij = a_ij + 1 and defining b and π accordingly.
  • B Proofs: Temperature scaling is a special case within the Dirichlet calibration map family and therefore functions as a general-purpose calibration method.
  • C Dirichlet calibration: The Dirichlet calibration section presents reliability diagrams and other plots to examine its representational power relative to other calibration methods.

C.1 Reliability diagram examples · D Experimental setup

Reliability diagrams show that Dirichlet L2 often improves calibration across datasets and classifiers, while temperature scaling can help selectively or worsen overall calibration. The experiments use cross-validated reliability analyses and implement Dirichlet calibration with Newton–Raphson, multinomial logistic loss, and L2 regularisation.

  • C.1 Reliability diagram examples: On balance-scale with an MLP, reliability diagrams compare the original classifier with six calibration methods using confidence- and classwise-reliability views.The plots aggregate all test partitions from 5 times 5-fold cross-validation.
  • C.1 Reliability diagram examples: Dirichlet L2 and One-vs.Rest isotonic regression obtain the lowest ECE on balance-scale, whereas One-vs.Rest frequency binning worsens the original calibration.Temperature scaling reduces high-probability overconfidence for classes 2 and 3 but also worsens calibration in another range.
  • C.1 Reliability diagram examples: For AdaBoost SAMME on car, the uncalibrated model is underconfident for classes 1, 2, and 3, with posterior probabilities never exceeding 0.7 despite higher true class proportions.The reliability diagrams evaluate three calibrators on a four-class problem.
  • C.1 Reliability diagram examples: On car, Dirichlet L2 and One-vs.Rest isotonic regression obtain the lowest ECE, while temperature scaling worsens the original calibration overall.Dirichlet L2 produces the largest probability spread and reduces the mean gap between predictions and true class proportions; temperature scaling helps class 1 but harms other classes.
  • C.1 Reliability diagram examples: Boxplots show that uncalibrated predictions for good and very good classes have low variance, while isotonic and Dirichlet L2 produce higher output variance than temperature scaling.The boxplots summarize the full posterior-probability distributions rather than binning them.
  • C.1 Reliability diagram examples: For the acceptable class, only Dirichlet L2 provides the highest mean probability for the correct class among the compared calibration approaches.The passage notes that this advantage is directly observable, unlike the broader variance observation, which lacks quantitative justification there.
  • D Experimental setup: The experimental setup covers varied non-neural classifiers and datasets, implementing Dirichlet calibration with Newton–Raphson, multinomial logistic loss, and L2 regularisation.Existing logistic-regression implementations applied to log-transformed predicted probabilities can also be used.

D.1 Datasets and performance estimation

The evaluation uses a broad set of datasets and estimates performance through nested cross-validation across classifier–calibrator combinations. Results are summarized with seven metrics by averaging measurements from 25 test partitions.

  • Datasets: Table 3 lists each dataset with its sample count, feature count, class count, and brief description.
  • Performance estimation: Each dataset uses 5 times 5-fold cross-validation, producing 25 test partitions for every dataset, classifier, and calibrator combination.
  • Performance estimation: Within each test partition, the corresponding training set undergoes 3-fold cross-validation to train classifiers and validate calibrators with hyperparameters.
  • Performance estimation: Performance is evaluated with 7 metrics: accuracy, Brier score, log-loss, maximum calibration error, confidence-ECE, classwise-ECE, and the ECE metrics’ p test statistic.
  • Performance estimation: The 25 resulting measures are averaged to obtain the reported performance estimates.

D.2 Full example of statistical analysis · E Results

The paper illustrates its statistical-analysis workflow using naive Bayes and log-loss, then reports final rankings, calibration hyperparameters, classifier comparisons, and deep-neural-network results.

  • D.2 Full example of statistical analysis: The worked example uses naive Bayes and log-loss to demonstrate how final rankings and statistical tests are computed.Estimated log-loss averages 5-times 5-fold cross-validation losses from inner 3-fold aggregated predictions.
  • D.2 Full example of statistical analysis: Table 4 ranks calibrators separately for each dataset, assigning tied methods their averaged rank.The resulting sub-indices form the basis for subsequent analysis.
  • D.2 Full example of statistical analysis: The same ranking and testing process is applied across all 11 classifiers and every metric.Table 6 summarizes the final average results, including rounded average rankings for naive Bayes from Figure 5a.
  • D.2 Full example of statistical analysis: For naive Bayes with log-loss, the Friedman statistic is 73.8 with p-value 6.71E-14.These values are reported in Table 4 for the calibration-method ranking comparison.
  • E Results: The results section presents ranking tables for every metric and critical-difference diagrams.It also reports selected hyperparameters for Dirichlet calibration with L2 regularisation, Frequency binning, and Width binning.
  • E Results: The results additionally compare calibration across the 11 classifiers and provide further analyses of deep neural networks.These comparisons are included alongside the ranking and hyperparameter results.

E.1 Final ranking tables for all metrics

This section presents final ranking tables across accuracy, log-loss, Brier score, MCE, confidence-ECE, and classwise-ECE. Rankings are interpreted so that lower values indicate better metric performance.

  • Final ranking tables: Final rankings are provided for accuracy, log-loss, Brier score, MCE, confidence-ECE, and classwise-ECE.The tables are Tables 5–12, with the listed metrics represented across the ranking tables.
  • Interpreting rankings: Lower rankings indicate better metric values across the reported comparisons.For example, higher accuracy and lower log-loss both correspond to lower rankings.
  • Metric coverage: The final tables include separate rankings for accuracy, Brier score, confidence-ECE, log-loss, MCE, and classwise-ECE.These are presented as Tables 5, 7, 9, 6, 8, and 10, respectively.

E.2 Final critical difference diagrams for every metric … E.5 Deep neural networks

Across final metric comparisons, hyperparameter selection, classifier baselines, and deep-network experiments, Dirichlet L2 is a statistically strong calibrator, while classifier performance varies substantially by model and metric. The analyses also examine regularization preferences, calibration-test behavior, and a class-specific failure of temperature scaling on CIFAR-10.

  • E.2 Final critical difference diagrams for every metric: Dirichlet L2 ranked best for accuracy, log-loss, and p-cw-ece with statistical significance, and joined the statistically best group for the remaining metrics.For log-loss, Dirichlet L2, OvR Beta, and OvR width binning differed significantly from the remaining calibrators.
  • E.3 Best calibrator hyperparameters: Dirichlet L2 most often selected λ = 1e−3 or lower, with λ = 1e−7 repeatedly chosen without harming overall experimental results.One-vs.-Rest frequency binning preferred 10 equal-frequency bins, whereas width binning preferred 5 equal-sized bins.
  • E.4 Comparison of classifiers: The classifier comparison excluded four datasets because at least one classifier could not complete the experiment, leaving 17 datasets for uncalibrated comparisons.The removed datasets were shuttle, yeast, mfeat-karhunen, and libras-movement.
  • E.4 Comparison of classifiers: MLP and linear-kernel SVC consistently occupied the higher-ranking classifier group and never the lowest, while random forest and radial-basis SVC varied across measures.Random forest appeared in the worst group for 4 measures, and radial-basis SVC appeared in the worst group 3 times.
  • E.5 Deep neural networks: The deep-network analysis evaluated calibration methods using error rate, log-loss, Brier score, MCE, confidence-ECE, classwise-ECE, p-conf-ECE, and p-cw-ECE.It also compared MS-ODIR, vector scaling, and MS-ODIR-zero on log-loss across three dataset-split replications.
  • E.5 Deep neural networks: Temperature scaling systematically under-estimated class 4 probabilities for c10_resnet_wide32 on CIFAR-10.The comparison used classwise reliability diagrams after temperature scaling and Dirichlet calibration.
Loading 1910.12656v1…