PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrating Deep Neural Networks using Focal Loss

Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H. S. Torr, Puneet K. Dokania

arXiv:2002.09437v2cs.LGcs.CVstat.ML

TL;DR

Miscalibration makes DNN confidence unreliable, and conventional NLL training is hypothesised to contribute through overfitting. The paper replaces cross-entropy with focal loss, analyses its regularisation effect, and introduces automatic γ selection. Across experiments, focal-loss models are more naturally calibrated while maintaining accuracy, with strong calibration results and improved out-of-distribution detection.

  • Problem

    DNNs can overestimate the correctness of their predictions, making deployed systems unable to trust model confidence.

  • Method

    The paper replaces cross-entropy with focal loss and provides a principled method to choose γ automatically during training.

  • Results

    Focal-loss models are more calibrated than models trained with cross-entropy, label smoothing, MMCE, or Brier loss while maintaining accuracy.

  • Takeaways & Limitations

    Using focal loss can produce naturally calibrated networks and improve out-of-distribution sample detection.

  • Takeaways & Limitations

    Temperature scaling does not calibrate models under data distribution shift, while focal loss in general is not a proper loss for arbitrary target distributions.

Abstract

from arXiv · show

Miscalibration - a mismatch between a model's confidence and its correctness - of Deep Neural Networks (DNNs) makes their predictions hard to rely on. Ideally, we want networks to be accurate, calibrated and confident. We show that, as opposed to the standard cross-entropy loss, focal loss [Lin et. al., 2017] allows us to learn models that are already very well calibrated. When combined with temperature scaling, whilst preserving accuracy, it yields state-of-the-art calibrated models. We provide a thorough analysis of the factors causing miscalibration, and use the insights we glean from this to justify the empirically excellent performance of focal loss. To facilitate the use of focal loss in practice, we also provide a principled approach to automatically select the hyperparameter involved in the loss function. We perform extensive experiments on a variety of computer vision and NLP datasets, and with a wide variety of network architectures, and show that our approach achieves state-of-the-art calibration without compromising on accuracy in almost all cases. Code is available at https://github.com/torrvision/focal_calibration.

1 Introduction

Deep networks can be overconfident because conventional training may overfit NLL, making predictions unreliable. The paper proposes focal loss, automatic γ selection, and experiments showing improved calibration across settings.

  • Motivation: Overconfident predictions are poorly calibrated because their confidence overestimates the likelihood of correctness, undermining downstream trust.
  • Existing approaches: Temperature scaling improves calibration without changing accuracy, but reduces confidence on correct predictions and fails under distribution shift.
  • Proposed approach: Focal loss replaces cross-entropy and regularises the KL divergence, increasing predictive entropy to prevent overconfidence.
  • Proposed approach: The paper provides a principled method to choose γ automatically for each sample during training.
  • Experiments: Experiments compare focal-loss models with cross-entropy, label smoothing, MMCE, and Brier-loss baselines across datasets and architectures.

2 Problem Formulation

The paper formalises calibration by comparing predicted confidence with empirical accuracy. It defines binned and adaptive metrics that quantify calibration for predicted classes and full class distributions.

  • Calibration: A model is perfectly calibrated when confidence equals accuracy conditional on that confidence.
  • Metrics: ECE measures the expected absolute difference between model confidence and accuracy using finite confidence bins.
  • Metrics: Each bin’s accuracy is the fraction of correct predictions, while its confidence is the average predicted confidence.
  • Metrics: MCE records the largest absolute accuracy-confidence difference across bins.
  • Metrics: AdaECE uses bins with evenly distributed sample counts, addressing ECE’s tendency to be dominated by high-confidence bins.
  • Metrics: Classwise-ECE evaluates calibration of probabilities for all classes rather than only the predicted class.

3 What Causes Miscalibration?

The analysis links miscalibration to continued NLL overfitting after classification accuracy saturates. Incorrect test predictions become increasingly confident as softmax distributions sharpen and weight norms grow.

  • Analysis setup: The study uses a ResNet-50 trained on CIFAR-10 with SGD to examine NLL, entropy, classification error, and ECE during training.
  • Curse of misclassified samples: After epoch 150, test NLL rises while train NLL continues decreasing, and the increase is caused by incorrectly classified samples.
  • Peak at the wrong place: Incorrectly classified test samples become increasingly confident because their softmax distributions grow peakier during training.
  • Weight magnification: Cross-entropy encourages weight magnification because its minimum requires the correct-class probability to approach 1, achievable only as ||W|| →∞.

4 Improving Calibration using Focal Loss

Focal loss improves calibration by regularizing confidence and network weights while preserving the objective’s focus on correct classification. Its empirical and theoretical behavior explains reduced NLL overfitting and improved calibration relative to cross-entropy.

  • Motivation: Cross-entropy can overfit NLL by increasing confidence on predictions irrespective of correctness, whereas focal loss changes this training behavior.This overconfidence is strongly related to poor calibration.
  • Mechanism: Focal loss upper-bounds a regularized KL divergence, effectively adding a maximum-entropy regularizer that increases the entropy of predicted distributions.For one-hot targets, minimizing focal loss still leads the predicted distribution toward the target distribution.
  • Empirical observations: On CIFAR-10 ResNet-50 experiments, focal-loss models maintained low test NLL and reduced the peaky wrong predictions associated with cross-entropy miscalibration.Across stopping criteria, focal loss outperformed cross-entropy in both test accuracy and ECE, whereas cross-entropy calibration gains could cost test accuracy.
  • Mechanism: For confident samples, focal loss reduces gradient magnitudes after p crosses a γ-dependent threshold, implicitly regularizing network weights.Initially, gradients can be larger than with cross-entropy for low-confidence samples, then become smaller after the threshold.
  • Hyperparameter choice: Higher γ values produce steeper reductions in g(p, γ), with γ = 3 outperforming γ = 2 and γ = 1 in calibration among the tested focal-loss models.The paper cautions that higher γ is not universally better, because behavior changes for γ ≥ 7.

5 Experiments

Experiments across image and document classification compare focal loss with established calibration baselines, temperature scaling, and multiple architectures. Focal loss generally achieves the lowest calibration errors while maintaining competitive accuracy and shows stronger OoD detection under distribution shift.

  • Experimental Setup: Experiments cover CIFAR-10/100, Tiny-ImageNet, 20 Newsgroups, and SST using several convolutional architectures.The evaluated models include ResNet-50, ResNet-110, Wide-ResNet-26-10, DenseNet-121, and Global Pooling CNNs.
  • Baselines and Calibration: The baselines include cross-entropy, MMCE, Brier loss, and label smoothing with LS-0.05 reported.Temperature scaling is selected by grid-searching temperatures from 0 to 10 using validation ECE.
  • Calibration Results: Focal loss with sample-dependent γ and γ = 3 broadly produces the lowest calibration errors before and after temperature scaling.FLSD-53 generally has the lowest ECE, AdaECE, and Classwise-ECE; differences from most approaches are mostly statistically significant in the reported bootstrap intervals.
  • Accuracy: Focal-loss models obtain competitive classification accuracy across dataset-network pairs.Test-set classification errors are reported for the major loss functions and additional focal-loss variants.
  • Out-of-Distribution Detection: Focal loss yields the highest AUROC scores on SVHN and Gaussian-noise CIFAR-10-C OoD data, including without temperature scaling.ROC plots compare ResNet-110 and Wide-ResNet-26-10 trained on CIFAR-10 before and after temperature scaling.
  • Confidence and Calibration: Sample-dependent focal loss has optimal temperatures mostly between 0.9 and 1.1, whereas cross-entropy and MMCE require temperatures between 2.0 and 2.8.Temperatures near 1 indicate innate calibration; much larger temperatures can make predictions generally underconfident.

6 Conclusion

The paper presents focal loss as an alternative to cross-entropy that produces naturally calibrated classification networks while maintaining accuracy. It attributes this behavior to entropy maximization and implicit weight regularization, and reports improved OoD detection alongside i.i.d. calibration.

  • Conclusion: Focal loss can produce more naturally calibrated classification networks than cross-entropy while maintaining accuracy.The conclusion frames focal loss as an alternative training loss for calibrated models.
  • Mechanism: Focal loss implicitly maximises entropy while minimising KL divergence between predicted and target distributions.This is presented as a theoretical property of the loss.
  • Mechanism: Focal loss naturally regularises network weights during training, reducing NLL overfitting and improving calibration.The conclusion links the loss design to reduced overfitting of negative log-likelihood.
  • Empirical Findings: Focal-loss models are better calibrated under i.i.d. assumptions and can be better at detecting OoD samples.The OoD experiments use CIFAR-10 as in-distribution data and SVHN and CIFAR-10-C as OoD datasets.

7 Broader Impact

The paper argues that calibrated neural networks can improve the reliability of real-world applications. It connects calibration with alerting users when predictions may be untrustworthy and discusses possible fairness benefits.

  • Impact: Calibrated neural networks can improve model reliability in real-world applications.The passage presents calibration as relevant to deployment.
  • Impact: Calibration can help deploy models that alert users when predictions may not be trustworthy.This is described as a practical use of calibrated confidence estimates.
  • Impact: The authors believe research on better-calibrated models may help improve fairness.They do not directly identify a situation in which calibrated neural networks negatively affect society.

B Relation between Focal Loss and Entropy Regularised KL Divergence

Focal loss can be interpreted as combining cross-entropy's KL-divergence minimization with an entropy-increasing regularization effect. In one-hot classification, this favors accurate but less overconfident predictions.

  • Focal loss trades off minimizing KL divergence against maximizing prediction entropy, with the trade-off controlled by γ.The loss is related to cross-entropy through an entropy term that becomes stronger as γ increases.
  • For one-hot targets, focal loss encourages high probability on the ground-truth class without making that probability excessively high.The KL term favors assigning probability to the correct class, while the entropy term discourages extreme confidence.
  • Focal-loss solutions in binary classification are more entropic and closer to 0.5 than cross-entropy solutions, depending on γ.The paper reports this behavior across different target probabilities q.
  • The binary focal-loss optimum is defined by minimizing the displayed objective over x in [0,1].The objective uses q and γ to determine the predicted probability x.

C Focal Loss and Cross-Entropy on a Linear Model

A simplified linear-model analysis examines how focal loss differs from cross-entropy in calibration and weight behavior. It links focal loss's smaller gradients on confident samples to lower weight magnification and lower confidence on mistakes.

  • Setup: The analysis uses a generalized linear model without batch normalization or weight decay to isolate how weight magnification affects miscalibration.The synthetic binary classification setup uses linearly separable Gaussian clusters with 10% label flips and a two-parameter logistic regression model.
  • High Confidence for mistakes: Larger weights produce larger logits and high confidence even on misclassified test points, making cross-entropy mistakes more confident than focal-loss mistakes.The paper uses this mechanism to connect weight magnification with miscalibration.
  • Weight Magnification: Focal loss implicitly regularizes the last-layer weights by reducing gradient norms for confident samples relative to cross-entropy.This provides the proposed explanation for its lower weight magnification.
  • Weight Magnification: Cross-entropy and focal loss learn the same decision regions, but cross-entropy produces much larger weight norms.The comparison separates classification boundaries from confidence calibration.
  • Principled γ selection: A sample-dependent γ can be selected using a Lambert-W-based threshold so that focal-loss gradients are no larger than cross-entropy gradients above p0.Proposition 2 gives γ* for p ≥ p0 and specifies when equality occurs.

E Dataset Description and Implementation Details

The experiments cover image and document classification datasets and use standard, dataset-specific training procedures. Models are evaluated with calibration and accuracy metrics under specified optimization settings.

  • Datasets: The image experiments include CIFAR-10, CIFAR-100, and Tiny-ImageNet, while NLP experiments use 20 Newsgroups and SST Binary.CIFAR-10 and CIFAR-100 each contain 60,000 images; the excerpt also specifies dataset-specific model and split details.
  • Image training: CIFAR models are trained with momentum SGD for 350 epochs using a staged learning-rate schedule, batch size 128, and crop and flip augmentation.The schedule is 0.1, 0.01, and 0.001 over successive training periods.
  • NLP training: The NLP models use Adam for 20 Newsgroups and AdaGrad with weight decay for SST Binary, with GloVe embeddings and validation-selected checkpoints.The excerpt specifies 50 epochs for 20 Newsgroups and 25 epochs for SST Binary.
  • Evaluation: The study reports ECE, Ada-ECE, Classwise-ECE, NLL, MCE, and classification error, including evaluations before and after temperature scaling.MMCE uses λ = 2 in the reported experiments.

F Additional Results

Additional experiments compare focal-loss variants, calibration metrics, and accuracy across datasets and architectures. Sample-dependent focal loss broadly achieves the lowest calibration errors while maintaining competitive accuracy.

  • Focal-loss variants: Fixed γ = 3 gives the best ECE among fixed-γ focal-loss models.This agrees with the paper's expectation that stronger regularization improves calibration in this comparison.
  • Focal-loss variants: The sample-dependent policy FLSD-532 uses γ = 5 for p̂_i,y_i ∈ [0, 0.2), γ = 3 for [0.2, 0.5), and γ = 2 for [0.5, 1].The policy is reported as producing competitive results.
  • Reported metrics: The additional tables report calibration metrics both before and after temperature scaling, with optimal temperatures selected by cross-validating ECE.They cover adaptive ECE, Classwise-ECE, ECE, NLL, MCE, error, and Top-1/Top-5 accuracy.
  • Accuracy: Sample-dependent γ produces the highest Top-5 accuracy on all CIFAR-100 models and the second-highest accuracy on Tiny-ImageNet.On Tiny-ImageNet, it is only marginally below the highest accuracy.
  • Calibration results: FLSD-53 broadly produces the lowest calibration errors across the additional comparisons.This pattern is reported for ECE, AdaECE, and Classwise-ECE bar plots with 90% bootstrap confidence intervals.
  • Calibration results: Several pre-temperature-scaling differences between cross-entropy and focal loss are statistically significant.The significance is reported in quite a few cases rather than universally.

H Focal Loss is Confident and Calibrated

The supplementary experiments examine confidence concentration, feature norms, and early stopping for focal-loss models versus cross-entropy and other calibration losses. Across these views, focal loss preserves confidence on correct predictions while reducing confidence on errors and improves the accuracy–ECE trade-off without relying on early stopping.

  • Confidence after temperature scaling: Focal-loss models have much lower |S99| before temperature scaling but significantly higher |S99| afterward than cross-entropy and MMCE models.With focal loss, the optimal temperature is 1.1, compared with 2.5–2.8 for cross-entropy and MMCE, so confidence decreases less after scaling.
  • Qualitative confidence: Focal loss produces high confidence on correctly classified samples and low confidence on misclassified samples, unlike cross-entropy, which remains highly confident on errors.The qualitative comparison uses ResNet-50 models trained on CIFAR-10 and reports confidence before and after temperature scaling.
  • Feature dynamics: Feature norms are evaluated throughout training for cross-entropy and focal loss with γ = 1, 2, and 3 to compare their training dynamics.The analysis uses the L2 norms of features from the final ResNet block before the linear layer.
  • Early stopping: Focal loss consistently outperforms cross-entropy in both test error and ECE across early-stopping criteria and without early stopping.The criteria select validation models by ECE, NLL, or classification error; validation ECE can improve test ECE while substantially increasing test error.

K Machine Translation: A Downstream Task

The paper evaluates focal loss for Transformer machine translation on WMT 2014 English-to-German, comparing it with hard-target and label-smoothed cross-entropy. Focal loss achieves the best reported calibration and a competitive translation score, with temperature scaling preserving this favorable result.

  • Results: Focal loss outperforms hard-target and label-smoothed cross-entropy on ECE and has a competitive edge on BLEU.The comparison uses Transformer models trained with cross-entropy, label smoothing α = 0.1, or focal loss γ = 1.
  • Results: 26.39 BLEU is the highest observed test-set score for focal loss, versus 26.33 for hard-target cross-entropy and 26.36 for label smoothing.These scores are reported on the English-to-German test set; validation-tuned optimal temperatures do not always maximize test BLEU.
  • Temperature scaling: The focal-loss model has an optimal temperature of 1, matching the label-smoothed model.Figure K.1 shows that low ECE and high BLEU occur at roughly similar temperatures, although validation-selected temperatures need not maximize test BLEU.
Loading 2002.09437v2…