PaperlayerView exact revision on arXiv

Source-linked AI summary

Confidence-Aware Learning for Deep Neural Networks

Jooyoung Moon, Jihyo Kim, Younghak Shin, Sangheum Hwang

arXiv:2007.01458v3cs.LGstat.ML

TL;DR

Overconfident neural-network predictions limit reliability in safety-critical applications, while existing confidence-estimation methods often add computational costs or require customized architectures. The paper introduces Correctness Ranking Loss, which regularizes class probabilities for ordinal confidence ranking, and reports well-ranked estimates that support OOD detection and active learning. The method has a training cost close to conventional classification, works with existing architectures, and uses a single inference.

  • Problem

    Overconfident predictive probabilities make neural networks unreliable for safety-critical applications, where confidence estimates should indicate when predictions are likely wrong.

  • Method

    Correctness Ranking Loss regularizes class probabilities by comparing sample pairs so confidence estimates follow their expected correctness ordering.

  • Results

    Experiments show that CRL produces well-ranked confidence estimates across benchmark datasets and supports related tasks including OOD detection and active learning.

  • Takeaways & Limitations

    CRL provides a computationally efficient, architecture-independent way to obtain confidence estimates from standard deep classifiers using a single inference.

  • Takeaways & Limitations

    CRL requires counting correct prediction events for all training samples, so incremental learning with newly labeled samples cannot be applied to CRL models.

Abstract

from arXiv · show

Despite the power of deep neural networks for a wide range of tasks, an overconfident prediction issue has limited their practical use in many safety-critical applications. Many recent works have been proposed to mitigate this issue, but most of them require either additional computational costs in training and/or inference phases or customized architectures to output confidence estimates separately. In this paper, we propose a method of training deep neural networks with a novel loss function, named Correctness Ranking Loss, which regularizes class probabilities explicitly to be better confidence estimates in terms of ordinal ranking according to confidence. The proposed method is easy to implement and can be applied to the existing architectures without any modification. Also, it has almost the same computational costs for training as conventional deep classifiers and outputs reliable predictions by a single inference. Extensive experimental results on classification benchmark datasets indicate that the proposed method helps networks to produce well-ranked confidence estimates. We also demonstrate that it is effective for the tasks closely related to confidence estimation, out-of-distribution detection and active learning.

1. Introduction

Deep neural networks often produce overconfident probabilities, limiting safety-critical deployment and motivating confidence estimates that reflect both correctness likelihood and ordinal ranking. The paper proposes Correctness Ranking Loss to regularize class probabilities efficiently, improving confidence ranking and related tasks such as OOD detection and active learning.

  • Motivation: Overconfident probabilities can make deep neural networks unreliable in safety-critical applications, where models must indicate when predictions are likely wrong.The paper frames this requirement as a model knowing what it does not know.
  • Confidence estimation: Confidence quality has two distinct aspects: calibration reflects correctness likelihood, while ordinal ranking orders predictions by their likelihood of being correct.The paper notes that calibration alone is insufficient because it is orthogonal to classification accuracy and ranking performance.
  • Confidence estimation: Ordinally ranked confidence estimates support related tasks including failure prediction, selective classification, and out-of-distribution detection.The paper focuses on obtaining good predictions in terms of ordinal confidence ranking.
  • Method: Correctness Ranking Loss compares randomly selected sample pairs and penalizes confidence estimates that violate their expected correctness ordering.The loss is minimized when samples more likely to be correct receive greater confidence estimates.
  • Method: CRL adds one loss value during training, works with existing architectures, and produces confidence estimates through a single inference.The paper presents this as providing high-quality confidence estimates with little computational increase.
  • Results: Experiments across benchmark datasets and architectures show that CRL produces well-ranked confidence estimates and performs effectively on OOD detection and active learning.The results compare CRL with methods designed specifically for confidence estimation.

2. Related Work

Prior confidence-estimation methods include Bayesian approaches, softmax-based techniques, architectural confidence heads, and ensembles. The paper positions its method as a standard-architecture alternative that avoids additional training or inference costs while producing strong confidence estimates.

  • Bayesian approaches: Bayesian methods represent predictive uncertainty through distributions over neural-network parameters and approximate posterior inference.The related work lists Laplace approximation and Markov-chain methods as examples.
  • Non-Bayesian approaches: Standard classifiers can use softmax outputs for confidence estimation, including threshold-based methods for ordinal ranking and OOD detection.The paper describes this approach as a simple baseline for both tasks.
  • Architectural methods: Some methods add a separate confidence-output node, but their estimates are generated independently of class probabilities and depend on that node’s predictive performance.These methods target OOD detection or ordinal ranking through augmented architectures.
  • Ensembles and training dynamics: Deep Ensembles provide another confidence-estimation approach, while Average Early Stopping addresses impairment of confidence for easily learnable samples during SGD training.These approaches rely on multiple models or predictions rather than a single standard-classifier inference.
  • Positioning: Compared with prior studies, the proposed method avoids architecture augmentation and additional computational costs for training and inference.It turns a standard classification network into a strong confidence-estimation baseline.

3. Confidence-Aware Learning

This section formalizes ordinal confidence ranking and motivates Correctness Ranking Loss (CRL), which uses training correctness frequencies to regularize confidence estimates efficiently.

  • 3.1. Problem Statement: Ordinal ranking evaluates whether higher-confidence predictions are more likely correct than lower-confidence predictions.Confidence functions include maximum class probability, negative entropy, and margin.
  • 3.2. Motivation: Estimating each sample’s true correctness probability is impractical because the data distribution is unknown and training progressively biases the classifier toward its dataset.The method therefore uses observed correctness frequencies during training as an approximation.
  • 3.3. Correctness Ranking Loss (CRL): CRL compares randomly selected sample pairs and penalizes confidence rankings that contradict their relative probabilities of being classified correctly.The loss is designed to enforce greater confidence for samples with higher correctness frequency.
  • 3.3. Correctness Ranking Loss (CRL): The pairwise margin is |c_i − c_j|, so pairs with larger differences in correctness frequency impose stronger ranking requirements.For c_i > c_j, no CRL penalty applies when κ_i exceeds κ_j + |c_i − c_j|.
  • 3.3. Correctness Ranking Loss (CRL): The training objective combines cross-entropy and CRL, while mini-batch sampling and a few pairs reduce the cost of evaluating all possible pairs.Confidence estimates are computed from softmax outputs using maximum probability, negative entropy, or margin; negative entropy is min-max normalized.

4. Experiments

Experiments evaluate CRL across ordinal ranking, calibration, OOD detection, and active learning using benchmark datasets and established architectures. CRL generally improves confidence quality and supports strong performance across these tasks.

  • Evaluation metrics: Evaluation measures confidence quality through ranking metrics including AURC and E-AURC, plus calibration metrics including ECE, Brier score, and NLL.AUPR-Error and FPR-95%-TPR are also used for ordinal ranking evaluation.
  • Ordinal ranking: CRL improves classification accuracy and confidence estimates, outperforming competing methods in 7 of 9 experiments on AURC.For DenseNet on CIFAR-100, CRL-softmax achieves 1.43% higher accuracy than Baseline and improves all reported confidence metrics.
  • Ordinal ranking: CRL regularizes training by improving ordinal ranking even when the classifier slightly overfits NLL on CIFAR-100.The NLL and AURC curves support the reported regularization effect.
  • Out-of-distribution detection: CRL models perform better in most OOD detection comparisons, while CRL combined with ODIN outperforms ODIN with Baseline in all but one experiment.For DenseNet with SVHN in-distribution and TinyImageNet OOD, CRL-softmax reaches FPR-95%-TPR 7.99 versus 19.93 for Baseline ODIN.
  • Active learning: CRL-softmax leads active learning performance on CIFAR-10, reaching 80.6% accuracy at stage two and 91.8% at stage six.On CIFAR-100, CRL performs better after stage four and is the only method exceeding 70%, reaching 70.4% accuracy.

5. Conclusion

The paper introduces CRL, a regularization method that improves confidence ranking in deep classifiers and supports OOD detection and active learning.

  • CRL is a regularization method implemented as a ranking loss to alleviate overconfident predictions.It is motivated by correct prediction events during SGD-based training.
  • CRL-trained networks produce well-ranked confidence estimates for tasks related to what the model does not know.The paper demonstrates this particularly for OOD detection and active learning.
  • The method is applied to image classification and could be extended to other classification tasks such as natural language processing.

S1. Experimental Details: Ordinal Ranking

The ordinal-ranking experiments evaluate confidence quality using risk-separation, calibration, likelihood, and error-detection metrics across image-classification datasets and models. CRL consistently improves confidence estimates across confidence functions and remains comparable to methods requiring multiple predictions.

  • Evaluation Metrics: AURC measures risk across coverage, with lower values indicating better separation of correct and incorrect predictions by confidence.E-AURC subtracts the optimal AURC to provide a unitless measure across models.
  • Evaluation Metrics: AUPR-Error evaluates failure prediction by treating misclassified samples as positives in a precision-recall curve.
  • Evaluation Metrics: FPR-at-95%-TPR measures the false positive rate when the true positive rate reaches 95%.
  • Evaluation Metrics: ECE approximates the expected difference between accuracy and confidence by partitioning predicted probabilities into bins.Each bin compares its accuracy with its average confidence score.
  • Experimental Setup: The experiments use CIFAR-10, CIFAR-100, and SVHN with multiple architectures and confidence-estimation methods, including stochastic and snapshot approaches.The datasets are image-classification benchmarks, while MCdropout uses 50 stochastic predictions.
  • Results: CRL improves ordinal confidence quality across maximum probability, negative entropy, and margin confidence functions, often matching or exceeding methods requiring multiple predictions.

S2. Experimental Details: Out-of-Distribution Detection

The OOD-detection experiments evaluate confidence-based separation of in- and out-of-distribution images using standard detection metrics and several datasets and comparison methods. Results include comparable performance across LSUN and its iSUN subset.

  • Evaluation Metrics: Detection error measures the minimum error rate over thresholds separating in-distribution from out-of-distribution samples.
  • Evaluation Metrics: AUROC measures the area under the true-positive-rate versus false-positive-rate curve.
  • Evaluation Metrics: AUPR-In and AUPR-Out use in-distribution and out-of-distribution samples as positives, respectively, in precision-recall evaluation.
  • Datasets: TinyImageNet, LSUN, and iSUN provide out-of-distribution datasets resized to 32 × 32.TinyImageNet contains 200 classes, LSUN contains 10 scenes, and iSUN is a subset of LSUN.
  • Comparison Methods: ODIN combines temperature scaling with small input perturbations to separate in- and out-of-distribution softmax scores.
  • Comparison Methods: Mahalanobis uses feature-based confidence scores, input perturbations, and weighted combinations across network layers for OOD identification.
  • Results: iSUN detection performance is similar to LSUN because iSUN is a subset of LSUN.

S3. Experimental Details: Active Learning

The active-learning experiments compare sampling strategies across stages, while controlling redundancy and initialization. CRL confidence estimates support strong active-learning performance, including the highest reported CIFAR-100 accuracy in the supplied results.

  • Experimental Settings: Active-learning query strategies select samples according to uncertainty, but redundant information can occur among selected samples.
  • Experimental Settings: To reduce redundancy, samples are selected from a random subset of 10,000 unlabeled examples at each stage.
  • Experimental Settings: CRL requires counting correct prediction events for all training samples, so incremental learning with newly labeled samples cannot be applied to CRL models.All models are therefore retrained from scratch at each stage with shared random seeds.
  • Query Strategy: The compared query strategies include random, entropy-based, core-set, and entropy-based MCdropout sampling.
  • Results: Active-learning results are reported over five runs, with stage percentages indicating the labeled-data proportion and means and standard deviations summarized.
Loading 2007.01458v3…