PaperlayerView exact revision on arXiv

Source-linked AI summary

Energy-based Out-of-distribution Detection

Weitang Liu, Xiaoyun Wang, John D. Owens, Yixuan Li

arXiv:2010.03759v4cs.LGcs.AI

TL;DR

OOD detection is difficult because softmax confidence can be overconfident on inputs outside the training distribution. The paper introduces energy scoring for pre-trained classifiers and energy-based fine-tuning, reporting improved benchmark performance, including an 18.03% FPR95 reduction on CIFAR-10.

  • Problem

    Softmax confidence can assign arbitrarily high confidence to inputs far from training data, limiting its suitability for OOD detection.

  • Method

    The framework uses energy as a non-probabilistic OOD score for pre-trained classifiers and as a trainable cost function that shapes energies for in-distribution and OOD data.

  • Results

    18.03% lower average FPR95 was achieved on CIFAR-10 than with softmax confidence, while energy fine-tuning reduced FPR95 by 5.20% versus OE on CIFAR-10 and 10.55% on CIFAR-100.

  • Takeaways & Limitations

    Energy scores provide a simple replacement for softmax confidence and are reported to outperform softmax and hybrid generative baselines across the evaluated OOD benchmarks.

Abstract

from arXiv · show

Determining whether inputs are out-of-distribution (OOD) is an essential building block for safely deploying machine learning models in the open world. However, previous methods relying on the softmax confidence score suffer from overconfident posterior distributions for OOD data. We propose a unified framework for OOD detection that uses an energy score. We show that energy scores better distinguish in- and out-of-distribution samples than the traditional approach using the softmax scores. Unlike softmax confidence scores, energy scores are theoretically aligned with the probability density of the inputs and are less susceptible to the overconfidence issue. Within this framework, energy can be flexibly used as a scoring function for any pre-trained neural classifier as well as a trainable cost function to shape the energy surface explicitly for OOD detection. On a CIFAR-10 pre-trained WideResNet, using the energy score reduces the average FPR (at TPR 95%) by 18.03% compared to the softmax confidence score. With energy-based training, our method outperforms the state-of-the-art on common benchmarks.

1 Introduction

OOD detection is important for reliable machine learning in an open world, but softmax confidence can be arbitrarily high for inputs far from training data. The paper proposes energy scores as a unified alternative that supports both pre-trained classifiers and fine-tuning.

  • OOD detection matters for deploying machine learning models in safety-critical applications such as rare disease identification.
  • Softmax confidence can become arbitrarily high for inputs far from the training data, making it suboptimal for OOD detection.
  • Energy scores are theoretically aligned with input probability density, whereas softmax confidence is mathematically shown to be density-misaligned.
  • The framework uses energy either as a scoring function for any pre-trained neural classifier or as a trainable cost function for fine-tuning.
  • 18.03% lower average FPR95 is achieved on CIFAR-10 with energy scoring than with softmax confidence on WideResNet.
  • The paper provides mathematical background, introduces energy-based OOD detection, evaluates it experimentally, reviews related literature, and discusses broader impact.

2 Background: Energy-based Models

Energy-based models map inputs to scalar energies and can connect those values to probability densities through the Gibbs distribution. In discriminative classifiers, logits provide the basis for defining an input energy without changing the network parameterization.

  • An energy-based model maps each input x to a single non-probabilistic scalar E(x).
  • The Gibbs distribution converts collections of energy values into a probability density p(x).
  • The partition function marginalizes over labels, while T denotes the temperature parameter.
  • A discriminative classifier f(x) maps an input to K real-valued logits used to derive a categorical softmax distribution.
  • For a labeled input, the energy is defined as the negative logit for its class, E(x,y) = −f_y(x).

3 Energy-based Out-of-distribution Detection

The paper uses energy scores to distinguish in- and out-of-distribution inputs, both by replacing softmax confidence at inference and by training models to shape an explicit energy gap. Energy is connected to input density and avoids the softmax score’s bias toward overconfident predictions.

  • Energy as an inference-time score: Negative log-likelihood training pushes down the energy of in-distribution examples through a ground-truth term and a contrastive term over labels.The energy function is a smooth approximation dominated by the ground-truth label energy.
  • Energy as an inference-time score: Energy-based OOD detection assigns lower energies to observed data and higher energies to unobserved data, enabling separation through an energy threshold.Negative energy scores can be used so in-distribution examples have higher conventional scores.
  • Energy versus softmax confidence: The energy score can replace softmax confidence for pre-trained neural networks because it operates in the original logit space rather than a maximum-shifted space.The paper argues that maximum-logit shifting biases softmax confidence and weakens its ability to distinguish in- and out-of-distribution examples.
  • Energy versus softmax confidence: For a CIFAR-10 WideResNet, energy scores separate an in-distribution sample from an SVHN sample more clearly than softmax confidence.Softmax confidence is 1.0 versus 0.99, whereas negative energy is 11.19 versus 7.11.
  • Energy-bounded learning: Energy-bounded learning fine-tunes a classifier to assign lower energies to in-distribution data and higher energies to auxiliary OOD data.The objective combines cross-entropy with separate squared hinge losses using in-distribution and OOD energy margins.
  • Energy-bounded learning: Energy scores form smoother, more distinguishable distributions than softmax scores, which concentrate at high values and exhibit spiky distributions.This pattern is reported for both pre-trained and fine-tuned networks, including energy-bounded learning.

4 Experimental Results

The experiments evaluate energy-based OOD detection across common benchmarks, comparing pre-trained and fine-tuned WideResNet models with softmax-based and competing methods. Energy improves OOD discrimination and often matches or exceeds alternatives while remaining parameter-free at inference.

  • Experimental Setup: The evaluation uses SVHN, CIFAR-10, and CIFAR-100 as in-distribution datasets and six common OOD benchmarks, reporting FPR95, AUROC, and AUPR.The six OOD datasets are Textures, SVHN, Places365, LSUN-Crop, LSUN-Resize, and iSUN.
  • Pre-trained Models: 18.03% lower average FPR95 is achieved by energy than by softmax confidence on CIFAR-10 with inference-time OOD detection.The comparison uses a WideResNet and a pre-trained model without fine-tuning.
  • Fine-tuning: 5.20% lower FPR95 is achieved by energy fine-tuning than by Outlier Exposure on CIFAR-10, increasing to 10.55% on CIFAR-100.Both approaches use the same fine-tuning data and training configurations.
  • Score Distributions: Energy scores produce more distinguishable in- and out-of-distribution distributions, whereas softmax scores concentrate at high values for both groups.This pattern appears for scores from both pre-trained and fine-tuned networks.
  • Comparisons and Practicality: Energy-based detection outperforms hybrid models incorporating generative modeling, and its pre-trained-network version is parameter-free and often comparable to or better than ODIN.The parameter-free score is described as easy to use and deploy.

5 Related Work

Related work includes softmax-based, fine-tuning, generative, and hybrid approaches to OOD detection. The paper positions energy scoring as a parameter-free alternative to methods that can require extensive tuning or difficult density estimation.

  • Pre-trained Models: Softmax confidence is a common OOD baseline, but neural networks can assign arbitrarily high confidence to OOD inputs.Prior methods also include learned confidence branches attached to pre-trained classifiers.
  • Hybrid Models: Table 3 compares the proposed method with hybrid models that combine discriminative and generative modeling.The paper states that energy-based detection outperforms these hybrid models with and without fine-tuning.
  • Practical Considerations: Previous OOD methods may be computationally expensive or require many hyperparameters, whereas the paper describes energy as parameter-free in an OOD-agnostic setting.The related-work discussion contrasts energy with approaches requiring additional tuning or data.
  • Generative Modeling: Generative methods detect OOD examples through estimated in-distribution density, but deep generative models can assign high likelihood to OOD data.This challenges the reliability of density estimates as OOD scores.

6 Conclusion and Outlook

The paper concludes that energy provides a promising framework for OOD detection because its scores align with input density and improve separation between in- and out-of-distribution data. It also identifies applications beyond image classification as future work.

  • Conclusion: Energy assigns lower values to in-distribution data and higher values to out-of-distribution data as a non-probabilistic OOD score.The framework is presented as a replacement for softmax confidence.
  • Conclusion: Energy scores are provably aligned with input density and yield substantially improved OOD detection performance.This alignment is contrasted with softmax confidence scores.
  • Outlook: Future work will explore energy-based OOD detection beyond image classification, including possible applications such as active learning.The paper frames these as directions for future research.

7 Broader Impact

The project aims to improve the dependability and trustworthiness of modern machine learning models, with potential benefits across applications from digital content understanding to transportation and health care.

  • The work targets improved dependability and trustworthiness for modern machine learning models.
  • The authors identify applications spanning digital content understanding, driver assistance, autonomous vehicles, and rare disease identification.
  • The project releases code and presents the framework as an easy-to-use tool for addressing anomalies in the open world.

A Detailed Experimental Results

Table 4 reports OOD detection performance with CIFAR-10 as the in-distribution dataset across individual OOD test datasets.

  • CIFAR-10 is used as the in-distribution dataset for evaluating each OOD test dataset.
  • The table reports OOD detection performance separately for each specific OOD test dataset.
  • Mahalanobis scores are calculated from features of the second-to-last network layer.
  • Bold numbers indicate superior results.

B Details of Experiments

The experiments specify hardware, evaluation procedures, energy-bound settings, and comparisons across datasets and temperature or margin parameters.

  • Details of Experiments: Experiments use PyTorch and NVIDIA Tesla V100 DGXS GPUs, with one fixed-seed fine-tuning run and performance averaged over 10 random batches per OOD dataset.
  • Details of Experiments: Energy fine-tuning takes around 6 minutes on one GPU, each epoch takes 34 seconds, and evaluating six OOD datasets takes approximately 4 minutes.
  • Details of Experiments: Table 5 reports CIFAR-100 OOD detection performance for each specific dataset, using second-to-last-layer features for Mahalanobis scores.
  • Energy Bound Parameters: The optimal min is −23 for CIFAR-10 and −27 for CIFAR-100, while the optimal mout is −5 for both datasets.
  • Details of Experiments: Table 6 compares softmax-based and energy-based approaches with pretrained and fine-tuned WideResNet models trained on SVHN.
  • Parameter Effects: For CIFAR-10 WideResNet, FPR at 95% TPR increases as temperature T becomes larger; margin-parameter effects are shown on log-scaled x-axes.
Loading 2010.03759v4…