Energy-based Out-of-distribution Detection

Introduction

• OOD detection matters for safely deploying models in open-world and safety-critical settings, where inputs may differ from training data and should not be predicted.The introduction gives rare disease identification as an example application.Source: §1 · p. 1
• Softmax-based OOD detection [13] is suboptimal because neural networks can assign arbitrarily high confidence to inputs far from training data, owing to a label-overfitted posterior output space [31].Source: §1 · p. 1

Contributions

• Energy is lower for observed data and higher for unobserved data, aligning with input likelihood, whereas softmax confidence is mathematically biased and misaligned with input density.The paper presents this alignment as a theoretical reason energy is desirable for OOD detection.Source: §1 · pp. 1–2
• The unified framework uses energy as a score for pre-trained classifiers and as a trainable cost for fine-tuning, with square hinge losses shaping an energy gap between in-distribution and OOD training data.Energy can be derived from a discriminative classifier without an explicit density estimator, and the framework can exploit auxiliary outlier data [11].Source: §1 · p. 2
• On CIFAR-100, energy fine-tuning reduces average FPR (at 95% TPR) by 10.55% versus softmax-based fine-tuning [14] and outperforms prior state of the art on six OOD datasets.The fine-tuned model maintains similar in-distribution classification accuracy.Source: §1 · p. 2
• On CIFAR-10 WideResNet, energy scoring reduces average FPR (at 95% TPR) by 18.03% versus softmax confidence [13].Energy can replace softmax confidence at inference for pre-trained networks without additional parameters to tune.Source: §1 · p. 2

• An energy-based model maps each input x to a scalar, non-probabilistic energy E(x), whose values can define p(x) through the Gibbs distribution.Source: §2 · p. 2
• The partition function marginalizes over labels, temperature controls the Gibbs distribution, and free energy is the negative log partition function.Source: §2 · p. 2
• The framework uses energy as a score for a pretrained network or a trainable cost for fine-tuning, and flags inputs as OOD when negative energy falls below a threshold.Source: Figure 1 · p. 3
• A discriminative neural classifier maps an input to K logits, which define a categorical distribution through softmax.Source: §2 · p. 3
• For a classifier f(x), E(x,y) = −f_y(x), while free energy E(x; f) uses the softmax denominator without changing the network parameterization.Source: §2 · p. 3

3.1 Energy as Inference-time OOD Score

• NLL training pushes down in-distribution energy, and the energy score can classify inputs by thresholding −E(x; f), with higher energies treated as OOD.The score is non-probabilistic, computed with logsumexp, and the threshold is selected using in-distribution data; NLL's contrastive term raises energies for other labels.Source: §3.1 · p. 4

3.2 Energy Score vs. Softmax Score

• Softmax confidence was 1.0 vs. 0.99, while energy scores were E(xin) = −11.19 and E(xout) = −7.11 for CIFAR-10 and SVHN examples, respectively.Softmax confidence is equivalent to a special free-energy case with logits shifted by their maximum, a bias that makes it less reliable for distinguishing in- from out-of-distribution examples.Source: Figure 2 · p. 5, §3.2 · p. 5

3.3 Energy-bounded Learning for OOD Detection

• Two squared hinge terms with separate margins train the model to penalize in-distribution energy above m_in and OOD energy below m_out, using unlabeled auxiliary OOD data.The objective combines standard cross-entropy with energy regularization; the authors report that separate margins make optimization more flexible and training more stable than a single hinge margin.Source: §3.3 · pp. 5–6

4.1 Setup

• Experiments use SVHN, CIFAR-10, and CIFAR-100 as in-distribution data and six OOD benchmarks, measuring FPR95, AUROC, and AUPR.The OOD benchmarks are Textures [5], SVHN, Places365, LSUN-Crop [48], LSUN-Resize [48], and iSUN [47]; 80 Million Tiny Images supplies auxiliary outliers, with CIFAR-10 and CIFAR-100 examples removed.Source: §4.1 · p. 6

4.2 Results

• Energy scoring lowers average FPR95 by 18.03% on CIFAR-10 and 8.69% on SVHN versus softmax scoring.These are inference-time comparisons using pre-trained models.Source: §4.2 · p. 7
• Energy fine-tuning lowers FPR95 by 5.20% on CIFAR-10 and 10.55% on CIFAR-100 versus Outlier Exposure (OE) [14].Both approaches use the same training data and learning-rate and batch-size configurations.Source: §4.2 · p. 7
• Pre-trained energy scores form smooth distributions, while softmax scores for in- and out-of-distribution data concentrate at high values.The experiments link energy scores’ greater separation between the two groups to more effective OOD detection.Source: §4.2 · p. 7
• On a pre-trained network, energy scoring is parameter-free and achieves comparable or better performance than ODIN [24] in many cases.Existing pre-trained approaches may require tuned hyperparameters, additional data, or a trained classifier.Source: §4.2 · p. 8, Table 2 · p. 8
• The energy-based method outperforms hybrid models both with and without fine-tuning.The hybrid baselines incorporate generative modeling and label information during training.Source: §4.2 · p. 8
• Increasing temperature makes energy scores less distinguishable; fine-tuning is relatively insensitive to the tested mout range, but too-small min can degrade performance.The results support using T = 1 for parameter-free energy scoring.Source: §4.2 · p. 8
• Energy fine-tuning yields CIFAR-10 test error of 4.98%, versus 5.32% for OE and 5.16% for the pre-trained model.Inference-time energy scoring preserves the pre-trained network’s accuracy because it does not change its parameters.Source: §4.2 · p. 8

Out-of-distribution uncertainty for pre-trained models

• Prior OOD methods are computationally expensive or require tuning many hyperparameters; in contrast, the energy score provides a parameter-free measurement that is easy to use in an OOD-agnostic setting.Source: §5 · p. 9

Out-of-distribution detection with model fine-tuning

• Unlike fine-tuning methods that use auxiliary OOD data, energy training raises anomalous-input energy without altering the semantic class space and works with or without such data.Prior approaches use synthesized GAN data [22] or unlabeled data [14], regularize predictions toward uniformity [14] [22], add background classes, or mine informative outliers [3]; fine-tuning methods lower anomalous-example confidence [2] [9] [27] [38].Source: §5 · p. 9

Generative Modeling Based Out-of-distribution Detection

• Although generative models estimate in-distribution density, they can assign high likelihood to OOD data; energy scores offer a density interpretation through discriminative classifiers that are easier to optimize.Improved metrics include likelihood ratios [34] [37], while generative-model OOD detection is difficult to optimize on large, complex datasets; the paper says standard SGD makes its discriminative method easier to optimize.Source: §5 · p. 9

Energy-based learning

• Unlike JEM’s joint-density objective and probabilistic OOD scores, this work directly optimizes an unnormalized, discriminative energy gap for energy-based OOD scoring.JEM [11] estimates p(x, y) and normalized densities, which can be challenging and unstable to compute; the paper’s formulation does not require proper normalization and allows greater optimization flexibility.Source: §5 · p. 9

• The proposed energy-based framework replaces softmax confidence with a non-probabilistic score that assigns lower values to in-distribution data and higher values to out-of-distribution data.Source: §6 · p. 9
• Energy scores are provably aligned with input density and yield substantially improved OOD detection performance.Source: §6 · p. 10
• Future work could explore energy-based OOD detection beyond image classification, including applications such as active learning.Source: §6 · p. 10
• The authors hope future research will broaden attention to OOD uncertainty estimation from an energy-based perspective.Source: §6 · p. 10

7 Broader Impact

• The project aims to improve the dependability and trustworthiness of modern machine learning models across a wide range of fields and societal activities.The authors view out-of-distribution uncertainty estimation as increasingly critical for consumer and business applications, transportation, and health care.Source: §7 · p. 10
• The authors hope releasing their code will give researchers a new methodological perspective and practitioners an easy-to-use tool for safety against open-world anomalies.Source: §7 · p. 10
• The authors anticipate no negative consequences from the work and hope to continue improving and building on the framework.Source: §7 · p. 10

A Detailed Experimental Results

• Table 4 reports OOD detection performance with CIFAR-10 as the in-distribution dataset.The caption says results are given for each OOD test dataset, but the supplied passage does not include the table values.Source: Table 4 · p. 14
• The Mahalanobis score is calculated using features from the second-to-last layer.Source: Table 4 · p. 14
• Bold numbers in Table 4 indicate superior results.The supplied caption does not identify which methods are superior or provide their values.Source: Table 4 · p. 14

B Details of Experiments

• Experiments used PyTorch on NVIDIA Tesla V100 DGXS GPUs, with one fixed-seed fine-tuning run and performance averaged over 10 random batches per OOD dataset.Energy fine-tuning took around 6 minutes per GPU, with 34 seconds per epoch; evaluating all six OOD datasets took approximately 4 minutes.Source: B Details of Experiments · p. 14
• Tables cover dataset-specific CIFAR-100 OOD performance, using second-to-last-layer features for Mahalanobis scores, and SVHN softmax-versus-energy comparisons with pretrained and fine-tuned WideResNets.Source: Table 5 · p. 15, Table 6 · p. 16
• Optimal min is −23 for CIFAR-10 and −27 for CIFAR-100, while optimal mout is −5 for both datasets.Source: B Details of Experiments · p. 15
• On a CIFAR-10 pretrained WideResNet, FPR at 95% TPR rises as T increases; the figure also examines min and mout during energy fine-tuning.The figure’s x-axes for the margin-parameter analysis are on a log scale.Source: Figure 4 · p. 16