Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One

Will Grathwohl; Kuan-Chieh Wang; Jörn-Henrik Jacobsen; David Duvenaud; Mohammad Norouzi; Kevin Swersky

Source-linked AI summary

Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One

Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, Kevin Swersky

arXiv:1912.03263v3Dec 6, 2019cs.LGcs.CVstat.ML

TL;DR

The paper targets the gap between generative models and strong discriminative classifiers by reinterpreting classifier architectures as joint energy-based models. JEM trains conditional classification and input-density modeling together, and achieves competitive generative and discriminative performance with improved calibration, out-of-distribution detection, and robustness. The main limitation is that EBM training can be unstable and may diverge without careful sampling and optimization.

Problem
Generative models have underperformed hand-tailored discriminative solutions on downstream tasks, despite potential benefits for calibration, missing-data problems, and semi-supervised learning.
Method
JEM reuses classifier logits to define a joint energy-based model, optimizing p(y|x) with cross-entropy and p(x) with SGLD-based training.
Results
JEM achieves performance near the state of the art in generative and discriminative modeling simultaneously, including 96.7% accuracy on SVHN and 72.2% on CIFAR100.
Takeaways & Limitations
The generative component improves uncertainty quantification, out-of-distribution detection, calibration, and adversarial robustness while preserving strong classification performance.
Takeaways & Limitations
EBM training uses unstable gradient estimators and can diverge unless sampling and optimization parameters are tuned correctly.

Abstract

from arXiv · show

We propose to reinterpret a standard discriminative classifier of p(y|x) as an energy based model for the joint distribution p(x,y). In this setting, the standard class probabilities can be easily computed as well as unnormalized values of p(x) and p(x|y). Within this framework, standard discriminative architectures may beused and the model can also be trained on unlabeled data. We demonstrate that energy based training of the joint distribution improves calibration, robustness, andout-of-distribution detection while also enabling our models to generate samplesrivaling the quality of recent GAN approaches. We improve upon recently proposed techniques for scaling up the training of energy based models and presentan approach which adds little overhead compared to standard classification training. Our approach is the first to achieve performance rivaling the state-of-the-artin both generative and discriminative learning within one hybrid model.

1 INTRODUCTION

The paper addresses the gap between generative models’ downstream performance and specialized discriminative solutions by reinterpreting classifier architectures as joint energy-based models. It reports strong generative and discriminative performance alongside improved calibration, out-of-distribution detection, and adversarial robustness.

Generative modeling has been motivated by benefits for semi-supervised learning, missing-data imputation, and uncertainty calibration, but recent deep generative work emphasizes samples and held-out log-likelihood.
Generative approaches lag hand-tailored downstream solutions partly because they diverge from state-of-the-art discriminative architectures.
JEM reinterprets classifier logits to define joint and marginal data densities while retaining modern classifier architectures.
The models outperform previous hybrid approaches and perform on par with or better than hand-tailored methods across generative and discriminative tasks.

2 ENERGY BASED MODELS

Energy-based models represent densities through scalar energies and a partition function, but their normalizing constants are generally intractable. Training therefore relies on sampling-based gradient estimates, including MCMC and SGLD.

An energy-based model expresses a probability density using a scalar energy function and a partition function that normalizes over inputs.
Because the partition function is usually intractable, normalized densities and standard maximum-likelihood training are not straightforward.
Likelihood gradients require expectations over the model distribution, so training commonly uses MCMC to draw approximate samples.
Recent large-scale EBM work uses Stochastic Gradient Langevin Dynamics to approximate these expectations for high-dimensional data.
SGLD commonly starts from a Uniform distribution and uses a polynomially decayed step size, while separately chosen noise and step sizes can bias sampling.

3 WHAT YOUR CLASSIFIER IS HIDING

A classifier’s logits can be reused to define joint and marginal energy-based densities without changing the classifier function. The resulting conditional class probabilities remain the standard Softmax probabilities, while input-density modeling uses the logits’ otherwise-unused offset.

A K-class classifier maps each input to K logits, which parameterize a categorical distribution through Softmax.
The paper reinterprets the same logits to define an energy-based joint distribution over data points and labels, then marginalizes labels to obtain an input density.
The LogSumExp of classifier logits can serve as the energy function for an input data point.
Unlike ordinary classification, shifting all logits for an input changes log pθ(x), allowing the model to represent input density using the logits’ extra degree of freedom.
Dividing the joint density by the marginal input density cancels the partition function and recovers the standard Softmax pθ(y|x).

4 OPTIMIZATION

JEM factorizes training into an exactly optimized conditional classification objective and an SGLD-based input-density objective. Persistent contrastive divergence reduces computation but decreases training stability.

The conditional likelihood is normalized and can be optimized with standard classifier training, whereas the joint and input densities are unnormalized.
JEM optimizes p(y|x) with cross-entropy and log p(x) with an SGLD-based gradient estimator to avoid finite-sampler bias in the conditional objective.
Persistent contrastive divergence provides an order of magnitude computational savings over restarting chains each iteration, at the cost of decreased training stability.

5 APPLICATIONS

JEM applies energy-based training to hybrid generative and discriminative tasks, including calibration, OOD detection, and adversarial robustness. Across these applications, it approaches state-of-the-art performance while retaining competitive classification.

5.1 HYBRID MODELING: JEM performs near the state of the art in generative and discriminative modeling simultaneously, outperforming other hybrid models.The comparison covers CIFAR10, SVHN, and CIFAR100 against hybrid, generative, and discriminative models.
5.1 HYBRID MODELING: JEM is competitive with state-of-the-art generative models on inception score and Fréchet Inception Distance, while achieving 96.7% accuracy on SVHN and 72.2% on CIFAR100.Normalized likelihoods are unavailable, so IS and FID are used as proxies for generative evaluation.
5.2 CALIBRATION: JEM produces nearly perfect calibration on CIFAR100 despite 72.2% accuracy, whereas the same architecture trained as a baseline achieves 74.2% accuracy but is highly overconfident.Unlike Platt scaling, JEM requires no additional training data for calibration.
5.3 OOD DETECTION: JEM outperforms its own and other generative models’ likelihoods for OOD prediction, while tractable likelihood methods are unreliable for this task.For energy-based models, the evaluated predictor is more effective than likelihood-based alternatives; likelihood can fail because high density does not ensure membership in the typical set.
5.4.1 IMPROVED ROBUSTNESS THROUGH EBM TRAINING: JEM delivers improved adversarial robustness over standard classifier training across both L2 and L∞ attacks, although it often remains below adversarial-training baselines.With zero refinement steps, EBM training alone improves robustness; additional refinement steps increase it further, subject to computational constraints.
5.4.1 IMPROVED ROBUSTNESS THROUGH EBM TRAINING: JEM avoids confidently classifying nonsensical images generated from noise to maximize the probability of the class “car.”The baseline confidently classifies unstructured noise, while adversarially trained ResNet images are somewhat structured but unrealistic.

6 LIMITATIONS

The paper identifies substantial practical difficulties in training energy based models, especially unstable gradient estimators and frequent training divergence. These issues require careful tuning and restarts, motivating further methodological improvements.

6 LIMITATIONS: Unnormalized likelihoods make it difficult to verify whether learning is taking place, limiting straightforward monitoring of EBM training.Image samples can help in some domains, but they only assess an approximation to the model and are not generally applicable.
6 LIMITATIONS: JEM’s gradient estimators are unstable and can diverge when sampling and optimization parameters are poorly tuned.The reported models regularly diverged during training and required restarts with lower learning rates or increased regularization.
6 LIMITATIONS: The authors argue that resolving EBM training difficulties could further improve the reported results.They present the existing results as motivation for developing more stable training methods.

7 RELATED WORK

Related work explores classifier-derived EBMs, scalable contrastive-divergence training, and score matching. These approaches differ in how they define densities, train high-dimensional models, and support downstream applications.

7 RELATED WORK: Prior work defined class-conditional EBMs from classifier logits, requiring additional parameters to derive both a classifier and an unconditional model.The paper distinguishes its joint-density interpretation from this alternative formulation.
7 RELATED WORK: Earlier classifier-derived generative models used an MCMC-like procedure to distinguish data from generated negative examples, with reported benefits for adversarial robustness.This line of work is presented as related to, but distinct from, the paper’s formulation.
7 RELATED WORK: Recent EBM methods scale to high-dimensional data with Contrastive Divergence and SGLD, but require many SGLD steps per training iteration.Each sampling step has approximately the computational cost of one standard discriminative-training iteration.
7 RELATED WORK: Score matching fits derivatives of model and data densities rather than the density itself, leaving its utility for the paper’s applications unclear.The authors nevertheless identify score matching as a promising direction for further research.

8 CONCLUSION AND FURTHER WORK

The paper presents JEM as a reinterpretation of standard classifier architectures that combines strong discriminative performance with generative modeling benefits. It also calls for improved EBM training methods.

8 CONCLUSION AND FURTHER WORK: JEM retains the strong performance of state-of-the-art discriminative models while adding benefits associated with generative modeling.The conclusion credits recent techniques for scaling EBM training to high-dimensional data.
8 CONCLUSION AND FURTHER WORK: The authors hope these results encourage improved approaches to training energy based models.They identify training difficulties as an ongoing issue despite the demonstrated utility of the framework.

A TRAINING DETAILS

The paper evaluates JEM’s training and sample quality using SGLD-based procedures, classifier and generative metrics, and conditional or unconditional sampling. Reported results include competitive generative scores and dataset-specific classification accuracies.

A TRAINING DETAILS: JEM training uses PCD with 20 SGLD steps per iteration and reinitializes chains from uniform noise with probability .05.Images are scaled to [−1, 1] and Gaussian noise is added during preprocessing.
A TRAINING DETAILS: Two sampling methods use either class-conditional SGLD followed by discarding labels or direct SGLD under the marginal pθ(x).The second method uses energy E(x) = −LogSumExp_y fθ(x)[y].
B SAMPLE QUALITY EVALUTION: The reported evaluation uses Inception Score and FID as proxies for likelihood, alongside accuracy and qualitative samples on datasets where those metrics are less commonly reported.The authors caution that IS and FID depend strongly on the pretrained classifier and implementation used.

D QUALITATIVE ANALYSIS OF SAMPLES

The analysis examines how JEM’s density and predictive scores relate to generated-sample characteristics and evaluates classifier calibration. Higher log p(x) varies by class and background structure, while higher p(y|x) corresponds to more centered objects.

Visual quality is difficult to quantify because IS and FID favor samples with higher p(y|x), but not necessarily samples with higher log p(x).The passage attributes this mismatch likely to limitations of the evaluation metrics rather than true sample quality.
Automobiles have the highest log p(x) distribution, and all top-100 generated samples by log p(x) belong to that class.
Within each class, higher log p(x) samples tend to have white backgrounds and centered objects, whereas lower-scoring samples have colorful, forest-like backgrounds.
Across classes, higher p(y|x) values correspond to clearly centered objects, while lower values do not.
Expected Calibration Error groups examples by confidence and compares average accuracy with average confidence, with zero indicating perfect calibration.The analysis uses 20 equally spaced confidence buckets.

E.2 FURTHER RESULTS

Further results report calibration, OOD detection, and evaluation details across additional datasets. JEM improves CIFAR10 calibration, remains usable for OOD scoring on a constant dataset, and is assessed with AUROC values.

JEM improves calibration on CIFAR10, including a noticeable boost when trained semi-supervised with 4,000 labeled examples and remaining data treated as unlabeled.In that setting, baseline and JEM accuracy are 78.0% and 74.9%, respectively, and semi-supervised training did not improve generalization.
The OOD experiments use pretrained Glow and IGEBM implementations for likelihood-based, likelihood-gradient, predictive-distribution, and class-conditional comparisons.
On the Constant dataset, generative models mainly fail, while JEM is the only model whose likelihoods yield a predictive score function for OOD detection.Approximate-mass scores could not be obtained from Glow models there because of numerical stability issues.
Table 7 reports OOD detection results using AUROC as the metric.

G ATTACK DETAILS AND FURTHER ROBUSTNESS RESULTS

The robustness experiments evaluate JEM under gradient-based and gradient-free attacks, compare sampling depths and models, and examine training stability. Robustness generally increases with refinement steps, although persistent-chain training introduces instability and trade-offs.

JEM is more robust than standard classifier training across gradient-free boundary and pointwise attacks, and its gradients are not vanishing through sampling.PGD was more successful than the gradient-free methods, indicating that attackers could use the sampling gradients.
Robustness increases as JEM adds sampling steps, with JEM-0 already more robust than the baseline.
Against IGEBM, JEM is more robust under L2 attacks until ϵ = 525, while IGEBM becomes more robust under L∞ attacks after ϵ = 12.IGEBM exceeds the adversarial-training baseline under L∞ after ϵ = 18 and under L2 above ϵ = 280.
The stochastic refinement attack averages predictions over multiple SGLD samples before applying PGD, using 5 samples in the main experiments because 10 was costly at ten refinement steps.Five and ten samples produced very similar results for JEM with one refinement step.
Refinement can correct adversarial perturbations generated against JEM-0 when evaluated with JEM-1 and JEM-10.The transfer-attack experiment tests whether additional refinement restores correct classification.
Persistent contrastive divergence reduces training steps from 80 to 20 and gives a 4x speedup, but it is less stable than the fastest short-run sampler.The model can also encode semantic information in chain age, with young CIFAR10 chains resembling frogs and old chains resembling cars.
Training remains unstable because high-energy replay-buffer samples can create gradients orders of magnitude larger than those from other examples and cause divergence.Heuristics including gradient and energy clipping did not stabilize training without harming generative or discriminative performance.