Maximum Entropy Generators for Energy-Based Models

• Energy-based model learning is challenging, and MCMC sampling can have long mixing times for high-dimensional data.Source: §1 · p. 1
• MEG trains an energy function together with an approximate sampler, using a generator network or using the generator to initialize a latent-space Markov chain.Generator training maximizes output entropy using nonparametric mutual-information techniques.Source: §1 · p. 1
• On CIFAR-10, MEG is competitive with WGAN-GP on Fréchet Inception Distance and Inception Score, and captures all 104 4-StackedMNIST modes at correct frequencies.Source: §1 · p. 2
• On KDD99, the learned energy functions perform as well as task-designed state-of-the-art anomaly detectors and outperform other energy-based and generative models.Source: §1 · p. 2
• MEG produces sharp images with competitive Inception and FID scores, captures more modes than standard GANs, and addresses mode dropping and spurious modes.The paper also reports competitive anomaly-detection results.Source: §1 · p. 2

• An energy function represents the negative log of an unnormalized estimated density, while the partition function normalizes the model distribution.Source: §2 · p. 2
• The maximum-likelihood gradient compares data-distribution and model-distribution energy gradients, which match in expectation at convergence.Source: §2 · p. 2
• Training requires samples from the model distribution, and converting an energy function into a generative process is difficult.Source: §2 · pp. 2–3
• MCMC mixing can be difficult when high-dimensional distributions concentrate near separated modes, because local moves rarely traverse low-probability regions between them.Source: §2 · p. 3

3 Maximum Entropy Generators for Energy-Based Models

• MEG replaces direct sampling from the energy model with an amortized generator G whose outputs define pG from a simple latent prior.The generator is tuned to minimize KL(pG||pθ), expressed as minimizing generated-sample energy while maximizing output entropy.Source: §3 · p. 3
• Because G is deterministic, mutual information between its latent input and output equals output entropy; the method estimates it with a Jensen-Shannon-based estimator.The authors report that this estimator works better in practice than the KL-based estimator.Source: §3 · p. 3

3.1 Improving training stability

• A zero-centered gradient penalty from the score-matching objective pushes data points near critical points, generally local minima, of the energy function.The full score-matching objective is hard to optimize with deep networks because estimating the Hessian diagonal gradient is difficult; the method uses its first term.Source: §3.1 · p. 4

3.2 Improving sample quality via latent space MCMC

• Latent-space MCMC uses MALA proposals with Metropolis-Hastings acceptance to bias sampling toward high-density regions, and the authors argue the latent manifold is flatter than data space.The MCMC-based approach is applicable to arbitrary latent distributions.Source: §3.2 · pp. 4–5

4.1 Visualizing the learned energy function

• On synthetic 2D datasets, MEG learns sharp distributions without spurious modes, as shown by the density visualizations.The datasets are swiss roll, 25-gaussians, and 8-gaussians; density estimates use a sample-based approximation of the partition function.Source: Figure 2 · p. 5, §4.1 · pp. 5–6

4.2 Investigating Mode Collapse

• MEG captures all tested StackedMNIST modes and has a significantly smaller KL divergence than WGAN-GP; the four-stack test covers 104 modes.The mode-count and KL comparison is reported in Table 1.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Modeling Natural Images

• On CIFAR-10, MEG samples are comparable to WGAN-GP, and latent-space MCMC improves their perceptual quality.The paper also reports qualitative samples on 64x64 cropped CelebA.Source: §4.3 · pp. 6–7, Table 2 · p. 7

4.4 Anomaly Detection

• On KDD99, MEG outperformed the previous state-of-the-art energy-based model DSEBM by +0.1990 F1 score and was comparable to DAGMM, the current state-of-the-art model specifically designed for anomaly detection.MEG is a general-purpose energy-based model. Its reported values are derived from 5 runs, with metrics averaged over each run’s last 10 epochs.Source: §4.4 · p. 7, Table 3 · p. 7
• On MNIST anomaly detection, MEG outperforms VAEs and is comparable to BiGAN-based methods using AUPRC.Each digit class is treated as anomalous in turn, with the other nine digits treated as normal.Source: §4.4 · p. 7, Table 4 · p. 8
• The anomaly-detection experiments demonstrate competitive energy-function quality; the authors do not claim state-of-the-art performance on that task.The evaluation covers KDDCUP and MNIST.Source: §4.4 · pp. 7–8

5 Related Work

• Contrastive divergence and persistent contrastive divergence approximate the negative-phase log-likelihood gradient, while score matching avoids computing the partition function.These approaches include CD-k, Persistent CD, and score matching for non-normalized graphical models.Source: §5 · p. 8
• Kim and Bengio maximize generator entropy through hidden-activation log-variance, whereas Dai et al. use inverse-model entropy minimization or isotropic Gaussian assumptions.MEG instead uses a tight mutual-information estimator without assumptions about the data distribution.Source: §5 · p. 8
• EBGAN’s discriminator signal may not vanish when generated and data distributions match, potentially driving the generator away from the true data distribution.Dai et al. argue that the EBGAN objective does not guarantee recovery of the true energy function.Source: §5 · p. 8

6 Conclusion

• MEG combines an energy model for energy estimates with a generator that produces fast approximate samples.The framework maximizes generator-output entropy using a nonparametric mutual-information lower-bound estimator.Source: §6 · p. 8
• MEG learns useful energy estimates, supported by toy 2D visualizations and performance in unsupervised anomaly detection.Source: §6 · p. 8
• MEG generates samples of high perceptual quality by Inception Score and Fréchet Inception Distance measures, and is reported robust to mode dropping and spurious modes.Source: §6 · p. 8

Appendix A Training Algorithm

• Algorithm 1 alternates nϕ energy-function updates with generator and entropy-statistics updates within each training iteration.The energy updates use score-penalty coefficient λ; the algorithm specifies λ = 0.1 and nϕ = 5 by default.Source: Appendix A · p. 12
• Each iteration samples real-data and latent minibatches, then uses per-dimension shuffling of latent samples before generating fake samples.The algorithm uses minibatch size m and forms generated samples as Gω(z).Source: Appendix A · p. 12
• Adam updates the energy function, generator, and entropy-statistics function using shared hyperparameters α, β1, and β2.The defaults are α = 0.0001, β1 = 0.5, and β2 = 0.9.Source: Appendix A · p. 12

Appendix B MCMC sampling in visible vs latent space

• Visible-space MALA chains mix poorly and are attracted to spurious modes, unlike latent-space chains, which alter image semantics without producing spurious modes.Source: Figure 3 · p. 13
• The figure compares samples at the beginning and end of MALA chains in visible space (top) and latent space (bottom).Source: Figure 3 · p. 13
• Latent-space MCMC changes semantic image attributes, while visible-space MCMC is reported to have poor mixing.Source: Figure 3 · p. 13