PaperlayerView exact revision on arXiv

Source-linked AI summary

How Good is the Bayes Posterior in Deep Neural Networks Really?

Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Świątkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, Sebastian Nowozin

arXiv:2002.02405v2stat.MLcs.LGstat.CO

TL;DR

The paper asks why increasingly accurate Bayesian deep-learning inference has not translated into better practical predictions. It uses careful SG-MCMC experiments and diagnostics to compare Bayes and tempered posteriors, finding that cold posteriors improve predictions while raising unresolved questions about priors, likelihoods, and inference.

  • Problem

    The paper examines whether accurate Bayes posterior predictives actually improve predictions in popular deep neural networks.

    Source: §1.2 · p. 2

  • Method

    The authors use SG-MCMC sampling, diagnostic tools, and experiments on deep networks to assess posterior accuracy and test hypotheses about cold posteriors.

    Source: §1.2 · p. 2, §3.1 · p. 3, §3.3 · p. 4

  • Results

    Cold posteriors improve predictions over the Bayes posterior, while existing SG-MCMC methods accurately approximate the posterior.

    Source: §1.1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The results shift attention from merely improving posterior approximations toward understanding why cold posteriors perform better.

    Source: §8 · p. 9

    The experiments implicate the prior in the cold-posterior effect but do not identify a simple variance-scaling fix, and the prior may not be its only cause.

    Source: §8 · p. 9

Abstract

from arXiv · show

During the past five years the Bayesian deep learning community has developed increasingly accurate and efficient approximate inference procedures that allow for Bayesian inference in deep neural networks. However, despite this algorithmic progress and the promise of improved uncertainty quantification and sample efficiency there are---as of early 2020---no publicized deployments of Bayesian neural networks in industrial practice. In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD. Furthermore, we demonstrate that predictive performance is improved significantly through the use of a "cold posterior" that overcounts evidence. Such cold posteriors sharply deviate from the Bayesian paradigm but are commonly used as heuristic in Bayesian deep learning papers. We put forward several hypotheses that could explain cold posteriors and evaluate the hypotheses through experiments. Our work questions the goal of accurate posterior approximations in Bayesian deep learning: If the true Bayes posterior is poor, what is the use of more accurate approximations? Instead, we argue that it is timely to focus on understanding the origin of the improved performance of cold posteriors.

1. Introduction

The paper questions whether Bayes posteriors improve predictions in deep networks despite theoretical and empirical reasons to expect posterior averaging to help. It studies posterior predictives, approximate inference, and the cold-posterior effect.

1.1. Bayesian Deep Learning

  • Bayesian deep learning averages predictions over likely parameter settings rather than selecting a single model.The posterior predictive is approximated by averaging predictions from sampled parameter values.Source: §1.1 · pp. 1–2
  • For deep neural networks, cooling the posterior below T = 1 significantly improves prediction performance over the Bayes posterior.The cold-posterior effect motivates the paper’s central investigation.Source: §1.1 · p. 2

1.2. Why Should Bayes (T = 1) be Better?

  • Posterior averaging is motivated by theory, classical empirical evidence, and successful deterministic model averaging.These motivations include theoretical dominance in some models, robustness in classical statistics, and strong predictive results from deep-learning ensembles.Source: §1.2 · p. 2
  • The paper tests the claim that Bayes posterior predictives are desirable through experiments on deep neural networks.It examines posterior quality, prior influence, and approximate inference rather than assuming the claim holds.Source: §1.2 · p. 2
  • The study demonstrates poor Bayes-posterior predictive performance versus SGD-trained models on ResNet-20/CIFAR-10 and CNN-LSTM/IMDB, and evaluates explanations for this behavior.It also introduces diagnostics indicating that existing SG-MCMC methods accurately simulate the posterior.Source: §1.2 · p. 2

2. Cold Posteriors Perform Better

Experiments on image and text classification show that tempered posteriors outperform the true Bayes posterior, while T < 1 has a problematic interpretation as artificially strengthened evidence. The effect is consistent with prior literature using cold posteriors.

2. Cold Posteriors Perform Better

  • Temperized Bayes ensembles obtained with T < 1 outperform the true Bayes posterior at T = 1.This is the main result of the section.Source: §2 · p. 2

2.1. Deep Learning Models: ResNet-20 and LSTM

  • For ResNet-20 on CIFAR-10, accuracy and cross-entropy improve for T < 1/10, and T = 10^-4 beats both T = 1 and a tuned SGD model.Brier score and expected calibration error also improve with cold posteriors.Source: §2.1 · p. 2
  • For CNN-LSTM on IMDB, the best predictive performance occurs with a tempered posterior in the approximate range 0.01 < T < 0.2.Figure 3 reports error bars of ± one standard error over three runs.Source: Figure 3 · p. 2, §2.1 · p. 3

2.2. Why is a Temperature of T < 1 a Problem?

  • Because T < 1 overcounts data by 1/T, cold posteriors give individual models overly strong evidence rather than representing the true Bayes posterior.At T = 0, posterior mass collapses onto MAP estimates; gains for T < 1 therefore implicate the prior, likelihood, or inference procedure.Source: §2.2 · p. 3

2.3. Confirmation from the Literature

  • Cold-posterior performance is not isolated: prior Bayesian deep-learning work also uses T < 1 posteriors.The paper presents this literature as broader evidence that the phenomenon requires explanation.Source: §2.3 · p. 3

3. Bayesian Deep Learning in Practice

The paper implements Langevin-based SG-MCMC for Bayesian neural networks and improves simulation accuracy with layerwise preconditioning and cyclical time stepping. The procedure addresses minibatch and discretization errors while retaining practical efficiency.

3. Bayesian Deep Learning in Practice

  • The section presents an efficient implementation of Bayesian posterior simulation by combining existing SG-MCMC techniques rather than introducing a major new contribution.The practical algorithm reduces simulation error through preconditioning and cosine-based cyclical step-size schedules.Source: §3 · p. 3, §3.3 · p. 4

3.1. Posterior Simulation using Langevin Dynamics

  • Langevin dynamics samples parameters from exp(−U(θ)/T), recovering the Bayes posterior at T = 1.The friction parameter affects convergence speed but not the limiting distribution.Source: §3.1 · pp. 3–4

3.2. Stochastic Gradient MCMC (SG-MCMC)

  • SG-MCMC replaces the full posterior gradient with an unbiased minibatch estimate and numerically discretizes the continuous-time dynamics.The minibatch gradient scales by the training-set size n, while the log prior retains 1/n scaling regardless of batch size.Source: §3.2 · p. 4
  • The method uses symplectic Euler updates with step size and friction parameters linked to SGD learning rate and momentum decay.The correspondence includes h = n/ℓ for training-set size n and learning rate ℓ.Source: §3.2 · p. 4, §3.3 · p. 5

3.3. Accurate SG-MCMC Simulation

  • Layerwise preconditioning and cyclical time stepping are combined to reduce SG-MCMC simulation errors from minibatch noise and discretization.The preconditioner is periodically updated and then held fixed, while the step size decreases within cycles to improve accuracy without making the entire procedure slow.Source: §3.3 · p. 4

4. Inference: Is it Accurate?

The paper tests whether poor Bayes-posterior predictions arise from inaccurate inference, SG-MCMC bias, minibatch noise, or ensemble variance. Across these tests, the reported evidence rules out each proposed explanation as sufficient for the cold-posterior effect.

4. Inference: Is it Accurate?

  • The study frames the remaining explanations around inference, the prior, or the likelihood because both Bayes and cooled posteriors are intractable and approximate SG-MCMC has finite-sample and minibatch limitations.The inference tests specifically examine whether simulation difficulty or approximation error produces the observed effect.Source: §4 · p. 5

4.1. Hypothesis: Inaccurate SDE Simulation

  • Accurate simulation diagnostics and comparisons with HMC rule out poor SDE simulation and missing accept-reject corrections as explanations for cold posteriors.The SG-MCMC experiments pass kinetic and configurational diagnostics, while HMC and SG-MCMC agree with optimal predictions at T = 1.Source: §4.1 · p. 5, §4.2 · pp. 5–6

4.2. Hypothesis: Biased SG-MCMC

  • The HMC comparison uses synthetic data generated from the prior and model, where Bayes optimality requires T = 1 in expectation, making inference error testable.The experiment uses non-minibatched MLPs with one to three layers and ten hidden units per layer.Source: §4.2 · pp. 5–6

4.3. Hypothesis: Stochastic Gradient Noise

  • For batch sizes 32, 64, 128, and 256, temperatures T < 1 remain optimal on ResNet-20/CIFAR-10 and CNN-LSTM/IMDB, so minibatch noise alone cannot explain T = 1 performance.The smallest batch sizes achieve the best cross-entropy, but their greater gradient noise does not eliminate the preference for cold posteriors.Source: §4.3 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6

4.4. Hypothesis: Bias-Variance Trade-off

  • Increasing the number of ensemble members cannot explain cold posteriors because the asymptotic analysis finds that residual variance is insufficient to account for the poor T = 1 performance.The bias-variance hypothesis predicts that S →∞ should remove the problem, but the appendix analysis rejects that explanation.Source: §4.4 · p. 6

5. Why Could the Bayes Posterior be Poor?

The paper examines whether the cold-posterior effect arises from the likelihood, prior, inference procedure, or SGD’s inductive bias. Experiments implicate the prior but find no simple variance fix, while SG-MCMC does not destroy SGD’s beneficial initialization bias.

5.1. Problems in the Likelihood Function?

  • The dirty-likelihood hypothesis attributes cold posteriors to batch normalization, dropout, and data augmentation, but cold posteriors remain when a clean likelihood is used in a modified ResNet.The CNN-LSTM already uses a clean likelihood, and the experiment is described as slightly inconclusive.Source: §5.1 · p. 7

5.2. Problems with the Prior p(θ)?

  • Typical N(0, I) prior draws for ResNet-20 produce concentrated class distributions across CIFAR-10 inputs, indicating that the prior is poor for the model likelihood.He-scaling priors produce similar results, and the authors interpret the prior as placing high probability on the same few classes for all inputs.Source: §5.2 · p. 7, Figure 7 · p. 8
  • Changing the Normal prior variance does not remove the cold-posterior effect, so an appropriate scaling of this simple prior is not a sufficient fix.The effect persists across the variances considered, while very small variance can make performance poor at every temperature.Source: §5.2 · p. 7, Figure 12 · p. 8
  • As training data shrink, Bayes and SGD/MAP behave differently across tasks: Bayes degrades gracefully on ResNet-20/CIFAR-10 but deteriorates quickly on CNN-LSTM/IMDB.The results are inconclusive but implicate the prior because its influence is strongest for small n.Source: §5.2 · p. 7, Figure 9 · p. 8, Figure 10 · p. 8
  • Increasing MLP depth or width makes the cold-posterior effect more prominent, indicating a connection between model capacity and effect strength.The capacity experiment varies depth and width on CIFAR-10 using an N(0, I) prior.Source: §5.2 · p. 7, Figure 11 · p. 8

5.3. Inductive Bias due to SGD?

  • SG-MCMC does not destroy SGD’s beneficial initialization bias: SGD recovers its previous performance after being initialized from the SG-MCMC state.The experiment alternates between converged SGD, 500 epochs of SG-MCMC, and subsequent SGD optimization.Source: §5.3 · p. 8

6. Alternative Explanations?

The paper considers alternative explanations for cold posteriors, including proposed non-Bayesian posterior targets and a claimed likelihood reinterpretation. It identifies promising directions while rejecting the latter argument as incorrect.

  • The paper leaves open whether explanations beyond those studied can account for the cold-posterior phenomenon.Source: §6 · p. 8
  • Masegosa-posteriors explicitly encourage diverse ensemble predictions and can outperform Bayes posteriors when the model is misspecified.The authors view them as a potentially better approximation target, compatible with variational and SG-MCMC algorithms.Source: §6 · p. 9
  • The claimed equivalence between cold posteriors and ordinary Bayes posteriors under a modified likelihood is incorrect.The paper demonstrates and discusses this error in Appendix M.Source: §6 · p. 9

7. Related Work on Tempered Posteriors

Related statistical work uses tempered or fractional posteriors for misspecification, slower concentration, or improved sampling, while distinguishing these purposes from changing the target distribution.

  • SafeBayes and fractional posteriors use T > 1 to address misspecified models or slow posterior concentration.The cited approaches include Grünwald et al. (2017), Jansen (2013), and Bhattacharya et al. (2019).Source: §7 · p. 9
  • Multiple-temperature Monte Carlo methods ordinarily improve sampling of rough energy landscapes rather than alter the desired target distribution.The passage contrasts this sampling purpose with the use of temperature to change a posterior.Source: §7 · p. 9
  • Variational inference can select temperatures different from one, according to Mandt et al. (2016).Source: §7 · p. 9

8. Conclusion

The conclusion argues that SG-MCMC can accurately approximate deep-network parameter posteriors, yet cold posteriors improve predictions and remain insufficiently explained. The appendices specify the experimental models, algorithmic connections, preconditioning, tempering, and bias–variance analysis supporting these claims.

8. Conclusion

  • SG-MCMC is robust, scalable, and accurate enough to approximate parameter posteriors in deep neural networks.This conclusion is supported by experiments in Sections 4–5 and diagnostics in Appendix I.Source: §8 · p. 9
  • Cold SG-MCMC posteriors improve predictions through ensembles, although their extra evaluation cost may make simpler ensemble methods more practical.The comparison is made with methods including Lakshminarayanan et al. (2017), Wen et al. (2019), and Ashukha et al. (2020).Source: §8 · p. 9
  • The experiments implicate the prior p(θ) in the cold-posterior effect, but scaling prior variance alone does not provide a simple fix.The authors conclude that suitable priors for Bayesian neural networks require further study.Source: §8 · p. 9

A.1. ResNet-20 CIFAR-10 Model

  • A ResNet-20 CIFAR-10 SGD baseline reaches 92.22 percent accuracy, close to the Keras reference of 92.16 percent.Source: §A.1 · p. 10

C. Connection to Stochastic Gradient Descent (SGD)

D. Semi-Adaptive Estimation of Layerwise Preconditioner M

  • The layerwise preconditioner ties parameter scales within each model variable, normalizes the least sensitive variable to scale one, and becomes M = I under equal sensitivity.Source: D. Semi-Adaptive Estimation of… · p. 12

E. Kullback-Leibler Scaling in Variational Bayesian Neural Networks

  • KL scaling in variational Bayesian neural networks corresponds to approximating a partially tempered posterior that scales the likelihood while leaving the prior unchanged.Source: E. Kullback-Leibler Scaling in… · p. 13

F. Inference Bias-Variance Trade-off Hypothesis

  • Asymptotic risk estimates show that longer SG-MCMC sampling improves performance, but not enough for T = 1 to surpass T < 1.The analysis uses end-of-cycle iterates and estimates C(∞) from ensemble-size regressions.Source: F. Inference Bias-Variance Trade-off… · p. 14

G. Cold posteriors improve uncertainty metrics.

Cold posteriors improve uncertainty metrics, while experiments indicate that SGD can recover after SG-MCMC initialization. The paper also develops temperature diagnostics for assessing simulation accuracy and convergence.

G. Cold posteriors improve uncertainty metrics.

H. Details on the Experiment for the Implicit Initialization Prior in SGD Hypothesis

I.1. Kinetic Temperature Estimation

  • Kinetic temperature estimates should equal the target temperature under perfect simulation and can diagnose discretization or convergence errors.Source: §I.1 · pp. 15–16
  • Per-layer kinetic temperature estimates help diagnose convergence issues and motivated the layerwise preconditioner.Source: §I.1 · p. 16

I.2. Configurational Temperature Estimation

  • Configurational temperature estimates also target the system temperature, but their sampling distribution lacks the simple known form available for kinetic estimates.Source: §I.2 · p. 16

J. Simulation Accuracy Ablation Study

The ablation study finds that combining cyclical time stepping with layerwise preconditioning yields accurate Langevin simulation, while deep-learning practices can be interpreted through Jensen posteriors. These interpretations connect dropout, batch normalization, and data augmentation to modified Bayesian procedures.

J. Simulation Accuracy Ablation Study

K. Dirty Likelihood Functions

  • The dirty-likelihood hypothesis treats batch normalization, dropout, and data augmentation as modifications that violate the likelihood principle.Source: K. Dirty Likelihood Functions · p. 17

K.2. Log-likelihood Bound and Jensen Posterior

  • The Jensen posterior is defined from a Jensen bound on the log-posterior and provides a probabilistic interpretation of several deep-learning methods.Source: §K.2 · pp. 17, 20
  • Any Jensen posterior can be interpreted as a full Bayesian posterior for the same model under a modified prior.Source: §K.2 · pp. 17–19

K.3. Deep Learning Techniques Optimize Jensen Posteriors

  • Dropout, batch normalization, and data augmentation fit the Jensen-posterior framework through latent randomness and corresponding test-time averaging procedures.Source: §K.3 · pp. 20, 22

K.4. Dirty Likelihood Experiment

Removing batch normalization and data augmentation makes the likelihood cleaner but does not eliminate the cold-posterior effect. The lower-temperature advantage is weaker in this experiment, leaving the dirty-likelihood explanation inconclusive but insufficient.

  • 87.8 ± 0.16% accuracy at T = 0.0193 exceeds 87.1 ± 0.13% at T = 1 in the clean-likelihood ResNet experiment.Source: §K.4 · p. 22, Figure 28 · p. 22
  • At T = 0.0193, cross entropy is 0.393 ± 0.015 versus 0.3918 ± 0.0021 at T = 1, making the two temperatures comparable on that metric.Source: §K.4 · p. 22
  • The clean-likelihood experiment is slightly inconclusive because lowering temperature produces a less marked improvement, while the CNN-LSTM already has a clean likelihood.Source: §K.4 · p. 22

L. Prior Predictive Analysis for Different Prior Scales

Prior-scale experiments show that the cold-posterior effect persists across Normal-prior scalings, while simple scaling corrections fail to produce sensible prior predictive distributions. Even near-zero bias priors leave predictions concentrated on the same class distribution.

L. Prior Predictive Analysis for Different Prior Scales

L.1. He-Scaled Normal Prior, N(0, I) for Biases

  • He-scaled priors leave prior predictive distributions highly concentrated around the same distribution for all training instances.Source: §L.1 · p. 23

L.2. He-Scaled Normal Prior, N(0, ϵI) for Biases

  • The He-scaled prior with N(0, ϵI) bias terms remains a bad prior because random draws assign mass to the same concentrated class distribution across training instances.Source: §L.2 · p. 23

M. Tempering the Observation Model?

The section examines whether tempering the observation model can provide a Bayesian interpretation of cold posteriors. It finds that normalization changes the posterior energy, while an exact construction requires assigning mass to an impossible pseudo-event.

Tempering the Observation Model?

  • Correct normalization adds a θ-dependent correction, so the proposed tempered likelihood does not produce the actual cold-posterior energy.The unnormalized derivation incorrectly treats the normalizing constant as parameter-independent; the resulting energy differs from the cold-posterior energy.Source: M. Tempering the Observation Model? · pp. 23–24
  • A He-scaled Normal prior can concentrate predictive mass on nearly constant label distributions, making it a bad prior, yet the cold-posterior effect remains for accuracy and cross-entropy.Source: Figure 30 · p. 24, Figure 29 · p. 24
  • For classification, an exact tempering construction introduces a pseudo-event that cannot occur and assigns it residual probability mass.The construction starts from a K-class observation model and redistributes probability through temperature scaling.Source: M. Tempering the Observation Model? · p. 24
  • The authors regard assigning probability to an impossible pseudo-event as non-Bayesian because the model violates known certainty about the event space.Source: M. Tempering the Observation Model? · p. 25

N. Details: Generation of a Synthetic Dataset Based on an MLP Drawn From its Prior Distribution

The appendix documents synthetic-data generation and HMC implementation details, then evaluates convergence and agreement between HMC and SG-MCMC. Diagnostics generally support convergence at moderate temperatures, while very low temperatures degrade convergence for deeper MLPs.

N. Details: Generation of a Synthetic Dataset Based on an MLP Drawn From its Prior Distribution

  • The synthetic dataset uses a depth-2, 10-unit ReLU MLP, five-dimensional standard-normal inputs, three labels, 100 inference points, and 10,000 evaluation points.Weights use a He normal prior and biases use standard deviation 0.05 to avoid degenerate or input-independent label distributions.Source: N. Details: Generation of a Synthetic… · p. 25

O. Details about Hamiltonian Monte Carlo

O.1. Hyperparameter choices

  • The appendix searches L in {5, 10, 100} and ε in {0.001, 0.01, 0.1} across five runs, using dual-averaging step-size adaptation.Source: §O.1 · p. 25, Figure 33 · p. 27

O.2. Convergence monitoring

  • Five chains generally converge across monitored statistics, although dispersion increases with MLP depth and temperature.Trace plots monitor cross-entropy, mean predictive entropy, and predictive-entropy standard deviation.Source: §O.2 · p. 26
  • For temperatures T in [0.001, 0.05], convergence diagnostics degrade substantially for depth-2 and depth-3 MLPs, with ESS reaching approximately 189 for depth 3.At T in [0.05, 1], diagnostics indicate approximate convergence across all three depths.Source: §O.2 · p. 26

O.3. KL divergence between predictive distributions

  • HMC and SG-MCMC show close cross-entropy agreement, with symmetrized predictive-distribution KL divergences around 10^-5 for depth 1 and 10^-3 for depths 2 and 3.Source: §O.3 · p. 26, Figure 32 · p. 27

Related papers

Loading 2002.02405v2…