PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian Neural Network Priors Revisited

Vincent Fortuin, Adrià Garriga-Alonso, Sebastian W. Ober, Florian Wenzel, Gunnar Rätsch, Richard E. Turner, Mark van der Wilk, Laurence Aitchison

arXiv:2102.06571v3stat.MLcs.LG

TL;DR

Isotropic Gaussian priors may not reflect the weight structures learned by different neural network architectures. The paper studies SGD-trained weights to design heavy-tailed and spatially correlated alternatives, finding improved performance and architecture-dependent effects on the cold posterior.

  • Problem

    It is unclear whether widely used isotropic Gaussian priors accurately represent neural-network weights or provide optimal Bayesian neural-network performance.

  • Method

    The paper analyzes SGD-trained weight distributions and builds their heavy-tailed or spatially correlated structure into Bayesian neural-network priors.

  • Results

    Architecture-informed priors improve performance across image-classification experiments, eliminating the cold posterior effect in FCNNs but slightly increasing it in ResNets.

  • Takeaways & Limitations

    Isotropic Gaussian priors are often non-optimal, but the appropriate prior depends heavily on the architecture and dataset.

  • Takeaways & Limitations

    Data augmentation strengthens the cold posterior effect, while removing it almost entirely eliminates the effect between T = 1 and lower temperatures in the reported setting.

Abstract

from arXiv · show

Isotropic Gaussian priors are the de facto standard for modern Bayesian neural network inference. However, it is unclear whether these priors accurately reflect our true beliefs about the weight distributions or give optimal performance. To find better priors, we study summary statistics of neural network weights in networks trained using stochastic gradient descent (SGD). We find that convolutional neural network (CNN) and ResNet weights display strong spatial correlations, while fully connected networks (FCNNs) display heavy-tailed weight distributions. We show that building these observations into priors can lead to improved performance on a variety of image classification datasets. Surprisingly, these priors mitigate the cold posterior effect in FCNNs, but slightly increase the cold posterior effect in ResNets.

1 INTRODUCTION

The paper questions the dominance of isotropic Gaussian priors because they can underperform SGD baselines and require cold posteriors. It proposes architecture-informed alternatives based on observed weight statistics, improving performance while affecting the cold posterior differently across architectures.

  • Motivation: Isotropic Gaussian priors are widely used, but Bayesian neural networks with them can perform worse than standard SGD baselines.Cold posteriors partly address this by artificially reducing posterior uncertainty, departing from the standard Bayesian solution.
  • Approach: The paper evaluates whether alternative priors better match SGD-trained weights and explain the cold posterior effect.Its experiments compare priors across architectures and datasets against empirical weight distributions.
  • Empirical observations: FCNNs exhibit heavy-tailed weights, whereas CNNs and ResNets exhibit significant spatial correlations in convolutional filters.These observations motivate heavy-tailed priors for FCNNs and spatially correlated priors for convolutional architectures.
  • Prior comparisons: Heavy-tailed priors improve Bayesian FCNN classification performance over widely used Gaussian priors.The contribution is reported for Bayesian FCNN experiments.
  • Prior comparisons: Spatially correlated Gaussian priors improve Bayesian ResNet classification performance over isotropic priors.The reported comparison concerns Bayesian ResNets.
  • Cold posterior effect: Better priors eliminate the cold posterior effect in FCNNs but slightly increase it in ResNets.The paper therefore treats the cold posterior as architecture-dependent rather than uniformly explained by prior misspecification.

2 BACKGROUND: THE COLD POSTERIOR EFFECT

The cold posterior effect occurs when lowering the posterior temperature below one improves BNN performance. Because this overcounts the data and departs from standard Bayes, it suggests misspecification in the prior, likelihood, or both.

  • Definition: A tempered posterior uses temperature T, with T = 1 recovering the standard Bayesian posterior.For neural network weights, the temperature scales the log-likelihood and log-prior terms, with a normalizing constant completing the posterior.
  • Definition: The cold posterior effect refers to improved performance when using T < 1.Cooling artificially reduces posterior uncertainty and can be implemented in Langevin dynamics and MCMC.
  • Possible causes: The cold posterior effect raises the possibility that Gaussian priors, likelihoods, or both are misspecified.Prior misspecification was proposed as one possible cause in earlier work using standard neural network architectures and benchmark tasks.

3 EMPIRICAL ANALYSIS OF NEURAL NETWORK WEIGHTS

The paper empirically studies SGD-fitted weights to identify prior structure beyond isotropic Gaussians. It finds heavy tails in FCNNs and spatial correlations in CNNs and ResNets, motivating heavy-tailed and correlated priors.

  • Empirical approach: SGD-trained FCNNs, CNNs, and ResNets provide empirical distributions of fitted weights for designing Bayesian priors.The distributions approximate the weight configurations reached by SGD as MAP solutions under a uniform prior.
  • Empirical approach: A human-in-the-loop expectation-maximization view alternates approximate weight inference with fitting prior parameters to sampled weights.SGD approximates the E-step, while the M-step fits a prior distribution to the resulting weights.
  • Heavy-tailed weights: FCNN and CNN weights trained on MNIST are heavier-tailed than Gaussian weights and are reasonably approximated in the tails by a Laplace distribution.This motivates considering heavy-tailed Bayesian priors rather than isotropic Gaussians.
  • Heavy-tailed weights: In ResNet20, fitted Student-t degrees of freedom increase in deeper layers, indicating progressively less heavy-tailed and more Gaussian-like weights.Lower layers are more heavy-tailed, while higher layers have larger degrees of freedom.
  • Spatial correlations: CNN filters show strong correlations between neighboring pixels, with weaker or negative correlations at more distant positions.The pattern differs by layer: distant correlations are lesser in layer 2 and can be negative in layer 1.
  • Spatial correlations: ResNet convolutional weights show significant spatial covariance across all layers, with stronger correlations in later layers.The strongest evidence concerns weights within the same convolutional filter rather than across channels.
  • Prior design: These observations motivate non-diagonal covariance priors for convolutional layers, especially when neighboring filter pixels have large covariances.The paper contrasts these correlated Gaussian candidates with heavy-tailed distributions including Laplace and Student-t priors.

4 EMPIRICAL STUDY OF BAYESIAN NEURAL NETWORK PRIORS

The study compares isotropic Gaussian, heavy-tailed, and spatially correlated Gaussian priors across Bayesian FCNNs, CNNs, and ResNets. Priors reflecting empirical weight structure improve some performance measures, but their effects on cold posteriors and uncertainty metrics depend on architecture, dataset, and temperature.

  • Experimental setup: SG-MCMC experiments compare Bayesian FCNNs, CNNs, and ResNets on MNIST, FashionMNIST, and CIFAR-10 using several alternative priors.The study includes predictive error, likelihood, calibration, and OOD detection, with SGD baselines shown where applicable.
  • Bayesian FCNNs: Heavy-tailed priors outperform Gaussian priors for FCNN test error and test NLL in all cases except Student-t on MNIST at low temperatures.Calibration and OOD results are less consistent across datasets and metrics.
  • Bayesian FCNNs: Heavy-tailed priors remove or invert the FCNN cold posterior effect, supporting prior misspecification as one explanation for the phenomenon.For FCNNs, the effect is mainly assessed through error and NLL; calibration and OOD results can behave differently.
  • Bayesian CNNs and ResNets: Correlated Gaussian priors improve performance over isotropic Gaussian priors, especially for ResNets at lower temperatures, but increase the ResNet cold posterior effect.The authors suggest this may reflect mitigation of low-temperature overfitting on CIFAR-10.
  • Bayesian CNNs and ResNets: Correlated priors considerably improve calibration and OOD detection over other methods for low-temperature ResNets on CIFAR-10.These uncertainty metrics need not track predictive performance, and temperature effects differ across datasets and architectures.
  • Scope and caveats: Data augmentation improves performance but strengthens the cold posterior effect; without augmentation, the effect between T = 1 and lower temperatures is almost entirely eliminated.The paper identifies the relationship between augmentation, likelihood specification, and priors as requiring further investigation.

5 RELATED WORK

Prior research has explored hierarchical, heavy-tailed, task-specific, and nonstandard-inference priors, but modern BNN applications have largely continued using simple Gaussian priors. Related work also studies empirical network statistics and inference fidelity, with limited coverage of how heavy-tailed priors interact with cold posteriors.

  • Empirical statistics and priors: Neuroscience and machine-learning research has used empirical statistics, including correlations and sparsity, to inform inductive priors for learning algorithms.This work provides historical context for using observed structure to design neural-network priors.
  • BNN priors: Although hierarchical and heavy-tailed BNN priors have a long history, most modern BNN applications still rely on simple Gaussian priors.Prior proposals have also included very heavy-tailed infinite-width models and other distributional choices.
  • BNN priors: Many earlier BNN priors were designed for specific tasks or depended heavily on non-standard inference methods.This distinguishes them from broadly evaluated priors studied in the present work.
  • Inference and related evaluations: Contemporaneous work found slight heavy-tailed-prior improvements for language modeling but did not assess their interaction with the cold posterior effect.It also compared gold-standard HMC with practical cyclical SG-MCMC and found high fidelity for the latter.

6 CONCLUSION

The paper finds that isotropic Gaussian priors are often non-optimal, but the best alternative depends strongly on architecture and dataset. Improved priors eliminate the cold posterior effect in FCNNs while increasing it in ResNets, leaving its broader origin unresolved.

  • 6 CONCLUSION: FCNNs learn heavy-tailed weights, whereas CNNs and ResNets learn spatial correlations in convolutional filters.These empirical SGD weight patterns motivate architecture-specific prior choices.
  • 6 CONCLUSION: Correlated Gaussian priors improve ResNet performance, while uncorrelated heavy-tailed priors improve fully connected neural network performance.The paper recommends these alternatives instead of widely used isotropic Gaussian priors.
  • 6 CONCLUSION: Heavy-tailed priors eliminate the cold posterior effect in FCNNs, but correlated priors increase it in ResNets.In ResNets, correlated priors produce much larger performance improvements at lower temperatures.
  • 6 CONCLUSION: The correct prior is highly task- and architecture-dependent rather than universal.Heavy-tailed priors offer little or no benefit for regression on UCI datasets.
  • 6 CONCLUSION: The study cannot draw strong conclusions about the origin of the cold posterior effect.A misspecified likelihood remains consistent with the ResNet findings, and an untested prior could still eliminate the effect.

A.3 THE INFLUENCE OF DATA AUGMENTATION ON THE COLD POSTERIOR EFFECT

Data augmentation appears to strengthen the cold posterior effect in Bayesian ResNets on CIFAR-10, while broader prior-related observations remain qualitatively stable across several settings. Warm posteriors generally do not improve performance.

  • A.3 THE INFLUENCE OF DATA AUGMENTATION ON THE COLD POSTERIOR EFFECT: Data augmentation significantly increases the cold posterior effect in Bayesian ResNets on CIFAR-10.The effect is observed when comparing experiments with and without augmentation.
  • A.3 THE INFLUENCE OF DATA AUGMENTATION ON THE COLD POSTERIOR EFFECT: Data augmentation substantially improves performance at colder temperatures but not at the true Bayes posterior T = 1.This pattern suggests augmentation may contribute to the cold posterior effect.
  • A.3 THE INFLUENCE OF DATA AUGMENTATION ON THE COLD POSTERIOR EFFECT: Almost all Bayesian neural network models outperform SGD baselines in likelihood, calibration, and out-of-distribution detection.The reported comparisons include FCNNs, CNNs, and ResNets.
  • A.3 THE INFLUENCE OF DATA AUGMENTATION ON THE COLD POSTERIOR EFFECT: FCNN input-layer covariances show spatial correlations, while later FCNN layers are less smooth than isotropic-Gaussian covariances without discernible structure.The same qualitative pattern appears across layers 1–3, with structure concentrated in the first layer.
  • A.3 THE INFLUENCE OF DATA AUGMENTATION ON THE COLD POSTERIOR EFFECT: CNN layer covariances show correlations in the spatial input direction but no discernible structure in the output direction.This is reported for both analyzed convolutional layers.
  • A.3 THE INFLUENCE OF DATA AUGMENTATION ON THE COLD POSTERIOR EFFECT: Warm posteriors generally do not improve performance, with some heavy-tailed FCNN priors performing optimally near T ≈1.This complements the focus on colder temperatures in the main experiments.
  • A.3 THE INFLUENCE OF DATA AUGMENTATION ON THE COLD POSTERIOR EFFECT: The ordering of priors and the cold posterior effect remain stable when prior variances are varied around the He initialization.The tested variances were four times larger and smaller than the He variance.

A.8 DIFFERENT FCNN ARCHITECTURES

The FCNN experiments retain their main prior-ordering and cold-posterior observations across architectures ranging from two to four layers.

  • A.8 DIFFERENT FCNN ARCHITECTURES: The ordering of different priors and the cold posterior effect persist in FCNNs with between 2 and 4 layers.This extends the main experiments beyond the three-layer FCNN architecture.

A.9 UCI REGRESSION

UCI regression experiments found no cold posterior effect: T = 1 performed best, while prior differences were generally small. However, elevated diagnostics and less stable low-temperature chains limit confidence in these results.

  • Experimental setup: The regression experiments used three-layer FCNNs and GGMC without minibatching, with 30,000 epochs for four datasets and 3,000 epochs for the others.The learning rate was fixed at 5 · 10^-5 without a cosine schedule.
  • Results: The UCI regression chains were less stable at lower temperatures, where the potential landscape is sharper.This instability occurred even with full-batch MCMC.
  • Results: T = 1 was best for every UCI dataset by median MSE, including the quantiles and outliers.No prior was consistently superior, and prior performances were generally similar.
  • Limitations: Higher split-bR diagnostics in regression than classification indicate that the chains did not explore the posterior as reliably.The authors therefore caution that results may differ with more accurate posterior-approximation methods.

A.10.2 BETWEEN-CHAIN AND WITHIN-CHAIN VARIANCES

The authors assess mixing with rank-normalized split-bR on symmetry-invariant predictive quantities rather than raw neural-network parameters. Diagnostics are generally favorable for smaller models but weaker for large CIFAR-10 ResNets and Student-t FCNNs.

  • Diagnostic definition: The analysis uses rank-normalized split-bR, which should be as close to 1 as possible for well-mixed independent chains.The implementation follows Vehtari et al. (2021) through ArviZ.
  • Diagnostic choice: The split-bR diagnostic is computed on symmetry-invariant functions such as logits, prior probability, log-likelihood, potential, and log-prior.This avoids treating parameter non-mixing caused by neural-network symmetries as predictive failure.
  • Results: The correlated Gaussian had the bR value closest to 1 within CIFAR-10 ResNets.This provides evidence that inference was easier with that prior in this setting.
  • Variational inference: For λ = 1, variational inference recovers the true posterior only when the approximating family contains all distributions over weights.Mean-field VI instead restricts the approximate posterior to a fully factorized Gaussian.

A.11.1 EXPERIMENTAL DETAILS AND RESULTS

The authors replicate the ResNet CIFAR-10 experiment with mean-field variational inference and find substantially worse performance than SG-MCMC, except for calibration. Prior choice has little apparent effect under the mean-field approximation.

  • Experimental setup: The replicated models were trained for 1,000 epochs with Adam on batches of 500 augmented CIFAR-10 datapoints.The learning rate was reduced from 0.01 to 0.001 after 500 epochs.
  • Results: MFVI performed far worse than SG-MCMC on all metrics except calibration.This comparison was made in the replicated ResNet experiment on CIFAR-10.
  • Results: Performance at λ = 1 was particularly poor, consistent with the reported need for λ < 1 tempering for decent MFVI performance.The passage describes this as documented behavior in MFVI.
  • Prior sensitivity: All tested priors performed similarly under MFVI, suggesting that prior choice had little effect in this experiment.The authors hypothesize that the mean-field assumption restricts posterior expressiveness and produces a poor posterior approximation.

B EVALUATION METRICS

The paper evaluates priors using predictive accuracy, likelihood, calibration, and OOD detection, while contrasting isotropic Gaussian, heavy-tailed, and spatially correlated Gaussian distributions. These choices reflect observed weight-distribution properties and differing uncertainty goals.

  • Evaluation metrics: The experiments evaluate test error, predictive log-likelihood, uncertainty calibration, and out-of-distribution detection.These metrics cover predictive performance and uncertainty quality under possible dataset shift.
  • Evaluation metrics: Predictive log-likelihood incorporates predictive posterior variance and trades off mean fit against uncertainty quality.It is also a proper scoring rule.
  • Evaluation metrics: Calibration measures whether stated uncertainty frequencies correspond to empirical predictive correctness.For example, 70% confidence should correspond to correctness with 70% probability.
  • Evaluation metrics: OOD detection measures whether uncertainty separates in-distribution from out-of-distribution examples so the model can reject shifted inputs.This matters when deployment involves dataset shift.
  • Considered priors: The compared priors include isotropic Gaussian, Laplace, Student-t, and spatially correlated Gaussian distributions.The heavy-tailed and correlated choices were motivated by empirical SGD-trained weight distributions and implementation practicality.
  • Considered priors: The correlated CNN prior uses a block-diagonal covariance with zero covariance across filters and a Matérn kernel over pixel distances within each filter.The experiments set µ = 0 and use σ = λ = 1.

D IMPLEMENTATION DETAILS

The appendix reports experimental setups, diagnostics, and supplementary performance analyses across Bayesian FCNNs, CNNs, and ResNets. These analyses examine priors, temperatures, activations, variances, depths, data augmentation, regression, and inference behavior.

  • Training setup: MNIST BNN experiments used 60 SG-MCMC cycles of 45 epochs, collecting 300 samples and discarding the first 50 as burn-in.Langevin noise was added only during the final 15 epochs of each cycle, with cosine learning-rate decay from 0.01 to zero.
  • Architectures: The FCNN used three layers with hidden width 100 and ReLU activations, while the CNN used two convolutional layers followed by one fully connected layer.CNN convolutional layers had 64 channels, 3 × 3 convolutions, ReLU activations, and 2 × 2 max-pooling.
  • Architectures and augmentation: ResNet experiments used a ResNet20 on CIFAR-10, with augmentation consisting of four-pixel padding, random 32x32 crops, and random horizontal flips.The listed implementation uses the same ResNet20 architecture as Wenzel et al. (2020a).
  • Supplementary performance analyses: Supplementary results show that correlated priors generally outperform isotropic priors in CNNs and ResNets, whereas heavy-tailed priors improve fully connected classification performance.The figures also report that heavy-tailed priors mitigate the cold posterior effect in FCNNs, while correlated CNN priors retain it and ResNet augmentation appears to increase it.
  • Robustness and diagnostics: Additional analyses find qualitatively similar behavior across sigmoid and tanh activations, prior variances, FCNN depths, and mean-field variational inference, while kinetic temperatures closely match sampler targets in augmented ResNet experiments.The appendix also includes temperature diagnostics for MNIST, FashionMNIST, and CIFAR-10 experiments.
  • Temperature analyses: Across supplementary temperature experiments, warm posteriors generally do not improve performance, with T ≈1 optimal for some priors and clearly best for UCI regression datasets.The regression box plots report median ±1.5 times the inter-quartile range and exclude runs where the potential diverges.
Loading 2102.06571v3…