PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian Deep Learning and a Probabilistic Perspective of Generalization

Andrew Gordon Wilson, Pavel Izmailov

arXiv:2002.08791v4cs.LGstat.ML

TL;DR

The paper asks how Bayesian reasoning can explain generalization in modern, underspecified neural networks and improve their predictive distributions. It studies marginalization over plausible solutions, function-space priors, and ensemble-based approximations, finding that deep ensembles and MultiSWAG support effective Bayesian averaging. It also cautions that priors engineered to resemble Gaussian processes may discard neural networks’ distinct inductive biases.

  • Problem

    Modern neural networks are underspecified and can represent multiple compelling solutions, so single-setting optimization can omit uncertainty relevant to accuracy and calibration.

  • Method

    The paper analyzes Bayesian marginalization, induced function-space priors, and ensemble-based approximations including deep ensembles and MultiSWAG.

  • Results

    Deep ensembles provide an effective approximate Bayesian marginalization mechanism, while MultiSWAG improves predictive distributions by marginalizing within attraction basins with similar training time.

  • Takeaways & Limitations

    A probabilistic view based on support and inductive biases explains several neural-network generalization phenomena, including fitting random labels while generalizing on noise-free data.

  • Takeaways & Limitations

    Engineering neural-network weight priors to resemble Gaussian-process priors may discard the distinct inductive biases that make neural networks useful as their own model class.

Abstract

from arXiv · show

The key distinguishing property of a Bayesian approach is marginalization, rather than using a single setting of weights. Bayesian marginalization can particularly improve the accuracy and calibration of modern deep neural networks, which are typically underspecified by the data, and can represent many compelling but different solutions. We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization, and propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction, without significant overhead. We also investigate the prior over functions implied by a vague distribution over neural network weights, explaining the generalization properties of such models from a probabilistic perspective. From this perspective, we explain results that have been presented as mysterious and distinct to neural network generalization, such as the ability to fit images with random labels, and show that these results can be reproduced with Gaussian processes. We also show that Bayesian model averaging alleviates double descent, resulting in monotonic performance improvements with increased flexibility. Finally, we provide a Bayesian perspective on tempering for calibrating predictive distributions.

1. Introduction

The paper frames generalization probabilistically through model support and inductive biases, arguing that Bayesian marginalization better represents uncertainty across plausible neural-network solutions.

  • Probabilistic perspective: Generalization depends on both a model’s support and its inductive biases, rather than flexibility or parameter count alone.Support measures which datasets receive nonzero marginal likelihood; inductive biases determine their relative prior probabilities.
  • Probabilistic perspective: Large support lets the posterior contract around sophisticated true solutions, while truncated or overly uniform support can produce erroneous or inefficient contraction.The model must represent plausible hypotheses while assigning higher prior probability to likely problem classes.
  • Bayesian deep learning: Bayesian inference marginalizes over parameter settings weighted by posterior probability instead of optimizing a single setting of weights.This distinction matters because modern neural networks are often underspecified and admit multiple high-performing solutions.
  • Bayesian deep learning: Deep ensembles provide an effective approximate Bayesian marginalization mechanism, while MultiSWAG additionally marginalizes within attraction basins with similar training time.The paper reports improved predictive distributions and significantly improved performance for MultiSWAG.
  • Function-space perspective: Function-space priors explain why models can fit random-label images yet generalize on noise-free problems: large support can coexist with reasonable inductive biases.The paper reports that Gaussian processes can reproduce this behavior, so it is not specific to neural networks.

2. Related Work

The related work situates the paper among Bayesian neural networks, PAC-Bayes, loss-geometry methods, deep ensembles, double-descent studies, and tempering approaches.

  • Bayesian neural networks: Earlier Bayesian neural-network work generally favored highly flexible model classes, including infinite-width limits that connect neural networks to Gaussian processes.The paper contrasts its probabilistic-generalization perspective with later work on scalable inference, function-space priors, and flat parameter-space objectives.
  • Loss geometry: Prior work connects flat solutions and loss valleys to generalization, while this paper emphasizes the functional diversity and Bayesian mass of solutions within such regions.Related studies include flat-minima optimization and mode-connecting curves between distinct solutions.
  • PAC-Bayes: The paper’s perspective differs from PAC-Bayes by favoring large support and multi-basin marginalization rather than compact priors or parameter-count reductions.It characterizes the contributions as orthogonal and complementary to PAC-Bayes.
  • Deep ensembles: Deep ensembles have been interpreted as approximate Bayesian inference, and this paper separates Bayesian model averaging from the Monte Carlo procedure used to approximate its integral.The target is an accurate predictive distribution, not necessarily a faithful posterior representation.
  • Double descent and tempering: The paper positions Bayesian model averaging as an alternative way to mitigate double descent and studies tempering as another Bayesian calibration perspective.These topics extend related work on regularization, double descent, and generalized or fractional Bayesian inference.

3. Bayesian Marginalization

Bayesian marginalization averages predictions across plausible parameter settings, which is especially relevant for underspecified neural networks with diverse high-performing solutions. The paper connects this view to calibration, accuracy, and deep ensembles as practical approximations to Bayesian model averaging.

  • Marginalization: Bayesian model averaging uses all parameter settings weighted by posterior probability, whereas classical training approximates the posterior with a point mass at a single optimum.This distinction becomes important when the posterior is diffuse and predictions vary across compelling solutions.
  • Why marginalization matters: Underspecified neural networks have diffuse likelihoods and diverse high-performing hypotheses, making single-setting predictions discard relevant epistemic uncertainty.Marginalization can therefore affect both predictive accuracy and uncertainty representation.
  • Loss valleys: Large, functionally diverse low-loss regions can dominate Bayesian model averages, favoring solutions that generalize well rather than redundant reparameterized solutions.The relevant Bayesian mass comes from both the volume of these regions and the diversity of their functions.
  • Accuracy: Bayesian model averaging can improve accuracy by averaging predictions from multiple high-performing models that disagree on some cases.The paper identifies deep ensembles as a compelling practical mechanism for this approximate averaging.
  • Deep ensembles: Deep ensembles retrain the same architecture into different attraction basins, enabling a BMA approximation that can outperform single-basin Bayesian approaches.Functional diversity is important because single-basin approximations may contain redundant functions.

4. An Empirical Study of Marginalization

The paper interprets deep ensembles as approximate Bayesian marginalization across posterior basins and introduces MultiSWAG to combine between-basin diversity with within-basin marginalization. Across predictive-distribution and corruption experiments, these multimodal approximations improve Bayesian prediction, accuracy, and negative log likelihood.

  • Multimodal marginalization: MultiSWAG combines independently trained SWAG approximations into a mixture of Gaussians centered on different posterior basins.It combines multimodal exploration with local posterior approximation without additional training time over standard deep ensembles.
  • Multimodal marginalization: Deep ensembles approximate Bayesian model averaging by representing functional diversity across multiple posterior basins.Within-basin predictive distributions vary little, whereas predictions can change substantially between basins.
  • Predictive distributions: Deep ensembles more closely approximate the true predictive distribution than single-basin variational inference, which becomes overconfident between data clusters.The Wasserstein distance decreases quickly with more deep-ensemble samples but remains roughly unchanged for variational inference.
  • Distribution shift: MultiSWAG and MultiSWA outperform deep ensembles for small model counts, while MultiSWAG significantly outperforms competing methods under severe Gaussian-blur corruption.MultiSWA retains the training and test-time computational requirements of deep ensembles.
  • Generalization: Multimodal posterior approximations can alleviate double descent and improve accuracy and negative log likelihood over SGD and single-basin marginalization.The paper reports these improvements as flexibility increases.

5. Neural Network Priors

The paper studies how neural-network weight priors induce function-space priors that determine support and inductive biases. Experiments find structured prior similarities, reasonable performance across prior scales, and that Gaussian processes and Bayesian neural networks can fit random labels while assigning them low marginal likelihood.

  • Function-space priors: The induced distribution over functions, rather than the parameter prior alone, controls a neural network’s generalization properties.The model architecture and parameter prior jointly determine this function-space distribution.
  • Random labels: Gaussian processes and Bayesian neural networks fit fully corrupted image labels while their marginal likelihood decreases as label corruption increases.Thus random-label datasets have support but become progressively less probable under the induced function priors.
  • Inductive biases: Randomly initialized convolutional networks can capture low-level image statistics before training, including for denoising, super-resolution, and inpainting.This provides evidence that neural-network function priors can encode useful image inductive biases.
  • Inductive biases: LeNet-5 assigns higher prior correlations to MNIST images from the same class than to images from different classes.This pattern holds across all tested prior scales, although correlations somewhat decrease as α increases.
  • Prior scale: Performance is near-optimal for α ∈[10^-2, 10^-1] and becomes mostly insensitive to larger α for both tested architectures.Smaller α constrains the weights too much; increasing α further leaves performance moderated by the network’s functional inductive biases.

6. Rethinking Generalization

The paper argues that fitting randomized labels is not mysterious or unique to neural networks: it reflects the interaction between model support and inductive biases. Gaussian processes reproduce this behavior, fitting random labels while generalizing on true labels.

  • The probabilistic explanation separates support, which permits noisy solutions, from inductive biases, which determine their prior probabilities.RBF-kernel GPs are flexible enough to support corrupted datasets while assigning higher prior plausibility to relatively simple functions.
  • Gaussian processes exactly reproduce the neural-network behavior of fitting images with random labels while generalizing on noise-free data.On CIFAR-10, the same GP fits random labels perfectly but performs at chance test accuracy, whereas training on true labels yields substantially better test accuracy.
  • 100% train accuracy and 10.4% test accuracy occur for GPs trained on CIFAR-10 images with random labels.
  • 72.8% training accuracy and 54.3% test accuracy occur for the same GP trained on true labels.

7. Double Descent

The paper argues that Bayesian model averaging should make performance improve monotonically with model flexibility, unlike the double descent observed with standard training. Experiments show that multimodal averaging removes double descent even under label corruption.

  • The results support the prediction that reasonable priors combined with effective Bayesian model averaging yield monotonic improvement with increasing flexibility.
  • Test error decreases monotonically as the number of independent SWAG models in MultiSWAG increases.
  • MultiSWAG completely mitigates double descent, with performance increasing monotonically as model size grows under substantial label corruption.This holds for both accuracy and negative log likelihood, whereas SGD suffers especially pronounced double descent with corrupted labels.
  • Deep ensembles show a similar double-descent-mitigation pattern to MultiSWAG, with approximately 1–2% worse accuracy.
  • SWAG reduces double descent, while MultiSWAG removes it by marginalizing over multiple posterior modes.
  • MultiSWAG significantly improves accuracy and NLL over both SGD and SWAG models.

8. Temperature Scaling

Temperature scaling adjusts how prior beliefs and likelihood evidence interact in the posterior. The paper argues that tempering can reflect model misspecification, improve practical Bayesian approximations, and remain consistent with Bayesian principles.

  • Temperature Scaling: T < 1 produces cold posteriors concentrated around high-likelihood solutions, T = 1 is standard Bayes, and T > 1 strengthens the prior effect.The temperature controls the interaction between prior and likelihood.
  • Temperature Scaling: Tempered posteriors are established in statistics as Safe Bayes, generalized Bayesian inference, and fractional Bayesian inference.These approaches have been motivated by prequential, learning-theoretic, and minimum-description-length perspectives.
  • Tempering Under Misspecification: The paper argues that T = 1 is unlikely to be optimal for realistic misspecified models and that tempering can better reflect beliefs about misspecification.The authors particularly advocate tempering when model capacity does not automatically scale with available information.
  • Prior Effects: A prior with an inappropriate scale can still produce a reasonable unconditional predictive distribution because the prior bias updates quickly with data.The paper reports posterior samples after observing 10, 100, and 1000 data points, while Figure 9 finds near-optimal NLL for α in [0.1, 0.3].
  • The Effect of Inexact Inference: Approximate Bayesian inference is practically useful but cannot be treated as independent sampling from the true posterior because neural-network posterior landscapes are highly multimodal and complex.The authors note that the specific inference technique may affect observed cold-posterior results.

9. Discussion

The discussion frames Bayesian deep learning around posterior-weighted marginalization and argues that neural-network priors should preserve their distinctive inductive biases and representation-learning abilities. It also identifies computational integration challenges and cautions against treating Gaussian-process limits or priors as universal replacements.

  • MultiSWAG extends deep ensembles through multimodal Bayesian averaging and improves generalization accuracy and log likelihood over SGD and single-basin marginalization.
  • Estimating Bayesian model averages remains difficult because modern neural networks have high-dimensional parameter spaces and complex posterior landscapes.
  • 9.1. The Future for BNN Priors: Function-space priors inspired by Gaussian processes may discard neural networks’ distinct inductive biases if parameter priors are engineered to imitate familiar kernels.
  • 9.1. The Future for BNN Priors: Infinite-width Bayesian neural networks recover Gaussian processes with fixed kernels, whereas neural-network training can learn similarity metrics through representation learning.
  • 9.1. The Future for BNN Priors: Generic parameter distributions can induce function distributions containing equivariance properties, representation-learning abilities, and other neural-network biases.
  • Bayesian approaches are distinguished by posterior-weighted marginalization rather than reliance on a single optimum.

D. Details of Experiments

The appendix details experimental procedures for predictive-function visualizations and comparisons with approximate Bayesian model averaging. It specifies network constructions, sampling and training protocols, and Wasserstein-distance evaluation against HMC predictive distributions.

  • The regression network uses three hidden layers of size 10, takes x and x^2 as inputs, and outputs y = f(x).
  • Data are generated from a randomly initialized network with isotropic Gaussian weight and output noise variances of 0.12.
  • 200 HMC chains produce 200 samples after SGD pretraining for 3000 steps and HMC runs of 2000 steps.
  • Deep Ensembles use 50 independently trained SGD networks, while SVI uses a fully factorized Gaussian approximation initialized at an SGD solution.
  • Predictive discrepancies are measured with one-dimensional Wasserstein distances between HMC and approximate predictive marginals, averaged over input locations.

D.2. Deep Ensembles and MultiSWAG

The appendix evaluates deep ensembles, MultiSWA, and MultiSWAG under corrupted-data and prior-function settings. MultiSWAG performs strongly under distribution shift, while neural-network priors induce class-structured correlations and varying function complexity.

  • The distribution-shift evaluation uses PreResNet-20 on CIFAR-10 with corruption types, and ensembles 20 samples from each SWAG model.
  • Frost corruption is omitted because of data-generation issues, while Gaussian blur is added to the evaluation.
  • LeNet-5 prior correlations are higher for images from the same MNIST class than for images from different classes, and sample complexity increases with α.

D.3. Neural Network Priors

These experiments examine how neural-network priors generate correlations and predictive behavior, how prior variance affects performance, and how Bayesian models respond to noisy labels and changing width. They also compare Bayesian neural networks and Gaussian processes in noisy-label settings and evaluate double descent with MultiSWAG.

  • 100 prior weight samples from LeNet-5 are used to compute pairwise class-0 logit correlations across 1000 MNIST datapoints and α values of 0.02, 0.1, and 1.
  • Functions sampled along an interpolated, norm-adjusted path between MNIST 0 and 1 images are evaluated through their logits.
  • Performance is poor for overly small α, near-optimal for intermediate α, and still reasonable for larger α on MNIST and CIFAR-10.
  • Gaussian-process and Bayesian-neural-network marginal likelihoods decrease as CIFAR-10 label corruption increases.
  • MultiSWAG experiments vary ResNet-18 width on CIFAR-100 with original labels and 10% or 20% label corruption, using ensembles of sampled SWAG models.
  • With 10% label reshuffling, MultiSWAG alleviates double descent in both test error and NLL, while NLL decreases monotonically as independent models increase under 20% corruption.

E. Analysis of Prior Variance Effect

The section analyzes how prior variance affects the function-space prior in ReLU networks. Without biases, variance changes output scale rather than sample complexity; more general scaling can also alter the relative influence of prior and variance terms.

  • The analysis reparameterizes spherical Gaussian weight and bias priors to expose how layerwise scales affect sampled network predictions.The resulting prediction expression makes the dependence on prior scales explicit.
  • For bias-free networks, prior weight scales affect only the network’s output scale.This follows by factoring the scales through the network using positive homogeneity of ReLU.
  • Increasing prior variance in bias-free ReLU networks therefore leaves sample complexity unchanged but produces increasingly confident softmax predictions.The confidence effect is observed when softmax is applied to the final-layer outputs.
  • With biases, a specific layerwise scaling rule makes the scale parameter affect only the predictive distribution’s scale.This result is established through an analogous positive-homogeneity argument.
  • More general prior scaling affects both output scale and the relative effect of prior and variance terms.The analysis does not cover residual connections or batch normalization, so it cannot be directly applied to the PreResNet-20 experiments.

F. Prior Correlation Structure under Perturbations

The section studies how a vague PreResNet prior correlates predictions on original and corrupted images. These correlations generally decay more slowly than matched linear and RBF-kernel baselines, indicating useful equivariance properties before training.

  • The experiment measures prior-logit correlations between original and corrupted CIFAR-10 images across 17 perturbations and five intensity levels.It samples 100 images and weights from a standard Gaussian prior, comparing PreResNet-20 with linear and RBF-kernel references.
  • For all corruptions except saturate, snow, fog, and brightness, PreResNet logits decay more slowly than those of the linear and RBF-kernel baselines.For many corruptions, including Gaussian noise, PreResNet prior correlations remain close to 1 across intensity levels.
  • The induced PreResNet function prior treats same-class images as highly correlated even after substantial real-world-like perturbations.The paper interprets these correlations as useful equivariance properties that contribute to neural networks’ effectiveness on high-dimensional data.
  • Figure 19 plots mean prior-logit correlation against corruption intensity for PreResNet-20, a linear model, and an RBF kernel, with shaded 1σ error regions.The RBF lengthscale is calibrated to match PreResNet correlations on uncorrupted images.
Loading 2002.08791v4…