Source-linked AI summary
The Case for Bayesian Deep Learning
Andrew Gordon Wilson
TL;DR
Standard neural-network training bets on one parameter setting, although the data can leave many high-performing explanations plausible. The paper argues that Bayesian marginalization better represents this uncertainty, and reports practical Bayesian methods with improved accuracy and calibration.
Problem
Classical training optimizes a single parameter setting even when deep networks have diffuse likelihoods and diverse compelling explanations.
Source: p. 2Method
Bayesian prediction marginalizes over parameter settings rather than betting everything on one optimized setting.
Source: pp. 2, 4Results
Bayesian model averaging across diverse high-performing models is described as improving accuracy and calibration over classical training.
Source: p. 2Takeaways & Limitations
Deep ensembles can serve as approximate Bayesian model averages rather than competing with Bayesian methods.
Source: p. 2Computing the integral remains challenging because posterior landscapes are difficult to navigate in enormously high-dimensional parameter spaces.
Source: p. 5
Abstract
from arXiv · showhide
The key distinguishing property of a Bayesian approach is marginalization instead of optimization, not the prior, or Bayes rule. Bayesian inference is especially compelling for deep neural networks. (1) Neural networks are typically underspecified by the data, and can represent many different but high performing models corresponding to different settings of parameters, which is exactly when marginalization will make the biggest difference for both calibration and accuracy. (2) Deep ensembles have been mistaken as competing approaches to Bayesian methods, but can be seen as approximate Bayesian marginalization. (3) The structure of neural networks gives rise to a structured prior in function space, which reflects the inductive biases of neural networks that help them generalize. (4) The observed correlation between parameters in flat regions of the loss and a diversity of solutions that provide good generalization is further conducive to Bayesian marginalization, as flat regions occupy a large volume in a high dimensional space, and each different solution will make a good contribution to a Bayesian model average. (5) Recent practical advances for Bayesian deep learning provide improvements in accuracy and calibration compared to standard training, while retaining scalability.