Source-linked AI summary
Priors in Bayesian Deep Learning: A Review
Vincent Fortuin
TL;DR
Bayesian deep learning often neglects the prior, despite evidence that commonly used priors can be misspecified. This review surveys alternative priors for Gaussian processes, variational autoencoders, and Bayesian neural networks, along with methods for learning priors from data, and highlights their potential to improve performance and enable applications.
Problem
The prior p(θ) is often neglected in Bayesian deep learning, although prior misspecification can require posterior tempering for optimal performance.
Method
The review surveys prior designs for Gaussian processes, variational autoencoders, and Bayesian neural networks, plus methods for learning priors from data.
Results
Well-chosen priors can improve performance and even enable novel applications, while commonly used isotropic Gaussian priors are usually misspecified.
Takeaways & Limitations
Practitioners should consider prior choices more carefully and draw on alternative priors or their principles when designing Bayesian deep learning models.
Takeaways & Limitations
Deep neural network mean-function parameters are difficult to choose truly a priori because deep neural networks are notoriously hard to interpret.
Abstract
from arXiv · showhide
While the choice of prior is one of the most critical parts of the Bayesian inference workflow, recent Bayesian deep learning models have often fallen back on vague priors, such as standard Gaussians. In this review, we highlight the importance of prior choices for Bayesian deep learning and present an overview of different priors that have been proposed for (deep) Gaussian processes, variational autoencoders, and Bayesian neural networks. We also outline different methods of learning priors for these models from data. We hope to motivate practitioners in Bayesian deep learning to think more carefully about the prior specification for their models and to provide them with some inspiration in this regard.
1 Introduction
The review argues that prior specification is often neglected in Bayesian deep learning even though misspecification can strongly affect inference and model selection. It surveys alternative priors and methods for learning them from data.
- Motivation: Prior choice is central to Bayesian modeling but is often treated as a nuisance, while meaningful prior specification can be difficult and objective alternatives may be intractable in Bayesian deep learning.The prior should reflect beliefs before observing data, but mapping subjective beliefs to tractable distributions is not straightforward.
- Motivation: In practical high-dimensional, non-asymptotic inference, priors can strongly influence the posterior because Bernstein-von-Mises regularity conditions often fail.This limits reliance on asymptotic consistency as a justification for uninformative priors.
- Consequences: Prior misspecification can make marginal likelihoods misleading for Bayesian model selection and undermine other properties motivating Bayesian inference.The review also discusses consequences for decision-theoretic guarantees under bad priors.
- Consequences: No prior is universally preferred: task-specific priors may improve inference performance or enable inference that would otherwise fail.This follows from applying the no-free-lunch theorem to Bayesian learning.
- Evidence: Empirical posterior tempering in modern Bayesian deep learning, including pT(θ | D) ∝ p(θ | D)1/T for T < 1, provides evidence consistent with prior misspecification.The review connects this observation to results in Bayesian linear models.
- Scope: The survey reviews prior designs for deep Gaussian processes, variational autoencoders, and Bayesian neural networks, then outlines methods for learning priors from data.It aims to encourage more careful prior selection and provide inspiration for alternatives to standard uninformative priors.
2 Priors in (Deep) Gaussian Processes
The review connects Gaussian-process priors to deep neural networks through learned kernels and means, stacked deep GPs, and infinite-width neural-network limits. These constructions increase expressivity or connect Bayesian models to neural-network behavior, but often require approximate inference or careful prior choices.
- Gaussian-process foundations: A Gaussian process specifies a prior over functions, with finite collections of function values jointly Gaussian under a mean function and kernel.The mean and kernel can be parameterized by hyperparameters and determine the finite-dimensional distributions.
- GPs parameterized by neural networks: Deep neural networks can parameterize GP mean functions and kernels, but neural-network kernels must be constructed carefully to remain valid kernels.Using the final network layer as a feature map yields kψ(x,x′)=⟨φ(x;ψ),φ(x′;ψ)⟩ and a Bayesian last-layer model.
- GPs parameterized by neural networks: Deep kernels apply a base kernel in neural-network feature space; linear bases reduce to Bayesian last-layer models, whereas RBF bases retain full infinite-dimensional GP structure.These models can be expressive and have been shown to improve properties such as adversarial robustness.
- Deep Gaussian processes: Deep Gaussian processes stack latent GP functions to represent increasingly complex, non-Gaussian output distributions while retaining a fully Bayesian treatment.Their additional flexibility removes closed-form posterior inference, requiring methods such as variational, expectation-propagation, amortized, or inducing-point approximations.
- Trade-offs: Deep GP flexibility comes with intractable closed-form posterior inference, creating a practical dependence on approximate inference techniques.This is a central computational trade-off relative to standard GPs.
- Deep Gaussian processes: Priors for individual deep-GP layers remain comparatively simple and understudied, often using RBF kernels with different lengthscales.Convolutional GP kernels are an exception that produces models similar in spirit to deep CNNs.
- Neural-network limits: Infinite-width BNNs with independent finite-variance parameter priors converge to GP function-space priors, with extensions to ReLU, deep, convolutional, correlated-weight, and attention architectures.Dependent and infinite-variance priors extend the limits to hyperprior-marginalized GPs and α-stable processes, while posterior convergence also holds under mild regularity assumptions.
- Neural-network limits: Infinite-width standard neural networks also yield GP descriptions, but the GP models training under gradient descent through the neural tangent kernel rather than an initialization prior.At finite width, the corresponding kernel does not exactly reproduce training behavior, though approximate corrections exist.
3 Priors in Variational Autoencoders
VAEs commonly use standard Gaussian priors, but alternative distributional and structural priors reshape latent representations and can improve performance for particular data structures and tasks. The review covers hyperspherical, mixture, Gaussian-process, hierarchical, discrete, and neural-process priors, along with their computational and modeling trade-offs.
- VAE foundations: VAEs use neural networks for the likelihood and variational posterior, with training based on optimizing the evidence lower bound.The ELBO combines expected reconstruction quality with a KL-divergence penalty to the prior.
- Distributional priors: Distributional priors replace the standard Gaussian within the usual VAE architecture, although the variational posterior may also need modification.The review discusses spherical, power-spherical, mixture, and Gaussian-process priors in this category.
- Distributional priors: Hyperspherical vMF priors improve benchmark VAE performance over standard Gaussians mostly for low-dimensional latent spaces up to d ≈20.For higher-dimensional spaces, products of lower-dimensional vMF distributions have been proposed; power-spherical distributions offer similar performance with greater numerical stability.
- Distributional priors: Mixture priors encode latent clusters, while Gaussian-process priors encode similarity between data points through kernels.Mixture priors face the challenge of choosing the number of clusters, whereas Gaussian-process priors require a kernel and can incur O(n^3) computation.
- Structural priors: Structural priors alter both the prior distribution and VAE architecture, including modified decoders or encoders for mixture, Gaussian-process, hierarchical, and discrete latent models.Hierarchical priors model dependent latent variables, while discrete priors quantize continuous variables against a finite prototype dictionary.
- Structural priors: Hierarchical and discrete latent structures support specialized capabilities including hierarchical data modeling, image generation, compression, clustering, and more interpretable representations.Neural processes are presented as another structural VAE prior that models more than one type of observed variable.
4 Priors in Bayesian Neural Networks
The review surveys weight-space and function-space priors for Bayesian neural networks, emphasizing alternatives to isotropic Gaussians because prior choice affects expressivity, sparsity, uncertainty, and inference behavior.
- Weight-space priors: Isotropic Gaussian priors are widely used in BNNs, but they can produce a cold posterior effect, suggesting prior misspecification.A tempered posterior with T ≪ 1 can outperform the true Bayesian posterior.
- Weight-space priors: Matrix-valued, hierarchical, heavy-tailed, and scale-mixture priors increase expressiveness through weight correlations or heavier tails.Marginalizing an inverse-Wishart covariance hyperprior yields a multivariate Student-t prior.
- Weight-space priors: Horseshoe and Indian buffet process priors encourage sparsity, while horseshoe priors can support feature selection and weight compression.Dropout can be interpreted as an approximation to related sparsity-inducing priors.
- Weight-space priors: Radial-directional priors separate weight-vector length from direction, allowing distinct distributions over the unit sphere and radial component.Suggested choices include von-Mises–Fisher or uniform directional distributions paired with half-Cauchy or Gaussian radial distributions.
- Weight-space priors: Other structured priors encode correlations among neighboring pixels, filter shapes, neural-network units, or implicit latent representations.These approaches may parameterize weights through neural networks or Gaussian processes, but selecting the function g is difficult and is usually meta-learned.
- Function-space priors: Function-space priors can be matched to Gaussian processes, but GP and BNN function-space distributions may have different supports, making the true KL divergence infinite or undefined.Proposed alternatives include Wasserstein distance, hypernetworks, and ridgelet-transform approximations.
5 (Meta-)Learning Priors
The review describes learning priors from marginal likelihoods, ELBO objectives, and related tasks, covering Gaussian processes, variational autoencoders, and Bayesian neural networks. These methods can make priors more expressive, but they introduce overfitting and task-selection concerns.
- Gaussian processes: Marginal-likelihood optimization can learn GP kernel and mean-function hyperparameters, including expressive spectral-mixture and neural-network kernels.For Gaussian-observation GPs, the objective combines goodness of fit with a log-determinant complexity penalty.
- Gaussian processes: GP meta-learning optimizes priors across related tasks, but too few or overly similar meta-tasks can cause overfitting.A hierarchical hyperprior can address this issue, and successful GP-prior meta-learning has been reported with as few as five meta-tasks.
- Variational autoencoders: For VAEs, optimizing the ELBO with respect to the prior yields the aggregated posterior as the optimal prior under the stated objective.The prior-dependent term is the KL divergence between the aggregated approximate posterior and the prior.
- Variational autoencoders: Flexible VAE priors include Gaussian mixtures, inducing-point parameterizations, implicit priors, and normalizing flows.Inducing-point mixture priors can improve performance without overfitting, while flow-based priors can produce expressive latent representations.
- Bayesian neural networks: Learning BNN priors is less common because weight spaces are large and the mapping from weights to functions is complex.The review discusses manual procedures based on empirical weight distributions alongside broader approaches to prior learning.
6 Conclusion
The review argues that prior selection is central to Bayesian deep learning because it affects uncertainty estimation, model selection, and decision support. It catalogs alternatives to standard Gaussian priors and notes that useful priors can sometimes be learned from data.
- Conclusion: Choosing good priors is important for obtaining Bayesian models’ uncertainty estimation, model selection, and optimal decision-support properties.The conclusion contrasts these goals with the frequent use of isotropic Gaussian or similarly uninformative priors.
- Conclusion: The review presents alternative priors for deep Gaussian processes, variational autoencoders, and Bayesian neural networks.It also discusses cases where priors can be learned from data alone.
- Conclusion: The authors acknowledge that the survey is necessarily incomplete while presenting it as an overview and source of guidance for prior selection.They encourage practitioners to consider alternatives to standard Gaussian priors and draw inspiration from the reviewed designs.