PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Bayesian Deep Learning

Hao Wang, Dit-Yan Yeung

arXiv:1604.01662v4stat.MLcs.AIcs.CVcs.LGcs.NE

TL;DR

Deep learning has advanced perception, but comprehensive AI also needs conditional and causal inference and uncertainty handling, areas where PGMs are useful. This survey reviews BDL as a unified framework combining deep-learning perception with task-specific probabilistic models. It describes applications across several areas and identifies scalability and information exchange as challenges.

  • Problem

    Deep learning has advanced perception, while comprehensive AI also needs conditional or causal inference and uncertainty handling.

    Source: §1 · p. 1

  • Method

    The survey synthesizes BDL models that combine a neural-network perception component with a task-specific component using probabilistic graphical models.

    Source: §6 · p. 31

  • Results

    BDL has found successful applications in recommender systems, topic models, stochastic optimal control, computer vision, natural language processing, and health care.

    Source: §6 · p. 32

  • Takeaways & Limitations

    The survey identifies efficient Bayesian neural networks as a foundation for further improving BDL’s scalability.

    Source: §6 · p. 32

    Efficient Bayesian neural-network formulations remain nontrivial, and early approaches have not been widely adopted because of limited scalability.

    Source: §1 · p. 3

Abstract

from arXiv · show

A comprehensive artificial intelligence system needs to not only perceive the environment with different `senses' (e.g., seeing and hearing) but also infer the world's conditional (or even causal) relations and corresponding uncertainty. The past decade has seen major advances in many perception tasks such as visual object recognition and speech recognition using deep learning models. For higher-level inference, however, probabilistic graphical models with their Bayesian nature are still more powerful and flexible. In recent years, Bayesian deep learning has emerged as a unified probabilistic framework to tightly integrate deep learning and Bayesian models. In this general framework, the perception of text or images using deep learning can boost the performance of higher-level inference and in turn, the feedback from the inference process is able to enhance the perception of text or images. This survey provides a comprehensive introduction to Bayesian deep learning and reviews its recent applications on recommender systems, topic models, control, etc. Besides, we also discuss the relationship and differences between Bayesian deep learning and other related topics such as Bayesian treatment of neural networks. For a constantly updating project page, please refer to https://github.com/js05212/BayesianDeepLearning-Survey.

1 INTRODUCTION

Deep learning handles perception well, while probabilistic graphical models (PGMs) support conditional or causal inference and uncertainty; the survey presents Bayesian deep learning (BDL) as a principled way to unite them. It reviews BDL’s components, benefits, real-world challenges, and applications.

1 INTRODUCTION

  • BDL unifies deep learning’s perception with PGM-based inference, addressing the gap between deep learning’s strength on high-dimensional signals and PGM’s strength in probabilistic or causal inference.The introduction also notes that perception alone is insufficient for a comprehensive AI system.Source: §1 · pp. 1–2
  • A unified framework supports information exchange between perception and task-specific inference, models conditional dependencies on high-dimensional data, and handles three kinds of uncertainty.These include uncertainty in neural-network parameters, task-specific parameters, and information exchanged between components.Source: §1 · p. 2
  • BDL priors provide implicit regularization for both components and can help avoid overfitting, especially when data are insufficient.Priors can encode expert knowledge or other prior information for the task-specific component.Source: §1 · p. 3
  • Real-world BDL faces challenges in scaling Bayesian neural networks and exchanging information efficiently, ideally including first- and second-order information in both directions.Training perception and task-specific components separately assumes no uncertainty in their information exchange.Source: §1 · p. 3
  • The survey covers BDL components, learning algorithms, and applications including recommender systems, topic models, and control.Its roadmap also includes deep-learning and PGM preliminaries and future research issues.Source: §1 · p. 3

2 DEEP LEARNING

The survey reviews deep learning models as background for Bayesian deep learning, moving from multilayer networks and autoencoders to convolutional and recurrent architectures. It highlights how these models transform inputs, learn representations, process spatial features, and model sequences.

2 DEEP LEARNING

  • Deep learning is introduced through multilayer networks and related models as background for Bayesian deep learning.The section is intended as background rather than a comprehensive survey of neural networks.Source: §2 · p. 3, §2.4.2 · p. 8

2.1 Multilayer Perceptrons

  • An MLP is a sequence of parametric nonlinear transformations trained by learning weights and biases through backpropagation and stochastic gradient descent.Its hidden activations are computed from the input and learned parameters, rather than learned as free parameters.Source: §2.1 · p. 4

2.2 Autoencoders

  • An autoencoder learns a compact representation by reconstructing its input, while an SDAE takes corrupted data as input and predicts the clean data.The middle-layer output can serve as the compact representation.Source: §2.2 · p. 5

2.3 Convolutional Neural Networks

  • CNNs combine convolution, which transforms inputs into feature maps, with max-pooling, which downsamples maps and provides local translation invariance.A typical CNN alternates convolutional and max-pooling layers before an MLP performs classification or regression.Source: §2.3 · pp. 5–6

2.4.1 Vanilla Recurrent Neural Network.

  • Feedforward networks do not naturally use preceding words to predict the next word, whereas RNN hidden states depend on current inputs and previous hidden states.The recurrent connection provides short-term memory; an unrolled representation shares weights and biases across time steps.Source: §2.4 · p. 6, §2.4.1 · pp. 6–7, Figure 2 · p. 6

2.4.2 Gated Recurrent Neural Network.

  • Vanilla RNN gradients can vanish or explode across time steps, and their signals decay exponentially, limiting long-term dependencies in long sequences.LSTMs are introduced as gated RNNs that can summarize or forget old states while processing a sequence.Source: §2.4.2 · p. 7
  • An encoder LSTM can pass its final output and cell states to a decoder LSTM, forming an encoder-decoder architecture.The encoder represents a sequence and the decoder recovers it from that representation.Source: §2.4.2 · p. 7, Figure 3 · p. 7

3 PROBABILISTIC GRAPHICAL MODELS

PGMs represent random variables and probabilistic relationships, with directed models as the survey’s focus. The section introduces model structure and extensions, then contrasts efficient point-estimate learning with Bayesian methods that account for uncertainty.

3.1 Models

  • PGMs use nodes for random variables and links for probabilistic relationships, and this survey focuses mainly on directed PGMs.Directed PGMs are also called Bayesian networks; undirected PGMs are Markov random fields.Source: §3 · p. 8, §3.1 · p. 8
  • LDA illustrates a PGM whose graphical representation and generative process describe how document words and topics are generated.The graph distinguishes observed variables from latent variables and parameters, which learning algorithms can estimate.Source: §3.1 · p. 8
  • LDA extensions incorporate temporal information, topic correlations, online processing, or rating and social information for recommendation [89].Collaborative topic regression incorporates rating information to make recommendations.Source: §3.1 · p. 8

3.2 Inference and Learning

  • Learning and inference are often intertwined: LDA alternates latent-variable updates with parameter updates, then can infer topics for a new document using learned parameters.The survey distinguishes parameter estimation as learning and latent-variable estimation given parameters as inference.Source: §3.2 · p. 9
  • MAP offers a cost-effective point estimate, while variational inference and MCMC provide Bayesian treatments that account for uncertainty in latent variables and parameters.Variational inference approximates the posterior with factorized distributions; learning then minimizes KL divergence to the true posterior.Source: §3.2 · pp. 9–10

4 BAYESIAN DEEP LEARNING

The survey introduces the general Bayesian deep learning framework and presents recent models applied to recommender systems, topic models, control, and other areas.

4 BAYESIAN DEEP LEARNING

  • The survey introduces the general BDL framework and concrete models for applications including recommender systems, topic models, and control.It points readers to Table 1 for a summary of these models.Source: §4 · p. 10

4.1 A Brief History of Bayesian Neural Networks and Bayesian Deep Learning

The survey distinguishes its broad use of Bayesian deep learning from Bayesian neural networks, which it treats as a possible perception component within the broader framework. It reviews the history and growing use of both areas.

4.1 A Brief History of Bayesian Neural Networks and Bayesian Deep Learning

  • A Bayesian neural network places a prior on neural-network parameters and marginalizes their posterior distribution to make predictions.This Bayesian model averaging can be viewed as aggregating results across a distribution of neural networks.Source: §4.1 · p. 10
  • This survey uses BDL broadly for a probabilistic framework that subsumes BNNs; a BDL model with a perception component and an empty task-specific component is equivalent to a BNN.The survey notes that BDL is sometimes used specifically to mean BNNs.Source: §4.1 · p. 10
  • BNN research dates to the 1990s, while broader BDL research began roughly in 2014, after the 2012 ImageNet LSVRC breakthrough.The survey describes later work on BNN scalability and incorporation of deep neural network advances.Source: §4.1 · p. 10
  • BDL has found applications in areas such as recommender systems and computer vision and has featured in conference workshops.Source: §4.1 · p. 10

4.2 General Framework

Bayesian deep learning integrates a probabilistic perception component with a task-specific component, linked through hinge variables. The framework supports different generative directions and levels of uncertainty modeling.

4.2 General Framework

  • BDL’s task-specific component can use complex distributions and relationships, including Bayesian networks, deep Bayesian networks, or stochastic processes.The perception component is typically a probabilistic deep-learning model with multiple nonlinear layers.Source: §4.2 · pp. 10–11
  • BDL has perception and task-specific components, with hinge variables connecting them to variables in the task-specific component.The perception variables include neural-network weights and neurons; hinge variables directly interact with the perception component, while task variables do not.Source: §4.2 · pp. 10–11
  • When edges point toward hinge variables, the model is usually supervised; when they originate from hinge variables, it is usually unsupervised.The supervised form uses perception as a probabilistic representation learner; the unsupervised form uses task-specific constraints and domain knowledge to help learn representations.Source: §4.2 · p. 11
  • Learnable variance is more flexible than hyper-variance, which is more flexible than zero-variance; a reported HV CDL significantly outperformed a ZV CDL [121].The authors state that properly regularized LV models normally outperform HV models, which outperform ZV models.Source: §4.2 · p. 12
  • Practical BDL learning algorithms should be online and scale linearly with the perception component’s free parameters.The section explains that these criteria rule out conventional variational inference or MCMC unless adapted for online use, and make Hessian-based Laplace methods impractical at scale.Source: §4.2 · p. 12

4.3 Perception Component

The perception component is ideally probabilistic, allowing it to represent uncertainty in parameters and outputs. The section surveys neural-network forms that can serve as this component, from RBMs and probabilistic autoencoders to natural-parameter networks.

4.3 Perception Component

  • A probabilistic or Bayesian neural network is the ideal perception component because it can represent uncertainty in parameters and outputs.The survey discusses recent scalable developments following early Bayesian neural-network work that was not widely adopted because of scalability.Source: §4.3 · pp. 12–13

4.3.1 Restricted Boltzmann Machine.

  • RBMs use binary hidden neurons, are not trained with back-propagation, and use Contrastive Divergence; trained models can infer visible or hidden neurons by marginalizing the others.RBMs can be stacked into deep belief networks or combined with convolutional layers, and multiple branches can support multimodal learning.Source: §4.3.1 · p. 13

4.3.2 Probabilistic Generalized SDAE.

  • Probabilistic generalized SDAE models the clean input probabilistically; as λ_s tends to infinity, it reduces to a Bayesian formulation of vanilla SDAE.Its generative process assumes both clean and corrupted inputs are observed; posterior maximization is equivalent to minimizing reconstruction error with weight decay.Source: §4.3.2 · p. 13

4.3.3 Variational Autoencoders.

  • A VAE maximizes the evidence lower bound using an encoder and decoder, with a KL-divergence term regularizing the encoder.The encoder outputs Gaussian mean and variance, and samples the latent variable using the reparameterization trick.Source: §4.3.3 · pp. 13–14

4.3.4 Natural-Parameter Networks.

  • NPNs propagate distributions through neural-network layers, producing output distributions rather than single values.They represent hidden neurons and weights as distributions; Gaussian NPNs can also learn when input variance is zero, and NPNs support other exponential-family distributions.Source: §4.3.4 · p. 14

4.4 Task-Specific Component

Task-specific components incorporate probabilistic prior knowledge and can take the form of Bayesian networks, deep Bayesian networks, or stochastic processes. These forms represent conditional dependencies, nonlinear relationships, or continuous-time dynamics.

4.4 Task-Specific Component

  • Task-specific components incorporate probabilistic prior knowledge, represented with Bayesian networks, bidirectional inference networks, or stochastic processes.Bayesian networks are the most common choice and naturally represent conditional dependencies and uncertainty.Source: §4.4 · p. 14, §4.4.1 · p. 14

4.4.1 Bayesian Networks.

  • Probabilistic matrix factorization uses a Bayesian network to describe conditional dependencies among users, items, and ratings.The model draws latent item and user vectors from zero-mean Gaussian distributions; a fully Bayesian treatment can also place priors on latent-factor precision matrices [97].Source: §4.4.1 · p. 15

4.4.2 Bidirectional Inference Networks.

  • Bidirectional inference networks replace shallow linear conditional structures with deep nonlinear ones parameterized by Bayesian neural networks [117].Inference and learning are performed by back-propagation across the BNNs; the framework retains the ability to incorporate prior knowledge as Bayesian networks.Source: §4.4.2 · pp. 15–16

4.4.3 Stochastic Processes.

  • Stochastic processes can model continuous-time dynamics, including topic evolution with a Wiener process and phoneme boundaries with a Poisson process.For phoneme boundaries, the Poisson-process intensity can be produced by a neural network taking raw speech as input [83] [99].Source: §4.4.3 · p. 16

5 CONCRETE BDL MODELS AND APPLICATIONS

This section discusses how BDL can facilitate supervised learning, unsupervised learning, and representation learning, using examples from several application areas.

5 CONCRETE BDL MODELS AND APPLICATIONS

  • The section considers BDL for supervised learning, unsupervised learning, and representation learning in general.Source: §5 · p. 16
  • The application examples include recommender systems and topic models.Source: §5 · p. 16
  • Control is also among the application domains considered.Source: §5 · p. 16

5.1 Supervised Bayesian Deep Learning for Recommender Systems

The section surveys recommender models that combine deep content representations with probabilistic modeling of relationships among users, items, and ratings. CDL and its variants couple these components, with extensions for Bayesian sampling, efficient learning, ranking, and alternative neural representations.

5.1.1 Collaborative Deep Learning.

  • CDL tightly couples deep content representation learning with collaborative filtering, and experiments report significant improvement over the state of the art.The probabilistic SDAE and graphical model exchange information in both directions.Source: §5.1.1 · p. 17
  • CDL uses implicit feedback and item content to predict unobserved ratings, with a middle representation linking content features to rating-based item vectors.The middle layer and latent item offset support learning content features while capturing item and user relationships.Source: §5.1.1 · pp. 17–18
  • When λ_n/λ_v approaches either zero or positive infinity, CDL degenerates to a model whose predictive performance suffers greatly.The two limits remove the decoder or place the SDAE representation directly into CTR, respectively.Source: §5.1.1 · p. 19

5.1.2 Bayesian Collaborative Deep Learning.

  • Bayesian CDL also offers sampling-based inference, whose weight sampling becomes a Bayesian generalized version of back-propagation when λ_s approaches infinity.Adaptive rejection Metropolis sampling uses gradients to approximate proposals while accounting for variance.Source: §5.1.2 · p. 20

5.1.3 Marginalized Collaborative Deep Learning.

  • Marginalized CDL uses mSDAE to marginalize corrupted inputs and obtain closed-form solutions, making learning more computationally efficient than SDAE.The exposition describes a one-layer case that can be generalized to nonlinear and multilayer cases.Source: §5.1.3 · p. 21

5.1.4 Collaborative Deep Ranking.

  • Collaborative Deep Ranking replaces exact-rating modeling with pairwise item preferences, and the ranking objective significantly improves recommendation performance.Preferences are formed from item pairs for which a user's observed rating difference is positive.Source: §5.1.4 · pp. 21–22

5.1.5 Collaborative Variational Autoencoders.

  • Collaborative variational autoencoders replace CDL's probabilistic SDAE with a VAE, while the broader framework also permits recurrent components and richer task-specific models.These alternatives are described as options for sequential data and complex scenarios such as cross-domain recommendation.Source: §5.1.5 · p. 22

5.1.6 Discussion.

  • CDL-based models jointly extract deep features from high-dimensional content and capture implicit relationships, and their learned representations may support tasks beyond recommendation.Their hinge-variable connections are described as simple, effective, and computationally efficient; six of eight listed models are HV models.Source: §5.1.6 · p. 22

5.2 Unsupervised Bayesian Deep Learning for Topic Models

The section reviews unsupervised topic models that combine neural representations with probabilistic structure. It covers models that incorporate item relations or deep latent topic structure, along with alternative components and inference approaches.

5.2.1 Relational Stacked Denoising Autoencoders as Topic Models.

  • RSDAE integrates a hierarchy of latent factors with item relations to learn content representations and relationships among items.Its relational latent vectors are regularized using graph information, encouraging linked items to have closer representations.Source: §5.2.1 · pp. 23, 25

5.2.2 Deep Poisson Factor Analysis with Sigmoid Belief Networks.

  • Deep PFA models word counts with Poisson factorization and places a deep sigmoid belief network prior on binary topic-activation variables.The SBN provides a deep structure for topic modeling.Source: §5.2.2 · pp. 25–26
  • For deep PFA, SGNHT is presented as a more accurate and scalable alternative to online MCMC-based BCDF, with momentum intended to help escape local optima.SGNHT generalizes SGLD and SGHMC.Source: §5.2.2 · p. 26

5.2.3 Deep Poisson Factor Analysis with Restricted Boltzmann Machine.

  • Deep PFA can also use an RBM instead of an SBN as its perception component, with RBM parameters updated by contrastive divergence.The survey reports that replacing SBN with RBM can achieve comparable performance.Source: §5.2.3 · p. 27

5.2.4 Discussion.

  • Topic-modeling components interact bidirectionally: topic-hierarchy knowledge informs word, topic, and relation modeling, while those models help discover the hierarchy and compact document factors.The survey notes that DPFA's probabilistic coupling needs no additional distributional assumptions, whereas naively making RSDAE zero-variance would violate the stated i.i.d. requirement.Source: §5.2.4 · p. 27

5.3 Bayesian Deep Representation Learning for Control

For control from raw images, Bayesian deep learning (BDL) combines learned perception with a dynamical control model so the two can exchange information. The survey describes a representation-learning model designed to capture image semantics while preserving local linearity in the state space.

5.3 Bayesian Deep Representation Learning for Control

  • Embed to Control combines an image-perception component and a control component to learn semantic representations that preserve local linearity for control.The perception and task-specific components provide the two respective properties within the BDL framework.Source: §5.3 · p. 28

5.3.1 Stochastic Optimal Control.

  • The control problem maps raw images to latent states and minimizes trajectory cost for given dynamics, which can be locally linearized around estimated states and controls.The latent states represent system states, and the control formulation includes system noise.Source: §5.3.1 · p. 28

5.3.2 BDL-Based Representation Learning for Control.

  • The model maps raw images to latent states, predicts state transitions from states and controls, and reconstructs images from the latent states.Its transition model uses an approximate posterior, and its reconstruction model specifies a Bernoulli distribution.Source: §5.3.2 · pp. 28–29

5.3.3 Learning Using Stochastic Gradient Variational Bayes.

  • Stochastic gradient variational Bayes learns the model parameters, with the first loss term providing a variational bound on each data point’s marginalized log-likelihood.The training data consist of image, control, and subsequent-image tuples.Source: §5.3.3 · p. 29

5.3.4 Discussion.

  • The representation must capture semantic information and preserve local state-space linearity, while perception and control interact through state estimates and control signals.The control component uses perception’s mapped states and confidence; its signals affect the incoming video.Source: §5.3.4 · pp. 29–30
  • The control model uses a VAE-style exchange mechanism with separately parameterized mean and covariance, giving it more free parameters than the cited CDL and CDR models.Its diagonal covariance satisfies the stated independence requirement.Source: §5.3.4 · p. 30

5.4 Bayesian Deep Learning for Other Applications

The survey describes BDL applications in link prediction, language, vision, speech, forecasting, and health care. Across these areas, models combine neural perception with task-specific probabilistic components to represent dependencies, latent structure, or domain knowledge.

5.4.1 Link Prediction.

  • For link prediction, surveyed BDL models combine neural perception with graph or community-structure models; one reported model achieves state-of-the-art performance.Graphite extends relational deep learning with graph-convolutional perception, while another model combines graph convolutional networks and a stochastic blockmodel.Source: §5.4.1 · p. 30

5.4.2 Natural Language Processing.

  • Language-revision models use recurrent neural networks for perception and relatively simple task-specific components to link input and output sequences.The survey cites models and.Source: §5.4.2 · p. 30

5.4.3 Computer Vision.

  • In unsupervised vision, BDL models represent scene objects and their attributes with latent variables, while neural networks model image features.Attend, Infer, Repeat [20] models object position, scale, appearance, and presence; variants seek computational-efficiency or performance improvements.Source: §5.4.3 · p. 30

5.4.4 Speech.

  • Speech applications use factorized latent-variable models, Gaussian mixtures, and stochastic processes for speech factors, controllable synthesis, and phoneme or utterance structure.The cited stochastic-process models report improved speech-recognition performance, including lower word error rate for recurrent Poisson process units.Source: §5.4.4 · pp. 30–31

5.4.5 Time Series Forecasting.

  • BDL forecasting models pair recurrent-neural-network perception with task-specific components for dependencies among variables, and the survey reports substantial improvement over previous non-BDL models.The surveyed models are applied across time-series forecasting domains.Source: §5.4.5 · p. 31

5.4.6 Health Care.

  • Health-care applications use Bayesian priors, model averaging, and task-specific components to incorporate human knowledge, address robustness for under-represented data, and support interpretability.Examples include electronic-health-record analysis, false-discovery-rate control in cancer-drug screening, and counterfactual reasoning.Source: §5.4.6 · p. 31

6 CONCLUSIONS AND FUTURE RESEARCH

The survey frames BDL as a probabilistic integration of neural networks and probabilistic graphical models, with perception and task-specific components. It reviews component variants and learning algorithms, and describes applications alongside future directions in application depth and scalability.

6 CONCLUSIONS AND FUTURE RESEARCH

  • BDL integrates neural networks and probabilistic graphical models through perception and task-specific components, with multiple instantiations of each surveyed.The survey also reviews parameter-learning algorithms including block coordinate descent, Bayesian conditional density filtering, stochastic gradient thermostats, and stochastic gradient variational Bayes.Source: §6 · p. 31
  • BDL addresses tasks that combine perception from high-dimensional signals with probabilistic inference, and the survey reports applications across several areas.Examples include recommender systems, topic models, stochastic optimal control, computer vision, natural language processing, and health care.Source: §6 · pp. 31–32
  • The survey identifies deeper study of existing applications, exploration of more complex tasks, and improved BDL scalability as future directions.It notes that progress in efficient Bayesian neural networks can support further scalability improvements.Source: §6 · p. 32

Related papers

Loading 1604.01662v4…