PaperlayerView exact revision on arXiv

Source-linked AI summary

Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks

José Miguel Hernández-Lobato, Ryan P. Adams

arXiv:1502.05336v2stat.ML

TL;DR

Backpropagation-based neural networks require hyperparameter tuning and do not naturally represent parameter uncertainty, while existing Bayesian methods lack scalability. The paper introduces probabilistic backpropagation, which propagates approximate distributions forward and gradients backward. Across ten datasets, PBP is fast, has competitive predictive performance, and estimates posterior weight uncertainty accurately, subject to approximation limitations.

  • Problem

    Backpropagation requires extensive hyperparameter tuning and yields point-estimate weights, while existing Bayesian neural-network methods do not scale well to large networks and datasets.

  • Method

    PBP approximates each weight's marginal posterior with a Gaussian and updates these distributions by propagating probabilities forward and marginal-likelihood gradients backward.

  • Results

    Across ten datasets, PBP makes accurate predictions, is fastest because it runs once without hyperparameter selection, and provides useful posterior-variance estimates.

  • Takeaways & Limitations

    PBP offers a scalable Bayesian-learning alternative that combines competitive predictive performance with calibrated or useful estimates of uncertainty in network weights.

  • Takeaways & Limitations

    The expectation-propagation approximation can underestimate posterior variance when too many iterations are performed over the data.

Abstract

from arXiv · show

Large multilayer neural networks trained with backpropagation have recently achieved state-of-the-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.

1. Introduction

Backpropagation has enabled expressive neural networks but requires extensive hyperparameter tuning and provides only point estimates of weights. Bayesian methods can address these issues, yet existing approaches struggle to scale, motivating probabilistic backpropagation.

  • Challenges of backpropagation: Backpropagation requires tuning many potentially layer-specific hyperparameters, including learning rate, momentum, and weight decay.Finding suitable values can be time-consuming on large datasets.
  • Challenges of backpropagation: Point-estimate weights prevent standard backpropagation networks from accounting for parameter uncertainty in predictions.The paper identifies uncertainty estimates as desirable when weights are poorly specified.
  • Bayesian motivation: Bayesian neural networks can marginalize or infer hyperparameters, propagate parameter uncertainty into predictions, and improve robustness to overfitting.These benefits arise from averaging over parameter values rather than selecting one point estimate.
  • Limitations of prior methods: Existing Bayesian neural-network methods have limited scalability across network architectures and dataset sizes.Scalability limitations have hindered widespread adoption of approaches including Laplace approximation, Hamiltonian Monte Carlo, expectation propagation, and variational inference.
  • Probabilistic backpropagation: PBP learns Bayesian neural networks by propagating probabilities forward to compute marginal likelihood and gradients backward to update posterior parameters.The method is designed to be fast, accurate in prediction, and calibrated in posterior weight uncertainty.

3. Probabilistic backpropagation

Probabilistic backpropagation replaces point weights with Gaussian marginal posteriors and mirrors backpropagation by propagating distributions forward and gradients backward. It uses approximate Bayesian updates to maintain tractability while producing posterior predictions and uncertainty estimates.

  • Probabilistic representation: PBP represents each synaptic weight with a one-dimensional Gaussian approximating its marginal posterior instead of a point estimate.Forward activations therefore become random distributions that are sequentially approximated by Gaussians matching marginal means and variances.
  • Two-phase computation: PBP propagates input distributions forward, computes the logarithm of the marginal target probability, and backpropagates its gradients to posterior means and variances.The backward phase uses reverse-mode differentiation analogous to classic backpropagation.
  • Approximate Bayesian updates: PBP updates approximate Gaussian beliefs by matching moments after incorporating likelihood factors, yielding the same mean and variance as the updated target approximation.These updates are described as an assumed density filtering procedure with improvements from expectation propagation.
  • Assumed density filtering: PBP factorizes the neural-network posterior and sequentially incorporates priors and likelihood factors by minimizing a KL divergence between the updated and approximating distributions.The procedure approximates an exact online learning algorithm while avoiding explicit storage of every expectation-propagation factor.
  • Likelihood incorporation: PBP approximates likelihood normalizers with Gaussian forms and propagates means and variances through layers, using Gaussian approximations for pre-activation distributions.The Central Limit Theorem supports approximate Gaussian pre-activations when the preceding layer is sufficiently wide.

4. Related Work

Existing Bayesian neural-network methods face scalability and approximation limitations, while PBP is positioned against these methods through online probabilistic propagation and posterior-variance estimation.

  • Limitations of prior methods: HMC, Laplace, and batch EP methods can be impractical for large neural networks or datasets because of computational, memory, or tuning requirements.HMC is batch and requires tuning; Laplace may require an infeasible inverse Hessian; batch EP is not expected to scale and requires numerical quadrature.
  • Limitations of prior methods: Scalable variational inference uses nested stochastic approximations, which can introduce noise into stochastic-gradient computations.The method approximates the marginal-likelihood bound with Monte Carlo sampling and then optimizes it using a second approximation.
  • Relation to PBP: Expectation-backpropagation uses forward probability propagation and backward gradient propagation but is limited to binary targets and does not update continuous-weight posterior variances.PBP differs by supporting continuous targets and updating both Gaussian posterior means and variances.

5. Experiments

Experiments evaluate PBP on predictive accuracy, toy-data uncertainty, deeper networks, and active learning. Across these settings, PBP is competitive with BP, faster than methods requiring hyperparameter searches, and produces useful uncertainty estimates.

  • Predictive performance: PBP obtains the best average test RMSE on 6 out of 10 datasets, while PBP and BP perform best overall.VI performs poorly, which the authors attribute to two stochastic approximations rather than one.
  • Predictive performance: PBP is the fastest method because it runs once without selecting hyperparameter values, whereas BP and VI repeat expensive Bayesian-optimization searches.The comparison reports average test log-likelihood for VI and PBP and average running time for each method.
  • Multiple hidden layers: With up to four hidden layers, PBP results are similar to the one-hidden-layer results and are usually best with two hidden layers.The deeper-network experiments use 50 units per hidden layer, except 100 for Year and Protein.
  • Toy-data predictive distributions: On toy data, PBP and BP predictions are much closer to ground-truth HMC than VI, while PBP and BP perform similarly.The toy experiment uses 20 inputs, a cubic data-generating function with Gaussian noise, and a one-hidden-layer network with 100 hidden units.
  • Active learning: Active-learning gains from PBP-A over PBP-R are significant on Boston, Energy, Power Plant, and Yacht, indicating useful posterior-variance estimates.The active-learning setup uses 20 training instances, 100 test instances, repeated pool additions, and comparisons with HMC, EP, and LA.
  • Active learning: PBP-A and HMC-A show similar improvements over their random counterparts during data collection, while active learning works less well for EP and LA.Figure 2 tracks average test RMSE and standard errors for Boston Housing, Yacht, and Energy.

6. Conclusions and future work

PBP is presented as a scalable Bayesian neural-network algorithm that combines probabilistic forward propagation with backward gradient updates. Experiments on ten datasets support accurate predictions, useful posterior-variance estimates, and fast performance, while future work targets broader problems and mini-batch training.

  • Conclusions and future work: PBP approximates the posterior over weights with products of Gaussians updated through forward probability propagation and backward gradient propagation.The gradients of the marginal likelihood update the Gaussian means and variances.
  • Conclusions and future work: Experiments on ten datasets show that PBP makes accurate predictions and produces useful estimates of posterior variance in network weights.
  • Conclusions and future work: PBP is characterized as fast and state-of-the-art for Bayesian neural-network learning, with future work planned for multilabel, multiclass, mini-batch, and model-evidence settings.

Supplementary Material

The supplied supplementary passages identify the paper’s authors and affiliations.

  • Supplementary Material: José Miguel Hernández-Lobato is affiliated with Harvard University, USA.
  • Supplementary Material: Both listed authors are affiliated with Harvard University, USA.
  • Supplementary Material: Ryan P. Adams is affiliated with Harvard University, USA.

1 Derivation of the gradients

The gradient derivation extends backpropagation to Gaussian weight distributions by tracking derivatives of the marginal likelihood with respect to activation and weight means and variances. The supplied material also specifies Gaussian and ReLU-based activation quantities used in these updates.

  • Gradient propagation: PBP backpropagation tracks separate gradients of log Z with respect to each neuron’s activation mean and variance.These are represented by δm_j and δv_j, respectively.
  • Activation moments: The derivation defines means and variances for neuron inputs, activations, and outputs while using the ReLU activation function.The standard Gaussian probability density and cumulative distribution functions appear in the resulting expressions.
  • Weight gradients: The recursive rules propagate gradients through connections and compute derivatives with respect to the means and variances of Gaussian network weights.The weight connecting neurons i and j is described by its mean and variance.

2 Results with neural networks including more than one hidden layer

Supplementary experiments evaluate PBP and BP on networks with two, three, and four hidden layers. PBP has the best overall predictive performance in these experiments, although the best depth depends on the dataset.

  • Experimental setup: PBP and BP were evaluated with neural networks containing 2, 3, and 4 hidden layers.Each hidden layer used 50 units, except Year and Protein, which used 100.
  • Results: PBP had the best overall predictive performance, with PBP2 achieving the best results on 5 datasets.Performance was measured using average test RMSE and corresponding standard errors.
  • Results: The optimal number of hidden layers for PBP was problem dependent.

3 Error in the second approximation in equation (12) in the main text

This section evaluates the error introduced by replacing a Student’s t density with a Gaussian density having the same mean and variance. The approximation becomes more accurate as the Student’s t degrees of freedom increase, which can occur as uncertainty about γ decreases.

  • PBP’s second approximation replaces a Student’s t density with a Gaussian density matching its mean and variance.
  • The Gaussian replacement becomes more accurate as the Student’s t degrees of freedom increase.
  • Iterating over the data can reduce uncertainty about γ, increasing the degrees of freedom and improving this approximation.

4 List of approximations

PBP relies on several approximations to remain scalable. These include expectation propagation, moment matching for Gamma parameters, Gaussian replacement of Student’s t densities, and memory-saving repeated data passes that can underestimate posterior variance.

  • PBP uses expectation propagation to fit a parametric approximation to the exact posterior distribution.
  • The implementation refines αγ, βγ, αλ and βλ by matching the first and second moments of λ and γ.Matching the Gamma sufficient-statistic expectations would minimize KL divergence, but has no analytical solution.
  • PBP approximates normalization constants by replacing Student’s t densities with Gaussian densities having the same mean and variance.
  • To avoid storing one approximate factor for every likelihood factor, PBP omits factor removal before processing each datapoint and makes multiple ADF passes.With too many iterations over the data, this can lead to underestimation of posterior variance.

5 Derivations of equations (9) and (10) in the main text

The derivation expresses moments of a Gamma density multiplied by an arbitrary function through normalization constants. Approximations to those constants then provide the moments needed to recover a matched Gamma distribution.

  • The Gamma density is defined using shape α and rate β, while H(α, β) denotes the normalization constant of f(x)Gamma(x|α, β).
  • The first and second moments of the normalized product f(x)Gamma(x|α, β) are expressed using H at shifted shape parameters.
  • Approximating H(α, β), H(α + 1, β) and H(α + 2, β) makes each moment computable.
  • The new Gamma parameters αnew and βnew are chosen so the resulting Gamma distribution matches the normalized product’s mean and variance.

6 EP updates for the approximate factors corresponding to the prior

This section describes expectation-propagation updates for prior-associated approximate factors. Factors are initialized, incorporated into the posterior approximation, and then repeatedly refined through cavity distributions and moment matching.

  • Gamma priors on λ and γ are incorporated into q only once because removing and reinserting them would not improve the approximation.
  • The prior factors are represented as exact factors and corresponding approximate factors used by the EP updates.
  • Approximate factors begin uniform, and EP first incorporates each prior factor after incorporating the Gamma priors for λ and γ.
  • The update rules match means and variances of wij,l after approximating the relevant Student’s t density with a Gaussian.
  • On later iterations, EP removes an approximate factor from q to form a cavity distribution before refining that factor.
  • The updated q parameters are obtained by moment matching using the cavity distribution and an approximation to the normalization constant.
Loading 1502.05336v2…