Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks

1. Introduction

• Backpropagation requires tuning many potentially layer-specific hyperparameters, yields point estimates that omit parameter uncertainty, and often relies on large networks plus regularization to control overfitting.Finding good hyperparameter values can take substantial time even with Bayesian optimization.Source: §1 · p. 1
• Bayesian neural networks can infer hyperparameters, propagate parameter uncertainty into predictions, and reduce overfitting by averaging over parameter values.The introduction describes these as potential advantages of Bayesian approaches.Source: §1 · p. 1
• Earlier Bayesian neural-network methods generally lacked scalability; Graves (2011) was scalable but reportedly performed poorly due to Monte Carlo noise, while Soudry et al. (2014) handled binary weights but not posterior variance for continuous weights.The introduction also names Laplace approximation, Hamiltonian Monte Carlo, expectation propagation, and variational inference as prior approaches.Source: §1 · pp. 1–2
• PBP propagates probabilities forward to obtain the marginal likelihood, then propagates its gradients backward; experiments show it is fast, predicts accurately, and estimates posterior weight uncertainty in a calibrated way.The paper presents PBP as a fast approach without the disadvantages of previous methods.Source: §1 · p. 2

2. Probabilistic neural network models

• Targets are modeled as network outputs plus Gaussian noise, with Bayesian inference over the weights and the noise and weight precisions.For each input x_n, y_n = f(x_n; W) + ϵ_n, where ϵ_n has variance γ^-1.Source: §2 · p. 2
• The network has fully connected layers with per-layer biases, ReLU hidden units, scaled layer inputs, and a one-dimensional output for scalar targets.The input scaling divides by the square root of the number of incoming connections.Source: §2 · p. 2
• A Gaussian weight prior uses precision λ, whose gamma hyperprior is weakly informative, equivalent to 12 prior samples with variance 1; the noise precision γ also has a gamma prior.The weakly informative characterization reflects the 12 equivalent samples being few relative to the observed data count N.Source: §2 · p. 2
• Although Bayes’ rule defines the parameter posterior and the predictive distribution for a new input, exact computation is intractable in most cases.The predictive distribution integrates over the posterior and uses a Gaussian observation model centered at the network output.Source: §2 · p. 2

3. Probabilistic backpropagation

• PBP represents each weight with a one-dimensional Gaussian marginal, propagates activation distributions forward, then backpropagates marginal-log-probability gradients to update posterior means and variances.The forward pass approximates layer distributions by matching marginal means and variances; the backward pass uses reverse-mode differentiation.Source: §3 · p. 3

3.1. PBP as an assumed density filtering method

• PBP applies assumed density filtering by sequentially incorporating posterior factors through KL-divergence projection, yielding an approximate online Bayesian learning algorithm.The approximation is initialized with uniform weight and hyperparameter beliefs before factors are incorporated one at a time.Source: §3.1 · p. 3

3.2. Incorporating the prior factors into q

• For prior factors, PBP updates weight and hyperparameter beliefs; because the normalizer lacks a closed form, it approximates it using a Student’s t density matched by a Gaussian.The hyperparameter updates match its first and second moments rather than exactly minimizing KL divergence.Source: §3.2 · pp. 3–4

3.3. Incorporating the likelihood factors into q

• For each likelihood factor, PBP approximates the network output as Gaussian to compute the normalizer, propagating marginal means and variances through Gaussian and rectified-linear layer approximations.The output-distribution approximation assumes Gaussian layer inputs and approximates the Student’s t likelihood density with a Gaussian of the same mean and variance.Source: §3.3 · p. 4

3.4. Expectation propagation

• To avoid storing a separate approximate factor for every likelihood, PBP repeats ADF passes without removing factors, while updating prior factors with EP after each pass.Too many passes can underestimate variance, but the method targets large datasets where fewer than 100 passes are possible.Source: §3.4 · p. 5

3.5. Implementation details

• PBP perturbs initially zero weight means to select a symmetric posterior mode and retains previous weight parameters when an approximate update produces negative variance.The mean perturbations are sampled from N(0, 1/(Vl + 1)); negative variances can arise from the approximate normalizer.Source: §3.5 · p. 5

Related Work

• HMC and Jylänki et al. (2014) are batch methods with scaling limitations, while Laplace inference can require infeasible Hessian computation.Diagonal Hessian approximations can deteriorate; Jylänki et al. also require numerical quadrature and per-datapoint factors, whereas PBP incorporates each likelihood factor in one step, expected to be more accurate.Source: §4 · p. 5
• Graves (2011) uses Monte Carlo and a second SGD approximation for variational inference, which can yield poor solutions on large data sets with few passes.The method's marginal-likelihood bound requires an intractable expectation under a factorized Gaussian approximation, and the initial approximation uses data inefficiently.Source: §4 · p. 5
• EBP is limited to binary targets, holds posterior variances constant with continuous weights, and fixes the prior-variance hyperparameter, unlike PBP.PBP assumes continuous targets and can extend to binary targets; EBP's unchanged variances also prevent accurate uncertainty estimates and make mean updates inefficient.Source: §4 · p. 6

5.1. Predictive performance

• PBP and BP performed best overall, with PBP best on 6 of 10 datasets; VI performed poorly.The paper attributes VI’s poor performance to two stochastic approximations: sampling from the variational approximation and subsampling the data.Source: §5.1 · p. 7
• PBP was the fastest method because it required no hyperparameter selection and ran only once, unlike BP and VI, which were rerun 30 times for tuning.Bayesian optimization also added considerable overhead on the smallest datasets.Source: §5.1 · p. 7

5.2. Multiple hidden layers

• With up to four hidden layers, PBP usually achieved the best results with two hidden layers.The reported results were similar to those in Table 1.Source: §5.2 · p. 7

5.3. Toy data set

• On the toy dataset, PBP and BP predictions were closer to HMC ground truth than VI’s, and PBP and BP performed similarly despite PBP’s automatic hyperparameter adjustment.Figure 1 displays the methods’ predictions and credible intervals alongside the ground truth.Source: Figure 1 · p. 6, §5.3 · p. 7

5.4. Active learning

• Active sampling significantly improved HMC over random selection in Boston, Energy, Power Plant, and Yacht, with similar PBP gains, indicating useful PBP posterior-variance estimates.All methods selected the next input with the highest predictive variance.Source: §5.4 · p. 8, Table 2 · p. 8
• Across Boston, Yacht, and Energy, PBP’s active-learning gains over random selection tracked HMC’s, while active learning worked less well for EP and LA.Figure 2 shows average test RMSE during data collection.Source: Figure 2 · p. 7, §5.4 · p. 8

• Across ten datasets, PBP made accurate predictions and produced useful estimates of posterior variance on network weights, while achieving fast, state-of-the-art Bayesian neural-network learning.Source: §6 · p. 8
• PBP approximates the weight posterior with a product of Gaussians, updating its parameters by propagating probabilities forward and gradients backward in a two-stage process.Source: §6 · p. 8
• Future work will extend PBP to multilabel and multiclass problems, mini-batch learning, and estimates of model evidence.Source: §6 · p. 8

1 Derivation of the gradients

• PBP computes gradients of log Z with respect to Gaussian weight means and variances, using two per-neuron signals instead of traditional backpropagation’s single error-gradient variable.The paired signals track gradients with respect to each neuron's activation mean and variance.Source: §1 · p. 10
• The derivation uses activation means and variances, invokes the ReLU activation and standard Gaussian pdf and cdf, and writes gradient rules recursively across connected neurons.It computes gradients of γ_j and α_j with respect to activation means and sums over neurons receiving signals from neuron j.Source: §1 · pp. 10–11

2 Results with neural networks including more than one hidden layer

• PBP had the best overall predictive performance, with PBP2 achieving the best results in 5 datasets.The comparison reports average test RMSE and standard errors for PBPx and BPx, where x is the number of hidden layers.Source: §2 · p. 12
• The optimal number of hidden layers for PBP was problem dependent: one layer was optimal for Wine and Year, while four worked better for Protein.Source: §2 · p. 12
• The experiments compared BP and PBP using 2-, 3-, and 4-hidden-layer networks with 50 units per layer, or 100 for Year and Protein.Source: §2 · p. 12

3 Error in the second approximation in equation (12) in the main text

• The relative error in log Z is very small during PBP’s last 100 iterations on Boston Housing.The section compares the first and last 100 iterations over individual datapoints.Source: §3 · p. 12
• Equation (12) approximates the Student’s t density with a Gaussian density having the same mean and variance.Source: §3 · p. 12
• The Gaussian approximation improves as the Student’s t degrees of freedom increase, which occurs as PBP reduces uncertainty about noise parameter γ.In the late iterations, posterior parameters αγ and βγ are relatively high, increasing the degrees of freedom.Source: §3 · p. 12

• PBP uses expectation propagation (EP) to adjust a parametric approximation to the exact posterior distribution.Source: §4 · p. 12
• The EP implementation refines posterior parameters by matching the first and second moments of λ and γ, since matching Gamma sufficient-statistic expectations lacks an analytical solution.Source: §4 · p. 12
• PBP approximates the normalization constants by replacing a Student’s t density with a Gaussian density having the same mean and variance.Source: §4 · p. 13
• To avoid storing an approximate factor for every likelihood factor, PBP uses repeated ADF passes that can underestimate posterior variance when too many data iterations are run.Each likelihood factor is treated as a novel example.Source: §4 · p. 13

5 Derivations of equations (9) and (10) in the main text

• H(α, β) is defined as the normalization constant of f(x)Gamma(x|α, β).The derivation also denotes H explicitly as a function of α and β.Source: §5 · p. 13
• The first and second moments use H(α, β), H(α + 1, β), and H(α + 2, β), with the constants replaced by their approximations.The derivation denotes these three values as Z, Z1, and Z2, respectively.Source: §5 · pp. 13–14
• The updated Gamma parameters αnew and βnew are obtained by matching the normalized product’s mean and variance to a Gamma distribution.The Gamma distribution’s mean and variance are α/β and α/β2, respectively.Source: §5 · p. 14

6 EP updates for the approximate factors corresponding to the prior

• Only the equation (2) prior factors require repeated EP updates; the Gamma priors on λ and γ are incorporated into q once.Removing and reinserting those Gamma priors would not improve q.Source: §6 · p. 14
• EP initializes each approximate factor as uniform, incorporates the Gamma priors first, and then updates each factor and q on its first incorporation.The initial parameters are ˜mij,l = 0, ˜vij,l = ∞, ˜αij,l = 1, and ˜βij,l = 0.Source: §6 · p. 14
• The moment-matching updates ensure that the Gaussian approximation to the Student’s t density in equation (11) has matching means and variances for wij,l.The passage describes this as the guarantee provided by the update rules.Source: §6 · p. 14
• On later iterations, EP removes the approximate factor to form a cavity, matches moments after multiplying the cavity by the exact factor, and updates q and the approximate factor.The normalization constant is approximated using equation (11) with the cavity distribution; the matching yields new parameters for q.Source: §6 · p. 15