The Bayesian Learning Rule

1.1 Learning-Algorithms

• Learning algorithms across optimization, deep learning, and graphical models can be derived as instances of the Bayesian learning rule (BLR).The paper presents the BLR as a common framework for unifying, generalizing, and improving algorithms from these fields.Source: §1.1 · p. 2, §1.2 · p. 2

1.2 The Bayesian Learning Rule

• The BLR optimizes a candidate distribution under a Bayesian objective, with different distribution classes and natural-gradient approximations yielding different algorithms and variants.The paper assumes a regular, minimal exponential family for its main discussion and notes that mixtures extend the framework.Source: §1.2 · pp. 2–3

1.3 Examples

• The paper derives gradient descent, Newton’s method, and ridge regression, and identifies new algorithms for multimodal optimization, uncertainty estimation, binary neural networks, and variational inference.The derivations illustrate how changing the candidate class and approximations recovers different methods.Source: §1.3 · p. 4, Table 1 · p. 7

1.3.1 Gradient Descent

• With a Gaussian candidate of fixed identity covariance, the BLR recovers gradient descent when the expected loss is approximated at the mean.The paper describes this delta-method step as a crude approximation of the Bayesian solution; it also cautions that a Dirac-delta candidate makes the objective meaningless because its entropy goes to −∞.Source: §1.3.1 · pp. 4–5

1.3.2 Newton’s Method

• Learning a Gaussian candidate’s precision as well as its mean raises the BLR derivation from a first-order method to a second-order method.The resulting online Newton variant uses an exponentially smoothed Hessian average as a preconditioner; recovering classical Newton’s method also uses the delta method and a unit learning rate, justified for strongly convex losses or initialization near the solution.Source: §1.3.2 · pp. 5–6

1.3.3 Ridge Regression

• For ridge regression, the BLR’s natural-gradient conditions recover the closed-form solution by assigning higher-order information to the Gaussian candidate’s natural parameters.The resulting conditions are S*m = X^T y and S = X^T X + δI, with the candidate mean equal to the ridge solution.Source: §1.3.3 · p. 6

1.4 Outline of the Rest of the Paper

• The paper proceeds from BLR derivations to optimization, deep-learning, and Bayesian-inference applications before concluding.The outline includes new uncertainty-estimation algorithms and algorithms for conjugate and non-conjugate inference.Source: §1.4 · p. 8

2.1 Bayesian Learning Rule as Natural-Gradient Descent

• A natural-gradient update penalizes changes according to distance between distributions, rather than the potentially misleading Euclidean distance between parameter configurations.A second-order expansion of the KL divergence yields the Fisher information matrix, and the resulting Fisher-preconditioned gradient is the natural-gradient direction.Source: §2.1 · pp. 8–9
• The BLR is more general than maximum-likelihood natural-gradient updates because its objective loss and candidate distribution can be chosen independently.The maximum-likelihood update is a special case using a Gaussian candidate.Source: §2.1 · p. 9

2.2 Bayesian Learning Rule as Mirror-Descent

• For exponential-family candidates, mirror descent in expectation-parameter space is equivalent to natural-gradient descent in natural-parameter space, without a second-order KL approximation.The derivation makes the approximation exact and indicates that some parameterizations are computationally preferable; the reverse mapping also holds.Source: §2.2 · pp. 9–10

3.1 First- and Second-Order Methods by using Gaussian Candidates

• Increasing Gaussian candidate complexity from fixed covariance to learned precision lets natural gradients retrieve higher-order loss information and yields second-order conditions.With fixed covariance the entropy is constant and the derivation recovers a first-order condition; learning precision recovers a second-order condition, and more complex sufficient statistics can retrieve higher derivatives.Source: §3.1 · pp. 10–11

3.2 Multimodal Optimization by using Mixtures-Candidate Distributions

• The mixture example illustrates locating multiple minima in one run, but the authors explicitly do not claim a complete solution to multimodal optimization.The example uses two Gaussian components to illustrate how responsibility assignment can associate components with distinct minima.Source: §3.2 · pp. 11–13
• A finite-mixture BLR uses Newton-like component updates, with expected gradients and Hessians updating each component’s mean and precision, respectively.When components take responsibility for separate minima, their means approximately satisfy first-order optimality conditions; entropy encourages diversity, but component count, initialization, and collapse remain practical difficulties.Source: §3.2 · pp. 12–13

4.1 Stochastic Gradient Descent

• With a Gaussian candidate, SGD is a BLR update using a mini-batch gradient; allowing its covariance to vary yields an automatically estimated preconditioner via a second-order method.The paper argues that first-order optimizers alone offer no clue for estimating this preconditioner, and that SGD iterates are affected by its choice.Source: §4.1 · p. 13

4.2 Adaptive Learning-Rate Algorithms

• Diagonal Gaussian candidates make exponential smoothing a consequence of the BLR update, yielding online second-order methods suited to stochastic learning and Adam-like variants.Smoothing stabilizes the scale estimate compared with directly using a mini-batch Hessian; Adam-like variants approximate the Hessian and use square-root scaling.Source: §4.2 · pp. 14–15

4.3 Dropout

• BLR extends dropout beyond SGD to Newton, RMSprop, and Adam variants using more flexible posterior approximations than the original MC-dropout derivation by Gal and Ghahramani (2016).The mixture-candidate extension can learn dropout probability and vary it by layer, but this leaves the original justification for dropout; Concrete relaxation can approximate gradients with respect to that probability.Source: §4.3 · pp. 16–17

4.4 Uncertainty Estimation in Deep Learning

• Variational Bayesian averaging over candidate parameters can favor flatter, more robust solutions, shifting away from high-loss regions and preferring wide minima over sharp ones.The BLR lets the noise distribution adapt from data, unlike unregulated noise in stochastic search and global-optimization methods.Source: §4.4 · p. 18, Figure 1 · p. 19
• VOGN matches Adam’s performance and gives better uncertainty estimates than Adam, OGN, and other Bayesian alternatives (Osawa et al., 2019, Fig. 1).OGN and OGGN are scalable BLR variants of Laplace’s method; OGN performs similarly to Adam at large scale while yielding a reasonable variance estimate.Source: §4.4 · pp. 18–19

4.5 Binary Neural Networks

• BayesBiNN uses Bernoulli candidate distributions to make gradients of the Bayesian objective well-defined for binary networks, and a Concrete approximation recovers a step similar to STE (Bengio et al., 2013).Noise and nonzero temperature soften binary weights to provide meaningful gradients; the entropy term produces exponential smoothing, and the solution minimizes the Bayes objective.Source: §4.5 · pp. 20–21
• As temperature τ approaches zero, BayesBiNN and STE solutions are expected to become similar when s* approaches infinity.In that condition, the left-hand side of the preceding equation goes to zero.Source: §4.5 · p. 21
• BayesBiNN is expected to share robustness properties with those discussed for Fig. 1, while its exponential smoothing resembles Helwegen et al. (2019), whose formulation lacks a well-defined objective.Source: §4.5 · p. 21

5.1 Conjugate Models and Bayes’ Rule

• For conjugate exponential-family models, one BLR step with ρt = 1 yields the exact posterior natural parameter; efficient marginal computations may still require methods such as message passing.This includes Gaussian linear models, Kalman filters and smoothers, PCA, factor analysis, mixture models, hidden Markov models, and Gaussian process regression.Source: §5.1 · p. 23

5.2 Expectation Maximization

• EM is a coordinate-wise BLR update with ρt = 1 for a candidate distribution factorized across latent variables and parameters.The derivation extends to generalized EM, parallel BLR steps, and online schemes.Source: §5.2 · pp. 23–24

5.3 Stochastic Variational Inference and Variational Message Passing

• Stochastic variational inference uses ρt = 1 for latent-variable updates and ρt < 1 for parameter-distribution updates, as a special case of the more general BLR.For conjugate models, coordinate-wise variational message-passing updates are also BLR special cases; their parameter update uses the conjugate update rather than the delta method.Source: §5.3 · p. 24

5.4 Non-Conjugate Variational Inference

• In non-conjugate variational inference, local natural-gradient parameters approximate factors while the update computes local and global parameters together.The local approximations are described as globally, rather than locally, tight; Gaussian posteriors yield quadratic loss surrogates.Source: §5.4 · p. 25
• For non-conjugate inference, BLR updates apply to conjugate and non-conjugate factors, as well as probabilistic and deterministic variables, and can integrate automatic differentiation.The framework is presented as suitable for practical probabilistic programming.Source: §5.4 · pp. 25–26
• BLR also generalizes online-learning derivations of online gradient descent, online Newton step, and online mirror descent by using expected loss and posterior-based surrogate choices.The authors state that theoretical guarantees from online learning can translate to BLR and the algorithms derived from it.Source: §5.4 · p. 26

6. Discussion

• The paper frames the learning objective as Bayesian: candidate distributions represent knowledge, and the variational formulation balances new information with prior information under computational limits.Exponential families and mixtures are presented as natural candidate choices that balance complexity and practicality.Source: §6 · p. 27
• Natural gradients encode information about the learning-loss landscape and connect seemingly different algorithms through information geometry.They can appear as loss derivatives or as messages in a graphical model.Source: §6 · p. 27
• The examples derive classical and modern learning algorithms from BLR, which the authors present as a principled framework for understanding and designing algorithms.Examples include ridge regression, Newton’s method, Kalman filters, stochastic-gradient descent, RMSprop, and Dropout.Source: §6 · p. 27

A. Bayesian Inference as Optimization

• With log-likelihood loss and a prior, the Bayesian learning problem’s minimizer is the posterior distribution.The appendix presents Bayesian inference as a special case of the Bayesian learning problem.Source: A. Bayesian Inference as Optimization · p. 28
• Choosing q(θ) as p(θ|D) minimizes the stated objective because the evidence Z(D) is constant and the KL divergence is zero.Source: A. Bayesian Inference as Optimization · p. 28

B. Natural Gradient of the Entropy

• When h(θ) is constant, the natural gradient of the entropy is −λ.Source: B. Natural Gradient of the Entropy · p. 28

C. The Delta Method

• The delta method approximates a function’s expectation using the expectation of its Taylor expansion.For a distribution with mean m, a first-order Taylor approximation provides one such estimate.Source: C. The Delta Method · p. 28
• The first-order delta method approximates the expected gradient by the gradient at the mean.The appendix identifies the derivation as using Bonnet’s theorem followed by a first-order Taylor expansion.Source: C. The Delta Method · p. 29
• The second-order delta method approximates the expected Hessian by the Hessian at the mean.The appendix identifies the derivation as using Price’s theorem followed by a second-order Taylor expansion.Source: C. The Delta Method · p. 29

C.1 Deterministic Estimates as crude-Bayesians

• As σ approaches zero, the Gaussian posterior’s entropy tends to −∞, making the Bayesian objective useless for continuous parameters in deterministic models.The section therefore rejects using a Dirac delta at the MAP estimate to justify deterministic solutions as Bayesian ones.Source: §C.1 · p. 29
• The Delta method treats a MAP estimate as a crude Bayesian approximation by using a posterior with pre-specified uncertainty rather than a Dirac delta.It replaces the expectation with the loss at the mean and ignores uncertainty around the estimate.Source: §C.1 · pp. 29–30
• Laplace’s method uses curvature around the MAP estimate to set uncertainty, using the Hessian or a positive-definite approximation.The section frames this as one example of the Delta method’s uncertainty choice.Source: §C.1 · p. 30
• The Delta-method view is more general than Laplace’s because posterior covariance may be arbitrary and the posterior need not be Gaussian.The section contrasts this with Opper and Archambeau (2009), who showed Laplace’s method as a crude approximation of variational inference.Source: §C.1 · p. 30

D. Newton’s Method from the BLR

• For Gaussian candidates, the derivation expresses natural gradients through gradients with respect to the mean and covariance, using the chain rule and Bonnet’s and Price’s results.The update derivation then obtains the covariance and mean updates from these relations.Source: D. Newton’s Method from the BLR · pp. 30–31
• A Gaussian candidate with constant h yields the BLR setup used to derive Newton’s method updates for covariance and mean.The section simplifies the covariance update first, then the mean update.Source: D. Newton’s Method from the BLR · p. 31

E. Natural-Gradient Updates for Mixture of Gaussians

• For mixture-of-Gaussians candidates, the joint distribution is minimal-conditional, ensuring a non-singular Fisher information matrix, and the derivation yields Newton-like updates.The update uses gradients expressible through the loss gradient and Hessian.Source: E. Natural-Gradient Updates for… · pp. 31–32
• When mixture components are centered at their means and far apart, the derivation simplifies the updates and assigns components responsibility for different local minima.The far-apart assumption makes other components’ responsibilities approximately zero.Source: E. Natural-Gradient Updates for… · p. 32

F. Deep Learning Algorithms with Momentum

• Momentum makes the effective gradient an average of previous gradients with decaying weights, with the hope of smoothing and reducing stochastic-gradient variance.The exact mechanisms behind momentum-based methods remain an open research question.Source: F. Deep Learning Algorithms with… · p. 33
• Adaptive-learning methods such as RMSprop and Adam use momentum variants with exponential smoothing, while their momentum scaling depends essentially on the ratio of successive scaling vectors.The section identifies deriving this dynamic scaling from Bayesian principles as a challenge for stochastic-gradient approximations.Source: F. Deep Learning Algorithms with… · p. 33
• The BLR incorporates momentum within mirror descent, where the momentum term obeys the geometry of the posterior approximation.The revised rule penalizes proximity to the previous mean using KL divergence, interpreted as ‘natural momentum’.Source: F. Deep Learning Algorithms with… · p. 33
• With a Gaussian candidate of fixed covariance, the update recovers the classical momentum form; a diagonal covariance candidate with mini-batch gradients yields another update.The section connects the resulting momentum variant to Eq. 78 under the assumptions stated for p37.Source: F. Deep Learning Algorithms with… · pp. 33–34
• A momentum-version of the update recovers the RMSprop-style form under the stated approximation, justifying scaling vectors and their exponential smoothing.This uses st ≈ st−1, replaces st by √st, and adds a constant c.Source: F. Deep Learning Algorithms with… · p. 34

G. Dropout from the BLR

• Dropout updates for the weight means and covariances reuse the mixture-of-Gaussians update derivation, with two components and one component’s parameters fixed.The resulting updates reduce to Eqs. 42 and 43.Source: G. Dropout from the BLR · pp. 34–35
• The loss-gradient expectation is moved inside the gradient because the entropy term depends only on θjl, although the loss gradient may depend on all of θ.Source: G. Dropout from the BLR · p. 34
• To recover dropout, the derivation uses reparameterized gradient samples but ignores the independent Gaussian noise because dropout weights are deterministic.Gradients are evaluated at the dropout mean zjlmjl using the Delta method.Source: G. Dropout from the BLR · p. 34
• The derivation assumes the two mixture components are not close, which is described as reasonable for dropout; under this assumption, the gradient with respect to mjl is approximately zero.This approximation is used for the gradients of log q(θjl).Source: G. Dropout from the BLR · pp. 34–35

• Table 2 lists each distribution’s natural parameters, sufficient statistics, expectation parameters, and relevant sections.Source: Table 2 · p. 36