PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian Recurrent Neural Networks

Meire Fortunato, Charles Blundell, Oriol Vinyals

arXiv:1704.02798v4cs.LGstat.ML

TL;DR

The paper addresses how to obtain useful uncertainty and regularisation for RNNs through variational Bayes. It adapts Bayes by Backprop to truncated training and introduces gradient-based posterior sharpening; Bayesian RNNs improve across language modelling and image captioning while offering stronger uncertainty properties.

  • Problem

    Existing Bayesian uncertainty methods had not directly applied variational Bayes to recurrent neural networks, despite RNNs' broad use in sequence prediction.

  • Method

    The paper applies Bayes by Backprop to truncated RNN training and sharpens the approximate posterior locally using gradients from each minibatch.

  • Results

    Bayesian RNNs improve performance over corresponding baselines on language modelling and image captioning, with better regularisation and uncertainty properties.

  • Takeaways & Limitations

    Posterior sharpening extends beyond recurrent networks, and Bayesian RNN uncertainty estimates can indicate when models are outside their reliable knowledge.

  • Takeaways & Limitations

    Posterior-sharpened inference requires an extra gradient and incurs a penalty in training speed; exact gradients for RNNs are unavailable efficiently, so BPTT is used.

Abstract

from arXiv · show

In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80\%. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.

1 INTRODUCTION

The paper applies variational Bayes to RNNs to combine parameter uncertainty with regularisation, introducing posterior sharpening as a more flexible approximation. It reports efficient BBB application, improved benchmark performance, and a new uncertainty benchmark.

  • Bayes by Backprop applies variational Bayes to RNN training, allowing parameter uncertainty while using a prior to regularise the network.The KL divergence acts as a regulariser, and averaging across models is part of the Bayesian training effect.
  • Posterior sharpening adapts the variational posterior locally to each data batch using batch gradients, forming a hierarchical approximation.This makes the posterior more flexible than a typical fixed Gaussian approximation.
  • Bayes by Backprop is applied efficiently to RNNs, while the proposed variance-reduction technique is intended to transfer to other maximum-likelihood frameworks.
  • The methods improve performance on language modelling and image captioning benchmarks, outperforming established regularisation techniques such as dropout.
  • The paper introduces a new benchmark for studying uncertainty in language models.

2 BAYES BY BACKPROP

Bayes by Backprop learns a variational posterior over neural-network weights by balancing expected negative log-likelihood against a KL complexity penalty. In the Gaussian formulation, the posterior uses diagonal covariance, and the KL term provides regularisation related to weight decay.

  • Bayes by Backprop learns a variational posterior q(θ) over neural-network weights θ using Gaussian mean and standard-deviation parameters.The covariance is diagonal, while θ can have dimensionality in the millions.
  • Minimising variational free energy is equivalent to maximising log-likelihood subject to a KL complexity term that regularises the network.
  • The variational free energy combines expected negative log-likelihood with KL[q(θ) || p(θ)], where p(θ) is the parameter prior.
  • With a zero-mean Gaussian prior, the KL term acts like weight decay on posterior means, with its rate automatically tuned by prior and posterior standard deviations.
  • Before this work, Bayes by Backprop uncertainty had been used for feedforward and reinforcement-learning models but not recurrent neural networks.

3 TRUNCATED BAYES BY BACKPROP THROUGH TIME

The paper adapts Bayes by Backprop to truncated backpropagation through time by handling shared recurrent weights, parameter sampling, and KL allocation across minibatches of truncated sequences. The resulting formulation keeps sampled parameters fixed across each sequence while distributing regularisation over training segments.

  • RNN unrolling: An RNN core maps state s_t and input x_t to the next state s_t+1, and unrolling applies this recurrence across a sequence.The unrolled representation is s_1:T = F_T(x_1:T, s_0).
  • RNN unrolling: Because recurrent weights are shared across unrolled steps, each weight receives T gradient contributions during backpropagation through time.
  • BBB adaptation: Applying BBB raises two design questions: when to sample recurrent parameters and how to weight the KL regulariser.Figure 1 illustrates the BBB adaptation to an RNN.
  • BBB adaptation: For a length-T sequence, the variational objective contains one KL penalty, so each weight is penalised once rather than T times; sampled parameters remain fixed throughout the sequence.
  • Truncated training: Long sequences and large models make full unrolling prohibitive, so practical training uses minibatches of truncated sequences.
  • Truncated training: The truncated objective distributes KL responsibility across B minibatches and C truncated sequences, with the KL penalty equally allocated across them.Subsequent truncated sequences carry forward the previous sequence's final RNN state, and fresh parameters are sampled for each minibatch.

4 POSTERIOR SHARPENING

Posterior sharpening makes the variational posterior batch-dependent by using local gradient information, aiming to improve its accuracy and stabilise Bayesian neural-network training. The method uses a hierarchical posterior and supports Bayesian model averaging, with inference and computational trade-offs.

  • Posterior sharpening: Posterior sharpening adds batch-derived gradient information to the variational posterior, making it more flexible and potentially more accurate.The posterior is adapted locally to each minibatch, forming a local approximation from a global posterior.
  • Outcome: The sharpened posterior yields more stable optimisation, addressing a stated common pitfall of Bayesian neural-network training.The paper connects this approach to hierarchical variational schemes and minibatch-conditioned proposals inspired by VAEs.
  • Posterior sharpening: A neural-network transformation of minibatches is impractical for million-dimensional parameter vectors, so the method parameterises the posterior using θ and its loss gradient.Both θ and gθ are d-dimensional vectors, avoiding an additional large neural network.
  • Posterior sharpening: The hierarchical posterior uses q(ϕ) as in standard Bayes by Backprop and learns η as a per-parameter learning rate.The gradient-based update shifts the sampled parameters according to η and the minibatch gradient.
  • Training: Training samples truncated-sequence minibatches, samples ϕ and θ, then updates µ, σ, and η using gradients of the posterior-sharpening loss.The loss combines negative log likelihood with a KL divergence between the batch-conditioned posterior and its prior.
  • Training: The lower bound improves when the likelihood gain along the gradient exceeds the added KL cost, while learned η controls the trade-off between curvature improvement and KL loss.The authors state that effectiveness depends on sufficiently large likelihood curvature and identify more powerful parameterisations as future work.
  • Inference: Posterior sharpening supports two inference choices: use q(ϕ), or use the batch-conditioned q(θ|ϕ, (x, y)) with an additional KL term and gradient computation.The second option yields an upper bound on perplexity and incurs a penalty in training speed; for RNNs, truncated BPTT is used because the exact gradient is inefficient.

5 RELATED WORK

The paper situates posterior sharpening among probabilistic line-search, dynamic-evaluation, learning-to-optimise, and earlier Bayesian neural-network and recurrent-network methods. Its distinguishing feature is using gradient information to inform a variational posterior while learning adaptation parameters.

  • Related optimisation methods: Posterior sharpening resembles line search because learned η moves the posterior along a gradient direction; it can also be interpreted as a trust-region method.The paper states that its probabilistic treatment is the first to combine a variational posterior with the reparameterisation-trick or perturbation-analysis gradient in this way.
  • Related recurrent methods: Dynamic evaluation trains an RNN during evaluation with a fixed learning rate and cumulative updates using only previously seen data.The cited approach is deterministic and ignores the KL between a posterior with privileged information and a prior.
  • Related optimisation methods: Learning-to-optimise learns update rules or learning rates, whereas this paper treats learning rates as free parameters that adapt more quickly to non-stationary parameter distributions.The paper highlights gradient information as an input to its variational posterior.
  • Bayesian neural networks: Earlier Bayesian neural-network work includes maximum-a-posteriori schemes, variational methods, variance updates, mixture posteriors, and variational interpretations of dropout.The related literature spans methods from Buntine and Weigend through Graves, Blundell and later dropout-based approaches.
  • Bayesian recurrent networks: Prior Bayesian recurrent-network studies include second-order online training, point-estimate weights, Monte Carlo dropout for LSTMs, and variational schemes with biased variance gradients.The paper explicitly compares its experiments with Monte Carlo dropout results for LSTMs.

6 EXPERIMENTS

Experiments apply Bayes by Backprop to established LSTM architectures for language modelling and image captioning, evaluating perplexity, uncertainty calibration, pruning, and caption quality. Bayesian approaches improve benchmark performance and calibration, while posterior sharpening further reduces perplexity and pruning removes many weights with little validation impact.

  • Language modelling: The experiments replace dropout with Bayes by Backprop in a strong two-layer, 650-unit LSTM configuration for Penn Treebank language modelling.The architecture and network configuration are kept unchanged to assess Bayesian training on an established model.
  • Language modelling: Bayes by Backprop is compared with the LSTM dropout baseline, Variational LSTMs, and dynamic evaluation using word-level perplexity.Table 1 reports Penn Treebank perplexity, with lower values better and DE denoting Dynamic Evaluation.
  • Language modelling: 258 perplexity after one epoch with standard BBB improves to 227 with posterior sharpening.This comparison evaluates the variance-reduction capability of posterior sharpening.
  • Language modelling: 80% of weights can be removed with little impact on validation perplexity when weights are pruned in reverse signal-to-noise-ratio order.Weights are ordered by |µ_i|/σ_i before being set to zero.
  • Uncertainty analysis: On the reversed Penn Treebank test set, BBB reaches about 0.67 nats/word with 10 samples, versus below 0.58 for MC Dropout.Using the posterior mean gives slightly below 0.65 nats/word, while turning off dropout gives below 0.62.
  • Image caption generation: For MSCOCO image captioning, BBB significantly improves BLUE and CIDER and outperforms the dropout baseline by a large margin.The evaluation reports perplexity, BLUE-4, and CIDER against the Show and Tell model.

7 DISCUSSION

The paper applies Bayes by Backprop to RNNs and extends it with posterior sharpening, reporting improvements in language modelling and image captioning alongside better regularisation and uncertainty properties.

  • Posterior sharpening creates a hierarchical weight posterior that adapts locally to each data batch using model gradients.
  • Bayes by Backprop RNNs improved over two open-source baseline models in language modelling and image captioning.
  • The reported Bayesian RNNs were better regularised and had stronger uncertainty properties on out-of-distribution data than corresponding baselines.
  • The approach is presented as compatible with enhancements to recurrent and non-recurrent neural architectures.

A.1 BAYES BY BACKPROP ALGORITHM

The Bayes by Backprop algorithm estimates gradients for variational posterior parameters by sampling network weights, running ordinary backpropagation, and incorporating KL-term gradients.

  • Bayes by Backprop minimizes variational free energy with respect to the posterior mean and standard deviation parameters.
  • The procedure samples ϵ from a standard multivariate normal and reparameterizes weights as θ = µ + σϵ.
  • After forward and backward propagation, the algorithm combines the network gradient with KL gradients to update the posterior parameters.

A.2 LSTM EQUATIONS

The appendix defines the recurrent and LSTM state transitions, including the gates that control how inputs affect outputs and internal cell state.

  • An RNN core maps the state s_t and input observation x_t to the next state s_{t+1}.
  • An LSTM state contains an internal cell state c_t and an exposed state h_t.
  • Input, forget, and output gates modulate the effects of inputs on an LSTM cell’s outputs.
  • The weight and bias pairs W_i/b_i, W_f/b_f, W_c/b_c, and W_o/b_o affect the input gate, forget gate, cell update, and output gate, respectively.

A.3 WEIGHT PRUNING

The paper analyzes pruning in a converged LSTM by removing weights with the lowest signal-to-noise ratios and visualizes the resulting pruning patterns.

  • Around 80% of network weights could be removed with little impact on validation perplexity.
  • Weights were ranked by |µ_i|/σ_i and removed in reverse order, setting selected parameters to zero.
  • Figure 5 shows pruning patterns for one 650-unit LSTM cell from the converged model.
  • The figure encodes dropped parameters with white dots, zeroed rows with horizontal white lines, and per-row removals in the final column.
Loading 1704.02798v4…