Source-linked AI summary
Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
Yarin Gal, Zoubin Ghahramani
TL;DR
CNNs overfit quickly when labelled data is limited, creating a need for methods that work with small datasets. The paper places probability distributions over CNN kernels and approximates the posterior with Bernoulli variational distributions, while interpreting dropout training as Bayesian inference. The resulting approach improves robustness to over-fitting and classification accuracy, including state-of-the-art CIFAR-10 results, without additional training-time computational cost.
Problem
CNNs overfit quickly on small datasets, while modelling distributions over CNN kernels had not previously been attempted successfully.
Method
The paper places a probability distribution over CNN kernels, approximates the intractable posterior with Bernoulli variational distributions, and casts dropout training as approximate Bayesian inference.
Results
The proposed models improve robustness to over-fitting and classification accuracy, with better test accuracy at no additional training-time computational cost and state-of-the-art CIFAR-10 results.
Takeaways & Limitations
MC dropout provides a theoretically justified way to approximately integrate over convolutional kernels, though the choice of inference approximation should depend on the problem because test time is slower.
Takeaways & Limitations
The Bernoulli variational distribution is a fairly weak approximation, and the approach showed no improvement on ImageNet with the same models.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) work well on large datasets. But labelled data is hard to collect, and in some applications larger amounts of data are not available. The problem then is how to use CNNs with small data -- as CNNs overfit quickly. We present an efficient Bayesian CNN, offering better robustness to over-fitting on small data than traditional approaches. This is by placing a probability distribution over the CNN's kernels. We approximate our model's intractable posterior with Bernoulli variational distributions, requiring no additional model parameters. On the theoretical side, we cast dropout network training as approximate inference in Bayesian neural networks. This allows us to implement our model using existing tools in deep learning with no increase in time complexity, while highlighting a negative result in the field. We show a considerable improvement in classification accuracy compared to standard techniques and improve on published state-of-the-art results for CIFAR-10.
1 INTRODUCTION
CNNs perform strongly but overfit quickly on small datasets, while Bayesian CNNs had not previously modeled distributions over convolutional kernels successfully. The paper connects dropout with Bernoulli variational inference to build practical Bayesian CNNs and reports improved accuracy without additional training cost.
- CNNs require large amounts of data for regularisation and quickly over-fit on small data, whereas Bayesian neural networks are robust to over-fitting and can learn from small datasets.
- Inferring Bayesian neural-network posteriors is difficult, and Gaussian variational approximations can substantially increase parameters without improving predictive performance over dropout.One cited example doubles the number of model parameters while reporting the same predictive performance as traditional dropout.
- Dropout training can be cast as approximate Bernoulli variational inference in Bayesian neural networks, enabling convolution and pooling in probabilistic models.
- Applying dropout after convolution layers approximately integrates over CNN kernels, with Monte Carlo dropout averaging stochastic forward passes at test time.
- The proposed dropout CNN architectures achieve better test accuracy than existing approaches with no additional computational cost during training and reduce over-fitting on small datasets.
- The paper identifies that dropout approximation fails in some network architectures, explaining why dropout is not commonly used with convolutions.
2. Casting dropout as variational inference in Bayesian neural networks,
The paper proposes MC dropout for convolutions by interpreting dropout as approximate Bayesian inference. It then compares the resulting techniques empirically.
- The Bayesian interpretation of dropout supports using MC dropout for convolutions to address failures of the standard dropout approximation.
- The resulting techniques are compared empirically.
- The paper derives connections between dropout and approximate inference in Bayesian neural networks before presenting and evaluating a Bayesian CNN.
2 BACKGROUND
Probabilistic modelling places distributions over functions or neural-network weights, then uses variational inference when posterior calculations are intractable. Dropout provides a stochastic regularisation mechanism related to this Bayesian formulation.
- Probabilistic modelling: Bayesian modelling places a prior over functions and seeks the posterior distribution given observed inputs and outputs.The posterior supports prediction for a new input by integrating over possible functions.
- Variational inference: Variational inference approximates an analytically intractable posterior with an easier distribution q(ω) by minimising KL divergence.The optimisation is equivalent to maximising the log evidence lower bound.
- Dropout: Dropout samples binary variables for network units, dropping a unit when its variable is zero and reusing the sampled values during backpropagation.Each variable takes value 1 with layer-specific probability p_i.
- Bayesian neural networks: Bayesian neural networks place prior distributions over weight matrices, while biases may use point estimates.A standard matrix Gaussian prior is given for the weights.
- Bayesian neural networks: Posterior inference in Bayesian neural networks remains difficult even when the model has relatively few parameters.This motivates approximate inference methods.
3 DROPOUT AS APPROXIMATE VARIATIONAL INFERENCE IN BAYESIAN NEURAL NETWORKS
The paper casts dropout training as Bernoulli variational inference in Bayesian neural networks. Sampling from the variational weight distribution is equivalent to dropout, while Monte Carlo integration yields MC dropout predictions.
- Variational approximation: Bayesian neural-network posterior inference is intractable, so the paper approximates it with a variational distribution over the weights.The variational distribution is defined separately for each layer.
- Bernoulli variational distributions: The approximating distribution uses Bernoulli variables z_i,j sampled with layer-specific probabilities p_i and optimised parameters M_i.The diagonal operator forms a diagonal matrix from the sampled variables.
- Optimisation: The variational objective is estimated with Monte Carlo integration because its integral cannot be evaluated analytically.The resulting estimator for the variational objective is unbiased.
- Dropout equivalence: Sampling weights from q(W_i) is identical to performing dropout on a network whose weights are M_i.This establishes the equivalence between dropout training and the variational Bayesian model.
- Prediction: At prediction time, the approximate posterior replaces the exact posterior, and Monte Carlo integration produces MC dropout.MC dropout averages stochastic forward passes.
4 RELATION TO GAUSSIAN PROCESSES
The work extends the Bayesian interpretation of dropout beyond Gaussian processes to Bayesian neural networks. This extension supports convolution operations that need not have a corresponding Gaussian-process interpretation.
- Relation to Gaussian processes: The paper extends prior work relating dropout to approximate inference in Gaussian processes.Earlier work showed that a specific approximating distribution produces dropout’s objective in deep networks.
- Gaussian processes: Gaussian processes model distributions over functions, including joint Gaussian distributions over function values in regression.Regression observations are generated from a normal distribution centred on the function values.
- Extension to convolutions: The paper’s Bayesian-neural-network extension represents convolution operations, which do not necessarily have a corresponding Gaussian-process interpretation.Convolutions can instead be modelled as Bayesian neural networks.
5 BAYESIAN CONVOLUTIONAL NEURAL NETWORKS
The Bayesian CNN places distributions over convolution kernels and implements Bernoulli approximate inference with dropout around convolution layers. Because standard test-time dropout performs poorly there, the method uses MC dropout to average stochastic predictions.
- Implementation: Bayesian neural networks can be implemented by adding dropout after selected weight layers and evaluating the predictive posterior at test time.The posterior distribution acts as a regulariser by approximately integrating over weights.
- Convolution reformulation: Convolution is reformulated as a matrix product over vectorised input patches, allowing kernel integration within the Bayesian framework.Each patch contains h × w × K_i−1 values and is processed with the layer’s kernels.
- Bernoulli convolutional inference: Bernoulli random variables are applied to kernel–patch pairs, equivalently implementing dropout after each convolution layer before pooling.The construction randomly sets kernels to zero for different patches.
- Test-time prediction: Standard test-time dropout performs poorly after convolutions, so the paper averages stochastic forward passes using MC dropout.This approximates the predictive distribution at test time.
6 EXPERIMENTS
Experiments evaluate Bayesian CNN dropout on MNIST and CIFAR-10, small-data over-fitting, and existing CNN architectures. MC dropout improves test performance over Standard dropout, while the method remains limited on sufficiently small datasets and showed no ImageNet improvement.
- Bayesian convolutional neural networks: MC dropout with dropout after every parameter layer outperformed all compared LeNet models on MNIST and CIFAR-10.Standard dropout made the same lenet-all architecture perform poorly, whereas averaging stochastic forward passes produced the best results.
- Bayesian convolutional neural networks: The dropout approximation can fail in some CNN architectures because Standard dropout’s weight averaging differs from Monte Carlo averaging of stochastic forward passes.The experiments motivate MC dropout after convolution layers rather than relying on Standard dropout at test time.
- Model over-fitting: 15,000 training examples triggered over-fitting for Standard dropout lenet-ip, while Bayesian CNN lenet-all achieved better classification accuracy on MNIST.With 1,875 examples, both methods over-fit and required other regularisation.
- Model over-fitting: The proposed Bayesian CNN uses additional dropout layers to approximate Bayesian integration over kernels, improving robustness to over-fitting without additional model parameters.The Bernoulli variational distribution is a weak approximation, which explains over-fitting on sufficiently small datasets.
- MC dropout in standard convolutional neural networks: MC dropout consistently reduced CIFAR-10 test error across NIN, DSN, and Augmented-DSN compared with Standard dropout.The lowest reported Augmented-DSN error across five repetitions was 7.51.
- MC dropout in standard convolutional neural networks: No improvement was observed on ImageNet, possibly because its larger dataset already provides sufficient regularisation.The authors leave testing stronger regularisation on ImageNet subsets for future research.
7 CONCLUSIONS AND FUTURE RESEARCH
The paper presents Bayesian CNNs that improve robustness to over-fitting on small data while using existing tools with almost no implementation overhead. It also justifies MC dropout for CNNs, while noting a test-time cost and problem-dependent inference choices.
- The proposed Bayesian CNN places probability distributions over CNN kernels and uses Bernoulli variational distributions without additional model parameters.
- The implementation uses existing tools and requires almost no overheads.
- MC dropout is theoretically justified as approximate integration of CNN kernels and empirically improves architectures where standard dropout fails.
- Test time is scaled by the number of averaged forward passes, so the optimal inference approximation should depend on the problem.
- Future research includes studying Gaussian process interpretations of convolution and pooling and the effect of dropout probabilities on learnt filters.
A.1 BAYESIAN CONVOLUTIONAL NEURAL NETWORKS
The appendix specifies the network configurations and common optimisation settings used for MNIST and CIFAR-10 models.
- MNIST uses the LeNet network with dropout probability 0.5 in every dropout layer.
- CIFAR-10 uses the same setup, with 192 outputs per convolution layer and 1000 units in the final inner-product layer.
- All models use the same optimisation settings, including momentum 0.9, base learning rate 0.01, and weight decay 0.0005.
- The models are optimised for 1e7 iterations on MNIST and 1e5 iterations on CIFAR-10.
A.2 TEST TIME COMPLEXITY
The Bayesian CNN keeps training time identical to existing models but increases test time through averaged stochastic forward passes. These passes can be run concurrently or in distributed hardware.
- Training time is identical to that of existing models, while test time is scaled by T, the number of averaged forward passes.
- MC dropout estimates are formed by running the same input through dropout multiple times and averaging the resulting probability rows.
- Distributed hardware can execute the stochastic forward passes concurrently, making the test-time cost nearly constant in practice.