PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference

Kumar Shridhar, Felix Laumann, Marcus Liwicki

arXiv:1901.02731v1cs.LGstat.ML

TL;DR

Point-estimate neural networks can become overconfident when data are limited, while Bayesian CNN posterior inference has been difficult for large convolutional models. The paper applies variational Bayesian CNNs across multiple tasks, estimates aleatoric and epistemic uncertainty, and reports frequentist-equivalent performance, natural regularization, and effective pruning.

  • Problem

    Point-estimate neural networks can be overconfident on limited data, while Bayesian posterior inference over CNN kernels has been difficult because practical models contain many parameters.

  • Method

    The paper uses Bayes by Backprop with variational inference to place distributions over CNN parameters, estimate aleatoric and epistemic uncertainty, and prune the resulting architectures.

  • Results

    Bayesian CNNs achieve state-of-the-art results comparable to the same architectures trained by frequentist inference across image recognition, super-resolution, and GAN applications.

  • Takeaways & Limitations

    The proposed Bayesian CNNs provide uncertainty estimates and natural regularization, while pruning can make parameter counts comparable to frequentist models without reducing performance.

  • Takeaways & Limitations

    Variational posterior approximation in Bayesian neural networks can be computationally expensive, and Bayesian CNNs use twice as many parameters as point-estimate models before pruning.

Abstract

from arXiv · show

Artificial Neural Networks are connectionist systems that perform a given task by learning on examples without having prior knowledge about the task. This is done by finding an optimal point estimate for the weights in every node. Generally, the network using point estimates as weights perform well with large datasets, but they fail to express uncertainty in regions with little or no data, leading to overconfident decisions. In this paper, Bayesian Convolutional Neural Network (BayesCNN) using Variational Inference is proposed, that introduces probability distribution over the weights. Furthermore, the proposed BayesCNN architecture is applied to tasks like Image Classification, Image Super-Resolution and Generative Adversarial Networks. The results are compared to point-estimates based architectures on MNIST, CIFAR-10 and CIFAR-100 datasets for Image CLassification task, on BSD300 dataset for Image Super Resolution task and on CIFAR10 dataset again for Generative Adversarial Network task. BayesCNN is based on Bayes by Backprop which derives a variational approximation to the true posterior. We, therefore, introduce the idea of applying two convolutional operations, one for the mean and one for the variance. Our proposed method not only achieves performances equivalent to frequentist inference in identical architectures but also incorporate a measurement for uncertainties and regularisation. It further eliminates the use of dropout in the model. Moreover, we predict how certain the model prediction is based on the epistemic and aleatoric uncertainties and empirically show how the uncertainty can decrease, allowing the decisions made by the network to become more deterministic as the training accuracy increases. Finally, we propose ways to prune the Bayesian architecture and to make it more computational and time effective.

1 Introduction

The paper addresses overconfident neural-network decisions and difficult Bayesian posterior inference by introducing Bayesian convolutional networks with variational inference. It develops efficient convolutional Bayesian learning and evaluates uncertainty, regularization, pruning, and applications across multiple tasks.

  • Problem: Convolutional neural networks can overfit small datasets and make overly confident decisions because they cannot correctly assess predictive uncertainty.These models require substantial training data and may fit training data without predicting well on new data.
  • Bayesian approach: Bayesian convolutional networks place probability distributions over weights to provide uncertainty estimates and regularization instead of relying on single point estimates.Averaging over parameter values supports robustness to overfitting, while priors provide a regularization effect.
  • Variational inference: Exact Bayesian inference is intractable for neural networks, so Bayes by Backprop approximates the true posterior with Gaussian variational distributions whose parameters are learned by minimizing KL divergence.The approximation uses distributions characterized by means and variances for the network parameters.
  • Method: The proposed CNN implementation applies two convolutional operations, one for the mean and one for the variance, and uses cheap model averaging to learn richer representations and predictions.The method is presented as an efficient application of Bayes by Backprop to convolutional architectures.
  • Results: The method is reported to match frequentist performance in identical architectures, estimate aleatoric and epistemic uncertainty, and support deterministic decisions as training accuracy increases.The paper also reports natural regularization, reduced epistemic uncertainty with more data, and applicability across CNN architectures without performance limitations.

2 Background

Neural networks learn task-relevant representations from examples, while Bayesian inference replaces fixed weights with distributions to represent uncertainty and reduce overfitting. Variational inference makes the otherwise intractable posterior computationally usable.

  • Neural Networks: Artificial neural networks learn complex data patterns through interconnected layers that transform inputs into predictions.Biological analogies motivate the use of interconnected processing elements and layered representations.
  • Convolutional Neural Networks: Convolutional networks learn progressively more complex features while sharing weights across image locations to reduce the number of learned parameters.Weight sharing avoids relearning the same features across different image regions.
  • Bayesian Inference: Bayesian inference places a prior over functions or weights and predicts new outputs by integrating over possible functions or parameters.The posterior predictive integral is introduced before approximation.
  • Variational Inference: Because exact posterior integration is intractable, variational inference approximates p(w|X, Y) with q(w) by minimizing KL(q(w) || p(w|X, Y)).The approximation produces a computable predictive distribution.
  • Local Reparameterization: The local reparameterization trick converts global weight uncertainty into local activation noise and reduces stochastic-gradient variance as 1/M.The method samples intermediate variables rather than directly sampling ε.
  • Uncertainty: Point-estimate networks can become overconfident on limited data, whereas Bayesian uncertainty separates irreducible data noise from reducible model uncertainty.Aleatoric uncertainty cannot be reduced by collecting more data, while epistemic uncertainty can.

3 Related Work

Earlier Bayesian neural-network work developed posterior approximations, stochastic optimization, dropout interpretations, and pruning methods, but scalability and convolutional applicability remained important concerns.

  • Bayesian Neural Networks: Earlier Bayesian neural-network methods included MAP schemes, smoothness-inducing priors, minimum-description-length losses, and posterior modelling.These approaches explored regularization and uncertainty in neural-network parameters.
  • Scalability: Hybrid Monte Carlo was difficult to scale to modern neural networks and large datasets, motivating more scalable approximate-inference methods.The cited work explicitly identifies limitations in both architecture and data-size scalability.
  • Approximate Inference: Variational inference and expectation propagation provided more scalable posterior approximations, while mixture-posteriors extended the variational approach.The cited methods differ in posterior form and computational strategy.
  • Dropout-Based Uncertainty: Dropout and Gaussian Dropout have been interpreted as approximate variational-inference schemes and provide uncertainty through predictions under different dropout masks.The paper compares its methodology with Gal and Ghahramani’s dropout-based framework.
  • Model Compression: Pruning research has used saliency, sparsity, reduced numerical precision, quantization, and factorization to reduce model size or computation.The cited approaches differ in their dependence on architecture and inference hardware.

4 Our Concept

The proposed Bayesian CNN places distributions over convolutional and fully connected weights, using local reparameterization and separate operations for mean and variance. It estimates predictive uncertainty and applies sparsification to control the additional parameter cost.

  • Bayesian CNN Architecture: A fully Bayesian CNN assigns probability distributions to weights in both convolutional filters and fully connected layers.Distributions only in convolutional layers do not constitute a fully Bayesian CNN for most architectures.
  • Local Reparameterization: The local reparameterization trick samples layer activations instead of weights to accelerate computation in convolutional layers.The convolutional formulation defines noise, receptive fields, convolution, and component-wise multiplication.
  • Mean and Variance: Bayesian convolution uses two operations: one learns the posterior mean and the other learns its variance parameter.The mean operation follows a point-estimate optimization, while the variance operation learns α.
  • Predictive Uncertainty: Predictive distributions for unseen classifications are estimated by sampling Gaussian variational weights and computing predictive variance.The predictive variance can be decomposed into aleatoric and epistemic uncertainty.
  • Uncertainty Decomposition: Separating aleatoric and epistemic uncertainty indicates whether poor performance is associated with noisy data or model uncertainty.Only epistemic uncertainty is described as reducible through additional data.
  • Parameter Efficiency: Bayesian CNNs learn mean and variance parameters, doubling parameters relative to point-estimate networks unless filters are reduced or weights are pruned.The paper halves filters for parameter parity and uses L1-based thresholding to retain important non-zero weights.

5 Empirical Analysis

The empirical analysis evaluates Bayesian CNNs with variational inference across classification, uncertainty, regularization, pruning, and training-time behavior. Results are generally comparable to frequentist architectures while providing uncertainty estimates and regularization effects.

  • Method: Bayesian CNNs store Gaussian-distributed weights using mean and variance parameters, optimized through Bayes by Backprop and variational free energy.Variance is parameterized with Softplus to prevent it from becoming zero.
  • Classification: Bayesian CNNs achieve validation accuracies comparable to corresponding frequentist architectures on MNIST and CIFAR-10.On CIFAR-10, Bayesian LeNet-5 with Dropout reported higher accuracy, but the authors could not reproduce that result.
  • Uncertainty: Decreasing posterior standard deviation over epochs indicates that Bayesian CNN parameters become more confident as more data is processed.This decrease continued even after Bayesian LeNet-5 reached 99% MNIST validation accuracy.
  • Regularization: Bayesian inference reduces overfitting on CIFAR-100, performing comparably to frequentist AlexNet with three dropout layers.Using 100 samples overfit slightly less than using 25 samples, but the authors retained 25 samples for other experiments.
  • Uncertainty: The aleatoric uncertainty of CIFAR-10 is about twenty times that of MNIST.The analysis relates the larger CIFAR-10 aleatoric uncertainty to its lower validation accuracy and irreducible variability.
  • Pruning: Halving AlexNet’s filters produced a 6 per cent accuracy gain on CIFAR-10 and equivalent performance on MNIST and CIFAR-100.The authors suggest reduced overfitting or improved generalization as possible explanations, without distinct filter-visualization evidence.
  • Computational Cost: Bayesian CNN training time is twice that of a comparable frequentist network when using one sample, because each weight has mean and variance parameters.The general expression defines training time as T = 2 ∗ number of samples ∗ t.

6 Application

The paper applies Bayesian CNNs to image super-resolution and generative image modeling. In super-resolution, the network extracts low-resolution features before efficient upscaling; in DCGANs, Bayesian architectures generate CIFAR-10 samples comparable in number and quality to the original model.

  • Image Super-Resolution: Super-resolution recovers a high-resolution image from a low-resolution input and is used in areas including medical imaging and face recognition.The paper reviews prediction-, edge-, and patch-based approaches before presenting its Bayesian CNN application.
  • Image Super-Resolution: The Bayesian super-resolution model extracts features in low-resolution space and uses efficient sub-pixel convolution to produce the high-resolution output.This avoids performing the main super-resolution computation directly in high-resolution space.
  • Image Super-Resolution: The super-resolution network replaces standard convolution with Bayesian convolution while retaining PyTorch’s PixelShuffle layer.The forward pass computes the mean, variance, and KL divergence, using an upscale factor of 3 in the experiments.
  • Image Super-Resolution: On BSD300, Bayesian super-resolution results were comparable to the original paper in the number and quality of generated images.The authors describe this experiment as demonstrating that Bayesian networks can be used for image super-resolution.
  • Generative Adversarial Networks: The Bayesian DCGAN follows the DCGAN architecture, using Bayesian convolutional layers in generator and discriminator networks.The experiments use CIFAR-10 images with normalization, batch size 64, Adam optimization, and standard DCGAN-style activations.
  • Generative Adversarial Networks: The paper presents fake CIFAR-10 samples generated by the Bayesian DCGAN and real CIFAR-10 samples for comparison.The figures provide qualitative visual examples rather than a reported quantitative metric.

7 Conclusion and Outlook

The paper presents Bayesian CNNs with variational inference, uncertainty estimation, pruning, and applications across image and GAN tasks. It also identifies future work involving posterior sharpening, prior design, and Bayesian generative networks.

  • Bayesian CNNs use Bayes by Backprop to place prior distributions over model parameters and estimate aleatoric and epistemic uncertainty.
  • The approach achieves state-of-the-art results comparable to identical architectures trained with frequentist inference.
  • Epistemic uncertainty decreases with more training data, while Bayesian methods provide natural regularization compared with dropout.
  • Pruning can make Bayesian CNN parameter counts comparable to frequentist models while maintaining equal or better performance.
  • Bayesian CNNs are applied to image recognition, image super-resolution, and GANs, with initial results for Bayesian discriminators in GANs.
  • Future work includes posterior sharpening, alternative initialization and prior designs, and investigation of Bayesian generative networks in GANs.

Bayesian Settings

The Bayesian settings use a sample size range selected for experimental performance.

  • Sample sizes range from 10 to 25, with most experiments using either 10 or 25 unless otherwise specified.

Non Bayesian Settings

The non-Bayesian settings use Normal initialization for consistency with the Bayesian networks.

  • Weights were initialized with a Normal distribution having mean = 0 and variance = 1.

Architectures

The architectures section lists the original LeNet and AlexNet configurations alongside AlexNetHalf, which halves AlexNet’s filters.

  • Table 11 presents the LeNet architecture with its original paper-defined configuration.
  • Table 12 presents the AlexNet architecture with its original paper-defined configuration.
  • Table 13 presents AlexNetHalf with the number of filters halved relative to the original architecture.

10 How to replicate results

To replicate the results, install PyTorch and the repository dependencies, then enter the folder for the selected task. Separate commands reproduce Bayesian and Frequentist CNN results, with additional guidance in the repository README.

  • Install PyTorch from the official website and install the repository requirements after cloning it.The setup uses the PyTorch-BayesianCNN GitHub repository and its requirements.txt file.
  • Enter the folder corresponding to the task to replicate: Image Recognition, Super Resolution, or GAN.
  • Use the Bayesian CNN replication procedure to reproduce the Bayesian CNN results.
  • Run python main_nonBayesian.py to reproduce the Frequentist CNN results.
  • Read the repository README sections for more replication details.
Loading 1901.02731v1…