Source-linked AI summary
The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning
Jiaming Zeng, Adam Lesnikowski, Jose M. Alvarez
TL;DR
Active learning uses model uncertainty to select unlabeled data for labeling, while Bayesian CNNs can be difficult and time-consuming to train. This paper varies the number and position of Bayesian layers in MNIST active learning and finds that a few near the output can capture uncertainty as well as fully Bayesian CNNs. The authors conclude this can combine deterministic CNN speed and accuracy with Bayesian uncertainty modeling.
Problem
Bayesian CNNs can estimate model uncertainty, but they are time-consuming and increasingly difficult to train as network size and parameter counts grow.
Source: §1 · p. 1Method
The study varies the number, position, and prior initialization of Bayesian layers in CNNs evaluated through active learning on MNIST.
Source: §1 · p. 1, §3 · p. 2, §4.1 · p. 3Results
Takeaways & Limitations
The authors conclude that deterministic CNN speed and accuracy can be combined with Bayesian CNN uncertainty modeling using a few Bayesian layers near the output.
Source: §5 · p. 4The experiments remain to be extended to larger networks, real-world applications, segmentation, and different prior types.
Source: §5 · p. 4
Abstract
from arXiv · showhide
One of the main challenges of deep learning tools is their inability to capture model uncertainty. While Bayesian deep learning can be used to tackle the problem, Bayesian neural networks often require more time and computational power to train than deterministic networks. Our work explores whether fully Bayesian networks are needed to successfully capture model uncertainty. We vary the number and position of Bayesian layers in a network and compare their performance on active learning with the MNIST dataset. We found that we can fully capture the model uncertainty by using only a few Bayesian layers near the output of the network, combining the advantages of deterministic and Bayesian networks.
1 Introduction
Active learning reduces labeling demands by training on a small labeled pool and selecting which unlabeled data to send to an oracle. The paper investigates whether Bayesian layers’ positions and number affect uncertainty capture, finding that a few near the network output can match traditional Bayesian CNNs on MNIST.
1 Introduction
- Active learning trains a model on a small labeled pool, then uses an acquisition function to choose unlabeled data for an external oracle to label.The acquisition function is often based on the model’s uncertainty.Source: §1 · p. 1
- Bayesian CNNs can estimate uncertainty, but they take a long time to train and become harder to train as networks and parameter counts grow.The cited work [2] reports improvements over existing active-learning approaches for high-dimensional data, while noting long training time.Source: §1 · p. 1
- A few Bayesian layers near the output can capture the same level of uncertainty as traditional Bayesian CNNs in active learning on MNIST.The study varies the number and position of Bayesian layers and reports comparable accuracy to [2].Source: §1 · p. 1
2 Bayesian Convolutional Neural Networks
Bayesian CNNs place prior probability distributions over model parameters. Because calculating the model evidence is complex, the paper describes using variational inference to approximate the posterior with a Gaussian distribution.
2 Bayesian Convolutional Neural Networks
- Bayesian CNNs place prior probability distributions over their model parameters.The passage also introduces a posterior distribution conditioned on data, though its displayed expression is incomplete.Source: §2 · pp. 1–2
- The paper describes the prediction from a Bayesian CNN as a distribution conditioned on a test input and the training data.The supplied passage gives only a partial prediction expression.Source: §2 · p. 2
- Variational inference approximates the true posterior with a simpler distribution because calculating the model evidence is complex.The approximate posterior q(w) is Gaussian, with parameters μq and σq.Source: §2 · p. 2
3 Experimental Setup
The experiments use the MNIST active-learning setup and network structure from [2], varying the number and position of Bayesian layers across eight architectures. Acquisition functions select data over repeated training cycles, with posterior predictions approximated using Monte Carlo samples.
3 Experimental Setup
- The experiments start with 20 labeled images and acquire 10 per cycle until reaching 1,000 images on MNIST.The setup follows [2].Source: §3 · p. 2
- Eight architectures vary the number and position of Bayesian layers in a CNN with two convolutional and two dense layers.The architecture uses 32 convolution kernels, a 4x4 kernel size, 2x2 pooling, and a 128-unit dense layer; Table 1 summarizes the Bayesian and deterministic layer combinations.Source: §3 · p. 2, Table 1 · p. 2
- Random, Max Entropy, and Variation Ratios serve as acquisition functions, with 100 Monte Carlo samples approximating class probabilities each cycle.Networks are trained for 200 epochs per acquisition cycle.Source: §3 · p. 2
- The Bayesian layers are trained with Flipout and ADAM, and each experiment is repeated and averaged over three runs.The stated ADAM learning rate is 0.001 and batch size is 64.Source: §3 · p. 2
4 Results
The results indicate that a Bayesian layer near the network output can capture model uncertainty in active learning, while adding more Bayesian layers may compromise accuracy without improving uncertainty modeling. Varying the initial variance further supports Dense2 as the most important layer for capturing uncertainty.
4 Results
- Input-side or no Bayesian layers underperform a fully Bayesian network, while a few output-side Bayesian layers outperform it; extra layers may compromise accuracy without added uncertainty benefit.Source: §4 · p. 3, Figure 2 · p. 3
- Bayesian CNN test errors are slightly lower than, but comparable to, MC Dropout results in [2].Source: Table 2 · p. 3, §4 · p. 3
4.1 Effect of Bayesian-ness of the Prior Initialization
- The authors tuned initial variance mean µ across −3, −5, −9, and −11; lower µ initializes closer to a deterministic CNN, while higher µ is more Bayesian.Table 3 compares test errors after optimizing the initial posterior variance.Source: §4.1 · p. 3, Table 3 · p. 4
- For max entropy, the fully Bayesian network is unaffected by initial µ; with one Bayesian layer, larger µ and Dense2 outperform Bayesian Conv1, and variation ratios show the same Dense2 advantage with larger variance.Source: §4.1 · p. 4, Figure 3 · p. 4
- A more Bayesian Dense2 layer captures the same uncertainty level as fully Bayesian CNNs while retaining CNN speed and accuracy.Source: §4.1 · p. 4
5 Conclusion
The experiments suggest that fully Bayesian CNNs are unnecessary for capturing model uncertainty: a few Bayesian layers near the output outperform the fully Bayesian network. The work identifies larger networks, real-world applications, other methods, and different priors as future research directions.
5 Conclusion
- One or two Bayesian layers near the output outperformed the fully Bayesian CNN in capturing model uncertainty.The authors conclude that fully Bayesian CNNs are unnecessary for this purpose.Source: §5 · p. 4
- Using fewer Bayesian layers near the output combines deterministic CNNs’ accuracy and speed with Bayesian CNNs’ ability to capture uncertainty.The authors state that this could increase the ease of implementing and using Bayesian deep learning.Source: §5 · p. 4
- The authors propose extending the experiments to larger networks, real-world applications, other methods such as segmentation, and different priors.Source: §5 · p. 4
A Variational Inference
Variational inference addresses the difficulty of calculating Bayesian CNN model evidence by approximating the intractable posterior with a simpler distribution. Minimizing the KL divergence is equivalent to minimizing the negative ELBO, which can serve as the loss.
A Variational Inference
- Because calculating the model evidence is difficult, Bayesian CNNs are typically solved with approximation methods such as variational inference.Source: A Variational Inference · p. 5
- Variational inference approximates the intractable posterior p(w|X, y) with a simpler distribution q(w) by minimizing their KL divergence.Source: A Variational Inference · p. 5
- Minimizing the KL divergence is equivalent to minimizing the negative ELBO, which can be used as the Bayesian CNN loss.Source: A Variational Inference · p. 5
B Estimating Model Uncertainty
The method estimates prediction uncertainty by marginalizing over an approximate posterior. It uses sampled weight sets in repeated forward passes and averages their predictions to approximate the predictive distribution.
B Estimating Model Uncertainty
- Prediction uncertainty is estimated by marginalizing over the approximate posterior using Monte Carlo integration.Source: B Estimating Model Uncertainty · p. 5
- Each of T forward passes samples a different set of weights from the Bayesian CNN.Source: B Estimating Model Uncertainty · p. 5
- Averaging the predictions from those passes approximates the predictive distribution.Source: B Estimating Model Uncertainty · p. 5