Source-linked AI summary
Large-Scale Visual Active Learning with Deep Probabilistic Ensembles
Kashyap Chitta, Jose M. Alvarez, Adam Lesnikowski
TL;DR
Active learning can use uncertainty to select training data, but applying Bayesian neural networks (BNNs) at scale requires approximations. This paper introduces regularized Deep Probabilistic Ensembles (DPEs) for visual active learning, reporting improvements over baselines across classification and segmentation, including up to 20% IoU improvement in underrepresented classes.
Problem
BNNs provide a principled approach to uncertainty estimation, but deep models require approximations and can be difficult to optimize and use at scale.
Source: §1 · p. 1Method
DPEs use a KL-regularized ensemble objective derived from variational inference to approximate BNNs for active visual learning.
Source: §1 · p. 1, §3.2 · p. 4Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Annotating the right data for training deep neural networks is an important challenge. Active learning using uncertainty estimates from Bayesian Neural Networks (BNNs) could provide an effective solution to this. Despite being theoretically principled, BNNs require approximations to be applied to large-scale problems, where both performance and uncertainty estimation are crucial. In this paper, we introduce Deep Probabilistic Ensembles (DPEs), a scalable technique that uses a regularized ensemble to approximate a deep BNN. We conduct a series of large-scale visual active learning experiments to evaluate DPEs on classification with the CIFAR-10, CIFAR-100 and ImageNet datasets, and semantic segmentation with the BDD100k dataset. Our models require significantly less training data to achieve competitive performances, and steadily improve upon strong active learning baselines as the annotation budget is increased.
1. Introduction
Active learning uses model uncertainty to select data for annotation, but Bayesian neural networks (BNNs) are difficult to apply at scale. The paper proposes Deep Probabilistic Ensembles (DPEs), combining ensemble training with BNN-inspired regularization for visual active learning.
1. Introduction
- BNNs offer a principled approach to uncertainty, but deep models require approximations and can be difficult to optimize, tune, and run at test time.Estimating uncertainty can require many test-time samples, which is computationally demanding.Source: §1 · p. 1
- DPEs regularize ensembles using a KL term derived from variational inference to approximate BNNs, with negligible training overhead and no inference overhead compared with ensembles.The approach is designed for large-scale visual classification and is also applied to active semantic segmentation.Source: §1 · pp. 1–2
- On BDD100k semantic segmentation, DPEs yield up to 20% IoU improvement in underrepresented classes.The paper also applies DPEs to active image classification on large-scale datasets.Source: §1 · p. 2
2. Related Work
Prior work regularizes ensembles mainly to promote member diversity, while Bayesian neural networks (BNNs) use several approximation strategies for uncertainty estimation. Scalable active-learning methods include batch selection and ensemble-based uncertainty approaches.
2. Related Work
- Ensemble regularization has often promoted diversity to improve accuracy, whereas this work targets ensemble uncertainty estimates; BNN approximations include MCMC, variational inference, MC dropout, and Bayes by Backprop.MCMC is described as sample-inefficient and not applied to deep BNNs; variational inference can be fast but may perform poorly on deeper networks.Source: §2 · p. 2
- MC dropout and Bayes by Backprop have shown similar active-learning uncertainty performance, while BNN approximation estimates can underperform ensembles on large-scale visual tasks.MC dropout also requires careful dropout-rate tuning for reliable uncertainty estimates.Source: §2 · p. 2
- Classical active learning selects uncertain samples individually and retrains after each query, a process described as infeasible for deep networks.Adding one sample may have no statistically significant performance impact, while repeated retraining becomes computationally intractable.Source: §2 · p. 2
- Scalable batch approaches use pool filtering and diversity, core-set loss, or learned sample interestingness; ensemble uncertainty underlies state-of-the-art active learning.The paper evaluates gains over ensembles trained with its proposed regularization.Source: §2 · p. 2
3. Deep Probabilistic Ensembles
The method uses variational inference to motivate a KL regularizer over ensemble parameters, retaining ensemble optimization while approximating aspects of Bayesian uncertainty. Its objective combines supervised loss with that regularizer.
3.1. Variational Inference
- Variational inference restricts an intractable posterior computation to a family of distributions and selects the member closest to the true posterior by KL divergence.The minimized objective is the negative Evidence Lower Bound (ELBO).Source: §3.1 · pp. 2–3
- The variational objective combines a KL divergence to a chosen prior with expected negative log likelihood; its expectation makes BNN optimization difficult in part because it prevents co-adaptation.The paper proposes regularization intended to retain ensemble optimization simplicity while training BNN-like models.Source: §3.1 · p. 3
- DPEs are presented as simple to optimize like ensembles, with a training objective similar to BNNs and high performance with good uncertainty estimates.These are the properties contrasted in Table 1.Source: Table 1 · p. 3, §3.1 · p. 3
3.2. KL Regularization
- DPEs apply the KL divergence regularizer to the values each parameter takes across ensemble members, approximating their distribution with Gaussian mean and standard deviation estimates.The method assumes independent network parameters to compute the Gaussian KL analytically; the prior depends on the task.Source: §3.2 · p. 3
- The KL penalty constrains ensemble-member variances relative to the prior and keeps parameter means near prior means, while also promoting diversity when variance is low.The stated terms respectively prevent extremely large variances, penalize variances below the prior, and keep means close to the prior, especially at low variance.Source: §3.2 · p. 4
- The ensemble objective sums cross-entropy losses across models and adds the aggregated KL penalty, scaled by β; its likelihood term averages only the current ensemble configuration.This subset of assignments distinguishes the objective from traditional variational inference.Source: §3.2 · p. 4
4. Uncertainty Estimation for Active Learning
DPE active learning alternates training on labeled data with selecting samples from the unlabeled pool for annotation. Predictive entropy ranks samples, and each query in a batch is selected independently, enabling parallel addition but not accounting for within-batch effects.
4. Uncertainty Estimation for Active Learning
- Each iteration trains the ensemble on the current labeled set to convergence, then selects a subset of the unlabeled pool for annotation.The selected samples move into the labeled set before the next iteration.Source: §4 · p. 4
- The process continues until the total annotation budget is met, with the number of samples added per iteration fixed or variable.The experimental setup hides pre-existing labels and reveals them as samples are queried.Source: §4 · p. 4
- Predictive entropy of the ensemble ranks unlabeled samples, and the highest-scoring samples are selected for annotation.Because query entropies are independent, a batch can be added in parallel, but the method does not account for how samples within the batch affect one another’s uncertainty.Source: §4 · p. 4
5. Active Image Classification
The classification experiments examine growth schedules and acquisition functions, then compare DPE active learning with random, single-model, and L2-regularized ensemble baselines across CIFAR and ImageNet.
5. Active Image Classification
- Experiments span CIFAR-10, CIFAR-100, and ImageNet with ResNet-18 DPEs; ensembles use eight members and are retrained from scratch each iteration.The CIFAR datasets have 50,000 training images each; ImageNet has 1.28 million.Source: §5 · p. 5
5.1. Results
- 99.2% of the all-data upper-bound accuracy is reached with 32% of labeling effort in the CIFAR-10 growth-parameter experiment.Across tested growth parameters, performance remains competitive with the all-data upper bound; Linear-8 only marginally outperforms the other schedules despite requiring more retraining.Source: §5.1 · pp. 5–6, Figure 1 · p. 5
- Acquisition functions show no significant performance differences on the chosen tasks; Hens is competitive and outperforms alternatives at larger annotation budgets.The comparison includes random, categories-first entropy, mutual information, variance, and variation ratios.Source: §5.1 · p. 6, Table 3 · p. 6
- DPEs outperform random baselines and improve on single models and L2-regularized ensembles; significant gains occur on CIFAR-10 at all three budgets and CIFAR-100 at the largest.On ImageNet, DPEs perform similarly to L2 ensembles while active learning improves 3%–4% over random baselines.Source: Table 4 · p. 6, §5.1 · p. 7
6. Active Semantic Segmentation
The section adapts DPEs for active semantic segmentation, where pixel-level annotations are costly, using crop-based queries and class-weighted acquisition. On BDD100k, DPEs improve mean IoU over random selection and ensembles, with larger gains as the labeled set grows.
6. Active Semantic Segmentation
- A convolutional encoder with an ensemble of decoders concentrates KL regularization on higher-level decoder concepts, with training memory scaling linearly in the decoder subset.The encoder uses common L2 regularization.Source: §6 · p. 7
- The framework queries 4 × 3 image crops by summed pixel-level acquisition scores, and uses class weights to target selected classes.Crop queries can use annotation budgets more efficiently than querying whole images; per-pixel losses support training from partial crop annotations.Source: §6 · p. 7
- The BDD100k experiments use 84k crops from 7,000 training images, with active-learning iterations at 8%, 16%, and 32% of the data.A fully supervised model reaches 52.1% mean IoU, 1% below the cited similar-architecture baseline.Source: §6 · p. 8
6.1. Results
- At 26.9k training crops, DPEs improve mean IoU by nearly 2% over random and 1% over ensembles, recovering 96.2% of upper-bound performance with 32% of labeling effort.The Table 5 caption reports up to 1% improvement over ensembles; Table 6 compares class-wise IoU after 26.9k crops.Source: Table 5 · p. 7, §6.1 · p. 8
- For the motorcycle class, which has 240 training instances, the IoU difference is nearly 20%; class-weighted targeting improves traffic-sign IoU by 2% over Hens.The traffic-sign targeting also improves the wall-class IoU by 3.6% and narrows the traffic-sign upper-bound gap from 3.7% to 1.7%.Source: §6.1 · p. 8
7. Conclusion
The conclusion presents DPEs as a KL-regularized ensemble approach for uncertainty estimation that improves active learning across classification and semantic segmentation. It emphasizes scalability and gains for underrepresented segmentation classes.
7. Conclusion
- DPEs use a novel KL regularization term to approximate variational inference for BNNs and improve active learning across three image-classification datasets; on BDD100k, underrepresented classes gain up to 20% IoU.The approach is described as easy to integrate into existing frameworks and scalable to large models and datasets without impacting performance.Source: §7 · p. 8