Bayesian Generative Active Deep Learning

Toan Tran; Thanh-Toan Do; Ian Reid; Gustavo Carneiro

Source-linked AI summary

Bayesian Generative Active Deep Learning

Toan Tran, Thanh-Toan Do, Ian Reid, Gustavo Carneiro

arXiv:1904.11643v1Apr 26, 2019cs.LGstat.ML

TL;DR

The paper addresses the resource demands of deep learning when large labeled datasets are costly to create and train on. It combines Bayesian active learning with VAE-ACGAN-based generative augmentation to create informative samples, and reports consistently better classification results and more efficient training than the compared approaches. The authors identify limited comparison with a closely related method and difficulty optimizing certain complex acquisition functions as limitations or boundaries.

Problem
Large labeled datasets require substantial human labeling and computational resources, motivating training methods that reduce this dependence.
Method
The method selects informative unlabeled samples with BALD, labels them, and uses VAE-ACGAN to generate informative artificial samples for iterative training.
Results
The proposed approach consistently outperforms other methods across three datasets, while producing better classification results than data augmentation for similarly sized training sets.
Takeaways & Limitations
Informative sample generation can reduce the human and computer resources needed for labeling and model training within the evaluated classification problems.
Takeaways & Limitations
The paper lacks a comparison with the only other literature method proposing a similar approach and notes that complex acquisition functions can be difficult to optimize in deep learning.

Abstract

from arXiv · show

Deep learning models have demonstrated outstanding performance in several problems, but their training process tends to require immense amounts of computational and human resources for training and labeling, constraining the types of problems that can be tackled. Therefore, the design of effective training methods that require small labeled training sets is an important research direction that will allow a more effective use of resources.Among current approaches designed to address this issue, two are particularly interesting: data augmentation and active learning. Data augmentation achieves this goal by artificially generating new training points, while active learning relies on the selection of the "most informative" subset of unlabeled training samples to be labelled by an oracle. Although successful in practice, data augmentation can waste computational resources because it indiscriminately generates samples that are not guaranteed to be informative, and active learning selects a small subset of informative samples (from a large un-annotated set) that may be insufficient for the training process. In this paper, we propose a Bayesian generative active deep learning approach that combines active learning with data augmentation -- we provide theoretical and empirical evidence (MNIST, CIFAR-$\{10,100\}$, and SVHN) that our approach has more efficient training and better classification results than data augmentation and active learning.

1. Introduction

Deep learning training can demand substantial labeling and computation, motivating methods that reduce reliance on large labeled datasets. The paper combines active learning with generative augmentation to produce training samples that are informative, addressing weaknesses of each approach alone.

Motivation: Large-scale deep learning requires substantial human labeling and computational resources, limiting classification problems where labeled data or computation is scarce.These constraints are identified as an important machine-learning problem.
Existing approaches: Active learning selects informative unlabeled samples for oracle labeling and iterative retraining, but small selected training sets may cause overfitting.The approach aims to reduce labeling and training requirements by using smaller training sets.
Existing approaches: Data augmentation generates artificial training points from labeled data, but many generated samples may be uninformative and waste computational resources.The paper notes that this inefficiency can be especially problematic later in training.
Proposed approach: The proposed Bayesian generative active deep learning method uses BALD to select informative unlabeled samples, then uses a VAE-ACGAN to generate similarly informative samples for later iterations.The selected samples are labeled by an oracle before conditioning generation and expanding the labeled set.
Proposed approach: Compared with generative adversarial active learning, the approach uses acquisition functions reported as more effective than simple information loss and jointly trains generative and classification models.Its potential disadvantage is processing the unlabeled dataset with the acquisition function at each iteration, mitigated by sampling a smaller fixed-size subset.

2. Related Work

Related work combines active learning’s informative-sample selection with data augmentation’s synthetic-example generation, motivating a Bayesian method that generates samples conditioned on informative data. The proposed approach combines BALD and Bayesian data augmentation, jointly trains learner and generator, and theoretically supports the informativeness of generated samples.

Active Learning: Active learning selects informative samples from an unlabeled pool for oracle labeling, but acquisition functions can be difficult to optimize in deep learning.BALD measures mutual information and uses model-uncertainty estimates; earlier objectives may require high-dimensional Hessian inverses.
Data Augmentation: Data augmentation generates synthetic training points, but likelihood-driven generation tends toward distribution modes that become correctly classified and uninformative later in training.This can make generated samples relatively ineffective as training progresses.
Generative Active Learning: Generative active learning produces synthetic samples that are informative for the current model, extending active learning when an unlabeled pool alone is insufficient.GAAL generates samples through optimization and depends on a pretrained GAN and efficient generation optimization.
Bayesian Generative Active Deep Learning: The learner and generator are jointly trained, allowing them to co-evolve while generated samples remain near the selected informative samples.Under the stated proposition assumptions, the acquisition value at x′ is approximately equal to that at x∗, so x′ is also considered informative.
Model Components: The framework uses ACGAN and VAE-GAN components with the aim of improving classification performance rather than only synthetic-image quality.Its architecture includes a classifier, encoder, decoder/generator, and discriminator.
Bayesian Generative Active Deep Learning: The proposed method combines BALD and Bayesian data augmentation by conditioning VAE generation on the most informative selected sample rather than an independent latent variable.The selected sample x∗ is passed through an encoder and decoder to produce x′.

4. Implementation

The model combines a classifier, encoder, decoder/generator, and discriminator in a VAEACGAN architecture. Its training jointly updates these components using reconstruction, prior-regularization, and adversarial classification objectives.

Architecture: The network comprises a classifier, encoder, decoder/generator, and discriminator.The classifier can use any modern deep convolutional architecture, while the generative component is based on ACGAN.
Loss function: The VAE-ACGAN loss combines VAE reconstruction and regularization terms with adversarial training.The VAE loss consists of reconstruction loss Lrec and prior regularization Lprior, with the prior expressed through a Kullback-Leibler divergence.
Training procedure: Algorithm 1 initializes the network parameters and pretrains the classifier before iterative active-learning updates.Each iteration selects an informative pool sample, requests its oracle label, generates a reconstruction, and updates the model components.
Optimization: The encoder, generator, discriminator, and classifier are updated jointly using their respective loss gradients.The generator update uses γLrec − LACGAN, with γ = 0.75 in the experiments.
Training procedure: The encoder maps samples to latent representations, while the generator reconstructs samples from encoded or sampled latent variables.The stochastic latent input is sampled as u ∼ N(0, I).

5. Experiments and Results

The experiments evaluate Bayesian generative active learning across four image datasets, two ResNet-based classifiers, and several active-learning and augmentation baselines. Performance is assessed over acquisition iterations and training-set coverage, with repeated final evaluations and an information-value comparison.

Experimental design: The study measures top-1 classification accuracy for AL w. VAEACGAN against active-learning, Bayesian augmentation, and random-selection baselines.Compared methods include AL w. ACGAN, AL without DA, BDA with full or partial training sets, and random selection.
Experimental design: Figure 5 varies the number of acquisition iterations and the percentage of original training samples used for modeling.BDA with the full training set and 10× augmentation is included as an upper bound for the other methods.
Evaluation protocol: Final classification results are reported as mean ± stdev after 150 iterations from three runs with different random initializations.These repeated evaluations use ResNet18 and ResNet18pa on MNIST, CIFAR-10, and CIFAR-100.
Information-value analysis: Figure 6 compares the average acquisition-function information value of samples generated by AL w. ACGAN and AL w. VAEACGAN on CIFAR-100.The comparison uses ResNet18.

6. Discussion and Conclusions

Across datasets and classification models, AL w. VAEACGAN outperforms the compared active-learning and data-augmentation methods while using a fraction of the training set. The authors note high generated-image quality but identify missing comparison with the closest related method as a weakness.

Classification performance: AL w. VAEACGAN consistently outperforms other methods across datasets and classification models, with significant gains over BDA partial training and AL w. ACGAN in specified comparisons.The reported significance holds against BDA partial training on all three datasets and against AL w. ACGAN on CIFAR-10 and CIFAR-100 for both models.
Resource efficiency: A fraction of the training set achieves classification performance comparable with BDA using 10× augmentation over the entire training set.The authors interpret this as evidence that informative sample generation can reduce labeling and training resources.
Training dynamics: AL w. VAEACGAN eventually surpasses random sampling on MNIST after roughly 75 acquisition steps, or 13% of the training set.The active-learning methods initially perform worse than random sampling on MNIST before improving after this point.
Generated samples: The generated images are described as having surprisingly high quality across the evaluated datasets.Figure 7 displays images generated by AL w. VAEACGAN for each dataset.
Limitations: A stated weakness is the lack of comparison with the only other literature method proposing a similar approach.The authors also report that the method’s empirical computational cost is slightly higher than BDA and BALD.