Source-linked AI summary
A Bayesian Data Augmentation Approach for Learning Deep Models
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, Ian Reid
TL;DR
Deep learning benefits from large annotated datasets, but dominant augmentation methods rely on untested transformation assumptions. This paper proposes Bayesian augmentation with jointly learned synthetic data and classification, reporting clear improvements across image-classification tasks.
Problem
Large annotated datasets are expensive, while dominant augmentation methods assume random transformations reliably generate new labeled samples; that assumption is insufficiently tested.
Method
The method treats synthetic data as latent variables sampled from an iteratively learned training distribution and optimizes generation and classification jointly using Generalized Monte Carlo EM, demonstrated with an extended GAN.
Results
The proposed Bayesian data augmentation improves deep-learning classification across MNIST, CIFAR-10, and CIFAR-100, outperforming PMDA and similar GAN models.
Takeaways & Limitations
Bayesian augmentation provides a theoretically sound alternative to fixed random transformations by adapting generated training data as learning progresses.
Takeaways & Limitations
The compared transformation-based augmentation methods assume sufficiently small perturbations preserve labels, an assumption described as restrictive and insufficiently tested.
Abstract
from arXiv · showhide
Data augmentation is an essential part of the training process applied to deep learning models. The motivation is that a robust training process for deep learning models depends on large annotated datasets, which are expensive to be acquired, stored and processed. Therefore a reasonable alternative is to be able to automatically generate new annotated training samples using a process known as data augmentation. The dominant data augmentation approach in the field assumes that new training samples can be obtained via random geometric or appearance transformations applied to annotated training samples, but this is a strong assumption because it is unclear if this is a reliable generative model for producing new training samples. In this paper, we provide a novel Bayesian formulation to data augmentation, where new annotated training points are treated as missing variables and generated based on the distribution learned from the training set. For learning, we introduce a theoretically sound algorithm --- generalised Monte Carlo expectation maximisation, and demonstrate one possible implementation via an extension of the Generative Adversarial Network (GAN). Classification results on MNIST, CIFAR-10 and CIFAR-100 show the better performance of our proposed method compared to the current dominant data augmentation approach mentioned above --- the results also show that our approach produces better classification results than similar GAN models.
1 Introduction
The paper proposes Bayesian data augmentation that learns a distribution of synthetic training points and generates them iteratively during deep-model training. It introduces GMCEM and demonstrates the approach with a GAN-based implementation for image classification.
- Motivation: Large annotated datasets support deep-model generalization but are expensive and difficult to collect through manual labeling.Manual annotation is described as costly, sometimes subjective, and prone to mistakes.
- Motivation: Conventional augmentation applies noisy geometric or appearance transformations to annotated samples under a label-preserving assumption.Examples include rotations, translations, color perturbations, and other predefined transformation spaces.
- Proposed Approach: The proposed Bayesian approach treats synthetic data points as latent variables drawn from a distribution learned from the annotated training set.New samples are generated as training progresses rather than before training from fixed transformations and noise models.
- Proposed Approach: GMCEM jointly augments training data and optimizes network parameters by sampling synthetic points, estimating expected parameters with Monte Carlo, and updating them with SGD.The method combines Monte Carlo EM with generalized EM optimization.
- Implementation and Evaluation: The GAN-based implementation uses a generator, a real-versus-fake authenticator, and a classifier, allowing different generative and discriminative models to be tested.Experiments evaluate classification on MNIST, CIFAR-10, and CIFAR-100.
2 Related Work
Related work commonly generates augmented samples through predefined transformations or deep generative models. The paper identifies limited diversity, restrictive noise assumptions, and non-adaptive generation as limitations motivating its iterative Bayesian approach.
- Statistical Learning: Latent-variable data augmentation has been used in statistical learning to facilitate posterior-distribution computation.These techniques are useful when original likelihood or posterior densities are difficult to maximize or sample.
- Data Augmentation: Common deep-learning augmentation applies Gaussian or uniform noise to predefined transformations of annotated samples.Examples include stroke warping, geometric changes, reflections, and color perturbations.
- Data Augmentation: Transformation-based augmentation may restrict diversity because perturbations must remain sufficiently small to preserve ground-truth labels.The paper characterizes this as an insufficiently tested noise model over a predetermined transformation space.
- Data Augmentation: These augmentation mechanisms generate data once before training and therefore do not adapt as learning progresses.The paper contrasts this non-iterative behavior with its iteratively learned generator.
- Deep Generative Models: GANs enable synthetic-image generation through adversarial training, but image quality can be low on datasets with high variability.Conditional GAN and AC-GAN are cited as variants designed to improve generative modeling.
3 Data Augmentation Algorithm in Deep Learning
The paper formulates data augmentation as Bayesian inference with synthesized data treated as latent variables, then develops an iterative Monte Carlo optimization procedure. The resulting GMCEM method alternates synthetic-data sampling with parameter updates and provides a weak-convergence guarantee under its improvement condition.
- Motivation: The approach targets over-fitting caused by high-dimensional model parameters and insufficiently large training sets through Bayesian augmentation of the training data.The paper presents augmentation as a way to target a more robust parameter-estimation process.
- Bayesian data augmentation: Data augmentation introduces latent synthesized data z=(t_a,x_a) to enlarge the observed training set and estimate the augmented posterior p(θ|y,z).The synthesized sample includes data x_a∈R^D and an associated class label t_a∈{1,...,K}.
- GMCEM algorithm: The E-step computes an expectation over the conditional predictive distribution p(z|θ_i,y), approximated by Monte Carlo samples z_m∼p(z|y,θ_i).The method uses M synthesized samples to estimate the expected objective.
- GMCEM algorithm: The M-step updates θ by maximizing the estimated Q function, but the paper uses a generalized update because the global maximizer has no analytical solution.The update only needs to satisfy Q-hat(θ_i+1,θ_i)>Q-hat(θ_i,θ_i), and is implemented with one differentiable gradient-descent step.
- GMCEM algorithm: GMCEM jointly augments training data and optimizes network parameters by iteratively sampling synthetic points, estimating expectations, and updating parameters with stochastic gradient descent.This combines Monte Carlo EM with generalized EM for deep learning models.
- Convergence: Under the condition Q-hat(θ_i+1,θ_i)>Q-hat(θ_i,θ_i), GMCEM weakly converges because the posterior satisfies p(θ_i+1|y)>p(θ_i|y).The proof takes expectations of the estimated Q-function and invokes the corresponding generalized-EM condition.
4 Implementation
The method supports differentiable generative and classification models, using a modular classifier–generator architecture and GAN-based components for Bayesian data augmentation.
- Model choice: The Bayesian DA algorithm can use any differentiable deep generative and classification models.This flexibility permits implementation with sophisticated models available in the field.
- Network Architecture: The architecture separates the classifier from the generator and authenticator networks.The classifier uses modern convolutional networks, while the generator uses a GAN with a deconvolutional generator and convolutional authenticator.
- Network Architecture: The classifier, authenticator, and generator are parameterized by θC, θA, and θG, respectively.The model also defines input x, latent variable u, and class label c.
- Optimization Function: The classifier optimization function is defined separately from the authenticator and generator optimization functions.The classifier objective is introduced before the GAN-related objectives.
- Optimization Function: Training alternates discriminative and generative optimization steps using complementary combinations of JC and JAG.The discriminative step minimizes JC + JAG, while the generative step minimizes JC − JAG.
- Optimization Function: GMCEM updates network parameters iteratively with gradient descent, while full-gradient evaluation is costly for large augmented datasets.The computational issue arises because training and augmented samples total N + M.
5 Experiments
Experiments compare the proposed Bayesian augmentation with PMDA across MNIST, CIFAR-10, and CIFAR-100, using multiple augmentation sizes and classifier architectures. The proposed method outperforms PMDA and AC-GAN, although PMDA can be better early in training; generated images are especially realistic for MNIST digits.
- Experimental setup: Experiments evaluate MNIST, CIFAR-10, and CIFAR-100 classification with top-1 accuracy at 2×, 5×, and 10× augmentation.PMDA uses random rotations and translations as its strongest tested transformation settings.
- Experimental setup: The implementation combines C-GAN generation with ResNet18 and ResNetpa classification models.The generator and authenticator architectures remain unchanged across the three datasets.
- Classification results: Larger augmented training sets generally produce more accurate classification.This pattern is reported for comparisons using both ResNet18 and ResNetpa.
- Classification results: The Bayesian DA algorithm outperforms PMDA by a large margin on all three datasets.The comparison uses the proposed method with the evaluated deep classifiers.
- Classification results: The proposed approach is far superior to AC-GAN in the reported ResNetpa comparison.The comparison is motivated by the similarity between the proposed model and AC-GAN.
- Training-time results: PMDA performs better during early training, but Bayesian DA becomes better after sufficient training time.With ResNet18, Bayesian DA surpasses PMDA after two hours on CIFAR-100 and five hours on MNIST.
- Generated images: The generator produces reasonably realistic synthetic images, particularly handwritten digits.The MNIST images are described as difficult to generate through fixed geometric and appearance transformations.
6 Conclusions
The paper presents Bayesian data augmentation that jointly learns the generator and classifier while sampling missing data from a distribution learned from annotated training data. Across image-classification tasks, it improves over standard augmentation and AC-GAN.
- Conclusion: The method treats missing data as samples from a distribution learned from annotated training data.Unlike dominant transformation-based methods, it jointly optimizes generator and classification models.
- Conclusion: Experiments show clear improvements over standard data augmentation methods and AC-GAN.The conclusion summarizes validation across several image-classification tasks.