PaperlayerView exact revision on arXiv

Source-linked AI summary

Smart Augmentation - Learning an Optimal Data Augmentation Strategy

Joseph Lemley, Shabab Bazrafkan, Peter Corcoran

arXiv:1703.08383v1cs.AIcs.LGstat.ML

TL;DR

Choosing augmentation strategies is difficult and often relies on trial and error. The paper introduces Smart Augmentation, a network that learns task-specific sample blending during target-network training, and reports improved accuracy and reduced overfitting across tested datasets, sometimes with smaller networks.

  • Problem

    Augmentation strategies are often chosen by trial and error, while blending samples raises unresolved questions about which samples to mix and how.

  • Method

    Smart Augmentation uses network A to blend multiple same-class samples into new data that reduces network B’s training loss.

  • Results

    Smart Augmentation improved accuracy and reduced error and overfitting across tested datasets, sometimes matching or outperforming larger networks.

  • Takeaways & Limitations

    The experiments indicate that augmentation can be automated for nonlinear multi-sample merging across varied image tasks and database complexity.

  • Takeaways & Limitations

    Future work is needed to test more sophisticated strategies, larger datasets with more classes, and channel counts associated with optimal results.

Abstract

from arXiv · show

A recurring problem faced when training neural networks is that there is typically not enough data to maximize the generalization capability of deep neural networks(DNN). There are many techniques to address this, including data augmentation, dropout, and transfer learning. In this paper, we introduce an additional method which we call Smart Augmentation and we show how to use it to increase the accuracy and reduce overfitting on a target network. Smart Augmentation works by creating a network that learns how to generate augmented data during the training process of a target network in a way that reduces that networks loss. This allows us to learn augmentations that minimize the error of that network. Smart Augmentation has shown the potential to increase accuracy by demonstrably significant measures on all datasets tested. In addition, it has shown potential to achieve similar or improved performance levels with significantly smaller network sizes in a number of tested cases.

I. INTRODUCTION · II. RELATED WORK

Limited labeled data can cause overfitting, motivating augmentation and regularization methods, but augmentation strategies are difficult to select manually and may harm performance. Smart Augmentation addresses this by learning how to combine samples to improve regularization rather than relying on fixed transformations or expert-designed strategies.

  • I. INTRODUCTION: Insufficient quality labeled data causes overfitting and prevents deep networks from generalizing beyond training samples.The introduction identifies access to enough labeled data as a central requirement for training deep neural networks.
  • I. INTRODUCTION: Data augmentation supplements datasets with similar samples, commonly using image rotation, translation, blurring, and related modifications to improve generalization.Augmentation is presented as a ubiquitous deep-learning technique for expanding image datasets.
  • I. INTRODUCTION: Augmentation reduces overfitting by extracting more general information, while supervised augmentation mixes same-label samples into recognizable class-representative data.The passage distinguishes label-independent transformations from label-guided sample generation.
  • I. INTRODUCTION: Augmentation choices cannot be applied blindly: rotation can make MNIST 6s and 9s difficult to distinguish, and effective strategies often require expert knowledge and costly trial and error.Researchers face constraints from limited training time, imagination, and experience when evaluating augmentation strategies.
  • I. INTRODUCTION: Blending samples is difficult because selecting which samples to mix, how many to use, and how to mix them is nontrivial.The introduction identifies sample blending as a particularly challenging form of data expansion.
  • I. INTRODUCTION: Smart Augmentation proposes using a CNN-learned feature space to merge same-class samples and generate data that improves target-network regularization.Unlike manual augmentation and simple learned transformations, the method learns combinations intended to improve regularization rather than appear natural.
  • II. RELATED WORK: Related approaches include manual transformations, dropout, adversarial examples, generative adversarial networks, transfer learning, and batch normalization for augmentation, generalization, or overfitting reduction.Smart Augmentation is positioned as distinct from manual augmentation, natural-image blending, and methods that learn natural-looking synthetic data.
  • II. RELATED WORK: +44% accuracy was reported for natural data blending when image layers were randomly blended with crops from training images.This related approach used randomly sampled image crops and synthetic blended images for natural scene text recognition.

III. SMART AUGMENTATION

Smart Augmentation learns suitable augmentation strategies during deep neural network training by merging samples from the same class and using the target network’s loss to guide the augmenter.

  • Smart Augmentation learns suitable augmentations when training deep neural networks.
  • It learns an augmentation strategy for a given input class by merging two or more samples from that class.
  • The merged samples train a target network, whose loss simultaneously informs the augmenter and enables generation of more training data.

A. Training Strategy for Smart Augmentation · B. The Generative Network A and Loss Function · C. How Smart Augmentation Works

Smart Augmentation jointly trains a generative Network A and task Network B, with A blending same-class samples to produce augmentations that reduce B’s loss. Backpropagation from B tunes A during training, after which A is removed and B is used for testing.

  • A. Training Strategy for Smart Augmentation: Network A generates new samples from several same-class inputs while Network B learns the desired task when representative dataset samples are insufficient.Sample selection may be random or based on clustering in pixel or feature space.
  • A. Training Strategy for Smart Augmentation: Network A is trained with loss L_A using its output and a selected same-class sample as inputs.The passage states that L_A accepts out1 and image i, where out1 is Network A’s output and image i is a selected sample.
  • B. The Generative Network A and Loss Function: The overall training loss is f(L_A, L_B), which may change with epoch number and can be parameterized by α and β.The experiments examine how α and β affect final accuracy.
  • B. The Generative Network A and Loss Function: Network A may be a single network or multiple networks, with multiple networks learning class-specific augmentations unsuitable for other classes.This design allows augmentations to work well for their given class.
  • B. The Generative Network A and Loss Function: Unlike a typical generative model, Network A accepts multiple samples simultaneously and is influenced by Network B during backpropagation.L_A controls the outputs so they remain similar to other members of the class while converging toward augmentations useful for Network B.
  • B. The Generative Network A and Loss Function: Error backpropagates from Network B to Network A, tuning A to generate augmentations that minimize Network B’s training loss.After training, Network A is cut out and Network B is used in the test process.
  • C. How Smart Augmentation Works: Network A learns task-specific sample blending by generating a new same-class sample for Network B, which should reduce Network B’s training loss.The generated sample is used as Network B’s input.
  • C. How Smart Augmentation Works: The generated image’s visual quality is not important; its value lies in helping Network B generalize, with the approach mainly targeting classification and potentially reducing overfitting and increasing accuracy.The passage illustrates blending features such as open eyes and an open mouth from two images.

IV. METHODS … 3) FERET:

The methods evaluate Smart Augmentation across four datasets selected to probe performance on different data characteristics, without attempting a comparative benchmark of database-specific techniques. The described preparation includes controlled face datasets and FERET, with subject-exclusive splits and normalized grayscale inputs.

  • IV. METHODS: Experiments ran on NVIDIA Titan X GPUs using Pascal architecture, Python 2.7, Theano, and Lasagne.
  • A. Data Preparation: Four datasets were selected to examine the algorithm on specific data characteristics, while comparisons with database-specialized techniques were deferred to prior work.
  • 1) Highly constrained faces dataset (db1):: db1 contained 4,000 frontal male and female faces from the AR faces database, split subject-exclusively into 70% training, 20% validation, and 10% testing.Images were converted to 96X96 grayscale with pixel values normalized between 0 and 1.
  • 2) Augmented, highly constrained faces dataset (db1a):: db1a expanded training to 48360 images using every combination of flipping, blurring, and rotations of -5,- 2,0,2,5 degrees.Validation and testing sets remained unaltered from db1.
  • 3) FERET:: FERET images were converted to grayscale and resized to 100X100, with pixel values normalized between 0 and 1.The dataset used subject-exclusive 70% training, 20% validation, and 10% testing splits.
  • 3) FERET:: Color FERET Version 2 was collected from December 1993 through August 1996 and labels images by gender, pose, and name.
  • 3) FERET:: FERET includes varied poses and obstructions, but similarities in quality, background, pose, and lighting make its images easy for modern methods to classify.Experiments used all FERET images with gender labels.

4) Adience: · 5) DB4:

The Adience experiments used normalized grayscale images with subject-exclusive splits, while DB4 used a normalized two-class subset of the MIT Places dataset. The DB4 experiments examined Smart Augmentation’s effectiveness, generality, behavior under unconstrained data, and effects on architecture, augmentation, parameters, overfitting, and model size.

  • 4) Adience:: Adience images were converted to 100x100 grayscale, normalized to [0,1], and split by subject into training, validation, and testing sets.The split was 70% training, 20% validation, and 10% testing.
  • 4) Adience:: Adience contains substantial variation in lighting, pose, subjects, backgrounds, and other photographic conditions.
  • 5) DB4:: DB4 was the MIT Places dataset, comprising 205 scene categories and 2.5 million labeled images.
  • 5) DB4:: The Places Dataset is unconstrained, spanning complex scenery across varied lighting conditions and environments.
  • 5) DB4:: The DB4 study asked whether Smart Augmentation improves accuracy, generalizes across datasets, and changes performance as datasets become more unconstrained.It also examined the effect of increasing the number of Smart Augmentation channels.
  • 5) DB4:: The study further evaluated Smart Augmentation against traditional augmentation, their combination, α and β settings, overfitting, and replacement of large networks by simpler ones.Overfitting was measured using train/test loss ratios, and the architectures were combined in varied ways.
  • 5) DB4:: The experiments used three neural-network architectures with varied parameters and connection mechanisms, including a small classifier CNN B1 and an unmodified VGG16 B2.B1 maps images to class labels through a softmax layer, while B2 is described as a large network.

A. Smart Augmentation with one Network A on the Gender Classification Task

On the gender-classification task, Smart Augmentation trains network A to generate samples that reduce network B’s error. Experiments varied input channels and datasets while evaluating validation loss only for network B to compare performance with and without Smart Augmentation.

  • Experiments: Experiments 1–8, 19, 22, and 24 trained network B for gender classification using Smart Augmentation.Network B was trained with the specified database using the technique illustrated in Figure 9.
  • Method: Network A receives k same-class grayscale images merged as k channels and produces an output image for network B.The first k randomly selected male or female images are mapped into separate channels of one input sample.
  • Method: The network A loss is the mean squared error between its output and an additional randomly selected same-class image.The target image must not be among the images selected for network A’s input.
  • Method: Smart Augmentation jointly trains network A to generate samples that reduce network B’s classification error.The total model loss is a linear combination of both networks’ losses, while validation loss is computed only for network B.
  • Experiments: Experiments varied network A’s input channels and datasets, including 1–8 channels on db1, 2 channels on db2, and 3 or 2 channels on db1a.Network B was trained from scratch under these configurations.

B. Smart Augmentation with two Network A’s on the Gender Classification Task · C. Training without Smart Augmentation on the Gender Classification Task

The paper evaluates Smart Augmentation with separate class-specific network A models for gender classification, including manually augmented data, and compares it against network B trained without Smart Augmentation as a baseline. The class-specific implementation uses merged same-class grayscale images, while the baseline retains network B’s architecture without network A.

  • B. Smart Augmentation with two Network A’s on the Gender Classification Task: Experiments 9–16 and 20 use a separate network A for each gender class, merging k randomly selected same-class grayscale images into k input channels.The first image maps to channel 0, the second to channel 1, and so on until the available channel count is reached.
  • B. Smart Augmentation with two Network A’s on the Gender Classification Task: All other loss functions in the two-network-A implementation are calculated as in the preceding implementation.This preserves the earlier loss-calculation procedure while changing the network-A arrangement.
  • B. Smart Augmentation with two Network A’s on the Gender Classification Task: 0.005 is the updated learning rate; rates above 0.005 caused dying RELU and halted effective learning within the first two epochs.The network is also more sensitive to batch-size variations.
  • B. Smart Augmentation with two Network A’s on the Gender Classification Task: The multiple-network-A experiments examine effects on accuracy and overfitting relative to one network A, including training on a manually augmented database in experiment 20.The stated goal is to identify whether multiple network A models change performance or overfitting behavior.
  • C. Training without Smart Augmentation on the Gender Classification Task: Experiments 23, 21, 18, and 17 train network B for gender classification without network A on db3, db2, db1a, and db1, respectively, establishing baseline comparisons.These baselines measure improvements attributable to Smart Augmentation on each dataset.
  • C. Training without Smart Augmentation on the Gender Classification Task: The no-Smart-Augmentation network has the same architecture as the preceding network B, except that it does not use network A.This isolates the contribution of Smart Augmentation in the comparison.
  • C. Training without Smart Augmentation on the Gender Classification Task: The baseline ends with dense layers of 1024 units and two units, uses dropout to avoid overfitting, and computes training, validation, and testing loss with categorical cross-entropy.Models are trained with Stochastic Gradient Descent using Nesterov Momentum, learning rate 0.01, and momentum 0.9, via Lasagne in Python.

D. Experiments on the places dataset

Experiments 25–30 evaluated Smart Augmentation on color scenes from the MIT Places dataset, varying loss parameters, input channels, and dataset configurations. The setup trained network A to generate samples that reduce network B’s error, with validation loss measured only for network B to compare augmentation against baseline performance.

  • Experimental scope: Experiments 25–30 tested Smart Augmentation on color scenes from the MIT Places dataset using two-class classification and varied α and β in the global loss.Unlike earlier face-dataset experiments, these experiments retained color information.
  • Baseline classifiers: Experiments 25–26 used simple classifiers to establish a baseline for evaluating other experiments on the same dataset.Experiment 25 used a VGG16 classifier trained from scratch on 256x256 RGB images, while experiment 26 used the prior subsection’s network B with a lower learning rate and place images.
  • Input configurations: The experiments varied network B’s input configuration from 1–8 channels on db1, 2 channels on db2 and 3, and 2 channels on db1a.These configurations were applied to networks trained from scratch as shown in the experiments table.
  • Smart Augmentation setup: Experiments 27–30 merged k randomly selected same-class images into k × 3 input channels before feeding generated and target images separately into network B.Network B used two convolutional layers with batch normalization and max-pooling after each convolutional layer, followed by two fully connected layers.
  • Optimization objective: The model’s total loss combined both networks’ losses, training network A to generate samples that reduce network B’s error while evaluating validation loss only on network B.This enabled comparison of validation loss with and without Smart Augmentation.

VI. RESULTS

Section VI reports the results of experiments 1–30 in Tables II and III, presented in the same order as the corresponding experiments in Table I. The following subsections explain these results in detail.

  • VI. RESULTS: Results from experiments 1–30 are presented in Tables II and III in the same order as the corresponding experiments in Table I.The results are explained in detail in the subsections below.

A. Smart Augmentation with one Network A on the Gender Classification Task · B. Smart Augmentation and Traditional Augmentation

Smart Augmentation reduced overfitting and improved gender-classification accuracy, including gains on Feret, Adience, and AR faces. Combined with traditional augmentation, it further increased accuracy, although channel results varied and may partly reflect random image selection.

  • A. Smart Augmentation with one Network A on the Gender Classification Task: Smart Augmentation greatly reduced overfitting by narrowing the difference between training and validation loss and helping Network B learn more general features.Network B also achieved higher test-set accuracy with Smart Augmentation.
  • A. Smart Augmentation with one Network A on the Gender Classification Task: 88.15%: Network B’s accuracy without Network A was the AR faces baseline for evaluating other results.The paper suspects much of the reported variation arose by chance because images were selected randomly, while 3- and 8-channel configurations exceeded 95%.
  • A. Smart Augmentation with one Network A on the Gender Classification Task: Network A learned to generate blended images, with the left image in the examples formed from the two images on the right.These examples illustrate the kinds of augmentations learned during training.
  • A. Smart Augmentation with one Network A on the Gender Classification Task: 4.94 percentage points: Smart Augmentation increased Feret accuracy from 83.52% to 88.46% with 2 inputs.Adience accuracy also increased from 70.02% to 76.06%.
  • A. Smart Augmentation with one Network A on the Gender Classification Task: 95.38%: 8 channels produced the best reported accuracy, followed by 95.09% with 3 channels, while 7, 5, and 4 channels performed worst.The corresponding lower accuracies were 91.62%, 91.04%, and 91.04%. No noticeable pattern emerged when varying Network A’s number of inputs.
  • B. Smart Augmentation and Traditional Augmentation: 95.66%: Combining traditional augmentation with Smart Augmentation increased accuracy from 88.15% to 89.08% with traditional augmentation alone, then to 95.66%.This demonstrates further improvement when Smart Augmentation is used with traditional augmentation.
  • B. Smart Augmentation and Traditional Augmentation: 95.66%: Using 2 Network A’s achieved the same accuracy as the corresponding combined-augmentation experiment, suggesting both configurations may have found the same optima.Across the examined cases, Smart Augmentation performed better than traditional augmentation, although manual augmentation could potentially find a better strategy.

C. Smart Augmentation with two Network A’s on the Gender Classification Task · D. Experiments on the Places Dataset

On gender classification, class-specific Smart Augmentation networks slightly improved accuracy overall but required at least two channels. On Places, multiple augmentation networks and α > β improved results, with Smart Augmentation enabling a smaller network to approach or exceed VGG-16 performance.

  • C. Smart Augmentation with two Network A’s on the Gender Classification Task: Distinct Network A’s for each class slightly outperformed using one Network A on the gender classification task.The result supports learning class-specific augmentations more efficiently.
  • C. Smart Augmentation with two Network A’s on the Gender Classification Task: 0.25 percentage points: excluding one- and zero-input-channel networks, Smart Augmentation increased average accuracy from 92.94% to 93.19%.The comparison is reported across the eligible gender-classification experiments.
  • C. Smart Augmentation with two Network A’s on the Gender Classification Task: 86.99%: Smart Augmentation underperformed without augmentation when only one channel was used, compared with 88.15% without Smart Augmentation.The paper attributes this to ineffective use of mutual information and emphasizes using at least two channels.
  • D. Experiments on the Places Dataset: Multiple Network A’s performed slightly better than single-network configurations on the Places dataset.This reproduces the pattern observed in the gender-classification experiments.
  • D. Experiments on the Places Dataset: Higher α than β increased accuracy on the Places dataset.The reported improvement occurred when α was greater than β.
  • D. Experiments on the Places Dataset: 96.5%: a small Network B trained alone achieved lower accuracy than VGG-16 trained without Smart Augmentation at 98.5%.This comparison is identified as one of the most significant Places-dataset results.
  • D. Experiments on the Places Dataset: 98.75% to 99%: the same small Network B achieved this accuracy range when trained with Smart Augmentation.These results indicate that Smart Augmentation can, in some cases, enable a much smaller network to replace a larger one.

VII. DISCUSSION AND CONCLUSION

Smart Augmentation automatically learns nonlinear augmentations during target-network training, improving generalization while reducing error and overfitting. Across tested datasets, it improved accuracy and sometimes matched or exceeded larger networks with significantly fewer parameters.

  • Method and scope: Smart Augmentation automatically learns suitable augmentations during deep-network training by exploiting mutual information within a class.The method was tested on progressively more difficult datasets, from constrained faces to unconstrained places.
  • Main conclusion: Smart Augmentation automated nonlinear merging of two or more same-class samples and improved target-network generalization while reducing overfitting.The augmentation task is learned concurrently with the target task.
  • Experimental observations: No linear correlation was found between the number of samples mixed by network A and accuracy when at least 2 samples were used.The absence of correlation applies under the stated minimum-sample condition.
  • Experimental observations: Smart Augmentation reduced error and overfitting regardless of how unconstrained the database was.This finding held across the tested databases.
  • Experimental observations: Better accuracy was achieved with Smart Augmentation than with traditional augmentation alone.Altering the α and β loss-function parameters slightly impacted results, but more experiments are needed to identify optimal values.
  • Practical implications: Smart Augmentation on a small network achieved better results than a much larger VGG 16 network, supporting practical CNN implementations in embedded systems and consumer devices.The paper also reports significantly improved accuracy across all tested datasets and similar or improved performance with significantly smaller networks in some cases.
  • Future work: Future work includes learning more sophisticated augmentation strategies, testing larger datasets with more classes, and studying channel counts associated with optimal results.The proposed statistical study would identify the number of channels giving the highest probability of optimal results.
Loading 1703.08383v1…