Source-linked AI summary
Understanding data augmentation for classification: when to warp?
Sebastien C. Wong, Adam Gatt, Victor Stamatescu, Mark D. McDonnell
TL;DR
The paper asks how synthetic data augmentation benefits classifier training and compares data-space warping with feature-space over-sampling across three classifiers on MNIST. It finds that known label-preserving transformations make data-space augmentation more beneficial than feature-space augmentation, while synthetic-data gains remain bounded by equivalent real-data training.
Problem
The paper investigates how and when synthetic data augmentation benefits machine-learning classifiers, including whether augmentation should occur in data-space or feature-space.
Method
The authors compare data-space warping and feature-space over-sampling for CNN, CSVM, and CELM classifiers using MNIST.
Results
Known label-preserving transforms made data-space augmentation more beneficial than feature-space augmentation, with benefit varying across classifiers.
Takeaways & Limitations
When plausible label-preserving transforms are available, augmentation should be performed in data-space; otherwise SMOTE can provide some feature-space benefit.
Abstract
from arXiv · showhide
In this paper we investigate the benefit of augmenting data with synthetically created samples when training a machine learning classifier. Two approaches for creating additional training samples are data warping, which generates additional samples through transformations applied in the data-space, and synthetic over-sampling, which creates additional samples in feature-space. We experimentally evaluate the benefits of data augmentation for a convolutional backpropagation-trained neural network, a convolutional support vector machine and a convolutional extreme learning machine classifier, using the standard MNIST handwritten digit dataset. We found that while it is possible to perform generic augmentation in feature-space, if plausible transforms for the data are known then augmentation in data-space provides a greater benefit for improving performance and reducing overfitting.
I. INTRODUCTION
The paper examines how and when synthetic data augmentation benefits classifiers, contrasting data-space warping with feature-space over-sampling across several classification systems.
- The study investigates the benefits and limitations of data augmentation for three machine-learning classifiers.The classifiers are a convolutional neural network, convolutional support vector machine, and convolutional extreme learning machine, evaluated on MNIST.
- Prior work used augmentation to increase training examples, reduce overfitting, improve imbalanced-class performance, and achieve strong handwritten-digit classification results.Warped training data contributed to a reported 0.4% MNIST error rate in 2003.
- Data warping creates augmented examples through transformations of existing data, whereas SMOTE creates synthetic examples in feature-space.Data warping has been used for handwriting distortions, while SMOTE was proposed as an application-independent feature-space method.
- Synthetic imagery can differ from the real images underlying it, creating a distributional discrepancy termed the synthetic gap.One cited approach trained a sparse auto-encoder on real and synthetic images to minimize this gap.
- A cited alternative argues that test samples should undergo the same warping process as training samples to maximize correct-classification probability.
III. METHOD AND DATA
The experiments separate feature generation from classification so augmentation can be compared in data-space and feature-space under a shared architecture.
- Each classifier used a two-stage architecture consisting of a common feature-generation stage followed by a classification stage.The shared convolutional and pooling layer generated features for the classification engine.
- The convolutional and pooling-layer weights were kept fixed so the classifiers used the same generated features.
A. Datasets
The experiments use MNIST handwritten digits and elastic deformations to generate label-preserving data-space augmentations, while recognizing that excessive deformation can destroy labels.
- Datasets: MNIST contains 60,000 labeled 28 by 28 grayscale training images, 10,000 test images, and 10 digit classes.The baseline varied the amount of available training data while keeping class counts equal.
- Augmentation in data-space: Elastic warping transforms pixel locations using a random displacement field, with α controlling displacement strength and σ controlling field smoothness.The displacement field is smoothed by convolving uniformly distributed random values with a Gaussian.
- Augmentation in data-space: Label integrity was assessed by whether a human observer could still recognize the transformed digit.
- Augmentation in data-space: For MNIST, α ≥8 pixels could occasionally make characters unrecognizable by shifting critical parts outside the image or introducing illegible kinks.The experiments used α = 1.2 pixels and σ = 20, selected using CELM performance.
- Augmentation in data-space: Elastic-warping samples were generated offline and reused across the experiments.
C. Augmentation in feature-space
Feature-space augmentation is useful when label-preserving raw-data transformations are difficult to validate, with SMOTE providing benefits but DBSMOTE increasing overfitting in these experiments.
- Feature-space augmentation is relevant when arbitrary raw-data transformations cannot be easily validated as label preserving.Some problems instead use hand-crafted features containing salient information for classification.
- SMOTE generates synthetic feature-space samples by selecting real examples and interpolating between them.For these MNIST experiments, k = 2 random samples were used.
- DBSMOTE generates synthetic samples within distance eps of each class cluster center.The experiments used eps ≤4 for DBSMOTE.
- DBSMOTE increased overfitting in the experiments, consistent with generating samples around class centers rather than reducing overfitting.
- SMOTE and DBSMOTE samples were generated online because the algorithms are embedded within the classification architecture.
IV. EXPERIMENTAL SYSTEM PARAMETERS
The experiments use a two-stage architecture with fixed convolutional and pooling parameters to generate features before classification.
- The experiments use the two-stage architecture illustrated in Figure 1 for each classifier.
- The convolution and pooling process is repeated for L = 96 filters.
B. Stage-2 classification.
Stage-2 evaluated three classifiers with fixed convolutional-layer weights: a backpropagation-trained neural network, a support vector machine, and an extreme learning machine. The baseline experiment varied real training samples to assess performance and overfitting.
- Stage-2 compared a backpropagation-trained neural network, support vector machine, and extreme learning machine while holding convolutional-layer weights constant.
- The neural network used a single hidden layer of 1600 sigmoid-activated neurons, with backpropagation updating only the hidden neurons.
- The support vector machine used a multiclass 1-vs-all L2-loss formulation.
- The extreme learning machine used least-squares regression over random weight-projection neurons and a single hidden layer of 1600 neurons.
- The baseline varied available real samples to test whether synthetic samples could approach the benefit of an equivalent number of real samples.
- Experiments were repeated three times; CNN and CELM varied because of randomized training or projections, whereas CSVM was deterministic.
A. Baseline Results
With more real MNIST samples, the classifiers generally reduced the training–test error gap, indicating less overfitting. CNN performance improved most clearly, while elastic data-space deformations outperformed feature-space augmentation for CNNs.
- As real samples increased from 500 to 5000 per class, the training–test error gap decreased, indicating reduced overfitting across classifiers.
- CNN benefited most from additional training samples, with both training and test error improving as sample counts increased.
- Lower test error indicates better performance, while a smaller training–test gap indicates reduced overfitting.
- CNN test error decreased steadily as the number of samples increased, and repeated runs produced similar error rates.
- Elastic deformations in data-space improved CNN error more than feature-space augmentation, although training and test errors decreased together.
- SMOTE produced marginally promising feature-space augmentation results, with test error continuing to decrease through 50,000 samples.
- DBSMOTE produced a slight test-error improvement, while its training-error curve closely followed SMOTE’s curve.
C. Convolutional SVM Results
For the convolutional SVM, feature-space DBSMOTE degraded performance as synthetic samples increased, while SMOTE and elastic warping yielded little or only modest improvement. DBSMOTE therefore did not reduce overfitting in this classifier.
- Increasing DBSMOTE samples caused CSVM performance to degrade, with error increasing as more synthetic samples were added.
- DBSMOTE was not effective at reducing overfitting in the SVM classifier.
- The CSVM results were presented as error percentages varying with the number of samples, with training and test errors distinguished by dashed and solid lines.
- SMOTE provided little to no CSVM performance improvement, while elastic warping produced only modest test-error improvement at very large augmentation amounts.
- With increasing elastic-warping augmentation, the training–test error gap decreased steadily.
D. Convolutional ELM Results
For CELM on MNIST, increasing synthetic samples generally improved performance, but data-space elastic deformation outperformed feature-space methods and remained inferior to adding real samples.
- Increasing synthetic samples generally improved CELM performance as measured by training and test error.Figure 6 distinguishes dashed training-error curves from solid test-error curves.
- Data-space augmentation with elastic deformations produced the best results, though slightly worse than adding equivalent real training samples.
- Elastic-deformation augmentation improved error markedly at 1000 samples per class before gains flattened.This condition comprised 500 real and 500 augmented samples per class.
- SMOTE initially improved performance from 500 to 1000 samples per class, but further synthetic samples reduced performance.The initial test-error decrease was not accompanied by a corresponding reduction in training accuracy, leaving a large train-test gap.
- DBSMOTE had a slightly negative impact on CELM test error.
VI. DISCUSSION
The discussion compares data-space and feature-space augmentation across classifiers and asks how much synthetic data helps. Data-space elastic distortions were strongest when label-preserving transforms were available, while benefits varied by classifier and could diminish or reverse with more synthetic samples.
- The experiments isolate whether augmentation is better performed in data-space or feature-space using a two-stage classification architecture.
- For handwritten digit recognition, data-space augmentation using elastic distortions was clearly better when label-preserving transforms were available.The authors expect this pattern to extend to other classification tasks under the same transform condition.
- When label-preserving transforms were unavailable, SMOTE provided some benefit to CNN or CELM classifiers, whereas DBSMOTE was not recommended.The architecture separating data-space and feature-space is described as artificial, but the results may still inform modern architectures.
- Synthetic-data performance was never better than training on an equivalent amount of real data, suggesting an upper bound set by real-data training.
- CNN test error consistently decreased with more SMOTE or elastic-warping data, although the experiments did not reach the point where improvement stopped.
- CSVM performance degraded when more synthetic data were added using DBSMOTE.
- CELM with SMOTE peaked at 1000 training samples per class, after which additional synthetic samples decreased performance.
- Data-augmentation benefit was greatest for CNN, followed by CELM and then CSVM.
VII. CONCLUSION
The paper concludes that data-space augmentation is preferable when label-preserving transforms are known, while SMOTE is a more robust feature-space option than DBSMOTE. Augmentation gains remain bounded by equivalent real data.
- Data augmentation can operate in data-space or feature-space, but data-space augmentation is better when label-preserving transforms are known.
- SMOTE can augment data in feature-space, while DBSMOTE may increase overfitting by generating samples near existing cluster centers.
- The improvement in error percentage from data augmentation is bounded by the equivalent amount of real data.