Source-linked AI summary
Dataset Augmentation in Feature Space
Terrance DeVries, Graham W. Taylor
TL;DR
Domain-specific augmentation is difficult to reuse across problems, motivating a more general approach. The paper transforms examples in learned feature space, using sequence-to-sequence context vectors, and reports that feature-space extrapolation improves supervised-learning performance across multiple domains, with near state-of-the-art results on two datasets.
Problem
Domain-specific augmentation requires transformations to be designed, implemented, and evaluated separately for each domain, while much available data remains unlabeled.
Method
The method projects examples into learned sequence-to-sequence context vectors and augments them by adding noise, interpolating, or extrapolating in feature space.
Results
Feature-space extrapolation improves supervised-learning performance across five datasets from different domains and achieves near state-of-the-art results on two of them.
Takeaways & Limitations
Feature-space extrapolation is presented as a domain-independent, general-purpose augmentation framework for improving generalization when labeled data is limited.
Takeaways & Limitations
Domain-specific augmentation methods may perform well in one domain but cannot necessarily be applied to other domains.
Abstract
from arXiv · showhide
Dataset augmentation, the practice of applying a wide array of domain-specific transformations to synthetically expand a training set, is a standard tool in supervised learning. While effective in tasks such as visual recognition, the set of transformations must be carefully designed, implemented, and tested for every new domain, limiting its re-use and generality. In this paper, we adopt a simpler, domain-agnostic approach to dataset augmentation. We start with existing data points and apply simple transformations such as adding noise, interpolating, or extrapolating between them. Our main insight is to perform the transformation not in input space, but in a learned feature space. A re-kindling of interest in unsupervised representation learning makes this technique timely and more effective. It is a simple proposal, but to-date one that has not been tested empirically. Working in the space of context vectors generated by sequence-to-sequence models, we demonstrate a technique that is effective for both static and sequential data.
1 INTRODUCTION
The paper proposes augmenting datasets through simple transformations in learned feature space rather than domain-specific input-space transformations. It focuses on feature-space extrapolation as a domain-independent way to improve supervised learning when labeled data is limited.
- Approach: Feature-space augmentation perturbs, interpolates, or extrapolates between existing examples instead of applying domain-specific input transformations.The proposed transformations operate on learned representations produced by unsupervised representation-learning models.
- Motivation and contribution: Learned feature spaces are motivated as regions where traversing between representations is more likely to produce realistic samples than traversing in input space.The paper connects this motivation to the resurgence of unsupervised representation learning.
- Approach: The study examines adding noise, interpolating, and extrapolating context vectors to determine which transformations most improve supervised learning.These operations manipulate vector representations of existing data points.
- Motivation and contribution: The method is domain-independent, requires no specialized knowledge, and is intended to improve generalization when labeled data is limited.The authors report that augmented models outperform models trained only on the original dataset.
- Motivation and contribution: The paper identifies feature-space extrapolation as an effective augmentation strategy and reports improved supervised-learning performance over training on original data alone.The authors present the technique as a general-purpose framework comparable in role to augmentation used in visual recognition.
2 RELATED WORK
Prior augmentation methods are especially established for images but are harder to reuse across domains. The paper positions learned feature-space augmentation as an alternative to repeatedly designing domain-specific transformations.
- Domain-specific augmentation: Dataset augmentation is a standard regularization technique, especially in visual recognition where image manipulations can readily generate new data.Examples include shifting, scaling, rotation, and other affine transformations.
- Domain-specific augmentation: Audio augmentation studies found that only pitch shifting and random frequency filtering improved singing-voice-detection performance among several tested transformations.The tested alternatives included Gaussian noise, pitch shifting, time stretching, loudness variation, frequency filtering, and input-space interpolation.
- Domain transferability: Effective audio transformations cannot necessarily be applied to other domains, requiring augmentation design, implementation, and evaluation to be repeated for each problem.This limits the straightforward reuse of domain-specific augmentation methods.
- Sequence-to-sequence representations: Sequence-to-sequence models convert input sequences into fixed-length context vectors that can be used to generate output sequences.The paper uses these context vectors as learned feature representations for augmentation.
3 MODEL
The model learns a feature space with a sequence autoencoder, transforms encoded context vectors, and uses the resulting vectors directly or decodes them into sequences for supervised learning.
- Architecture: The approach first learns a feature representation with a sequence autoencoder, then applies simple transformations to encoded context vectors.The sequence autoencoder provides a generic feature space for both time series and static data.
- Architecture: A sequence autoencoder uses recurrent encoder and decoder layers, with the encoder’s final hidden state serving as the context vector that seeds decoding.The implementation uses stacked LSTMs with two layers in both encoder and decoder.
- Architecture: The decoder is conditioned on the context vector at every time step, which improves reconstructions and is critical to the augmentation process.This differs from using the context vector only at the decoder’s first time step.
- Augmentation in feature space: Each example is encoded into feature space before augmentation, where noise, interpolation, or extrapolation can be applied to its context vector.Noise is generated from a zero-mean Gaussian distribution, while interpolation and extrapolation use neighboring context vectors.
- Augmentation in feature space: New context vectors can be used directly for supervised learning or decoded into full sequences for training a sequence classifier.Interpolated sequences use the average input length, whereas extrapolated sequences use the length of one input sequence.
4 EXPERIMENTS
Experiments evaluate feature-space augmentation across sequential and static data, comparing noise, interpolation, and extrapolation after learning representations with sequence autoencoders. Extrapolation generally improves classification, while reconstruction fidelity limits some image-based settings.
- Experimental procedure: A sequence autoencoder learns the feature space, and nearest-neighbour interpolation or extrapolation generates synthetic examples for classification.Each sample is paired with its 10 nearest in-class neighbours in feature space unless otherwise stated.
- Sinusoids: Noise-augmented sinusoid context vectors decode into valid samples with small changes in amplitude, frequency, and phase.The noise magnitude was γ = 0.5.
- Sinusoids: Interpolated sinusoids transition smoothly between parent characteristics, whereas extrapolation exaggerates differences such as amplitude and phase shift.Both transformations vary λ from 0 to 1.
- UJI Pen Characters: For handwritten characters, interpolation preserves parent characteristics and produces limited variety, while extrapolation produces a wider variety of samples.The authors hypothesize that this added variability is necessary for useful augmentation.
- Image classification: On MNIST and CIFAR-10, feature-space extrapolation improves over no augmentation and can outperform manual input-space augmentation, including a 1.4% CIFAR-10 error reduction.The CIFAR-10 baseline already used image transformations, suggesting the two augmentation types could complement each other.
- Reconstructed images: With reconstructed CIFAR-10 images, extrapolated examples worsen performance when combined with original images, but improve reconstructed-only training by 1.0% or 1.6% depending on augmentation.The authors attribute the discrepancy to possible fidelity loss when mapping between image and context-vector spaces.
5 DISCUSSION
The discussion finds that feature-space extrapolation is useful mainly for datasets with complex class boundaries, whereas interpolation can help when boundaries are simple.
- Extrapolation improved performance for complex class boundaries but hindered it for simple boundaries.Interpolation helped in simple cases such as linear separability or one class encircling another.
- Noise and interpolation did not generate useful synthetic examples in the reported feature-space experiments.The authors identify extrapolation as the effective operator overall.
- Interpolation may tighten class boundaries and increase confidence, potentially causing overfitting and overly smooth decision boundaries.The authors present this as a current hypothesis explaining interpolation’s behavior.
6 CONCLUSION
The paper presents domain-independent augmentation by extrapolating between samples in a learned feature space. Across five domains, the technique improves supervised learning performance, reaches near state-of-the-art results on two datasets, and may complement domain-specific augmentation.
- The method trains a sequence autoencoder to construct a learned feature space, then extrapolates between samples to augment datasets.The same simple architecture is used across the reported datasets.
- The technique increases dataset variability and is intended to produce more robust models.
- Quantitative experiments cover five datasets spanning speech, sensor processing, motion capture, and images.
- The approach achieves near state-of-the-art results on two datasets.
- Feature-space augmentation may complement domain-specific augmentation when used with Wide ResNets.