Source-linked AI summary
Dreaming More Data: Class-dependent Distributions over Diffeomorphisms for Learned Data Augmentation
Søren Hauberg, Oren Freifeld, Anders Boesen Lindbo Larsen, John W. Fisher, Lars Kai Hansen
TL;DR
Manual data augmentation requires pre-specifying transformations, motivating a learned alternative. The paper aligns within-class image pairs, models class-specific diffeomorphisms probabilistically, and uses sampled transformations for augmentation. It reports significant gains over manual schemes, including state-of-the-art results on small MNIST subsamples.
Problem
Existing augmentation schemes assume transformations are known a priori, making their specification an implicit form of feature engineering.
Method
The method aligns within-class image pairs under diffeomorphic transformations and fits a class-specific multivariate normal distribution in the tangent space at the manifold mean.
Results
Both the MLP and ConvNet significantly improve on the 1.5% MNIST error rate reported for 500 images per class.
Takeaways & Limitations
Learned augmentation can support strong deep-learning results with fairly small datasets and can use different transformation schemes for different classes.
Takeaways & Limitations
The approach requires observations that can be aligned to build statistical models of dataset deformations.
Abstract
from arXiv · showhide
Data augmentation is a key element in training high-dimensional models. In this approach, one synthesizes new observations by applying pre-specified transformations to the original training data; e.g.~new images are formed by rotating old ones. Current augmentation schemes, however, rely on manual specification of the applied transformations, making data augmentation an implicit form of feature engineering. With an eye towards true end-to-end learning, we suggest learning the applied transformations on a per-class basis. Particularly, we align image pairs within each class under the assumption that the spatial transformation between images belongs to a large class of diffeomorphisms. We then learn a class-specific probabilistic generative models of the transformations in a Riemannian submanifold of the Lie group of diffeomorphisms. We demonstrate significant performance improvements in training deep neural nets over manually-specified augmentation schemes. Our code and augmented datasets are available online.
1 Introduction
Data augmentation improves classification by applying transformations that preserve labels, but conventional schemes manually specify a small, shared set of transformations. The paper proposes learning class-specific augmentation distributions from data and reports significant improvements over manual schemes, especially for small datasets.
- Data variation includes label-relevant differences and irrelevant factors such as location, posture, and illumination.
- Data augmentation improves classification by generating new training observations through label-preserving image transformations.
- Conventional augmentation manually specifies simple transformations and often applies the same scheme across all classes.
- Class-specific differences can make a transformation useful for some labels but unsuitable for others, motivating learned augmentation schemes.
- The proposed approach aligns within-class image pairs, models diffeomorphisms per class, and samples transformations to create training data.The sampled transformations are applied to training images before training multilayer perceptrons and convolutional neural networks.
- The method significantly improves deep-network training over manually specified augmentation schemes, with the largest benefit reported for small datasets.
2 Background and Related Work
Prior approaches encode known transformation invariances through features, kernels, or augmented observations, but they generally assume the transformations are specified beforehand. This paper frames that specification as feature engineering and motivates learning transformations from data instead.
- A transformation T_θ is label-preserving when applying it to an image leaves the image’s class unchanged.
- Invariant features satisfy F(x) = F(x ◦T_θ), while related invariant-kernel and orbit-based approaches encode the same transformation knowledge differently.
- Invariant-feature, kernel, and orbit methods can be laborious or computationally expensive and often restrict transformations to simple families.
- Approximate methods include image pyramids with known scales and Tangent Prop, which locally linearizes transformations and therefore handles infinitesimal changes.
- Data augmentation commonly synthesizes observations by applying known transformations such as scaling, rotation, translation, flips, or color changes.
- These approaches generally assume transformations are known a priori, so specifying them constitutes an implicit form of feature engineering.
- The paper investigates whether transformations themselves should be learned from data rather than manually specified.
3 Diffeomorphisms: Representation, Inference, and Learning
The paper represents image deformations with a tractable finite-dimensional Riemannian submanifold of diffeomorphisms, estimates transformations by class-specific image alignment, and models their distributions for sampling. The learned transformations are non-rigid and capture deformations that manual schemes would be difficult to specify, while preserving topology and failing on topological differences.
- 3.1 Representing Diffeomorphisms: First-order diffeomorphisms provide a broad transformation class whose finite-dimensional subset makes inference computationally tractable.The subset is built from continuous piecewise-affine velocity fields on a triangular tessellation with boundary constraints.
- 3.1 Representing Diffeomorphisms: The transformation manifold contains diffeomorphisms, includes the identity, is closed under inversion, and is a connected Riemannian manifold with the velocity-field space as its tangent space.The exponential map links velocity fields in the tangent space to transformations on the manifold and has an efficient numerical implementation.
- 3.2 Estimating Transformations by Aligning Images: Within each class, image pairs among K = 5 nearest neighbors are aligned by finding a transformation T θmn such that xm ◦T θmn ≈xn.The resulting transformations are inferred with a Bayesian model using Gaussian intensity-difference likelihoods, smoothness priors, and MCMC sampling.
- 3.3 Statistical Models of Transformations: The aligned transformations are represented in the tangent space, where their intrinsic mean is the identity transformation and class-specific distributions can be modeled.The inverse exponential map supplies tangent-space coordinates, and the identity is established as the intrinsic mean of paired transformations.
- 3.4 Sampling New Data: The learned distributions generate realistic non-rigid deformations that capture class-specific variations difficult to model manually, while diffeomorphic alignment cannot capture topological differences.Examples include tail direction in “2” and self-intersection position in “8”; the full alignment process takes approximately 10 seconds on GPU hardware.
4 Experiments
The experiments evaluate learned augmentation on MNIST using nearest-neighbor, MLP, and ConvNet classifiers, comparing AlignMNIST with manually specified InfiMNIST schemes. AlignMNIST achieves the strongest reported performance, including with only 500 images per class.
- The augmentation datasets are built from 500 images per class by estimating transformations between nearest-neighbor image pairs and sampling class-specific tangent-space Gaussians.
- AlignMNIST achieves 1.4% test error with a nearest-neighbor classifier, versus 2.6% for InfiMNIST and 3.1% for MNIST.
- AlignMNIST gives the best MLP predictive model, with AlignMNIST500 within 0.2 percentage points of full-data InfiMNIST and 1.6 points better than InfiMNIST500.
- Additional ConvNet gains indicate that the augmentation benefits extend to networks with spatial structure.
- The study could not reproduce previously reported ConvNet results for InfiMNIST, while using the same implementation and evaluation scheme across datasets.
5 Discussion
The learned augmentation scheme reduces modeling dimensionality while producing well-behaved transformations and supports class-specific augmentation. It improves performance with smaller training sets, but requires alignable observations and extends naturally to several aligned data domains.
- 5 Discussion: Modeling diffeomorphisms in a finite-dimensional space lowers dimensionality and ensures that sampled transformations are well-behaved.The representation is needed for efficient inference over latent diffeomorphisms.
- 5 Discussion: The learned scheme achieves state-of-the-art results on small MNIST subsamples and can outperform classifiers trained on the full dataset or manually specified augmentations.The paper reports that learned augmentation allows significantly smaller training sets.
- 5 Discussion: Pairwise alignment provides O(N_y^2) potential transformations within a class, supporting estimation of a useful augmentation distribution.The number of available pairs is given for a class containing N_y observations.
- 5 Discussion: Learned augmentation can use different transformation schemes for different classes, unlike manually designed schemes that generally use one scheme across classes.Class-specific schemes are presented as a further benefit of learning from the data.
- 5 Discussion: The approach requires observations to be alignable, although the authors identify medical imaging, acoustic signals, and 3D meshes as relevant extension domains.The method is exemplified on MNIST, where observations often have well-defined alignments.