Learning to Compose Domain-Specific Transformations for Data Augmentation

Alexander J. Ratner; Henry R. Ehrenberg; Zeshan Hussain; Jared Dunnmon; Christopher Ré

Source-linked AI summary

Learning to Compose Domain-Specific Transformations for Data Augmentation

Alexander J. Ratner, Henry R. Ehrenberg, Zeshan Hussain, Jared Dunnmon, Christopher Ré

arXiv:1709.01643v3Sep 6, 2017stat.MLcs.CVcs.LG

TL;DR

Data augmentation helps address limited labeled data, but manually composing and tuning effective transformations is difficult. The paper learns a generative sequence model over user-specified, potentially stochastic black-box transformations using a generative adversarial approach and reinforcement learning. Across image and text tasks, it improves over standard heuristic augmentation, including gains of 4.0 accuracy points on CIFAR-10, 1.4 F1 points on ACE, and 3.4 accuracy points on mammography.

Problem
Manually finding effective compositions and parameterizations of class-preserving transformations is often infeasible when labeled data is limited.
Method
The method learns a generative sequence model over user-specified black-box transformation functions with a GAN-like objective and reinforcement-learning training.
Results
4.0 accuracy points on CIFAR-10, 1.4 F1 points on ACE relation extraction, and 3.4 accuracy points on mammography exceeded standard heuristic augmentation approaches.
Takeaways & Limitations
The approach supports domain-specific transformation operations across image and text modalities and can learn from unlabeled data.
Takeaways & Limitations
The approach relies on a weaker class-invariance assumption rather than assuming every provided transformation preserves class labels.

Abstract

from arXiv · show

Data augmentation is a ubiquitous technique for increasing the size of labeled training sets by leveraging task-specific data transformations that preserve class labels. While it is often easy for domain experts to specify individual transformations, constructing and tuning the more sophisticated compositions typically needed to achieve state-of-the-art results is a time-consuming manual task in practice. We propose a method for automating this process by learning a generative sequence model over user-specified transformation functions using a generative adversarial approach. Our method can make use of arbitrary, non-deterministic transformation functions, is robust to misspecified user input, and is trained on unlabeled data. The learned transformation model can then be used to perform data augmentation for any end discriminative model. In our experiments, we show the efficacy of our approach on both image and text datasets, achieving improvements of 4.0 accuracy points on CIFAR-10, 1.4 F1 points on the ACE relation extraction task, and 3.4 accuracy points when using domain-specific transformation operations on a medical imaging dataset as compared to standard heuristic augmentation approaches.

1 Introduction

Data augmentation addresses labeled-data scarcity, but manually choosing transformation compositions and parameterizations is difficult. The paper learns generative sequences of user-specified transformation functions to automate this process across domains and modalities.

Motivation: Data augmentation expands labeled training sets with class-preserving transformations, helping address the scarcity of labeled data for high-capacity models.It is also used as a form of weak supervision by leveraging domain knowledge about invariances.
Motivation: Selecting effective transformation parameterizations and compositions is time-consuming, difficult, and associated with large variations in end performance.Common practice uses transformations in random order with random parameterizations drawn from hand-tuned ranges.
Motivation: Existing automation methods may require large labeled datasets, restrict transformations to simple operations, or ignore domain knowledge.The paper instead targets settings where labeled training data is limited and expert invariances are available.
Approach: The proposed method learns a generative sequence model over user-specified black-box transformation functions to automate their composition and parameterization.The functions may represent incremental operations and need not be differentiable or deterministic.
Approach: The method uses a generative adversarial objective and reinforcement-learning training to produce transformed data that remains within the target data distribution using unlabeled data.The approach supports arbitrary transformation functions across image and text modalities.
Results: 4.0 accuracy points on CIFAR-10, 1.4 F1 points on ACE relation extraction, and 3.4 accuracy points on a mammography task exceeded standard heuristic augmentation approaches.The mammography result used domain-specific transformation operations developed with radiology experts.

2 Modeling Setup and Motivation

The paper represents augmentation as learned sequences of user-specified transformation functions, relaxing strict label preservation to allow unlabeled-data training while avoiding out-of-distribution mappings. A generative adversarial model selects useful compositions and parameterizations for downstream augmentation.

Generative adversarial objective: Figure 2’s pipeline trains a null-class discriminator and sequence generator adversarially, then uses the generator for augmentation of an end discriminative model.The generator produces sequences hτ1, ..., hτL from user-provided functions and unlabeled data.
Transformation representation: Users provide incremental black-box transformation functions that may be stochastic or non-differentiable, and the model learns their compositions.The representation supports operations such as image rotations, word substitutions, and tumor translations.
Weak class invariance: The method assumes transformations rarely change class but may map examples into an out-of-distribution null class.This weaker assumption is intended to better match practical augmentation settings and enables learning from unlabeled data.
Unlabeled-data learning: The objective minimizes the probability that generated transformation sequences map unlabeled examples into the null class.The optimization uses a distribution U of unlabeled data and trains the generator with respect to θ.
Sequence model classes: A diversity objective mitigates repetitive null transformations, while independent and state-based models provide alternative sequence representations.The state-based LSTM captures context-dependent and non-commutative transformation effects that mean-field choices cannot represent.

3 Learning a Transformation Sequence Model

The sequence model is trained with reinforcement learning because user-defined transformations may be stochastic or non-differentiable. States retain intermediate transformed data, and incremental discriminator-loss changes provide rewards for policy-gradient training.

Reinforcement learning formulation: Reinforcement learning handles transformation sequences whose black-box operations are not necessarily differentiable or deterministic.This preserves usability by allowing arbitrary scripts, libraries, and subfunctions in user-defined transformations.
State representation: The state after t operations includes the original input, intermediate transformed data, and previously selected transformation indices.Intermediate data are retained because transformations may be stochastic.
State representation: An LSTM updates its state from the previous state and the latest transformation index, then represents the sequence policy.The LSTM-based representation is one of the model classes used for Gθ.
Policy-gradient rewards: The incremental reward is the difference in discriminator loss between successive transformation steps.The first objective term is recast as an expected sum of these incremental rewards.
Policy-gradient optimization: Policy-gradient estimates are approximated by sampling batches of data points and multiple action sequences, with discounting over future rewards.The procedure uses n data points, m sampled sequences per point, and discount factor γ.

4 Related Work

Related work includes heuristic, interpolation-based, adversarial, and GAN-based augmentation methods. The paper distinguishes its approach by directly exploiting user-provided class-invariant transformations rather than relying solely on labels, restricted operations, or generated data.

Heuristic data augmentation: Heuristic augmentation commonly applies crops, flips, or small affine transformations with hand-tuned random parameters and orders.Related heuristic approaches have also been applied to audio and text.
Interpolation-based techniques: Interpolation-based methods generate augmented examples by interpolating between labeled points or learned feature representations.The paper views these approaches as complementary but orthogonal to direct use of domain-specific transformation operations.
Adversarial data augmentation: Adversarial augmentation methods adaptively select transformations to maximize the end classifier’s loss, often assuming label preservation or restricting operations.These assumptions distinguish them from the paper’s weaker invariance formulation.
GAN-based augmentation: Class-conditional GAN augmentation requires large labeled datasets, whereas standard unsupervised GANs generate additional out-of-class data for augmentation.The paper compares these GAN-based approaches with its transformation-focused formulation.

5 Experiments

Experiments evaluate the augmentation framework across benchmark and real-world image and text tasks, comparing learned generators with heuristic and GAN-based baselines. The learned approach generally improves performance, models sequential structure, incorporates domain-specific transformations, and avoids misspecified or destructive operations.

Datasets and experimental setup: The framework is evaluated on benchmark image datasets, ACE relation extraction, and mammography tumor classification, including robustness and hyperparameter sensitivity experiments.Experiments use MNIST, CIFAR-10, ACE, and DDSM, with unlabeled data available for generator training.
Evaluation: Experiments compare mean field and LSTM generators with Basic and heuristic augmentation baselines, plus related GAN methods on CIFAR-10.Table 1 reports task performance, while Table 2 compares comparable GAN methods using averaged CIFAR-10 results.
Results: Across the studied applications, learned augmentation outperforms the heuristic approach, with LSTM generally outperforming the mean field model.The authors also report smaller variance than comparable semi-supervised GAN approaches on CIFAR-10.
Results: On DDSM, adding segmentation-based domain-specific transformations harms heuristic augmentation, while the LSTM avoids destructive subsequences and achieves a 9.0 point boost over the comparable heuristic approach.The failure modes arise from combinations of segmentation-based TFs with standard operations such as zoom and rotate.
Robustness and calibration: On MNIST, the mean field generator learns to avoid misspecified transformation functions, while longer sequences improve CIFAR-10 end-model accuracy in calibration experiments.The calibration caption notes that random augmentation performs best with shorter sequences.

6 Conclusion and Future Work

The paper presents learned composition and parameterization of user-provided black-box transformations as a flexible augmentation method. It reports gains across modalities and domain-specific settings while identifying conditioning and dynamic sequence lengths as future directions.

Conclusion: The method learns to parameterize and compose user-provided black-box transformation operations for data augmentation.The framework supports arbitrary transformation functions, including operations supplied by domain experts.
Conclusion: A reinforcement-learning-trained generative sequence model in a GAN-like framework produces realistic transformed data useful for augmentation.The approach is designed to handle arbitrary transformation functions within the proposed framework.
Conclusion: The method yields strong gains over standard heuristic augmentation across applications, modalities, and complex domain-specific transformation functions.The paper frames domain knowledge about invariants as a form of weak supervision.
Future Work: Future work includes conditioning the generator on features of the transformed data point and generating transformation-function sequences with dynamic length.The authors also propose further formalizing data augmentation as weak supervision.

A.1 Review of Data Augmentation Use in State-of-the-Art

A review of leading CIFAR results illustrates that data augmentation is widespread and diverse, while its particular strategy can materially affect reported performance. The review is illustrative rather than comprehensive and emphasizes limits in reporting transparency for some tasks.

Use in state-of-the-art results: Data augmentation appears in 10 of 10 top CIFAR-10 results and 9 of 10 top CIFAR-100 results, with reported average boosts of 3.71 points.The review compiles examples of augmentation use in highly ranked results.
Variation in strategies: The reviewed papers use a wide variety of augmentation strategies, although some ResNet variants inherit simple strategies from prior work.The passage also reports that augmentation choice is widely reported to affect performance.
Scope and reporting caveats: The table is illustrative and not necessarily comprehensive, and published ImageNet results often describe augmentation implementation details opaquely.CIFAR-10 and CIFAR-100 were selected as representative, well-studied tasks with available published results.

B.1 Synthetic Data Examples

Synthetic experiments use two-dimensional points inside a unit ball to test whether learned transformation sequences preserve the data distribution. The setup compares mean-field and LSTM generators across transformation scenarios with an oracle discriminator.

Synthetic Setup: The synthetic dataset contains two-dimensional points uniformly sampled within a ball of radius r = 1 around the origin.Vector displacement transformations are applied to these points.
Synthetic Setup: The experiments compare mean-field and LSTM generators with either a fully connected discriminator or the oracle discriminator f(x) = 1{||x|| < 1}.The oracle discriminator identifies points inside the unit ball.
Synthetic Setup: The learned generator draws length-10 sequences of vector displacement transformations to produce augmented points.Figure 6 depicts original points in blue and transformed points in red.
Synthetic Scenarios: The toy scenarios test whether the model avoids individually harmful transformations and handles transformations that become irrecoverably lossy outside the unit ball.The first scenario contrasts displacement-magnitude distributions, while the second applies exponentially decaying magnitudes outside the ball.

B.2 Robustness to Transformed Test Data

The robustness evaluation tests classifiers on CIFAR-10 examples transformed by individual operations and compares absolute accuracy with relative score changes. The learned approach is described as most absolutely robust, while random augmentation is relatively robust for larger transformations.

Evaluation: Figure 7 reports accuracy on random 10% test subsamples and on versions augmented with a single transformation whose parameters are sampled uniformly at random.Dotted lines represent subsampled test accuracy, while vertical bars represent single-transformation accuracy.
Robustness Metrics: Absolute robustness is measured by model accuracy, whereas relative robustness is the change in model score on transformed test data.These metrics distinguish overall transformed-data accuracy from degradation relative to the original score.
Findings: The learned approach is reported as most absolutely robust, while random augmentation is most relatively robust, especially for larger transformations.The authors hypothesize that the learned approach mostly avoided applying larger transformations during training.

D.1 Benchmark Image Datasets

The benchmark image experiments use MNIST and CIFAR-10 with deliberately selected transformation sets and standard classifier architectures. The implementations also apply shared padding-and-cropping augmentation during training.

MNIST: MNIST uses 5000 training examples as a validation set and has a dataset-specific transformation-function set.The supplied passage introduces the MNIST transformations without enumerating them.
CIFAR-10: CIFAR-10 uses its own transformation-function set, including zoom factors of 0.9x, 1.1x, 0.75x, and 1.25x.The zoom values are listed as one component of the CIFAR-10 transformations.
Shared Training Augmentation: Both datasets use random four-pixel padding on each side followed by random crops back to the original dimensions during training.The choice of transformations is deliberate across datasets, while parameterization choices partly reflect separate implementations.
ACE: The ACE benchmark represents relation extraction as binary classification over Person-Organization entity pairs occurring in the same sentence.The corpus contains news articles and broadcast transcripts with pretagged entity mentions.
ACE: The ACE transformation set includes replacing nouns, verbs, or adjectives to the left, between, or to the right of the two entities.The listed operations cover the supplied noun, verb, and adjective replacement variants.

D.3 DDSM Mammography Task

The DDSM mammography task uses standard and domain-specific image transformations, including mass transplantation with Poisson blending. Generator training uses adversarial optimization, sequence length L = 10, and downstream validation-based model selection.

Transformations: The DDSM task uses five transformations: image rotation, image zoom, contrast enhancement, translation-and-transplantation, and rotation-and-transplantation.The operations are designed around mammogram geometry, tumor appearance, and mass context.
Domain-Specific Transformations: The transplantation transformations move or rotate a segmented mass and blend it onto randomly sampled normal tissue using Poisson blending.The transformed mass retains information about the mass and nearby tissue context.
Transformation Constraint: Poisson-blending transformation domains must avoid excessive proximity to destination-image boundaries to prevent spurious gradient information.This constrains the allowed translation and rotation ranges.
Training and Selection: The generator and discriminator are trained jointly with momentum SGD in an adversarial procedure, with sequence length fixed at L = 10 after an initial search.Final generators are selected using downstream validation performance and filtered partly by discriminator loss relative to random sequences.
Diversity: The diversity objective evaluates pairwise generalized Jaccard distance and the ratio of unique transformation n-grams to total possible n-grams.For CIFAR-10, learned sequences show lower values than random sequences on both reported diversity measures.

D.6 End Model Training

End-model training applies transformation regularization to transformed examples, with the calibration study showing benefits for both random and learned transformation sequences, especially the latter.

The training procedure transforms all minibatch examples during augmentation training, then disables transformation augmentation for the final ten epochs.The supplied training passage specifies ptransform = 1.0 initially and ptransform = 0.0 for the last ten epochs.
Transformation regularization adds a loss term measuring the distance between each example’s pre-softmax logits and those of its transformed copy.The term is applied to transformed data points in image experiments and adds little overhead because those points are already generated.
Adding transformation regularization improved performance for both heuristic random sequences and sequences generated by the trained LSTM model.The comparison was evaluated on a labeled validation set.
1.2 accuracy points versus 0.1 points: transformation regularization helped trained LSTM sequences more than random sequences.The calibration used coefficient 0.1 and an unlabeled-data batch size equal to 20% of the labeled-data batch size.