PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Data Manipulation for Augmentation and Weighting

Zhiting Hu, Bowen Tan, Ruslan Salakhutdinov, Tom Mitchell, Eric P. Xing

arXiv:1910.12795v1cs.LGcs.CLcs.CVstat.ML

TL;DR

Data manipulation can improve training, but prior approaches are generally designed for specific schemes. This paper adapts reward learning to jointly learn manipulation and model parameters, and reports improvements across low-data and imbalanced image and text classification.

  • Problem

    Data augmentation, weighting, and other manipulation can help with limited or low-quality data, but existing learning-based approaches are usually designed for particular manipulation types.

    Source: §1 · p. 1

  • Method

    The approach adapts reward learning to supervised learning, using different data-reward parameterizations to learn manipulation schemes jointly with the target model.

    Source: §1 · pp. 1–2, §4.1 · pp. 4–5

  • Results

    Both augmentation and weighting significantly improve over strong base models and dedicated methods in low-data and imbalanced image and text classification; augmentation excels with little data, while weighting better addresses imbalance.

    Source: §1 · p. 2, §5.2 · pp. 7–8

  • Takeaways & Limitations

    A shared algorithm can instantiate both augmentation and weighting, with their relative suitability varying by setting.

    Source: §1 · p. 2, §6 · p. 9

    On imbalanced SST-2, augmentation yielded around 50% accuracy because its language model was fit to imbalanced data, making label preservation inaccurate and introducing noise.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Manipulating data, such as weighting data examples or augmenting with new instances, has been increasingly used to improve model training. Previous work has studied various rule- or learning-based approaches designed for specific types of data manipulation. In this work, we propose a new method that supports learning different manipulation schemes with the same gradient-based algorithm. Our approach builds upon a recent connection of supervised learning and reinforcement learning (RL), and adapts an off-the-shelf reward learning algorithm from RL for joint data manipulation learning and model training. Different parameterization of the "data reward" function instantiates different manipulation schemes. We showcase data augmentation that learns a text transformation network, and data weighting that dynamically adapts the data sample importance. Experiments show the resulting algorithms significantly improve the image and text classification performance in low data regime and class-imbalance problems.

1 Introduction

Data manipulation can improve training, but existing learning-based methods are usually tailored to particular schemes. The paper adapts reward learning to learn different schemes with one gradient-based algorithm, reporting gains across low-data and imbalanced classification settings.

1 Introduction

  • Data augmentation, weighting, and synthesis address training-data quantity or quality, and different manipulation types can suit different settings.The paper highlights low-data regimes and datasets with imbalanced labels as motivating cases.Source: §1 · p. 1
  • Prior automated methods alleviate manual engineering, but are usually designed for either augmentation or weighting, limiting their application scope.Examples include reinforcement-learning-based augmentation, validation-based meta-learning weights, and curriculum-based weighting networks.Source: §1 · p. 1
  • One gradient-based algorithm learns different manipulation schemes by adapting reward learning to supervised learning and parameterizing a data reward.Manipulation parameters are learned jointly with the target model through gradient descent on validation examples.Source: §1 · pp. 1–2
  • Both augmentation and weighting improve over strong base models and dedicated manipulation methods across image and text classification; augmentation excels with little data, while weighting better addresses imbalance.The experiments include large-scale pretrained initializations such as BERT and ResNet.Source: §1 · p. 2
  • The authors frame their derivation as algorithm extrapolation between learning paradigms.They refer readers to Section 6 for further discussion.Source: §1 · p. 2

2 Related Work

Prior work covers augmentation, weighting, and synthesis, often with methods tailored to a particular manipulation type. This paper frames manipulation as reward learning and uses stochastic gradient descent to learn its parameters.

2 Related Work

  • For augmentation, the paper treats manipulation as a reward function rather than an RL policy and learns its parameters with stochastic gradient descent.Its text contextual-augmentation instantiation fine-tunes the augmentation network jointly with the target model, extending prior work [49].Source: §2 · p. 2
  • Data-weighting methods include loss-based rules and learned networks, while this framework iteratively updates sample weights as parameters rather than re-estimating them from scratch.The paper contrasts its reward-learning perspective with validation-set meta-learning.Source: §2 · p. 2
  • Data synthesis creates artificial samples from scratch, and the paper leaves adaptive synthesis as a possible future instantiation.Prior examples include conditional-image GANs [3] [34] and cyclic text generator-classifier training.Source: §2 · p. 2

3 Background

The background recasts supervised learning as policy optimization, then adapts reward learning so manipulation parameters can be optimized through their effect on model training. The resulting procedure alternates model and manipulation updates.

3 Background

  • The framework views the supervised model pθ(y|x) as a policy choosing output y for input x, and uses a variational distribution q(x,y) in its policy-optimization objective.The distribution factorizes as q(x,y) = p(x)q(y|x).Source: §3 · p. 3
  • The objective maximizes expected reward under q while keeping pθ close to q and regularizing q with entropy; an EM procedure alternates q and θ optimization.In the stated limiting case, the procedure reduces to maximum-likelihood learning on the empirical data distribution.Source: §3 · p. 3
  • Reward learning updates policy parameters using intrinsic and extrinsic rewards, then adjusts reward parameters to maximize the resulting policy’s extrinsic reward by backpropagating through the policy update.The policy parameter update depends on the intrinsic-reward parameters, enabling that gradient path.Source: §3 · p. 3
  • The joint algorithm alternates optimizing the model on manipulated training data and optimizing manipulation parameters using validation-set data log-likelihood.It returns the learned model and manipulation function.Source: §3 · p. 4

4 Learning Data Manipulation

The section formulates data manipulation as a parameterized reward learned alongside the target model using validation performance. It demonstrates this framework with text augmentation and data weighting.

4.1 Method

  • A relaxed, parameterized data reward can encode distinct schemes, including weighting matched examples and augmenting or synthesizing samples beyond the training set.The reward’s parameterization determines the manipulation scheme.Source: §4.1 · p. 4
  • Manipulation parameters are learned to maximize held-out validation performance, with gradients propagated through the model update to the reward parameters.The algorithm uses a very small validation set, such as 2 labels per class, in the low-data setting.Source: §4.1 · pp. 4–5
  • Unlike policy-gradient augmentation, the framework treats manipulation as a reward and uses stochastic-gradient updates; it is described as more broadly applicable than.Source: §4.1 · p. 5

4.2 Instantiations: Augmentation & Weighting

  • Text augmentation fine-tunes a label-conditional BERT substitution model jointly with target-model training, addressing a fixed augmenter that cannot adapt to the target model’s state.When task data are small, the fixed language model may also produce label-noisy samples; Gumbel-softmax approximates discrete substitutions for gradient propagation.Source: §4.2 · p. 5
  • Data weighting assigns each training example a learned importance weight; weights are maintained and updated during training rather than re-estimated from scratch each iteration.With minibatch optimization, weighted sampling is approximated by taking the softmax over the minibatch’s weights.Source: §4.2 · p. 6

5 Experiments

The experiments test augmentation and weighting on text and image classification under scarce data and class imbalance. Both methods improve on strong baselines in low-data settings, while weighting is effective under imbalance and text augmentation is not.

5.1 Experimental Setup

  • Experiments use small labeled datasets and validation sets to learn manipulation parameters, with binary classification used for the imbalance setting.The low-data text tasks use 40 training instances per class, and the imbalance experiments vary one class between 20 and 100 instances against 1,000 of the other.Source: §5 · p. 6, §5.2 · p. 7, §5.3 · p. 9

5.2 Low Data Regime

  • Both augmentation and weighting improve base-model performance in low-data classification, and augmentation performs better than weighting.The experiments cover text and image classification.Source: §5.2 · p. 7

5.3 Imbalanced Labels

  • On imbalanced SST-2, weighting performs best across settings, with gains over the base model ranging from around 6 accuracy points at 100:1000 to over 20 at 20:1000.The proportion-based weighting offers only limited improvement; the base model trained on joint training-validation data for fixed steps performs poorly.Source: Table 3 · p. 8, §5.3 · p. 9
  • On imbalanced CIFAR10, learned weighting outperforms the comparison approaches, while fixed proportion-based weighting harms performance at 50:1000 and 100:1000.Table 4 reports results for varying class counts.Source: Table 4 · p. 8, §5.3 · p. 9

5.2 Low Data Regime

  • Adaptive text augmentation improves over the base model, synonym substitution, and a fixed augmentation network across three text datasets.The adaptive method fine-tunes the augmentation network jointly with model training; heuristic synonyms sometimes harm performance.Source: Table 1 · p. 7, §5.2 · p. 8
  • Learned data weighting consistently improves over the base model and on text, while augmentation consistently outperforms weighting in this low-data setting.The weights are maintained as manipulation parameters throughout training rather than re-estimated from scratch each iteration.Source: Table 1 · p. 7, §5.2 · p. 8
  • Data weighting improves over the base model and on image classification with both randomly initialized and pretrained ResNet-34 models.Table 2 covers the balanced CIFAR10 subset with 40 training and 2 validation examples per class.Source: Table 2 · p. 7, §5.2 · p. 8

5.3 Imbalanced Labels

  • On imbalanced SST-2, text augmentation yields around 50% accuracy, which the authors attribute to inaccurate label preservation and noise from fitting the augmentation LM to imbalanced data.The authors note that augmenting only rare classes could potentially help.Source: §5.3 · p. 9

6 Discussions: Algorithm Extrapolation between Learning Paradigms

The paper frames its method as extrapolating an algorithm across learning paradigms: it adapts reward learning from reinforcement learning to supervised data manipulation. More broadly, the authors argue that shared formalisms can support systematic transfers of methods between contexts.

6 Discussions: Algorithm Extrapolation between Learning Paradigms

  • The method adapts an off-the-shelf reinforcement-learning reward algorithm to supervised learning through the connection between the paradigms.The authors describe this as extrapolating algorithms between learning paradigms.Source: §6 · p. 9
  • A shared formalism can offer new understanding of distinct paradigms and support systematic transfer of solutions between similar problems.The authors cite transferring reward learning to structured knowledge and exchanging techniques between GANs and VAEs as examples.Source: §6 · p. 9
  • The authors argue that algorithm extrapolation can replace some from-scratch design with repeatable transfers of ideas across contexts.They contrast this with work that requires deep expertise and yields solutions for dedicated contexts.Source: §6 · p. 9

Related papers

Loading 1910.12795v1…