PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Augmentation using Pre-trained Transformer Models

Varun Kumar, Ashutosh Choudhary, Eunah Cho

arXiv:2003.02245v2cs.CLcs.LG

TL;DR

Low-data tasks need larger training sets to reduce overfitting and improve robustness. This paper unifies conditional augmentation with pretrained transformers by prepending class labels, finding that Seq2Seq models perform best among the studied approaches in low-resource classification.

  • Problem

    Data augmentation is needed to increase training data and support robust machine-learning models in low-data tasks.

  • Method

    The paper conditions AE, AR, and Seq2Seq pretrained transformers for augmentation by prepending class labels to training examples and compares these methods across three classification tasks.

  • Results

    Pretrained Seq2Seq augmentation provides the best performance among the three methods, while generated data improves classification performance in low-data settings.

  • Takeaways & Limitations

    Label-conditioned augmentation offers a unified way to use different pretrained models and can be combined with other text and latent-space augmentation methods.

  • Takeaways & Limitations

    Augmentation performance varies across tasks and datasets, while shared hyperparameters may not be optimal for every setting.

Abstract

from arXiv · show

Language model based pre-trained models such as BERT have provided significant gains across different NLP tasks. In this paper, we study different types of transformer based pre-trained models such as auto-regressive models (GPT-2), auto-encoder models (BERT), and seq2seq models (BART) for conditional data augmentation. We show that prepending the class labels to text sequences provides a simple yet effective way to condition the pre-trained models for data augmentation. Additionally, on three classification benchmarks, pre-trained Seq2Seq model outperforms other data augmentation methods in a low-resource setting. Further, we explore how different pre-trained model based data augmentation differs in-terms of data diversity, and how well such methods preserve the class-label information.

1 Introduction

The paper presents a unified conditional data augmentation approach using pre-trained transformers, addressing label preservation and comparison inconsistencies in prior methods. It evaluates autoencoder, autoregressive, and seq2seq models in low-resource classification, where seq2seq augmentation performs best.

  • Motivation: Prior word-replacement and non-conditional language-model augmentation methods can alter class-relevant meaning and fail to preserve class labels.A sentiment example changes from “a small impact with a big movie” to “a small movie with a big impact.”
  • Motivation: Conditional BERT incorporates class labels through segment embeddings, but this conditioning does not generalize directly to models without segment embeddings.
  • Motivation: Earlier GPT-2 augmentation generated and selected examples differently from other models, preventing fair comparison across augmentation methods.
  • Approach: The paper compares BERT, GPT2, and BART as autoencoder, autoregressive, and seq2seq models for augmentation across sentiment, intent, and question classification.
  • Evaluation: The study simulates a low-resource classification setting using 10 training examples per class.
  • Results: All three pre-trained model types improve low-data classification performance, with the pre-trained seq2seq model achieving the best performance.
  • Contributions: The contribution is a seq2seq augmentation implementation, a conditional comparison of augmentation methods, and practical guidance for different pre-trained models.

2 DA using Pre-trained Models

The paper situates data augmentation within autoencoder, autoregressive, and seq2seq pre-training paradigms, using a denoising objective for seq2seq models.

  • Pre-trained Models: Pre-trained language models are trained in autoencoder, autoregressive, or seq2seq settings.
  • Pre-trained Models: The paper explores how these pre-training paradigms can be used for data augmentation.

1 Fine-tune G using Dtrain to obtain Gtuned

The procedure synthesizes augmented examples from the fine-tuned generator.

  • Data Generation: The algorithm synthesizes examples {x̂_i, ŷ_i} from the fine-tuned generator.

6 end

The augmentation formulation generates synthetic labeled data with a pre-trained model conditioned by prepended labels. It compares prepend and expand conditioning across model types, using model-specific generation procedures for BERT and GPT2.

  • Data Augmentation Formulation: Given training pairs and a pre-trained model G, the method generates a synthetic dataset for augmentation.
  • Data Augmentation Formulation: One synthetic example is generated for every training example, so augmented data matches the original dataset size.
  • Label Conditioning: Prepend conditions a model by adding the label to each sequence without expanding the model vocabulary, whereas expand also adds the label as a vocabulary token.
  • Label Conditioning: In prepend, labels may split into multiple subword units; expand treats each label as a single token.
  • Fine-tuning: All models are fine-tuned by updating their learnable parameters with their default task and loss function.
  • Autoencoder Models: BERT augmentation uses masked language modeling, randomly masking tokens and predicting them from context.
  • Autoregressive Models: GPT2 is fine-tuned on label-separated sequences, using SEP between labels and sentences and EOS at sentence ends.
  • Autoregressive Models: GPT2 generation prompts the model with a label and optionally the first three words of an input sequence to improve label quality.

2.2 Conditional DA using Pre-trained Seq2Seq model

The paper uses BART as a representative pre-trained seq2seq model for data augmentation and conditions it by prepending class labels. It trains BART with word-level masking and denoising reconstruction before generation.

  • BART is selected as a pre-trained seq2seq representative because of its relatively lower computational cost.
  • Class labels are prepended to examples of each class to condition BART during augmentation.
  • Masking performs better than insertion and deletion in preliminary experiments, and word-level masking outperforms subword masking consistently.
  • BARTword replaces one word with a mask token, whereas BARTspan replaces a continuous chunk with one mask token.
  • 40% of words are masked, and BART is fine-tuned to decode the original sequence from the masked sequence.

2.3 Pre-trained Model Implementation

The implementation compares BERT, GPT2, and BART using their respective toolkits and generation settings. Models are fine-tuned with task-specific procedures, while validation performance selects the best model.

  • BERT experiments use bert-base-uncased with separate prepend and expand settings, which require different training durations and learning rates.Prepend uses 10 epochs and 4e−5; expand uses 150 epochs and 1.5e−4.
  • GPT2 experiments use GPT2-Small with default fine-tuning parameters, SEP as a separate token, and nucleus sampling with top p = 0.9.The generation configuration also uses top k = 0 and an end-of-text token.
  • BART experiments use fairseq’s implementation and bart large model weights.
  • BART fine-tuning uses denoising reconstruction with 40% masking, prepended labels, label smoothing of 0.1, and a learning rate of 1e−5.The decoder reconstructs the original sequence while producing the prepended label as a sequence token.
  • The classifier uses BERT’s pooled [CLS] representation, applies 0.1 dropout, and passes the result to a Softmax layer.
  • Validation-set performance selects the best model across all augmentation experiments, which run on a single Nvidia Tesla V100 GPU.BART uses f16 precision.

3 Experimental Setup

The experiments compare three baseline augmentation methods across three text-classification datasets in simulated low-resource settings. Evaluation measures downstream classification performance, semantic fidelity, and generated-text diversity.

  • 3.1 Baseline Approaches for DA: The baselines are EDA, backtranslation, and CBERT.EDA uses word replacement, backtranslation uses EN-DE and DE-EN models, and CBERT is a model-based augmentation method.
  • 3.2 Data Sets: The datasets are SST-2 for sentiment classification, SNIPS with 7 intents, and TREC with six question types.SST-2 has Positive and Negative labels; TREC covers question categories such as person and location.
  • 3.2 Data Sets: Class labels are replaced with lowercase text labels, although byte-pair encoding may split them into multiple tokens.
  • 3.3 Evaluation: Low-resource training sets are created by randomly selecting equal numbers of examples per class, focusing on a setting with 10 examples per class.The study also considers 50 examples per class; 100 examples per class was judged comparatively moderate for SNIPS.
  • 3.3 Evaluation: Extrinsic evaluation adds generated examples to low-resource training data and evaluates the full test set across 15 repetitions.Training and development subsets are randomly resampled for each experiment.
  • 3.3 Evaluation: Intrinsic evaluation measures semantic fidelity through retained meaning and class information, and measures diversity using type token ratio.Type token ratio divides unique n-grams by all n-grams in generated text.

4 Results and Discussion

The experiments compare conditional augmentation with BERT, GPT2, and BART across classification, fidelity, and diversity. BART performs best for classification, while model families trade off label preservation, generation freedom, semantic fidelity, and diversity.

  • Generation by Conditioning on Labels: BERTprepend, BERTexpand, and CBERT have similar performance across three datasets, but BERTexpand requires more than 100 epochs to converge versus less than 8 for the others.The paper attributes the slower convergence to learning new label representations from limited fine-tuning data.
  • Pre-trained Model Comparison: BART outperforms other data augmentation approaches on all datasets in low-data classification evaluation.The comparison uses 10 examples per class for augmentation-model fine-tuning and reports mean (standard deviation) accuracy over repeated experiments.
  • Generated Data Fidelity: Classifier-based fidelity evaluation measures whether generated text retains the input class label, with higher accuracy indicating greater semantic fidelity.A classifier trained on labelled data predicts the labels of generated examples.
  • Generated Data Fidelity: AE methods outperform GPT2 and BART in semantic fidelity, while back translation outperforms all other methods on two datasets.These results emphasize that strong classification performance and label-preserving generation are distinct evaluation dimensions.
  • Generated Data Diversity: EDA produces the most diverse trigrams, whereas back translation produces the most diverse unigrams according to type-token ratios.EDA’s random token changes increase n-gram diversity without necessarily preserving input semantics, while back translation often introduces unseen unigrams.
  • Guidelines For Using Different Types Of Pre-trained Models For DA: RoBERTaprepend outperforms BERTprepend on TREC, achieving 66.12 versus 64.74 mean accuracy.The paper also reports that GPT2 generates coherent text but does not preserve labels well without a few starting words and the label.
  • Guidelines For Using Different Types Of Pre-trained Models For DA: Seq2seq models perform better with word or span masking than with other denoising objectives for data augmentation.The paper presents masking-objective choice as a practical guideline for seq2seq augmentation.
  • Guidelines For Using Different Types Of Pre-trained Models For DA: AE models preserve labels well, AR models support unconstrained generation but may lose label information, and seq2seq models balance diversity with semantic fidelity.For seq2seq models, generated-data diversity can be controlled by varying the masking ratio.

5 Conclusion And Future Work

The paper shows that AE, AR, and Seq2Seq pre-trained models can be conditioned on class labels for effective data augmentation. These methods can also be combined with other text and latent-space augmentation techniques.

  • AE, AR, and Seq2Seq pre-trained models can be conditioned on class labels by prepending label information.
  • Label-conditioned pre-trained models provide an effective way to augment training data.
  • The augmentation methods can be combined with co-training of the data generator and classifier.
  • The methods can also be combined with latent space augmentation.

A.1 Classification performance on 50 examples per class

With 50 examples per class, the evaluation reports that data augmentation does not improve classification performance overall and often reduces accuracy.

  • Table 7 reports mean and standard deviation accuracy on the full test set across 15 random training and development-data samplings.
  • Data augmentation does not improve classification performance when 50 examples per class are selected.
  • In many cases, data augmentation hurts classification accuracy in this setting.
Loading 2003.02245v2…