PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation

Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, Woomyeong Park

arXiv:2104.08826v2cs.CLcs.AI

TL;DR

Prompt-based large-language-model classification is difficult to scale because of prompt-length, inference-cost, and fine-tuning-compatibility limits. GPT3Mix instead uses prompted mixtures of real examples to generate synthetic text and soft labels for downstream training, and experiments show consistent gains across classification tasks, including up to 18.6% over the BERTbase baseline. The approach can improve pretrained-transformer robustness without online inference on heavy language models, but may propagate or amplify linguistic bias and toxicity.

  • Problem

    Prompt-based approaches face prompt-length limits, expensive and slow online inference, memory overhead, and poor compatibility with established fine-tuning methods.

  • Method

    GPT3Mix prompts large-scale language models with task-specific real samples to generate mixed synthetic text and uses their soft-label predictions for knowledge distillation.

  • Results

    GPT3Mix provides the most consistent performance boost across diverse classification tasks, improving average accuracy over the BERTbase baseline by as much as 18.6%.

  • Takeaways & Limitations

    GPT3Mix can improve pretrained-transformer robustness through mix-based perturbation and knowledge distillation without online inference on heavy language models.

  • Takeaways & Limitations

    GPT3Mix may propagate linguistic biases and toxicity, which can be amplified when augmented samples are reused in iterative applications.

Abstract

from arXiv · show

Large-scale language models such as GPT-3 are excellent few-shot learners, allowing them to be controlled via natural text prompts. Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability. This paper proposes a novel data augmentation technique that leverages large-scale language models to generate realistic text samples from a mixture of real samples. We also propose utilizing soft-labels predicted by the language models, effectively distilling knowledge from the large-scale language models and creating textual perturbations simultaneously. We perform data augmentation experiments on diverse classification tasks and show that our method hugely outperforms existing text augmentation methods. Ablation studies and a qualitative analysis provide more insights into our approach.

1 Introduction

Prompt-based large-language-model methods offer strong few-shot control but face scalability and compatibility limits. GPT3Mix addresses these limits by generating mixed synthetic text from real samples and transferring soft-label knowledge for downstream classification.

  • Motivation: Prompt-based approaches are constrained by prompt length, expensive online inference, and limited compatibility with established fine-tuning methods.These constraints make deployment in real-world use cases difficult because inference is slow and incurs substantial memory overhead.
  • Approach: GPT3Mix uses large-scale language models to imbue an existing training set with synthetic text rather than serving as the fine-tuned model or prompt-based backbone.This design retains the original downstream-training setup while using the language model for augmentation.
  • Approach: GPT3Mix generates hyper-realistic samples by embedding a few task-specific training sentences in a prompt and producing an augmented sentence influenced by their mixture.The method applies prompt-based generation to mix existing examples into new text.
  • Approach: Soft-label predictions enable knowledge distillation while the generated examples provide textual perturbations for training smaller classification models.GPT3Mix therefore combines synthetic-example augmentation with probability-based knowledge transfer from the large language model.
  • Evaluation: The authors introduce RT20, a post-cutoff movie-review benchmark designed to test augmentation without the possibility of language-model memorization.Results on RT20 attribute the method’s benefit to mix-based text synthesis rather than pretraining memorization.

2 Related Work

Prior NLP work includes knowledge distillation, shallow and deep text augmentation, external-language-model generation, and statistical mixing. GPT3Mix extends this landscape by using prompts from large-scale language models to generate synthetic augmentation samples.

  • Knowledge Distillation: Knowledge distillation trains a smaller student classifier using the outputs of a larger teacher classifier.GPT3Mix uses soft-label predictions from a large-scale language model to transfer knowledge to downstream classifiers.
  • Text Augmentation: Text augmentation perturbs linguistic space without changing class labels to improve downstream-model robustness and generalizability.Shallow methods inject locally plausible noise, including word or phrase substitutions such as EDA and synonym replacement.
  • Text Augmentation: External language models generate globally coherent augmentation samples through methods such as back-translation, BART-based generation, and masked-language-model denoising.These approaches target diverse and linguistically correct synthetic texts, including samples for data-scarce labels.
  • Text Augmentation: Other NLP methods mix existing text samples using statistical techniques and use pseudo-labeling to annotate unlabeled data with model predictions.These directions connect text mixing with semi-supervised learning practices.
  • Large-scale Language Models: Large-scale language models support prompt-based NLP because they are few-shot learners controllable through natural text.This capability motivates using prompts for synthetic text generation rather than only direct task solving.
  • Large-scale Language Models: The paper presents prompt-based generation of synthetic samples from large-scale language models specifically for text augmentation as a first contribution.GPT3Mix applies this prompt-based paradigm to producing augmentation data.

3 GPT3Mix

GPT3Mix adapts mix-based augmentation to text by prompting large-scale language models with selected labeled examples and task metadata. It generates synthetic text while using language-model label-token probabilities as soft labels for downstream training.

  • Motivation: GPT3Mix generates fluent synthetic text samples influenced by existing data distributions, adapting mix-based augmentation to text.The method addresses the difficulty of interpolating language from latent spaces by using large-scale language models for controlled generation.
  • Method: The augmentation pipeline selects examples, constructs a GPT3Mix prompt, and extracts augmentation from the language-model generation.Prompt construction includes a description header, selected text-label pairs, an augmentation prefix, and task-specific information.
  • Method: GPT3Mix specifies each task with text type, label type, and a one-to-one mapping from label classes to language-model vocabulary tokens.The task specification is S = (T, L, v), and the generic default is Sgeneric = (text, label, I).
  • Method: The language model jointly generates augmented text x′ and label y′, with the label token produced after the text to associate the output with its label.The prompt function P constructs the prompt from the task specification, while pLM denotes language-model likelihood.
  • Method: GPT3Mix combines text perturbation, pseudo-labeling, and knowledge distillation by training on augmented samples with soft labels alongside real samples.Soft-label probabilities are obtained by normalizing the likelihoods of generated label tokens and used with cross-entropy loss.

4 Experiments

Experiments evaluate GPT3Mix across data-scarce and full-data classification settings, including controlled RT20 tests, ablations, and comparisons with established augmentation methods. GPT3Mix generally provides consistent gains, with larger benefits for more expressive classifiers and language models.

  • Data-scarce experiments: BERTlarge matches fully trained BERTbase performance using just 1.0% of the original SST-2 training data with GPT3Mix augmentation.BERTlarge has 335M parameters, enabling it to exploit the synthetic samples more effectively.
  • Data-scarce experiments: GPT3Mix improves average classification accuracy across data-scarce tasks by as much as 18.6% for BERTbase, while other methods show nearly no improvement.It also provides more stable performance across repeated trials and tasks, although EDA and back-translation outperform it in some configurations.
  • Data-scarce experiments: BERTbase accuracy increases from 67.4 to 69.2 with GPT3Mix despite DistilBERTbase and BERTbase having similar non-augmented baselines.The paper attributes the larger effect to BERTbase having 65% more parameters and better utilizing GPT3Mix samples.
  • Ablation studies: Ablations show that larger augmenting language models provide greater data-augmentation benefits in data-limited settings.The ablation results use SST-2, while the main ablation experiments use DistilBERTbase and may underperform because they use smaller models and lower augmentation ratios.
  • Full-dataset experiments: GPT3Mix improves full-dataset SST-2 accuracy from 90.28% to 90.70% for DistilBERTbase and from 90.33% to 93.25% for BERTbase.The corresponding improvements are 0.42% and 2.92%, respectively.

4.4 Ablation Studies

The ablations examine how prompt composition, language-model capacity, task specification, and label choice affect GPT3Mix augmentation. Results show benefits from mixing examples, larger models, task-specific prompts, and soft labels, with trade-offs in inference cost and prompt overfitting.

  • Number of Prompt Examples: GPT3Mix requires at least two prompt examples to effectively mix existing samples and generate interpolated text samples.Using one example can instead produce perturbations or paraphrases.
  • Number of Prompt Examples: Under severely limited data availability, mixing at least two examples improves performance more than point-wise perturbation.The comparison concerns the 0.1% and 0.3% data-availability settings.
  • Number of Prompt Examples: Increasing prompt examples provides marginal benefits with more data but raises GPT-3 inference costs and can cause overfitting.The authors use k = 2 as a balance between inference cost and performance gains.
  • Language Model Capacity: Larger and more expressive augmenting language models benefit data augmentation.The capacity comparison uses GPT-3 models ordered from ada through davinci.
  • Language Model Capacity: Open-source GPT-like models, including GPT-2 and GPT-neo, provide comparable performance gains, supporting flexibility in the augmentation model choice.Even the smaller GPT-2 model provides performance gains.
  • Task Specification: Task-specific prompt specifications generate better samples, while even generic specifications outperform other augmentation baselines.The study compares optimal task specifications with generic specifications lacking task-specific meta-information.
  • Pseudo-Labels: Soft labels outperform sequence-optimized hard labels, indicating a strong advantage for retaining the language model's class distribution.The comparison is reported on SST-2.

5 Conclusion

The paper presents GPT3Mix as prompt-controlled text augmentation using large-scale language models, and reports improved pretrained-transformer robustness without online inference on heavy language models.

  • Conclusion: GPT3Mix uses large-scale language models and prompt-based controlled generation to augment text.The method is presented as a novel text augmentation technique.
  • Conclusion: The method improves pretrained-transformer robustness through mix-based perturbation and knowledge distillation without online inference on heavy language models.The authors position it as an alternative to prompt-based task-solving or direct fine-tuning.

6 Ethical Considerations

GPT3Mix can propagate biases and toxicity from pretrained language models, including when source examples are unbiased, and iterative use can amplify linguistic bias.

  • Risks: GPT3Mix may propagate linguistic biases and toxicity present in pretrained language models, even from unbiased real training examples.The concern arises because pretrained models are trained on untreated corpora and may degenerate during generation.
  • Risks: Iterative applications of GPT3Mix can amplify linguistic bias when augmented samples become source examples for later iterations.This limitation concerns repeated augmentation rather than a single augmentation pass.
  • Mitigations: Debiased models, decoding strategies, and temporary use for discriminator augmentation are proposed to reduce ethical concerns.For general-purpose dataset population, the authors recommend mitigating bias through existing debiasing work.

A Prompts

GPT3Mix constructs prompts from a task specification, label vocabulary, and labeled examples, then uses the prompt to generate augmented text.

  • Prompt Template: The prompt-constructing function P uses a task specification ST = (T, L, v).The template is illustrated for a sentiment-classification task.
  • Prompt Template: Each prompt lists examples containing a text type, example text, and corresponding label type.The label type is restricted to one of the specified label tokens.
  • SST-2 Example: For SST-2, the task specification identifies movie reviews and sentiment labels, with positive and negative as the allowed labels.The example prompt includes movie-review demonstrations paired with sentiment annotations.

B Task Specifications

GPT3Mix relies on task-specific prompt specifications, whose quality can materially affect augmentation and downstream performance.

  • Table 9 reports the optimal task specifications selected after validating candidate specifications for each task.These specifications are used for the experiments.
  • 38.8% downstream performance results when COLA uses “linguistic acceptability” instead of the optimal “grammar” label type on the 0.1% sub-dataset.This is below the non-augmented baseline of 68.80%.
  • 68.80% is the non-augmented COLA baseline exceeded by the incorrectly specified augmentation setting.

C RT20 Dataset

RT20 is a newly collected binary sentiment benchmark designed to test GPT3Mix without contamination from GPT-3’s training data.

  • RT20 is a binary sentiment dataset of movie reviews posted for movies released in 2020 or thereafter.
  • RT20 excludes the GPT-3 training dataset, eliminating performance improvements attributable to memorization.
  • The dataset uses critic reviews from Rotten Tomatoes, with fresh and rotten reviews sampled at a 1:1 ratio and labeled positive and negative.Critic reviews were selected partly to control data quality because they generally have higher linguistic acceptability than user reviews.

D GPT3Mix Samples

GPT3Mix generates samples that combine patterns from prompt examples while producing soft-label confidence estimates across several classification tasks.

  • SST-2: GPT3Mix generates synthetic text influenced by multiple example sentences, including their linguistic patterns and class-label probabilities.On SST-2, examples are generated with GPT-3 davinci.
  • SST-2: 75% positive and 25% negative are GPT3Mix’s predicted label probabilities for “Groundbreaking, disturbing.”
  • SST-2: 98% positive and 2% negative are GPT3Mix’s predicted label probabilities for a generated “Gosford Park” review.
  • RT20: On RT20, GPT3Mix mixes linguistic knowledge, diverse perturbations, and surface patterns without needing factual knowledge of newer films.The passage uses Tenet as an example of a movie GPT-3 would not have learned about.
  • SUBJ: GPT3Mix can generate low-confidence labels on SUBJ because subjectivity is vague and debatable.The generated example assigns 59% subjective and 41% objective probabilities.
Loading 2104.08826v2…