Source-linked AI summary
AugGPT: Leveraging ChatGPT for Text Data Augmentation
Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, Xiang Li
TL;DR
Limited and lower-quality data constrain few-shot NLP classification, while existing augmentation methods may sacrifice label faithfulness or sample diversity. AugGPT uses ChatGPT to generate semantically consistent but distinct auxiliary samples for downstream training. Experiments report superior few-shot classification performance and augmented-sample distributions, while medical-text augmentation remains vulnerable to incorrect outputs from insufficient domain knowledge.
Problem
Few-shot NLP classification is constrained by scarce, lower-quality data, while existing augmentation methods may lack label faithfulness, sample diversity, or both.
Method
AugGPT uses ChatGPT to rephrase each training sentence into multiple conceptually similar but semantically different auxiliary samples for few-shot text classification.
Results
AugGPT achieves superior testing accuracy and augmented-sample distributions over existing text data augmentation methods on few-shot text classification tasks.
Takeaways & Limitations
Semantic-level augmentation can improve downstream few-shot classification while increasing sample diversity and maintaining semantic similarity to original labels.
Takeaways & Limitations
AugGPT may produce incorrect medical-text augmentations because ChatGPT lacks sufficient domain knowledge.
Abstract
from arXiv · showhide
Text data augmentation is an effective strategy for overcoming the challenge of limited sample sizes in many natural language processing (NLP) tasks. This challenge is especially prominent in the few-shot learning scenario, where the data in the target domain is generally much scarcer and of lowered quality. A natural and widely-used strategy to mitigate such challenges is to perform data augmentation to better capture the data invariance and increase the sample size. However, current text data augmentation methods either can't ensure the correct labeling of the generated data (lacking faithfulness) or can't ensure sufficient diversity in the generated data (lacking compactness), or both. Inspired by the recent success of large language models, especially the development of ChatGPT, which demonstrated improved language comprehension abilities, in this work, we propose a text data augmentation approach based on ChatGPT (named AugGPT). AugGPT rephrases each sentence in the training samples into multiple conceptually similar but semantically different samples. The augmented samples can then be used in downstream model training. Experiment results on few-shot learning text classification tasks show the superior performance of the proposed AugGPT approach over state-of-the-art text data augmentation methods in terms of testing accuracy and distribution of the augmented samples.
1 INTRODUCTION
NLP models are difficult to train and generalize when data are scarce, particularly in few-shot learning. AugGPT addresses this challenge by using ChatGPT to generate auxiliary samples that improve diversity while preserving semantic consistency.
- Motivation: Limited training-data quality and quantity make it challenging to train NLP models that generalize well to unseen samples.The shortage commonly arises from privacy concerns or annotation costs.
- Few-shot learning: Few-shot learning is especially constrained because target-domain models must generalize from only a few examples, while data limitations affect both source and target domains.Existing approaches improve architectures or training procedures, but remain intrinsically limited by data quality and quantity.
- Data augmentation: Text data augmentation offers a model-agnostic way to overcome sample-size limits without changing the underlying model architecture.This makes augmentation practical across a wide range of NLP tasks.
- AugGPT: AugGPT uses ChatGPT to generate auxiliary samples for few-shot text classification across general-domain and medical datasets.The approach is motivated by large language models’ text-generation capabilities.
- Reported outcome: AugGPT improves sentence-classification accuracy by double digits while generating more diverse samples that maintain semantic similarity to the original labels.The paper evaluates both the faithfulness and compactness of generated samples.
2 RELATED WORK
Related work spans character-, word-, sentence-, and document-level augmentation, as well as few-shot learning and pretrained language models. These methods seek diverse, semantically consistent samples, but face semantic, training, accessibility, and domain-specific limitations.
- Data augmentation: NLP augmentation methods operate at character, word, sentence, and document granularity, generally aiming to produce sensible, diverse samples that preserve semantic consistency.Character-level methods include edits and OCR-error simulation; sentence-level methods include back translation, while document-level methods paraphrase whole documents.
- Data augmentation: Word-level methods use swaps, deletions, synonyms, embeddings, or masked language models to generate new text while attempting to preserve meaning or grammatical consistency.Contextual augmentation predicts masked words from surrounding context.
- Data augmentation: Embedding-based augmentation can substitute semantically opposite words because nearby embedding vectors are not necessarily semantically similar.The example contrasts “hot” and “cold,” whose embeddings may be close despite opposite meanings.
- Few-shot learning: Few-shot learning leverages prior knowledge to generalize to new tasks with only a few labeled samples, including settings where supervision is difficult because of privacy, safety, or ethical concerns.Data augmentation can complement model-based few-shot methods.
- Few-shot learning: Prompt-tuning and meta-learning face cumbersome prompt engineering, training instability, hyperparameter sensitivity, and substantial expertise requirements.These limitations motivate practical augmentation-based alternatives such as AugGPT.
- Language models: Pretrained language models include autoregressive, masked, and encoder-decoder architectures, while very large language models extend transformer-based representation learning to much larger scales.Representations can remain context-dependent and domain-dependent, creating adaptation requirements for smaller models.
- ChatGPT: ChatGPT has been studied as a data-augmentation model because pretrained language models can generate samples with similar semantic meaning.ChatGPT is based on GPT-3 and additionally trained with reinforcement learning from human feedback.
- ChatGPT: ChatGPT performs poorly on low-resource languages and faces additional challenges with distant-language translation.It performs better at understanding non-Latin scripts than generating them, according to the cited study.
3 DATASET
The experiments use an open-domain Amazon dataset and clinical or biomedical text classification benchmarks, including a small symptom dataset and PubMed20K.
- Datasets: Experiments evaluate AugGPT on Amazon reviews, two clinical NLP benchmarks, and PubMed20K sentence classification.The Amazon task classifies reviews by product category; PubMed20K classifies sentences into five abstract-section categories.
- Amazon: Amazon contains reviews from 24 product categories, with 300 sampled examples per category.
- Clinical dataset: The clinical symptom dataset contains 231 deduplicated transcript samples across seven symptom categories.Each example is a sentence describing symptoms, classified into its corresponding category.
- PubMed20K: PubMed20K contains around 20,000 annotated biomedical abstracts segmented into sentences labeled background, objective, method, result, or conclusion.
4 METHOD
AugGPT uses ChatGPT to expand few-shot samples before BERT training, combining pretrained language modeling, human-feedback alignment, and sentence rephrasing.
- Problem setup: Few-shot learning uses a larger labeled base dataset and a novel dataset with only a few labeled samples, with evaluation on the novel dataset.
- Framework: The framework fine-tunes BERT on base data, generates augmented novel-domain data with ChatGPT, then fine-tunes BERT using the augmented data.
- ChatGPT: ChatGPT is described as an autoregressive language model with transformer decoder blocks as its backbone.
- RLHF: RLHF aligns ChatGPT with user intent through supervised fine-tuning, reward modeling, and reinforcement learning from human feedback.
- Motivation: The authors motivate ChatGPT augmentation through its broad semantic expression space, human-like language, and reinforcement-learning-based quality assessment.
- ChatGPT augmentation: AugGPT rephrases each input sentence into six additional sentences to augment few-shot samples.
4.3 Few-shot Text Classification
The few-shot classifier uses BERT representations, cross-entropy classification, and contrastive learning to exploit base-domain knowledge while separating novel-task classes.
- Classification: BERT feeds the CLS representation into a task-specific classifier for final text-classification prediction.
- Few-shot challenge: Few-shot BERT fine-tuning can overfit because the small sample size limits generalization ability.
- Related approaches: Few-shot text-classification approaches are grouped into meta-learning, prompt-tuning, model design, and data augmentation.
- Classification: The classifier output dimension equals the union of the base and novel label spaces.
- Contrastive learning: The contrastive loss makes representations of same-category samples more compact and representations of different-category samples more separate.
- Objective: Few-shot BERT fine-tuning combines cross entropy with contrastive learning loss, whereas base-dataset fine-tuning uses cross entropy alone.
4.4 Baseline Methods
The evaluation compares AugGPT with character-, word-, and context-based augmentation methods implemented through established open-source libraries.
- Evaluation setup: Baseline implementations are drawn from the open-source nlpaug and textattack libraries.
- Character methods: Character-level baselines randomly insert, replace, exchange, or delete characters, including OCR, spelling, keyboard, and swap operations.
- Character methods: Keyboard augmentation simulates typos by replacing selected characters with adjacent keys on the QWERTY keyboard.
- Word methods: Word-level baselines include random deletion, word swapping, synonym replacement, embedding-based substitution, and random insertion.
- Contextual methods: Contextual baselines use BERT, DistilBERT, or RoBERTa to insert or substitute tokens predicted from masked contexts.
- Sentence methods: Back translation converts text to German and back to English to produce semantically equivalent but different text.
4.5 Prompt Design
The study uses separate single-turn and multi-turn dialogue prompts for AugGPT data augmentation, assigning prompt type by dataset.
- Amazon uses the multi-turn dialogue prompt, while Symptoms and PubMed20K use the single-turn dialogue prompt.
4.6 Evaluation Metrics
The evaluation measures augmented-data faithfulness with cosine similarity and compactness or learnability with TransRate. These metrics are computed from embeddings and labels to characterize similarity and class discrimination.
- Embedding Similarity: Cosine similarity assesses whether generated samples are close to actual samples in embedding space.The study uses embedding similarity between generated samples and actual test-dataset samples.
- Embedding Similarity: Higher cosine similarity indicates greater similarity between the compared embedding vectors.The compared vectors are denoted A and B.
- TransRate: TransRate quantifies transferability using mutual information between pretrained-model features and labels in the target data.It is computed with a single pass through the target data.
- TransRate: Higher TransRate could indicate better learnability, whereas its minimum occurs when class covariance matrices are identical and classes cannot be distinguished.
- TransRate: In the TransRate formulation, Y denotes augmented-example labels, Z denotes extracted embedding features, and H denotes Shannon entropy.
4.7 Direct Classification Performance by ChatGPT
The paper investigates direct ChatGPT classification with tailored prompts and few-shot examples. For Symptoms, the prompt maps descriptions to illness categories and includes two examples for downstream adaptation.
- The authors developed tailored prompts for ChatGPT to perform downstream classification through the API.
- The Symptoms prompt asks ChatGPT to predict an illness category from a person’s health description or symptom.
- Each Symptoms example combines a description with a typical corresponding class, and two few-shot examples are included.
5 EXPERIMENT RESULTS
Experiments use BERT in 2-shot classification with six augmented samples per class, comparing vanilla and contrastive-loss settings. AugGPT achieves the highest reported accuracies and favorable latent-space evaluations across datasets.
- Experimental Setup: The experiments use BERT with two real target-domain samples per class and six augmented samples per class.Models are fine-tuned on the few-shot and augmented samples.
- Experimental Setup: The evaluation compares vanilla BERT with BERT trained using an added contrastive loss.
- Classification Performance Comparison: AugGPT achieves the highest accuracy on Amazon, Symptoms, and PubMed20K among the compared augmentation methods.
- Classification Performance Comparison: 88.9% and 89.9% accuracy are achieved by AugGPT on Symptoms with vanilla BERT and contrastive loss, respectively.Without data augmentation, the corresponding accuracies are 63.6% and 60.6%.
- Classification Performance Comparison: 83.5% accuracy is achieved by AugGPT on PubMed20K with both vanilla BERT and contrastive loss.Without data augmentation, the corresponding accuracies are 79.2% and 79.8%.
- Evaluation of Augmented Datasets: AugGPT produces high-quality samples for both faithfulness and compactness on Symptoms and PubMed20K, and higher TransRate on Amazon.
6 CONCLUSION AND DISCUSSION
AugGPT improves few-shot classification by expanding limited data at the semantic level, while its broader applicability includes summarization and computer vision. The approach remains constrained in medical settings by ChatGPT’s lack of domain knowledge.
- Conclusion: AugGPT expands limited few-shot classification data at the semantic level to enhance consistency and robustness.
- Limitations: Medical-text augmentation may produce incorrect results because ChatGPT lacks sufficient domain knowledge.
- Future Work: AugGPT could support domain-specific science-paper and clinical-report summarization by generating diverse samples in different representation styles.
- Future Work: AugGPT may extend to computer vision by using language descriptions or existing images to guide generative-model augmentation in few-shot tasks.
- Future Work: Connections between LLMs and human cognition, including Theory of Mind capabilities, offer another direction for improving LLM design.