PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations

Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, Ming Yin

arXiv:2310.07849v2cs.CLcs.AI

TL;DR

High-quality text-classification training data are costly, motivating alternatives based on LLM-generated synthetic data. The paper evaluates zero-shot and few-shot generation while examining task- and instance-level subjectivity. It finds that models trained on synthetic data perform worse as subjectivity increases.

  • Problem

    Training-data collection and curation are costly, while the effectiveness of LLM-generated synthetic data varies across classification tasks.

  • Method

    The study evaluates GPT-3.5-Turbo synthetic-data generation under zero-shot and few-shot settings and relates model performance to task- and instance-level subjectivity.

  • Results

    Performance of models trained on synthetic data decreases for classification tasks and instances with higher subjectivity.

  • Takeaways & Limitations

    Real-world examples can increase synthetic-data diversity and boost resulting model performance, while synthetic-data model performance may help approximate subjectivity or label reliability.

  • Takeaways & Limitations

    The evaluation uses GPT-3.5-Turbo only, so the conclusions may not generalize to other LLMs.

Abstract

from arXiv · show

The collection and curation of high-quality training data is crucial for developing text classification models with superior performance, but it is often associated with significant costs and time investment. Researchers have recently explored using large language models (LLMs) to generate synthetic datasets as an alternative approach. However, the effectiveness of the LLM-generated synthetic data in supporting model training is inconsistent across different classification tasks. To better understand factors that moderate the effectiveness of the LLM-generated synthetic data, in this study, we look into how the performance of models trained on these synthetic data may vary with the subjectivity of classification. Our results indicate that subjectivity, at both the task level and instance level, is negatively associated with the performance of the model trained on synthetic data. We conclude by discussing the implications of our work on the potential and limitations of leveraging LLM for synthetic data generation.

1 Introduction

High-quality training data is essential but costly to collect and curate, motivating LLM-based synthetic data generation. This study evaluates whether classification subjectivity moderates the usefulness of such data.

  • Training-data collection and curation are costly, time-consuming, and complex, especially for novel domains or classification categories.
  • LLM-based data augmentation often improves model performance, but studies report mixed results for generating synthetic datasets from scratch.
  • The study investigates whether classification subjectivity helps explain variation in models trained on LLM-generated synthetic data.
  • The authors evaluate GPT-3.5-Turbo synthetic-data generation using both zero-shot and few-shot prompting settings.
  • The study provides experimental evidence about the potential and limitations of LLM-generated synthetic data for text classification.

2 Related Work

Prior work has used generative models and language models to create synthetic training data, but reported effectiveness remains mixed. This study focuses on task subjectivity as a moderator of synthetic-data usefulness.

  • Generative models have been used to create synthetic data for computer-vision recognition and segmentation tasks.
  • NLP studies have explored language-model-generated synthetic data for various text-classification tasks, with mixed effectiveness results.
  • The paper aims to clarify when language-model-generated synthetic data supports effective model training.
  • The study examines the role of task subjectivity in moderating synthetic-data effectiveness.
  • The work situates GPT-style language models within Transformer-based advances in natural-language processing.
  • The evaluation uses GPT-3.5-Turbo as the foundational model for synthetic-data generation.

3 Methodolgy

The methodology generates synthetic text under zero-shot and few-shot settings using contextual and data-generation prompts. Few-shot generation additionally supplies real examples, while diversity prompts encourage variation.

  • The study compares zero-shot and few-shot settings for generating synthetic training data with an LLM.
  • Zero-shot generation assumes no real-world text-label pairs are available and uses sequential prompts to produce labeled task-specific text.
  • A customized context prompt establishes the target domain, while a data-generation prompt specifies style, label, and word limit.
  • A diversity prompt is supplied after every n generated data points to encourage synthetic examples that differ from earlier outputs.
  • Few-shot generation provides randomly sampled real-world text-label examples before generating each synthetic text.
  • Prompt details for each classification task are provided in the appendix.

4 Evaluation I: Comparison Across Different Types of Tasks

Across ten text-classification datasets, models trained on real-world data generally outperform those trained on synthetic data, although few-shot prompting narrows the gap. Synthetic data is most effective for less subjective tasks, while highly subjective tasks show larger declines.

  • Evaluation setup: The first evaluation compares real-world, zero-shot synthetic, and few-shot synthetic training across 10 representative text-classification datasets.
  • Evaluation setup: The datasets span tasks including news classification, sentiment, spam detection, emotion, relations, irony, sarcasm, and humor.
  • Subjectivity measurement: Task subjectivity is estimated through a crowdsourced comparative study and ranked using a directed-graph procedure.
  • Performance comparison: Models trained on real-world data consistently outperform synthetic-data counterparts in Macro-F1 and accuracy across almost every task.
  • Performance comparison: Few-shot synthetic data almost always outperforms zero-shot synthetic data, indicating that real examples improve synthetic-data training effectiveness.
  • Performance comparison: For the BERT model, few-shot generation improves average performance over zero-shot generation by 10.6% in Macro-F1 and 8.8% in accuracy across 10 tasks.
  • Subjectivity effects: Synthetic-data models perform relatively close to real-world-data models on lower-subjectivity tasks such as AG’s news, relation classification, IMDB, and SMS spam.
  • Subjectivity effects: For the six most subjective tasks, BERT models trained on zero-shot synthetic data show average decreases of 27.4% in Macro-F1 and 24.2% in accuracy versus real-world-data models.

5 Evaluation II: Comparison Across Different Task Instances

The evaluation examines whether models trained on zero-shot LLM-generated synthetic data perform differently on task instances with varying subjectivity. Across most tasks, accuracy increases as instance-level agreement rises, while this relationship is weaker for models trained on real-world data and absent for two datasets.

  • 5.1 Instance-level Subjectivity Determination: Instance subjectivity is estimated from the fraction of annotators agreeing with the majority label, with lower agreement indicating greater subjectivity.Each instance received at least three annotations from unique workers, and the agreement measure is compared with task-level annotation agreement.
  • 5.1 Instance-level Subjectivity Determination: The instance-level agreement measure closely aligns with Krippendorff’s α and can serve as a reasonable proxy for instance subjectivity.Tasks with higher subjectivity generally exhibit higher average agreement values in the reported evaluation datasets.
  • 5.2 Evaluation Results: For most tasks, model accuracy has a strong monotonically increasing relationship with the agreement threshold γ, with Spearman’s ρ often exceeding 0.85.Raising γ filters out more highly subjective instances, so the observed pattern indicates better performance on less subjective instances.
  • 5.2 Evaluation Results: Models trained on zero-shot synthetic data perform better on instances with lower subjectivity, even within the same task type.The analysis trains BERT on zero-shot synthetic data and evaluates accuracy on subsets exceeding successive agreement thresholds.
  • 5.2 Evaluation Results: Models trained on real-world data show a similar but usually weaker relationship between decreasing instance subjectivity and increasing performance.The reported comparison describes smaller β and ρ values for real-world-data models than for synthetic-data models.

6 Conclusions and Discussions

The paper finds that higher subjectivity is associated with lower performance when models are trained on LLM-generated synthetic data, while several task-specific exceptions and limitations qualify this pattern.

  • Higher task-level and instance-level subjectivity is associated with lower performance for models trained on LLM-generated synthetic data.
  • Highly subjective tasks may expose limited synthetic-data diversity because LLMs struggle to capture nuanced emotions, contextual subtleties, and varied perspectives.The authors suggest increasing diversity so synthetic data better reflects real-world distributions.
  • Gold labels based on annotator majority votes may fail to represent individual perspectives, making highly subjective instances difficult for LLMs to reproduce faithfully.The paper questions whether a single majority label is appropriate when humans cannot reach consensus.
  • 6.2 Explaining a few exceptions: On Tweet irony detection, few-shot synthetic-data models outperform models trained on real-world data, plausibly because irony generation permits creative writing with few formality constraints.
  • 6.2 Explaining a few exceptions: For Financial Phrasebank and Sarcasm, instance-level subjectivity has little effect, while zero-shot synthetic-data models perform very poorly regardless of subjectivity.The authors relate these exceptions to specialized financial and political knowledge requirements.
  • 6.3 Limitations and future work: The conclusions may not generalize beyond GPT-3.5-Turbo, and subjectivity may not be the only moderator of synthetic-data effectiveness.The study also notes variability from relying on crowd workers to assess subjectivity.
  • 6.3 Limitations and future work: Incorporating real-world examples and human feedback or intervention may increase synthetic-data diversity and improve the resulting models.

A.1 Dataset and Task Descriptions

The appendix describes several text-classification datasets spanning news topics, entity relations, emotions, irony, and sarcasm, with task-specific label sets and dataset sizes.

  • Relation Classification identifies four relationships between entities: country, league, screenwriter, and tributary.The passage states that the dataset comprises English text, but the remainder is truncated.
  • Reddit Emotion classifies comments into joy, sadness, and surprise using a subset of Go Emotions containing 58,009 Reddit comments.
  • Tweet Irony Speech classifies English-language tweets as irony or non-irony, using 2,862 training instances and 784 test instances.
  • Tweet Emotion classifies English-language tweets into anger, joy, optimism, and sadness, with 3,257 training and 1,421 test instances.
  • Sarcasm News Headlines distinguishes sarcastic from non-sarcastic news headlines.The passage begins describing a dataset of 26,709 headlines but is truncated before further details.

B.1 Convergence Analysis

Synthetic-data-trained models converge faster than models trained on real-world data but show greater overfitting, suggesting limited diversity and weaker coverage of complex language patterns. Combining few-shot synthetic data with limited real-world examples can improve performance for many tasks.

  • Convergence and overfitting: Synthetic-data-trained models exhibit faster convergence and a greater propensity to overfit than models trained on real-world data.This pattern appears under both zero-shot and few-shot generation settings.
  • Convergence and overfitting: The synthetic data may lack diversity and fail to fully capture complex patterns in real-world language contexts.The convergence and overfitting behavior is interpreted as evidence for this limitation.
  • Few-shot augmentation: Models trained exclusively on limited real-world data versus few-shot synthetic data show task-dependent performance differences.The comparison uses the small real-world set provided as examples for guiding generation.
  • Few-shot augmentation: Combining few-shot synthetic data with a small amount of real-world data can outperform training only on the real-world data for many tasks.This result highlights the potential of few-shot synthetic data for data augmentation.

B.3 Similarity between the Synthetic Data and the Real Data

The study compares few-shot and zero-shot synthetic data with real-world examples using embedding-based cosine similarity. Few-shot synthetic data is generally more similar to real-world examples and tends to support better model performance than zero-shot synthetic data.

  • Similarity measurement: Cosine similarity is computed between embeddings of real-world examples and synthetic texts, using the mean similarity to each example’s top five synthetic matches.Texts are embedded with the all-MiniLM-L6-v2 Sentence Transformer, and similarity scores are rescaled to [0, 1].
  • Similarity comparison: Few-shot synthetic data consistently shows higher similarity to real-world examples than zero-shot synthetic data.The comparison is based on average top-five cosine similarity across classification tasks.
  • Similarity comparison: The similarity difference between few-shot and zero-shot synthetic data is statistically significant for every task except IMDB review classification.The significance was assessed with t-tests for each classification task.
  • Connection to performance: Greater similarity between few-shot synthetic data and real-world data may partly explain why few-shot-trained models tend to outperform zero-shot-trained models.The passage presents this as a partial explanation rather than a complete account.

B.5 Additional Results of More LLMs

Additional experiments show that the main pattern is stable across larger and decoder-based language models: synthetic-data effectiveness tends to be higher for less subjective tasks. Direct prompting can nevertheless underperform training a smaller classifier on generated synthetic data.

  • Additional LLMs: Across GPT2-large and Llama2 evaluations on six datasets, synthetic-data effectiveness remains higher for tasks with lower subjectivity.The overall trend shows only slight variations among different LLMs for each task.
  • Additional LLMs: Models trained on LLM-generated synthetic data exhibit only slight performance variations among different LLMs for each respective task.This result supports consistency of the overall subjectivity pattern across the tested LLMs.
  • Direct prompting comparison: For many tasks, directly prompting GPT-3.5 Turbo for classification yields poorer results than training a smaller model on zero-shot synthetic data.The zero-shot synthetic-data and real-data comparisons use BERT as the classification base model.
  • Direct prompting comparison: The discrepancy may arise because prompt constraints defining the label space can be too lax for accurate classification.This is presented as a possible explanation for the direct-prompting gap.

C Evaluation II: Comparison Across Different Task Instances (Additional Results)

For models trained on real-world data, performance is generally less sensitive to the subjectivity of individual task instances than for models trained on zero-shot synthetic data. Scarcasm News and Financial Phrasebank are exceptions.

  • Instance-level subjectivity: Real-world-data-trained models are less affected by instance-level subjectivity than zero-shot-synthetic-data-trained models.The relationship coefficients β and ρ are smaller for real-world-data-trained models, except on two datasets.
  • Instance-level subjectivity: Scarcasm News and Financial Phrasebank are exceptions to the weaker subjectivity relationship observed for real-world-data-trained models.The passage identifies these datasets as exceptions without specifying the direction of their deviation.

D Additional Details on the Generation of Synthetic Data

This section provides the prompts used to generate synthetic data under zero-shot and few-shot settings, alongside a figure examining how annotation-agreement thresholds relate to BERT accuracy on real-world data.

  • Synthetic-data generation prompts: The zero-shot and few-shot synthetic-data generation prompts are documented in Tables D.1 and D.2.Table D.2 continues the detailed task-specific prompts.
  • Instance-level agreement analysis: Figure C.1 shows changes in BERT accuracy as the instance-level annotation-agreement threshold varies.The figure concerns BERT models trained on real-world data.
  • Instance-level agreement analysis: The figure uses fitted linear regressions, R-squared scores, and Spearman’s ρ to characterize the relationship between agreement thresholds and model accuracy for each task.Higher R-squared and Spearman’s ρ values indicate stronger fit and rank correlation, respectively.
Loading 2310.07849v2…