PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Data Generation Using Large Language Models: Advances in Text and Code

Mihai Nadas, Laura Diosan, Andreea Tomescu

arXiv:2503.14023v2cs.CL

TL;DR

The survey addresses the lack of a comprehensive review of LLM-based synthetic data generation across natural language and code, especially as real labeled data remains costly, scarce, or sensitive. It synthesizes prompting, retrieval, self-refinement, and feedback-based methods, finding that synthetic augmentation is particularly useful in low-data settings while requiring safeguards for fidelity, bias, diversity, and reliability.

  • Problem

    High-quality labeled data is often scarce, expensive, or sensitive, while comprehensive reviews of LLM-based synthetic generation across text and code remain limited.

  • Method

    The survey organizes text and code techniques including prompt-based generation, retrieval-augmented methods, self-instruction, iterative refinement, and feedback-based validation.

  • Results

    Synthetic augmentation improves low-resource text classification accuracy or F1 by 3–26% in reported studies, while benefits diminish as real-data quantity increases.

  • Takeaways & Limitations

    LLM-generated data can provide scalable, controllable support for text and code training, particularly when data is limited, if outputs are filtered, weighted, or validated.

  • Takeaways & Limitations

    Simple or repeated synthetic generation can limit diversity, and synthetic-data workflows must address fidelity, factuality, bias, distributional drift, and model collapse.

Abstract

from arXiv · show

This survey reviews how large language models (LLMs) are transforming synthetic training data generation in both natural language and code domains. By producing artificial but task-relevant examples, these models can significantly augment or even substitute for real-world datasets, particularly in scenarios where labeled data is scarce, expensive, or sensitive. This paper surveys recent advances in leveraging LLMs to create synthetic text and code, highlighting key techniques such as prompt-based generation, retrieval-augmented pipelines, and iterative self-refinement. We examine how these methods can enrich low-resource tasks (e.g., classification, question answering) and facilitate code-centric applications (e.g., instruction tuning, code translation, bug repair) through automated verification of functional correctness. Alongside potential benefits - cost-effectiveness, broad coverage, and controllable diversity - we discuss the accompanying challenges, including factual inaccuracies in generated text, insufficient stylistic or distributional realism, and risks of bias amplification. Proposed mitigation strategies range from filtering and weighting synthetic outputs to reinforcement learning with execution feedback in code domains. We conclude by outlining open research directions, such as automated prompt engineering, cross-modal data synthesis, and robust evaluation frameworks, underscoring the growing importance of LLM-generated synthetic data in accelerating AI development while emphasizing ethical and quality safeguards.

1 Introduction

This survey addresses the growing use of LLMs to generate synthetic text and code when real training data is scarce, costly, or constrained. It organizes techniques, applications, benefits, challenges, and future directions across both domains.

  • Motivation: LLM-generated synthetic data can augment training for text and code when real data is scarce, expensive, or privacy-constrained.The survey frames synthetic generation as a response to limitations in acquiring high-quality supervised corpora.
  • Survey scope: The survey reviews prompt-based augmentation, retrieval-augmented generation, self-instruction, and reinforcement learning with feedback.Methods are categorized by prompting strategy, knowledge integration, and iterative refinement.
  • Text applications: Synthetic text generation supports classification, question answering, and instruction-following, with reported low-resource improvements of 3–26%.Prompt design and data curation are examined alongside effects on performance, diversity, and efficiency.
  • Code applications: Synthetic code generation supports code intelligence through prompted examples, executable validation, and large-scale instruction-tuning data.Examples include programming problems, diverse code edits, and datasets such as Code Alpaca and WizardCoder.
  • Challenges and mitigations: Key challenges include factuality, bias, distributional drift, insufficient diversity, and model collapse from iterative synthetic-data training.Mitigations include filtering, weighting, mixing synthetic with real data, retrieval, and execution feedback.
  • Future directions: The survey proposes future work on rigorous evaluation, controllable generation, multimodal and low-resource settings, and safe privacy-preserving applications.Its taxonomy distinguishes text and code tasks such as generation, translation, instruction tuning, and repair.

2 Related Work

Prior surveys cover NLP synthesis, general synthetic-data pipelines, or synthetic code separately, leaving limited systematic comparison across text and code. This survey unifies the modalities through a shared taxonomy, comparative analysis, and practical research agenda.

  • Prior survey coverage: Existing reviews emphasize either NLP applications, broad lifecycle frameworks, or code generation rather than systematically comparing text and code.The survey positions its contribution against these separate emphases.
  • Positioning: The survey bridges natural language and programming code through a unified treatment of LLM-driven synthetic-data methods.Text and code are treated as related symbolic modalities that can share fundamental generation and processing methods.
  • Framework: Its cross-domain taxonomy includes prompt-based, retrieval-augmented, and iterative self-refinement pipelines.The framework organizes techniques across both modalities rather than restricting analysis to one task family.
  • Comparative analysis: The survey compares empirical results, quality assurance, distributional realism, bias, and model collapse across text and code.These issues are identified as topics often handled separately in earlier surveys.
  • Research agenda: It distills practical recommendations and proposes future directions involving evaluation, human-in-the-loop workflows, and ethical considerations.The agenda is intended for both NLP and code-generation communities.

3 Survey Methodology

The survey uses a semi-automated, PRISMA-style literature review covering recent LLM-based synthetic-data work in text and code. Searches, snowballing, screening, and explicit inclusion criteria support the selection process.

  • Review process: The literature review follows a PRISMA-style process to make study identification, screening, and selection transparent.Figure 1 summarizes the review flow.
  • Inclusion criteria: The review includes work published from January 2020 through April 2025 on LLM-based synthetic generation, augmentation, or instruction tuning for text or code.The time window targets the period of rapid advancement in LLM-driven synthetic data.
  • Model scope: The review covers large models with at least several hundred million parameters, including proprietary and open-source systems.Examples span GPT-3/4, Claude, Gemini, Llama, CodeLlama, and StarCoder.
  • Scope: Included studies address tasks such as classification, question answering, instruction following, code synthesis, translation, and bug repair.The survey also requires empirical evidence, systematic evaluation, or substantial methodological innovation.
  • Exclusions: Studies focused exclusively on non-text or non-code modalities and methodologically weak non-peer-reviewed materials are generally excluded.Exceptions apply when outside modalities provide transferable methodology or insights.
  • Search strategy: Researchers searched Google Scholar, arXiv, and Semantic Scholar using synthetic-data, augmentation, instruction-tuning, prompting, and task-specific terms.Reference lists from surveys and benchmark papers were also examined through snowballing.

4 Background and Motivation

Synthetic data is algorithmically generated to mimic real data and offers controllable, scalable alternatives to costly collection and annotation. LLM prompting expands this capability, but generated data still requires quality and reliability safeguards.

  • Synthetic data: Synthetic data is artificially generated by algorithms or generative models to mimic characteristics of real-world data.This definition distinguishes generation from direct collection or annotation.
  • Earlier augmentation: Traditional rule-based augmentation often provides limited diversity because it introduces few new linguistic patterns or semantic variations.These limitations motivated the use of more capable generative models.
  • LLM generation: LLMs can generate coherent, task-specific examples through zero-shot and few-shot prompts, making them flexible synthetic-data generators.Prompted generation can produce additional plausible examples for a target task.
  • Cost and scalability: LLMs offer a cheaper and faster alternative to manual labeling, while synthetic data can scale to thousands or millions of examples.The motivation includes addressing scarcity, annotation cost, class imbalance, and data-hungry training.
  • Controllability: Synthetic generation can target edge cases, counterfactuals, styles, reading levels, low-resource languages, and rare contexts.Prompt and generation criteria provide fine-grained control over dataset composition.
  • Cost-quality trade-off: For 3,000 SST-2 samples, GPT-3 generation sharply reduces time and monetary cost but produces slightly lower accuracy than human-curated data.The comparison captures the central trade-off between efficiency and outcome quality.
  • Prompting: Few-shot prompting uses multiple task examples to guide additional outputs toward the demonstrated structure and label space.It generally improves task specificity, though repeated prompting can constrain diversity.
  • Caveat: LLM-generated data remains subject to a reliability requirement: it must be evaluated and curated to be as effective and dependable as real data.The survey frames filtering and related safeguards as necessary for practical use.

5 Synthetic Data Generation for Text Tasks

LLM-based text synthesis spans prompt strategies and augmentation pipelines that target diversity, relevance, and low-resource performance. Its benefits depend on quality control, distribution alignment, and balancing synthetic examples with real data.

  • Prompt-based generation: Prompting strategies—including zero-shot, few-shot, and topic-controlled approaches—trade off the diversity and relevance of generated text.Topic-controlled or randomized prompts target broader subtopic coverage, while few-shot and instruction-based prompts support format precision and label alignment.
  • Empirical findings: In low-resource classification, adding 100 GPT-3.5-generated samples to 100 real samples improved accuracy or F1 by 3–26%.Benefits were strongest under severe underfitting and diminished as the amount of real data increased.
  • Empirical findings: Synthetic augmentation is most useful as a supplement in low-data regimes rather than a replacement for extensive real corpora.Studies reported negligible improvements when synthetic data was added to large real datasets.
  • Representative approaches: Representative methods use rephrasing, LLM annotation, mixing synthetic and real examples, challenging NLI generation, and synthetic query–passage pairs across NLP tasks.These approaches support classification, information retrieval, and other task-specific augmentation settings.
  • Quality control: Quality improvement relies on external knowledge or constraints, post-generation filtering, duplicate and leakage checks, format validation, and learned critics.These measures address incorrect outputs and improve the usefulness of generated datasets.
  • Quality and distribution: Synthetic data must remain diverse, relevant, correct, and aligned with the real task distribution to avoid out-of-distribution examples.Retrieval, iterative prompt refinement, and filtering are presented as ways to improve final dataset quality; some scenarios approach real-data effectiveness when small curated datasets are augmented.

6 Synthetic Data Generation for Code Tasks

Synthetic code generation supports instruction tuning, translation, refactoring, problem synthesis, and bug-related data creation. Code's executable semantics enable automated correctness checks, while coverage, stylistic bias, and representation remain important concerns.

  • Synthetic code tasks: LLM-generated code data supplements real corpora for instruction datasets, translation, refactoring, programming problems, and bug detection or repair.The surveyed forms target specific gaps and specialized code intelligence tasks.
  • Instruction tuning: Code Alpaca generated 20K coding instruction-following examples from seed prompts using Self-Instruct.WizardCoder increased task complexity through iterative evolutionary prompting, while Magicoder produced 75K instruction-output pairs from open-source code.
  • Translation and refactoring: Refactoring pipelines create functionally identical variants through variable renaming, formatting, and comments, improving stylistic diversity.A CodeLLaMA-7B study reported up to 30% better performance after fine-tuning on cleaned-up versions than on unprocessed datasets.
  • Verification: Execution signals provide a relatively definitive pass/fail oracle for retaining correct and relevant synthetic programs.This enables high-precision filtering in code data generation, unlike the often-fuzzy assessment of factual correctness in language tasks.
  • Quality and properties: Code synthetic data raises bias and diversity concerns through over-represented libraries, coding patterns, or styles inherited from LLM training data.Reported gains also include improved benchmark results for code generation and completion after synthetic instruction tuning, and higher text-to-SQL accuracy with validated generated pairs.
  • Coverage: Synthetic code can target the long tail of programming tasks, where real datasets contain few examples of niche problems.The surveyed literature spans more than 50 works across 23 code-synthesis topics.

7 Challenges and Considerations

LLM-generated synthetic data introduces risks involving correctness, realism, distribution alignment, and bias. The survey discusses retrieval, execution, filtering, weighting, and dataset balancing as mitigation strategies.

  • Correctness: Synthetic text can contain hallucinated facts, while synthetic code can contain invalid programs that teach models spurious patterns or degrade coding ability.The paper distinguishes factual correctness for text from functional correctness for code.
  • Mitigation: Retrieval can ground text generation, execution can validate code, and critics, perplexity, static analyzers, or linters can filter low-quality outputs.These checks target factuality, functional correctness, and obvious code errors.
  • Realism and distribution: Distribution shift occurs when synthetic data does not match real-data distributions, potentially causing overfitting to cleaner text or narrower coding styles.Examples include synthetic text lacking typos and slang and code overrepresenting particular constructs.
  • Mitigation: Mixing synthetic data with real data can anchor models in real distributions, while filtering, weighting, and adversarial distribution matching address remaining gaps.The paper presents real-data mixing as a common response to distribution mismatch.
  • Bias: Biases in the generating LLM may transfer into synthetic datasets and amplify through recursive reuse, requiring diverse prompts and post-hoc balancing.The cited examples include gender representation and political slants.

7.3 Evaluation of Synthetic Data and Models

Evaluating synthetic data requires measuring both data quality and downstream model performance against real-data references. The survey recommends comparisons, diversity and realism measures, human assessment, robustness testing, and statistically rigorous reporting.

  • Evaluation criteria: Downstream performance is assessed on real, human-annotated test sets using task metrics such as accuracy, F1, BLEU, ROUGE, and pass@k.For code, the survey cites HumanEval and related benchmarks.
  • Evaluation criteria: Controlled comparisons of real-only, synthetic-only, and mixed training data quantify the contribution of synthetic examples.These ablations isolate how synthetic data changes performance.
  • Evaluation criteria: Diversity, realism, distribution shift, and human judgments can be evaluated with Distinct-n, Self-BLEU, perplexity, classifier discrimination, manual inspection, and Likert ratings.Code evaluation additionally includes execution-based correctness and expert review.
  • Evaluation criteria: Robustness testing should include out-of-distribution, rare, and edge-case samples to detect overfitting to synthetic patterns.The survey frames these tests as checks of generalization.
  • Code evaluation: Code-specific evaluation can use HumanEval, CoderEval, MultiPLE, DS-1000, execution accuracy, logical-form accuracy, and detailed syntax or logic error analysis.The recommended measures cover both benchmark success and error types.
  • Reporting and rigor: Future evaluations should report confidence intervals, cross-dataset validation, formal statistical tests, real-data baselines, synthetic-data composition, and both benefits and risks.The survey especially emphasizes monitoring distribution shift and overfitting in closed-loop pipelines.

7.4 Overfitting and Distribution Shift Risks in Closed-Loop Synthetic Data Generation

Closed-loop synthetic-data generation can amplify errors, biases, and artifacts through recursive retraining, producing overfitting, distribution drift, and model collapse. Maintaining real data, filtering outputs, and monitoring generalization are presented as safeguards.

  • Risks: Recursive training on model-generated data can amplify errors, biases, and artifacts, degrading generalization and specializing models to synthetic patterns.The risk is strongest when real data is scarce or absent.
  • Risks: Synthetic-only multi-generation corpora can cause robustness loss, distributional drift, and spurious correlations absent from real-world data.The survey describes these as empirical outcomes of closed-loop generation.
  • Mitigation: Mixing real and synthetic data, filtering and validating examples, and using curriculum or adversarial training can reduce distribution drift and model-collapse risk.The paper states that preserving a core of real data and using augmentation judiciously can minimize catastrophic collapse.
  • Practice: Practitioners should conduct ablations and explicitly monitor overfitting and loss of generalization when deploying synthetic-data pipelines.These checks are recommended as operational safeguards.

7.5 Scale and Cost Trade-offs

Synthetic data can reduce labeling costs but still requires computational time, model expenditure, quality control, and reproducibility measures. The section also highlights privacy risks and model-collapse concerns when generated data is reused recursively.

  • Cost and scale: Generating large synthetic datasets is cheaper than manual labeling but still incurs API costs and time, with uncertain returns from additional examples.More generation can increase coverage while also increasing redundancy and eventually producing diminishing gains.
  • Cost and scale: Choosing between larger and cheaper generators creates a quality-versus-quantity trade-off that can be addressed by generating broadly and filtering with stronger models or humans.The survey notes that larger models may produce more nuanced data with fewer mistakes.
  • Reproducibility: Reproducibility is difficult because repeated prompting can produce different datasets, motivating standardized generation tools such as DataDreamer.The stated goal is consistent experimentation and repeated results.
  • Ethical and legal risks: Synthetic outputs may reproduce bias, hate speech, private information, or copyrighted material from model training data, so datasets require leakage and sensitive-content checks.Synthetic data is not automatically safe merely because it is artificial.
  • Closed-loop risks: Replacing real data entirely with successive generations of synthetic data can reduce performance, whereas retaining real data while adding synthetic examples can maintain or improve it.The survey presents real-data retention as a way to avoid the degenerative feedback loop.

7.7 Task-Specific Nuances

Synthetic data challenges vary by task, affecting realism, diversity, niche-domain coverage, and trust in high-stakes applications. Addressing these issues requires task-sensitive evaluation and transparent validation.

  • Dialogue: Synthetic dialogue may miss topic shifts, misunderstandings, multi-turn coherence, and realistic user difficulty.User simulators can become overly cooperative when an LLM generates both sides of a conversation.
  • Summarization: Summarization requires diverse outputs that emphasize different interpretations or focus points rather than producing many similar summaries.The goal is to expose models to ambiguity or preference beyond a small set of reference summaries.
  • Code: Niche code domains, including cybersecurity and esoteric languages, may require specialized strategies such as transpiling from related languages.LLMs may have limited knowledge of very niche programming languages.
  • Trust and deployment: Practitioner trust depends on transparency about synthetic-data creation and evaluation, especially when high-stakes use requires evidence that outcomes do not degrade.Medical and legal applications may additionally require regulatory approval.
  • Overall: Synthetic data generation remains an active research area because task-specific challenges require careful application and continuing mitigation work.The paper frames future research as addressing these challenges while extending synthetic-data capabilities.

8 Future Directions

Future work centers on organizing the field, automating generation control, specializing models, and evaluating synthetic data through realistic benchmarks. These directions also include human-guided and domain-specific approaches for difficult settings.

  • Frameworks and taxonomies: Unified frameworks and taxonomies could organize pipelines from prompt design through generation, filtering, integration, and evaluation.Standardized workflows would help researchers compare methods and identify gaps.
  • Prompting and active learning: Automated prompt engineering could optimize prompts for synthetic-data utility while reducing manual trial-and-error.Candidate approaches include learnable prompts and evolutionary prompt search.
  • Prompting and active learning: Active-learning loops could ask LLMs to generate examples in regions of model uncertainty, potentially with human vetting.This connects generation to the needs identified during model training.
  • Specialized generators: Domain-specific LLMs may produce more realistic synthetic data for fields such as medicine and law than general-purpose models.Such specialization targets domains where real data is especially difficult to obtain.
  • Benchmarking: Synthetic-data benchmarks could compare generation strategies by measuring downstream performance on hidden test sets.Proposed competitions include challenges for both general datasets and code generation.
  • Overview: Table 6 groups future directions by central themes and representative subtopics.The themes include evaluation, ethics, prompting, multimodality, human-in-the-loop learning, domain-specific generators, and benchmarking.

9 Conclusion

The survey presents LLM-based synthetic data as a promising way to address scarcity in text and code, with gains in low-resource tasks and correctness-filtered code data. It also emphasizes that quality, distribution, bias, evaluation, and feedback-loop risks require safeguards and continued research.

  • Overall contribution: LLM-based synthetic data addresses scarcity and can enhance training in both natural-language and programming-language domains.The survey describes this as a promising paradigm for obtaining training data.
  • Text data: Synthetic text data is especially useful in low-resource classification and question-answering settings with limited human-labeled data.Prompt-based augmentation, retrieval augmentation, and iterative refinement are highlighted as important techniques.
  • Code data: Execution-based verification enables large-scale synthetic code datasets containing functionally correct programs.The survey links this capability to code snippets, programming instructions, problem solutions, and datasets such as Code Alpaca and WizardCoder.
  • Safeguards: Combining synthetic with real data can help avoid model collapse, while filtering and reinforcement-learning methods aim to improve dataset signal.The survey presents these as strategies for managing distribution shift, bias amplification, and feedback-loop risks.
  • Open challenges: Factual accuracy, diversity, and evaluation metrics remain open research issues in this rapidly evolving field.The survey notes continuing development of human-in-the-loop generation and domain-specific synthetic-data models.
  • Implications: With rigorous evaluation and ethical safeguards, synthetic data may complement real data and accelerate AI development where real data is scarce.The conclusion frames this as a future direction rather than an established replacement for manual data collection.
Loading 2503.14023v2…