Source-linked AI summary
Best Practices and Lessons Learned on Synthetic Data
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, Andrew M. Dai
TL;DR
AI models need large, diverse, high-quality datasets, but scarcity, privacy concerns, and collection costs make them challenging to obtain. This paper surveys synthetic-data research and prior-art evidence, including examples of geometry reasoning and TruthfulQA results. It concludes that synthetic data should be used responsibly, with attention to fidelity, controllability, and evaluation.
Problem
Large, diverse, high-quality datasets are important for AI, but data scarcity, privacy concerns, and collection and annotation costs make them difficult to obtain.
Source: §1 · p. 1Method
The paper surveys the current state of synthetic-data research, covering generation techniques, applications, empirical case studies, challenges, and future directions.
Source: §1 · p. 2Results
AlphaGeometry used 100 million synthetic data points with symbolic verification to reach geometry problem-solving comparable to a human Olympiad gold medalist, while synthetic hallucination data improved TruthfulQA performance (Zhang et al., 2023d).
Source: §2.1 · p. 3, §2.5 · p. 6Takeaways & Limitations
The paper recommends improving synthetic-data fidelity and controllability and developing standardized evaluation and contamination protocols, while using synthetic data responsibly to build more powerful, inclusive, and trustworthy AI systems.
Source: §6 · p. 10Synthetic alignment data may not represent the nuances and complexity of human values and preferences, and discrepancies can lead models to learn from biased, ungrounded, or unrepresentative data.
Source: §4 · p. 8
Abstract
from arXiv · showhide
The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs. Synthetic data has emerged as a promising solution by generating artificial data that mimics real-world patterns. This paper provides an overview of synthetic data research, discussing its applications, challenges, and future directions. We present empirical evidence from prior art to demonstrate its effectiveness and highlight the importance of ensuring its factuality, fidelity, and unbiasedness. We emphasize the need for responsible use of synthetic data to build more powerful, inclusive, and trustworthy language models.
1 Introduction
AI models depend on large, diverse, high-quality datasets, but scarcity, privacy concerns, and collection costs make them difficult to obtain. Synthetic data offers a potential alternative, with benefits that depend on maintaining factuality, fidelity, and fairness.
1 Introduction
- AI model training and evaluation rely on large, diverse, high-quality datasets that can be difficult to acquire because of scarcity, privacy concerns, and collection costs.The passage also notes predictions of future shortages of fresh text and image data.Source: §1 · p. 1
- Synthetic data is artificially generated to mimic real-world characteristics and patterns, using algorithms, generative models, or simulations rather than direct human creation.The paper presents it as a potential response to data-access challenges and a way to develop more robust, reliable, and fair AI models.Source: §1 · p. 1
- Synthetic data can scale where real data are scarce, be tailored to balance classes, and provide de-identified alternatives where privacy is important.Controlled variation can improve model performance and generalization.Source: §1 · p. 2
- False, hallucinated, or biased synthetic data may fail to generalize; factuality, fidelity, and bias therefore require careful evaluation.The paper calls for sophisticated generation and evaluation methods, rigorous testing, and fairness assessments.Source: §1 · p. 2
- The paper surveys synthetic-data research, including generation techniques, training and evaluation applications, challenges, limitations, and future directions.Its overview includes case studies and empirical evidence.Source: §1 · p. 2
2 Synthetic Data in Training
Synthetic data is presented as an effective, relatively low-cost alternative to real data, with training applications spanning reasoning, tool use, multimodality, multilingual learning, and alignment.
2 Synthetic Data in Training
- Synthetic data is presented as an effective, relatively low-cost alternative to real data across several training domains.The section surveys applications in reasoning, tool use and planning, multimodality, multilingual learning, and alignment.Source: §2 · p. 2
2.1 Reasoning
- Reasoning applications use synthetic questions, filtered rationales, and training examples; AlphaGeometry combines 100 million synthetic points with symbolic verification.The section also describes synthetic data for math, algorithmic reasoning, and physics reasoning.Source: §2.1 · pp. 2–3
2.2 Tool-using and Planning
- Synthetic trajectories teach tool use through simulated interactions, while simulated environments also provide feedback for planning (Thoppilan et al., 2022; Huang et al., 2022).Tool-use examples include generated or templated API calls; planning examples use environment feedback to train agents.Source: §2.2 · pp. 3–4
2.3 Multimodality
- Rendered HTML, charts, and webpages provide fine-grained vision-language training data, while image attributes can prompt synthetic multimodal instruction-response pairs (Lee et al., 2023; Liu et al., 2024b).Models finetuned on synthetic webpage data generalize reasonably well to realistic Internet-scraped data.Source: §2.3 · p. 4
2.4 Multilingual
- Multilingual synthetic data includes back-translated parallel samples and generated question-answer pairs, with reported gains on multilingual QA and information-retrieval benchmarks (Sennrich et al., 2016; Shakeri et al., 2021).Back-translation gains depend on synthetic-data quality and diversity.Source: §2.4 · pp. 4–5
2.5 Alignment
- Instruction-following data can be generated from seed examples, but sycophancy remains a concern; synthetic finetuning data has been used to reduce it on held-out prompts (Wang et al., 2022a; Wei et al., 2023b).The section also describes methods that revise instructions or increase their complexity.Source: §2.5 · pp. 5–6
- Synthetic SFT data can induce hallucinations when it contains hallucinated answers or trains models to answer unknown questions, while other synthetic-data methods have shown promise for mitigation (Zhang et al., 2023d).Mitigation approaches described include reward-model training, synthetic tasks, and fact-checking with confidence scores; scalable hallucination-evaluation tasks remain limited.Source: §2.5 · p. 6
- Constitutional AI uses AI-generated critiques and feedback in an RLAIF pipeline, but low-fidelity preference data may miss nuanced human judgment; real-data mixtures and interactive environments are proposed remedies (Bai et al., 2022; Argyle et al., 2023).The section describes synthetic data as a way to create diverse, controlled alignment datasets and support testing for bias and fairness.Source: §2.5 · pp. 6–7
3 Synthetic Data in Evaluation
Synthetic data supports evaluation of factuality, safety, and human judgments. The paper describes methods and examples across these uses.
Factuality
- Factuality evaluation has moved from lexical-overlap statistics to model-based methods and synthetic datasets grounded in knowledge graphs or web search.The passage notes that model-based methods can still struggle to identify specific errors.Source: §3 · p. 7
Safety
- Synthetic red-teaming datasets exposed inverse scaling, while synthetic backdoor attacks revealed deceptive behavior that standard safety training could not easily remove.One study produced 154 human-verified datasets; the passage presents these approaches as feasible AI assistance for scaling human oversight.Source: §3 · p. 7
Assisting human evaluation
- Synthetic LLM judgments and execution-based code evaluations show strong correlation with real human judgments.Examples include chatbot benchmarks and code-reasoning evaluations built from synthetic functions and execution tasks.Source: §3 · p. 7
4 Challenges and Limitations of Synthetic Data
The paper highlights risks from synthetic data misuse, its potential mismatch with human values, and harder benchmark decontamination.
Misuse of synthetic data might proliferate misinformation
- Synthetic data can be misused to impersonate people or manipulate opinion, and its misinformation may erode trust and make truth harder to distinguish.The paper calls for ethical-use guidance and mechanisms to detect and counter synthetic misinformation.Source: §4 · p. 8
Synthetic data might cause ambiguity in AI alignment
- Synthetic alignment data may miss the nuances of human values, leading to biased, ungrounded, or misrepresentative training signals and potentially misaligned behavior.The paper also notes that this ambiguity can complicate interpretation of model decisions.Source: §4 · p. 8
Training with synthetic data makes evaluation decontamination harder
- Rephrased benchmark examples in synthetic training data can make token-level contamination checks ineffective.The paper recommends more advanced detection methods and protected in-house evaluation benchmarks.Source: §4 · p. 8
5 Directions for Future Work
The paper identifies four future research directions: scaling, data quality and diversity, scalable oversight, and synthetic-data self-improvement.
Synthetic data scaling
- Whether synthetic-data training follows familiar scaling patterns remains open because synthetic-data quality may be less consistent than real-world data.Future work should study scaling laws and the balance between the quantity and quality of synthetic samples.Source: §5 · p. 9
Further improving quality and diversity of synthetic data
- Future generation methods should improve sample quality, diversity, and attribute control while incorporating domain knowledge and constraints.The paper points to applications including privacy-preserving analysis and training in healthcare, finance, and social sciences.Source: §5 · p. 9
Towards high-fidelity and more efficient scalable oversight
- Synthetic oversight research should cover more scenarios and modalities, while investigating whether controlled generation can deliver targeted oversight with less data.The passage notes that existing simulated scenarios can be narrow or oversimplified.Source: §5 · p. 9
The emergent self-improvement capability
- A key open question is whether a model can generate data better than its training data and thereby improve itself.The paper identifies the upper bound, underlying reasons, conditions, feasibility, and risks of this process as questions for future study.Source: §5 · pp. 9–10
6 Conclusion
Synthetic data can provide realistic, diverse, relatively low-cost alternatives for AI training and evaluation, but factuality, fidelity, and bias remain challenges. The paper calls for improved generation and evaluation practices and responsible use.
Conclusion
- Synthetic data is presented as a realistic, diverse, relatively low-cost alternative to real data for training and evaluation across domains.Its potential is paired with the continuing challenge of ensuring factuality, fidelity, and lack of bias.Source: §6 · p. 10
- Future work could improve generative-model fidelity and controllability and develop standardized evaluation and contamination protocols and tools.The conclusion also suggests integrating synthetic data with other techniques and applying it in additional domains.Source: §6 · p. 10