PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Learning for Synthetic Data Generation: A Review

Yingzhou Lu, Lulu Chen, Yuanyuan Zhang, Minjie Shen, Huazheng Wang, Xiao Wang, Capucine van Rechem, Tianfan Fu, Wenqi Wei

arXiv:2302.04062v10cs.LG

TL;DR

Machine-learning applications face data-quality, scarcity, access, privacy, and regulatory challenges. This paper systematically reviews machine-learning approaches for synthetic data generation across applications, models, privacy, fairness, evaluation, and future research. It concludes that synthetic data generation has broad potential while important development, deployment, privacy, fairness, and evaluation challenges remain.

  • Problem

    Machine learning is constrained by poor-quality or scarce data and by limited access to real data because of privacy, safety, and regulatory concerns.

  • Method

    The paper presents a high-level overview of state-of-the-art synthetic data generation approaches, applications, deep generative models, privacy, fairness, evaluation, and future research.

  • Results

    The review covers diverse application domains and deep generative models, while reporting privacy, fairness, evaluation, and deployment challenges in synthetic data generation.

  • Takeaways & Limitations

    Synthetic data generation is presented as a promising approach for addressing machine-learning data obstacles across diverse sectors.

Abstract

from arXiv · show

Machine learning heavily relies on data, but real-world applications often encounter various data-related issues. These include data of poor quality, insufficient data points leading to under-fitting of machine learning models, and difficulties in data access due to concerns surrounding privacy, safety, and regulations. In light of these challenges, the concept of synthetic data generation emerges as a promising alternative that allows for data sharing and utilization in ways that real-world data cannot facilitate. This paper presents a comprehensive systematic review of existing studies that employ machine learning models for the purpose of generating synthetic data. The review encompasses various perspectives, starting with the applications of synthetic data generation, spanning computer vision, speech, natural language processing, healthcare, and business domains. Additionally, it explores different machine learning methods, with particular emphasis on neural network architectures and deep generative models. The paper also addresses the crucial aspects of privacy and fairness concerns related to synthetic data generation. Furthermore, this study identifies the challenges and opportunities prevalent in this emerging field, shedding light on the potential avenues for future research. By delving into the intricacies of synthetic data generation, this paper aims to contribute to the advancement of knowledge and inspire further exploration in synthetic data generation.

I. INTRODUCTION

The paper frames synthetic data generation as a response to machine-learning data challenges involving quality, scarcity, privacy, and fairness. It surveys applications, generative models, evaluation strategies, and deployment challenges.

  • Data collection and annotation are time-consuming and expensive, while poor-quality data can produce incorrect or imprecise predictions.
  • Data scarcity arises from insufficient accessible datasets or costly manual labeling, creating a major obstacle for contemporary AI systems.
  • Synthetic data can support use cases where real data is unavailable or must remain private because of privacy or compliance risks.
  • The review provides background on synthetic data and surveys real-world applications, emphasizing opportunities associated with GANs and synthetic data generation.
  • It examines deep generative models, addresses privacy and fairness risks, reviews evaluation strategies, and identifies deployment challenges and future research directions.

II. APPLICATION

Synthetic data is presented across applications including computer vision and speech, where it can provide labeled or generated data for machine-learning systems. The section also surveys representative works in synthetic data generation.

  • Computer Vision: Synthetic data can improve labeling for difficult computer-vision tasks such as segmentation, depth estimation, and optical-flow estimation.
  • Computer Vision: GAN-based visual domain adaptation narrows embedding gaps and enables semantic segmentation across different domains.
  • Computer Vision: A 3D face model combined with hand-crafted assets produced realistic, diverse training images and comparable accuracy to real data for face-related tasks.
  • Voice: Machine learning and deep learning have expanded synthetic-voice applications in video production, digital assistants, and video games.
  • Voice: Speech-synthesis research uses graphical models with hidden variables, while conventional HMM-based systems can improve naturalness and reduce oversmoothing.
  • Voice: Synthetic speech supports text-to-speech naturalness and automatic speech recognition when speech data are sparse or limited.

C. Natural Language Processing (NLP)

Synthetic data generation in NLP spans models and applications that address context-sensitive language challenges and produce diverse sequence data. The section also connects synthetic data methods to healthcare and drug-discovery use cases.

  • NLP research applies deep generative models to categorization, routing, filtering, and searching relevant information across domains.
  • Context-dependent meanings and homonyms remain challenges for synthetic text generation and evaluation.BLEURT addresses these challenges by modeling human judgments with BERT-based training and millions of synthetic examples.
  • RelGAN combines a relational memory-based generator, Gumbel-Softmax relaxation, and embedded discriminator representations to improve sampling quality and diversity.
  • In healthcare, synthetic data supports patient confidentiality, reproducibility, disease understanding, and large-scale digital-twin generation.
  • Generative models support de novo drug design by learning molecular distributions and sampling novel structures from an enormous molecule space.The molecular search space is around 10^60, making brute-force enumeration computationally prohibitive.
  • MedGAN generates realistic high-dimensional synthetic patient records, including binary and count features, from real patient records.

E. Business

Business and education applications use synthetic data to model organizational and learning processes while reducing exposure of sensitive personal information. The section also describes privacy-preserving trajectory generation and AI-generated content applications.

  • Business data sharing is constrained by confidentiality and re-identification risks, motivating synthetic datasets that emulate real-data properties while protecting individuals.
  • Synthetic data enables organizations to refresh and model original data for continuous insights that support performance improvement.
  • Education: Education uses synthetic student data for privacy-conscious research, teaching-strategy design, teacher training, and personalized instructional planning.
  • Synthetic location and trajectory generation must resemble genuine user behavior while providing practical privacy protection against sensitive inference.
  • Trajectory methods combine n-gram or mobility models with differential privacy, discretization, or plausible-deniability constraints.
  • AI-generated content applications include interactive fiction, news reporting from structured earnings data, and scalable marketing or literary content production.

I. Finance

Synthetic data supports finance and other business functions by enabling analysis, testing, risk assessment, and privacy-conscious data use. The section extends these applications to retail, environmental studies, urban planning, and software development.

  • Financial institutions use synthetic data to comply with privacy regulations while retaining detailed datasets for analysis, development, testing, and validation.
  • Simulated transactions and market behaviors support risk assessment, anomaly identification, and fraud detection.
  • Synthetic workforce data can model employee performance, engagement, turnover, and the effects of HR policies on morale and performance.
  • Beyond finance, synthetic data models customer behavior and inventory, simulates environmental conditions, and supports urban planning decisions.
  • Synthetic code generation simulates coding scenarios, bug patterns, and software behaviors for testing, debugging, coding assistance, performance optimization, and release reliability.

III. GENERATIVE AI

The review surveys generative AI methods that learn data distributions and generate novel structured objects, emphasizing language models, VAEs, GANs, reinforcement learning, and diffusion models. It also compares methods across multiple aspects and explains core VAE and language-model mechanisms.

  • Generative AI models learn distributions from existing data and sample novel structured data objects, generally within unsupervised learning.
  • The review covers language models, variational autoencoders, generative adversarial networks, reinforcement learning, and diffusion models.
  • Language Model: Language models factorize sequence probability into conditional token probabilities and can generate natural-language or electronic-health-record sequences.
  • Variational Autoencoder (VAE): VAEs encode data into continuous latent variables, decode those variables to reconstruct data, and regularize the latent distribution toward a normal prior.
  • Variational Autoencoder (VAE): VAE inference samples latent variables from the normal distribution to generate novel data objects, with variants including disentangled, hierarchical, and sequence VAEs.

D. Generative Adversarial Network (GAN)

GANs formulate synthetic data generation as an adversarial learning task involving a generator and discriminator. Their alternating optimization trains the generator to resemble real data while the discriminator distinguishes real from synthetic samples.

  • GANs comprise a generator that produces data and a discriminator that classifies samples as real or synthetic.The discriminator performs binary classification, while the generator seeks outputs close to real data.
  • The generator is denoted G(z), and the discriminator assigns each data object a probabilistic score D(x).
  • The latent variable z is sampled from a normal distribution to enhance the diversity of generated data objects.
  • Training alternates between optimizing the generator with the discriminator fixed and optimizing the discriminator with the generator fixed.
  • During generator optimization, outputs are encouraged to be close to real data by receiving higher discriminator scores.
  • Wasserstein GAN aims to improve learning stability, accelerate training, and address problems such as mode collapse.

E. Reinforcement Learning (RL)

The paper describes reinforcement learning as sequential decision-making for constructing synthetic data step by step, then surveys diffusion models that transform data into noise and reverse that process to generate samples.

  • E. Reinforcement Learning (RL): RL generates synthetic data sequentially by growing a basic component at one time and formulates this process as a Markov decision process.
  • E. Reinforcement Learning (RL): At each step, the RL agent selects an action from the current state, transitions to a next state, and receives an environment reward.
  • E. Reinforcement Learning (RL): Policy gradients provide an unbiased estimator of the objective gradient, enabling stochastic optimization to maximize expected reward.
  • F. Diffusion Model: Diffusion models add Gaussian noise to data during a forward process and reconstruct data from noise during the reverse process.
  • F. Diffusion Model: Diffusion models estimate a variational lower bound of the negative log-likelihood and have achieved wide success in downstream synthetic problems.
  • F. Diffusion Model: The reverse process uses a neural network to approximate conditional distributions and predict the parameters needed for denoising.

G. Multimodal Learning

Synthetic multimodal data generation creates datasets combining text, images, audio, and numerical data to simulate complex real-world scenarios. GANs support cross-modal generation, while ChatGPT supports multimodal generation across several feature types.

  • Multimodal datasets integrate text, images, audio, and numerical values to provide a comprehensive view of complex scenarios.
  • Synthetic multimodal generation creates artificial datasets combining heterogeneous data types for fields including healthcare, finance, and education.
  • GANs can generate one data type from another, including images from text and audio from images.
  • ChatGPT supports multimodal data generation involving image, text, and numerical features.

IV. PRIVACY RISKS AND PREVENTION

Synthetic data can enable sharing of regulated datasets, but preserving the source distribution creates privacy risks. The review surveys anonymization, differential privacy, federated, and post-processing strategies for balancing privacy with utility.

  • Regulated clinical and genomics datasets may be withheld from public release, making synthesized data a possible sharing alternative.
  • Because synthetic data preserves the original distribution for utility, models may leak membership, sensitive attributes, or individual training instances.Repeated sampling can recover training samples, attributes, or training-data membership.
  • Anonymization methods such as K-anonymity and nearest marginal replace sensitive data with fictitious yet realistic data, but may lack rigorous privacy guarantees.
  • Differentially private synthesis adds noise through approaches including DPGAN, teacher-student generators, GANobfuscator, local perturbation, federated training, and Bayesian marginals.
  • Post-processing can resample synthetic data to remove samples that fail selected utility measures, improving downstream utility.

C. Privacy Threats in Foundation Models

Foundation models create emerging privacy risks because they can memorize and reproduce training data, while generated content may inherit bias and harmful stereotypes. Existing privacy protections may be inadequate, and evaluation of synthetic data remains costly and difficult.

  • Privacy threats: Sequences repeated more often in training data are more likely to be generated, enabling attackers to identify memorized training examples.The paper reports exposure from large language models and stable diffusion through generated sequences.
  • Privacy threats: Foundation-model approaches for data synthesis remain insufficiently explored, and existing privacy measures may not meet their demands.The paper calls for countermeasures addressing memorization and generalization problems.
  • Fairness: Synthetic data can inherit bias from preprocessing, collection, and algorithms, leaving minority groups under-represented.Fairness-aware methods include preprocessing, while balanced GAN-generated datasets have reduced disparate impact from subgroup imbalance.
  • Fairness: Differential privacy can increase the influence of majority subgroups and reduce the quality of images generated by GANs, lowering downstream utility.The cited findings distinguish effects on unfairness, group influence, and image quality.
  • Societal risks: Foundation-model outputs may reproduce stereotypes, exclusion, toxicity, misinformation, and sensitive information from their training data.The paper links generated-data quality to the quality of training corpora and calls for regulating real-world data used in synthesis.
  • Evaluation: Human evaluation directly assesses quality, similarity, or usability but is expensive, time-consuming, error-prone, difficult for high-dimensional data, and not scalable.Statistical comparison and downstream evaluation are presented as additional strategies.

VII. CHALLENGES AND OPPORTUNITIES

Synthetic data generation is surveyed as an increasingly viable alternative to real-data training, but its broader use remains constrained by evaluation, coverage, bias, security, and standardization challenges. The paper identifies foundation models as both a source of new risks and an opportunity for future research and defensive applications.

  • Scope and opportunities: The survey covers synthetic-data applications, state-of-the-art generation approaches, and future research directions across real-world domains.It examines deep neural network architectures and deep generative models for producing high-quality synthetic data.
  • Evaluation: Healthcare lacks readily available clinical quality metrics, and clinicians may struggle to interpret probability likelihood and divergence scores.The paper also calls for regulations ensuring quality standards and minimizing risks in medical synthetic data.
  • Coverage: Current methods may miss outliers and corner cases because limited attention and domain coverage constrain representation of the original data.The paper proposes studying outliers, regular instances, and their effects on model parameterization.
  • Bias and deployment: Biases from sample selection and class imbalance can cause algorithms to underperform when deployment conditions differ substantially from data-collection conditions.The paper calls for methods that address these biases.
  • Foundation models: Foundation models introduce risks including jailbreak attacks, deepfakes, discrimination, toxicity, misinformation, sensitive-information disclosure, and malicious use.Their realistic outputs and accessibility can challenge traditional detection systems and enable more widespread attacks.
  • Open challenges: Synthetic data is becoming a viable alternative to real-data training, but standard tools, synthetic–real distribution differences, and imperfect-data utilization remain open challenges.The paper expects greater future impact as models, metrics, and technologies mature.
  • Motivation: Synthetic data addresses data quality, scarcity, and governance challenges when real data is unavailable or restricted by privacy or compliance risks.The paper describes synthetic data as artificially annotated information generated by algorithms or simulations.
  • Future work: Future research can enhance synthetic-data functionality while supporting more ethical, secure, and transparent solutions.The conclusion frames these improvements as dependent on proactively addressing machine-learning risks.
Loading 2302.04062v10…