PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Data for Deep Learning

Sergey I. Nikolenko

arXiv:1909.11512v1cs.LGcs.CRcs.CV

TL;DR

The paper surveys how synthetic data can address limited or costly labeled data across computer vision and other machine-learning applications. It organizes methods and datasets, examines synthetic-to-real adaptation and privacy guarantees, and highlights future directions, while noting that real-data augmentation can outperform GAN-refined synthetic data in one gaze-estimation case.

  • Problem

    Deep-learning applications face datasets that are too small or expensive to label, while the field lacks a satisfactory general overview of synthetic data.

  • Method

    The paper provides a comprehensive survey of synthetic datasets, generation methods, synthetic-to-real adaptation, and privacy-related synthetic data across vision and other fields.

  • Results

    The survey identifies procedural generation, feedback loops, domain knowledge, and additional modalities as promising directions for synthetic-data research.

  • Takeaways & Limitations

    Synthetic data can provide pixel-perfect labels and support transfer to real tasks, but its usefulness depends on the application and adaptation strategy.

  • Takeaways & Limitations

    In a gaze-estimation case, affine augmentation of real images outperformed both synthetic UnityEyes data and GAN-based refinement.

Abstract

from arXiv · show

Synthetic data is an increasingly popular tool for training deep learning models, especially in computer vision but also in other areas. In this work, we attempt to provide a comprehensive survey of the various directions in the development and application of synthetic data. First, we discuss synthetic datasets for basic computer vision problems, both low-level (e.g., optical flow estimation) and high-level (e.g., semantic segmentation), synthetic environments and datasets for outdoor and urban scenes (autonomous driving), indoor scenes (indoor navigation), aerial navigation, simulation environments for robotics, applications of synthetic data outside computer vision (in neural programming, bioinformatics, NLP, and more); we also survey the work on improving synthetic data development and alternative ways to produce it such as GANs. Second, we discuss in detail the synthetic-to-real domain adaptation problem that inevitably arises in applications of synthetic data, including synthetic-to-real refinement with GAN-based models and domain adaptation at the feature/model level without explicit data transformations. Third, we turn to privacy-related applications of synthetic data and review the work on generating synthetic datasets with differential privacy guarantees. We conclude by highlighting the most promising directions for further work in synthetic data studies.

1 Introduction

The survey examines synthetic data as a response to limited or costly labeled data, covering its development, applications, synthetic-to-real transfer, privacy, and future research directions. It illustrates both the value of pixel-perfect annotations and the remaining challenges of realism, domain transfer, and data ownership.

  • Motivation: Synthetic data addresses insufficient or expensive labeled datasets by producing artificial training examples, including potentially unlimited pixel-perfect annotations from rendered 3D environments.For segmentation, rendering requires a one-time 3D-modeling investment and can produce RGB images, segmentation maps, and other labels.
  • Future directions and limitations: The survey identifies procedural generation, feedback from generation, domain knowledge, and additional synthetic modalities as promising directions, while noting that synthetic assets can retain intellectual-property constraints.Synthetic data may alleviate some legal issues, but handmade or manually collected 3D models can create ownership problems unless properly released.
  • Scope: The survey covers synthetic datasets and environments across low-level and high-level vision, objects, humans, robotics, neural programming, bioinformatics, and other applications.Its examples include optical flow, stereo disparity, object detection, segmentation, and visual reasoning, alongside broader machine-learning domains.
  • Synthetic-to-real transfer: Realism requirements vary by task: low-level vision can work with non-photorealistic data, while camera artifacts, photorealistic rendering, and hybrid datasets can improve real-world performance.Reported examples include Flying Chairs for low-level vision, simulated lens and sensor artifacts, and hybrid synthetic-plus-real training for viewpoint estimation and recognition.
  • Synthetic-to-real transfer: Synthetic data can transfer to real tasks through dataset combinations, domain randomization, feature or model adaptation, and refinement methods.The surveyed approaches include freezing lower detector layers while fine-tuning upper layers and combining randomized with photorealistic images.

3 Synthetic simulated environments

Synthetic environments extend synthetic data from static datasets to interactive training and evaluation settings for autonomous cars, robots, indoor agents, and aerial vehicles. The surveyed systems support simulation-based learning, synthetic-to-real transfer, and controlled benchmarking across navigation tasks.

  • Motivation: Interactive simulation is essential for training autonomous vehicles and robots because real-world reinforcement-learning experiments are impractical.Synthetic environments allow agents to learn through interaction without conducting costly physical experiments.
  • Scope: Section 3 groups environments and static datasets across outdoor, indoor, robotic, aerial, and game-based simulation settings.The survey also covers applications to navigation, localization, mapping, motion planning, and control.
  • Limitations: The survey notes that synthetic environments do not eliminate intellectual-property concerns because handmade or manually collected 3D models can remain protected.The SUNCG dataset became unavailable during a legal controversy over its data.
  • Indoor environments: Habitat-Sim renders thousands of frames per second, reported as 10-100x faster than previous simulators.The authors claim rendering can be faster than loading images from disk, reducing simulation as a training bottleneck.
  • Indoor environments: Indoor navigation studies report that reinforcement-learning agents outperform SLAM-based agents and generalize best across datasets, including synthetic-to-real transfer.The reported transfer includes SUNCG to Matterport3D and Gibson.
  • Robotic simulation: Dactyl policies trained entirely in ORRB simulation transferred successfully to a physical multi-finger robotic hand.This is presented as an example of synthetic-to-real transfer in robotic manipulation.
  • Aerial environments: Air Learning combines pluggable simulators, learning algorithms, control policies, and hardware-in-the-loop flight controllers for autonomous aerial robots.Its obstacle-avoidance benchmarks found that more onboard compute produced almost 2x shorter trajectories.
  • Aerial environments: Synthetic and real data can be combined for UAV wire detection by overlaying rendered 3D wire models onto real background images.Wire properties such as material, sag, and camera angle are varied without adapting the wires to background semantics.

4 Synthetic data outside computer vision

Synthetic data is used beyond computer vision in neural programming, bioinformatics, molecular design, and limited NLP applications. These areas use synthetic examples both to train models and, in some cases, as the direct target of generation.

  • Neural programming: Neural program induction and synthesis rely heavily on automatically generated programs because available tasks and programs are often simple.Training examples commonly consist of randomly generated programs or algorithmically generated symbol sequences.
  • Neural programming: Synthetic program-generation algorithms can omit important regions of program space, limiting semantic coverage despite good test-set performance.A proposed dataset-distribution methodology improved results in Calculator and Karel domains.
  • Bioinformatics: In de novo drug design, generated molecules are often the end goal rather than training data for another model.Candidate molecules are generated from a search space of about 10^60 synthesizable molecules and later explored in laboratory studies and clinical trials.
  • Bioinformatics: Molecular-generation research includes conditional adversarial autoencoders, VAE comparisons, new molecular representations, and the MOSES benchmarking platform.One reported generated inhibitor of Janus kinase 3 showed good activity and selectivity in vitro.
  • Bioinformatics: Bioinformatics uses synthetic data for biological images, live experiments with synthetic genetic material, privacy-sensitive datasets, and drug design.The survey describes synthetic data as routinely generated and used across bioinformatics.
  • Natural language processing: Synthetic data remains less widely used in NLP because generating meaningful text with predefined variables such as topic or sentiment remains difficult.The survey states that near-human text generation could make synthetic data less useful for discriminative target-variable models.

5 Directions in synthetic data development

The survey reviews methods for making synthetic data more useful, especially through domain randomization, improved CGI realism, compositing, and generative models. These approaches vary scene content, rendering conditions, realism, and data distributions to support transfer and downstream performance.

  • Domain randomization: Domain randomization broadens the synthetic distribution so models can transfer more directly from synthetic training data to real applications.Randomization can occur during scene construction and image rendering.
  • Domain randomization: Synthetic generators randomize objects, positions, distractors, backgrounds, textures, lighting, resolution, rendering type, and image noise.These controls operate at both 3D scene-construction and 2D rendering levels.
  • Domain randomization: Domain-randomized images can be completely unrealistic yet diverse enough to make networks focus on object shape.Tremblay et al. reported improved car detection for R-FCN and SSD, but not Faster R-CNN, compared with Virtual KITTI.
  • Domain randomization: Later work constrains randomization to realistic ranges and scene context, while other work learns parameters for transferring reinforcement-learning policies.The learned parameters include those controlling sampling over Markov decision processes.
  • CGI development: CGI realism can be improved by modeling object deformations, estimating rendering parameters, simulating sensor noise, and automating coherent scene composition.Examples include mesh deformation, realistic sensor postprocessing, and Meta-Sim for urban environments.
  • Compositing: Progressive and Selective Instance Switching improves results on MS COCO for Faster-RCNN, FPN, Mask R-CNN, and SNIPER detectors.The cited passage reports improved results but does not provide a numerical metric.
  • Generative models: GAN-conditioned synthetic images and geometry-preserving augmentation improve object-counting performance, including leaf counting.Generation can be conditioned on object counts or segmentation masks.

6 Synthetic-to-real domain adaptation and refinement

The survey groups synthetic-to-real adaptation into data-level refinement and feature/model-level approaches, then reviews applications across vision, robotics, and medical imaging. Reported studies generally improve transfer, while feature-level methods remain work in progress and GAN refinement is not uniformly superior to augmenting real data.

  • Approach taxonomy: Synthetic-to-real adaptation is divided into data-level refinement and feature/model-level methods that leave the data unchanged.The survey focuses mainly on recent deep-neural-network approaches.
  • Synthetic-to-real refinement: GAN-based refiners make synthetic images more realistic, with SimGAN introducing a refiner network and adversarial architecture for synthetic eye images.GazeGAN extends this direction with unpaired image-to-image translation and refines surrounding facial regions beyond the eye model.
  • Applications: Other refinement studies report improved classification and pose estimation, depth estimation without paired real data, sim-to-real policy transfer, and urban-scene segmentation.These results span synthetic object imagery, T2Net, robotic platforms, SYNTHIA-to-Cityscapes adaptation, and related segmentation systems.
  • Applications: Synthetic data and adaptation also support broader transfer gains, including better face verification and identification, end-to-end driving, and real-world robotic navigation or grasping.One robotic grasping study used 5,000 additional real episodes versus 580,000 for the comparison policy.
  • Approach taxonomy: Feature- and model-level adaptation is described as simpler and more robust in practice, but the survey characterizes the area as still mostly work in progress.The survey recommends this family when refined images themselves are not required.

A and Dvol

The framework uses discriminators in domains A and B to distinguish real from synthetic images, while cycle consistency regularizes the mappings.

  • Discriminators in domains A and B distinguish real images from images generated by Gvol using the standard GAN loss.
  • The objective includes a cycle consistency loss to constrain the transformations between domains.

A , Gvol

The framework uses segmentors to produce 3D segmentation masks and includes a shape consistency loss to preserve structural agreement.

  • Segmentors B produce 3D segmentation masks in the target space Y.
  • A shape consistency loss is included to maintain consistency of the generated shapes.

A , Gvol

The survey covers synthetic-data applications spanning medical imaging, privacy-preserving data release, and evaluation of financial strategies. It also identifies substantial variation among GAN architectures and a lack of direct evidence that synthetic financial time series improve forecasting or strategy development.

  • Medical imaging: Synthetic-to-real domain adaptation has been applied in medical imaging, including digitally reconstructed radiographs and transfers between imaging modalities.Reported applications include ultrasound transducer localization and synthetic cross-modal generation for segmentation or lesion detection.
  • Medical imaging: GAN-based medical-imaging methods are promising, but comparisons across architectures are usually lacking and larger experimental studies are needed.Existing studies commonly compare each architecture only with selected baselines.
  • Privacy: Differentially private synthetic datasets aim to preserve statistics and structure while protecting information about individual records in real training data.Reviewed approaches include noisy Bayesian networks, perturbed query outputs, and privacy-preserving deep-learning training.
  • Finance: Synthetic financial data has been used to test strategy robustness under altered abnormality distributions rather than to train financial strategies.Fraud-detection algorithms identify anomalies, after which synthetic data balances the detected abnormalities for evaluation.
  • Finance: No direct applications are reported in which synthetic financial time series improve forecasting or financial-strategy development.The survey attributes this boundary to the difficulty of predicting or accurately modeling financial time series.

8 Promising directions for future work

The survey identifies procedural generation and automatically learned data-generation policies as promising ways to scale and improve synthetic data, alongside richer domain-adaptation architectures and probabilistic-GAN combinations.

  • 8.1 Procedural generation of synthetic data: Qi et al. learn a stochastic scene grammar from SUNCG and use MCMC sampling to generate indoor layouts difficult for an automated classifier to distinguish from the originals.The grammar models object, furniture, and human-centric relations.
  • 8.1 Procedural generation of synthetic data: Procedural generation can scale synthetic data beyond manually constructed scenes by learning probabilistic models that generate new samples from real or synthetic data.This is intended for cases where randomly placing objects is insufficient.
  • 8.2 From domain randomization to the generation feedback loop: Generation feedback loops can learn domain-randomization parameters from real observations instead of relying on manually tuned simulation distributions.Related approaches tune synthetic-data distributions for both real-data similarity and downstream task performance.
  • 8.2 From domain randomization to the generation feedback loop: Adversarial deception networks transform synthetic inputs while preserving or predictably changing labels, achieving state-of-the-art-comparable domain adaptation and better generalization.Training alternates recognition updates with reversed-gradient deception updates.
  • 8.2 From domain randomization to the generation feedback loop: VADRA learns rendering policies with REINFORCE so synthetic-data parameters optimize downstream model performance rather than merely matching real data.The survey presents this meta-approach as a broader direction for automatically learning how to generate synthetic data.
  • 8.4 Additional modalities for domain adaptation architectures: Hierarchical generative models combined with GAN-based architectures are highlighted as a promising direction, including gaze-estimation models whose synthetic data improves standard estimators.The hierarchical model incorporates domain knowledge through eye-related landmarks and anatomical structure.
  • 8.4 Additional modalities for domain adaptation architectures: GIO-Ada combines synthetic images with segmentation and depth maps, using image refinement and output-level adaptation to improve segmentation on real images.Its discriminator aligns predicted segmentation and depth outputs across synthetic and real domains.

9 Conclusion

The survey presents synthetic data as a broadly effective source of labeled training data across tasks and domains, while identifying dataset bias, labeling costs, coverage gaps, and missing modalities as continuing motivations for its use.

  • 9 Conclusion: Synthetic data provides virtually limitless, perfectly labeled data and has been explored across many deep-learning problems, with additional use cases still remaining.The survey covers direct applications, synthetic-to-real adaptation, privacy guarantees, and improvements to synthetic-data generation.
  • 9 Conclusion: The survey reports synthetic data working well across diverse tasks and domains, including computer vision, autonomous driving, aerial navigation, robotics, and privacy-related data generation.It also reviews synthetic-to-real refinement, feature-level adaptation, and promising directions for improving generation.
  • 9 Conclusion: Synthetic data remains important for reducing dataset bias, covering corner cases, and supporting problems involving modalities unavailable in real datasets.The conclusion notes that many real datasets cannot fully cover broad underlying distributions or afford expensive manual labeling.
Loading 1909.11512v1…