Synthetic Data for Deep Learning

1 Introduction

• Synthetic-data generation addresses data scarcity and prohibitive labeling costs by creating artificial or manipulated examples, but synthetic-to-real domain transfer remains a central challenge.Rendering a modeled 3D environment can produce potentially unlimited pixel-perfect RGB, depth, stereo, point-cloud, and video labels after a one-time modeling investment.Source: §1 · p. 3
• The survey covers direct synthetic training, synthetic augmentation of real datasets to reduce bias, and privacy-oriented synthetic data, while distinguishing synthetic data from ordinary augmentation.It also highlights on-the-fly generation during training and organizes the computer-vision discussion from low-level problems to specialized applications.Source: §1 · pp. 3–5

2.1 Low-level computer vision

• Low-level vision benefits from synthetic data because optical flow and stereo ground truth are difficult to label or measure, while transfer to real images can succeed.Synthetic data need not be globally realistic for optical flow, but combining datasets and simulating camera-specific artifacts improves real-test performance; underwater disparity data also transferred successfully to real images.Source: §2.1 · pp. 7, 10

2.2 Basic high-level computer vision

• Synthetic data supplies pixel-perfect labels for detection, segmentation, depth, and 3D understanding, addressing tasks whose manual annotation is especially difficult.These datasets support applications ranging from everyday-object and retail recognition to small-object detection and semantic or instance segmentation.Source: §2 · p. 7, §2.2 · pp. 10–11

2.2.1 Datasets of basic objects

• ShapeNet and PartNet provide large 3D object resources with increasingly detailed geometric and hierarchical part annotations for synthetic-data generation and 3D understanding.ShapeNet contains more than three million models, while PartNet contains 573,585 part instances across 26,671 models from 24 categories.Source: §2.2.1 · p. 12
• Synthetic object datasets provide diverse environments and higher-quality labels than real RGBD scans, while placing synthetic objects on real surfaces improves detection and segmentation on real validation data [5] [198].Georgakis et al. [198] report significant gains from hybrid real+synthetic training, and Abu Alhaija et al. [5] found manually placed synthetic cars improved instance segmentation and object detection.Source: §2.2.1 · p. 13

2.2.2 Improving high-level computer vision with synthetic data

• Synthetic data improves detection, segmentation, pose, depth, and alignment, with hybrid datasets, domain randomization, frozen feature layers, and realistic rendering helping bridge synthetic-to-real gaps.Hybrid training can outperform purely synthetic or purely real data; photorealism is beneficial in some settings, while diversity remains important and DepthSynth improves recognition through realistic sensor simulation.Source: §2.2.2 · pp. 15, 17, 18
• For object detection, fine-tuning synthetic-trained models with small amounts of real data is preferable to equivalent mixed training, and synthetic-data diversity can matter more than photorealism.Nowruzi et al. report this result using SSD-MobileNet [254], contrasting with conclusions emphasizing realism.Source: §2.2.2 · p. 15

2.3 Synthetic people

• Synthetic people data helps alleviate real-dataset bias, cover corner cases, and support human-vision problems involving alternative modalities such as infrared face recognition.Crowd-analysis datasets also model agent trajectories and realistic rendered scenes, including bottlenecks, corridors, and obstacle flows.Source: §2.3 · p. 23

2.4 Character and text recognition

• Synthetic text composited onto real images is widely used for OCR, text detection, layout analysis, and text-line segmentation, including document digitization with mixed handwritten and printed text.Published generators include MJSynth [276] and SynthText in the Wild [222], while GANs support realistic text refinement and handwriting generation [14].Source: §2.4 · pp. 23–24

2.5 Visual reasoning

• CLEVR uses 100K rendered images and about 1M automatically generated questions to isolate visual reasoning from visual recognition in VQA.Its scene graphs encode object attributes and spatial relations, while executable functional programs generate questions with ensured, unique answers across querying, comparison, existence, and counting tasks.Source: §2.5 · p. 24
• COG extends synthetic visual-question generation to video, with functional-program questions involving time and a generator designed to reduce response bias.The generator begins with balanced target responses and then produces progressively more challenging video-question pairs.Source: §2.5 · p. 25

3 Synthetic simulated environments

• Interactive simulators enable reinforcement-learning agents to train without the expense or impracticality of real-world vehicle and robot experiments.Many systems transfer policies directly to the real world or fine-tune them on real data, without explicit domain adaptation.Source: §3 · p. 26

3.1 Urban and outdoor environments: learning to drive

• Outdoor synthetic environments support autonomous-driving perception and control through datasets and simulators such as Virtual KITTI, GTAV, SYNTHIA, CARLA, and TORCS.Reported benefits include improved real-set detection or segmentation, rare-condition coverage, multimodal sensing, and transfer to real driving data.Source: §3.1 · pp. 26, 28, 29, 30, 31

3.2 Datasets and simulators of indoor scenes

• Indoor simulation spans SUNCG, House3D, AI2-THOR, and Habitat for annotated scene understanding, navigation, embodied question answering, and interactive object manipulation.SUNCG contains over 45,000 annotated scenes, while House3D supports large-scale reinforcement learning and natural-language navigation.Source: §3.2 · pp. 35–37
• Indoor rendering studies find that increased geometric, material, camera, and lighting realism produces significant quality gains, while SUNCG’s legal controversy illustrates synthetic data’s intellectual-property risks.Synthetic data can avoid some real-data legal issues, but manually created or collected 3D assets may remain restricted.Source: §3.2 · pp. 35, 37
• Habitat-Sim renders thousands of frames per second, 10-100x faster than previous simulators, and supports synthetic and real datasets for scalable embodied-AI training.Its study found reinforcement-learning agents outperform SLAM-based agents and generalize best across datasets, including SUNCG to Matterport3D and Gibson.Source: §3.2 · p. 37

3.3 Robotic simulators

• Robotic simulators accommodate constrained hardware through ROS integration and sometimes hardware-in-the-loop control, with Gazebo and MuJoCo serving as major research platforms.Gazebo combines physics, lighting, and hardware integration and often supports transfer to real robots; MuJoCo specializes in contact-rich behaviors.Source: §3.3 · p. 38
• Dactyl’s reinforcement-learning policies were trained entirely in ORRB simulation and successfully transferred to the physical robot.ORRB renders robotic environments with depth and segmentation, emphasizes diversity through domain randomization, and supports distributed cloud environments; VRGym adds VR human input, multisensor body sensing, and ROS integration.Source: §3.3 · pp. 38–39

3.4 Vision-based applications in unmanned aerial vehicles

• UAV research commonly uses synthetic datasets or simulators for localization, obstacle detection and avoidance, visual servoing, and landing-site selection because real-world training or testing can be expensive or restricted.For landing-site selection, Castagno et al. [88] used a high-fidelity visual synthetic model of Manhattan rooftops rendered with the Unreal Engine and provided to simulated robots via AirSim.Source: §3.4 · pp. 39–40
• Air Learning benchmarks showed that more onboard compute produced almost 2x shorter trajectories in point-to-point obstacle avoidance.Air Learning combines pluggable environments and physics engines, learning algorithms, robot-control policies, and hardware-in-the-loop flight controllers.Source: §3.4 · p. 42
• Synthetic pretraining followed by fine-tuning on a small real dataset improved UAV wire detection using rendered wires over real background images.Madaan et al. varied wire material, sag, and camera angle without adapting wires to background semantics.Source: §3.4 · p. 42

3.5 Computer games as virtual environments

• An RL navigation agent trained in Doom transferred to a real Turtlebot by freezing most DQN weights and fine-tuning only a small subset.Karttunen et al. demonstrate transfer from computer-game environments to real-world navigation, while procedurally generated Atari levels improved generalization to unseen levels.Source: §3.5 · p. 43

4.1 Synthetic data for neural programming

• Synthetic data is indispensable for neural programming because large program datasets do not exist, so models are trained on automatically generated programs, inputs, and outputs [143, 308].Program induction and synthesis use randomly generated sequences, executable programs, abstract syntax trees, and simple tasks such as copying, sorting, and arithmetic.Source: §4.1 · pp. 44–45
• Shin et al. found that biased program-generation algorithms miss important regions of program space and proposed dataset-distribution generation that significantly improved Calculator and Karel results.The findings show that synthetic-data generation quality, not merely model architecture, materially affects neural program induction and synthesis.Source: §4.1 · p. 45

4.2 Synthetic data in bioinformatics

• Synthetic biological data supports drug and molecule generation, cell-image modeling, and experimental genomics, including a synthetic-sequence approach that outperformed deep learning models on in vivo alternative-splicing prediction.Generative models produce candidate molecules and plausible cell images, while synthetic DNA libraries measured through massively parallel reporter assays improved prediction quality over models trained on the human genome.Source: §4.2 · pp. 46–47

4.3 Synthetic data in natural language processing

• Synthetic data has seen little NLP use because controllable meaningful text generation remains difficult, and increasingly capable generative models have not produced significant training use beyond restricted domains such as electronic medical records [216].NLP augmentation includes word deletion, replacement, synonym substitution, and noising, while GAN-based methods address discrete text through reinforcement learning, continuous relaxations, or latent-space generation.Source: §4.3 · pp. 47–49

5.1 Domain randomization

• Domain randomization improves synthetic-to-real transfer by broadening scene and rendering variability, with reported gains for several detectors and successful transfer of grasping models to realistic objects.Randomization spans objects, textures, cameras, lighting, rendering, and noise; Tremblay et al. improved R-FCN and SSD but not Faster R-CNN relative to Virtual KITTI, while Tobin et al. transferred grasping from procedurally generated objects.Source: §5.1 · pp. 50–51

5.2 Improving CGI-based generation

• CGI-based generation can improve realism through deformable objects, sensor-noise simulation, and learned scene composition that aligns synthetic scenes with real data or task objectives.Meta-Sim uses scene graphs and probabilistic scene grammars to learn urban-environment transformations, while other methods model object deformation, camera noise, depth noise, and domain-specific dynamics.Source: §5.2 · pp. 51–52

5.3 Compositing real data to produce synthetic datasets

• Compositing real images can expand training data, but preserving visual context is crucial: context-aware placement and same-class instance switching improve detection, including across multiple MS COCO detectors.Random placement can degrade VOC’12 detection, whereas a context-prediction model improves it; PSIS preserves context, shape, and scale while rebalancing class distributions.Source: §5.3 · pp. 53–54

5.4 Synthetic data produced by generative models

• Generative models are most useful when conditioned on structure or labels: they improve leaf counting and handwritten-text recognition, whereas generated driving videos preserve road structure without yet improving driving-agent training.Conditions include object counts, segmentation masks, character sequences, and vocabulary; purely from-scratch generation remains difficult for classical vision applications.Source: §5.4 · pp. 54–55

6 Synthetic-to-real domain adaptation and refinement

• The survey divides synthetic-to-real adaptation into data-level refinement that produces improved synthetic data and feature/model-level methods that leave the data unchanged.The section focuses mainly on recent deep-neural-network approaches.Source: §6 · p. 56

6.1.1 Case study: GAN-based refinement for gaze estimation

• UnityEyes increased gaze-estimation robustness through randomized environmental variability, while SimGAN refined synthetic eye images using adversarial realism and similarity losses that preserve gaze direction.UnityEyes rendered each image in a different randomized environment, contrasting with MPIIgaze’s 15 participants; SimGAN used a refiner and discriminator with a target-preserving similarity objective.Source: §6.1.1 · p. 57
• GazeGAN extended refinement to unpaired synthetic-to-real translation with cycle and gaze-cycle consistency, refining 128 × 128 color images and surrounding facial regions rather than only 30 × 60 grayscale eyes.Its losses preserve gaze direction while translating between synthetic and real domains.Source: §6.1.1 · pp. 58–59

6.1.2 Refining synthetic data with GANs

• Refinement methods preserve task-relevant structure through content, cycle-consistency, foreground/background, or SSIM-based losses, adapting translation to recognition, depth, vending-machine detection, and crowd-counting requirements.Examples include mask-based content preservation in PixelDA, minimal translation and depth preservation in T2Net, separate foreground/background losses for vending machines, and SSIM for local crowd-counting patterns.Source: §6.1.2 · pp. 60–63
• PixelDA improved classification and pose estimation over purely synthetic training and prior adaptation methods, while T2Net achieved good depth estimation without paired real data and favored end-to-end training.T2Net also reportedly outperformed some, but not all, supervised approaches.Source: §6.1.2 · pp. 60, 62

6.1 Synthetic-to-real refinement

• GAN-based refinement improved downstream performance across gaze estimation, object classification and pose estimation, depth estimation, and object detection, though gains were not uniformly superior to real-data augmentation.Reported benefits include improved classification and pose metrics, good depth estimation without paired real data, and gains over basic synthetic data across three detection architectures; affine augmentation of real images remained the clear winner in one gaze study.Source: §6.1.1 · p. 59, §6.1.2 · pp. 60, 62, 63

T and DSSIM

• For person re-identification, Bak et al. selected the nearest synthetic illumination domain, translated it toward the target domain with CycleGAN, and fine-tuned the network, outperforming each individual pipeline component.The approach used M real source domains and N synthetic domains with N ≫ M.Source: T and DSSIM · p. 64
• CycleGAN-based adaptation improved sim-to-real policy transfer from a non-differentiable physics simulator to a real robotic platform.The model learned a transformation between simulator observations and real robotic data.Source: T and DSSIM · p. 64

6.1.3 Making synthetic data from real with GANs

• Conditional GANs can augment real datasets or fill missing regions of their distributions, including frontalization and pose balancing for face recognition [264].TP-GAN uses global and local face pathways with pixel-wise, symmetry, identity-preserving, and adversarial objectives, while DA-GAN uses adversarial, identity, and pose-preserving losses.Source: §6.1.3 · pp. 64–66
• DA-GAN refined real-face pose distributions and won both face verification and identification tracks in the large-scale NIST IJB-A competition.The model was designed to fill underrepresented viewing-angle regions, particularly where 3D simulation quality degrades at large yaw angles.Source: §6.1.3 · pp. 65–66
• Other data-level methods map synthetic and real images into a common pseudo-synthetic space or canonical representation, including VAE-based alignment and randomized-to-canonical robotic grasping [273].RCAN uses segmentation and depth alignment without cycle-consistency losses, while VR-Goggles uses two generators to translate between synthetic and real image domains.Source: §6.1.3 · pp. 66–67, S (xS), f VRG · p. 68

S (xS), f VRG

• VR-Goggles combines CycleGAN/CyCADA-style translation with GAN, semantic, and shift losses, and significantly improved performance on the CARLA navigation benchmark [153].The semantic loss preserves segmentation results during translation, while the shift loss encourages translation invariance to image shifts.Source: S (xS), f VRG · pp. 67–68, S , DVRG · p. 67, §6.1.3 · p. 67

6.2 Domain adaptation at the feature/model level

• Feature- and model-level adaptation changes representations or weights instead of generating realistic images, using weight regularization, real-data priors, pyramid consistency, or adversarial gradient reversal [190,191,504,698,699].The framework of Ganin and Lempitsky trains a task predictor while reversing gradients from a domain classifier to encourage domain-invariant features; curriculum methods estimate real label distributions from image features.Source: §6.2 · pp. 69–70, Figure 21 · p. 70
• Feature/model-level adaptation separates shared and domain-private representations, combining task, reconstruction, difference, and similarity losses to promote domain-invariant features.Domain separation networks use shared and private encoders, reconstruction constraints, orthogonality between shared and private features, and adversarial or MMD-based similarity alignment.Source: §6.2 · pp. 71–72, S (xS) and EDSN · p. 72
• Cross City Adaptation achieves noticeable segmentation gains when adapting SYNTHIA models to Cityscapes, while Structural Adaptation significantly improves urban-scene segmentation from SYNTHIA or GTA to Cityscapes.Feature-level methods align source and target representations using global, class-wise, or adversarial objectives; Structural Adaptation instead transforms low-level synthetic features toward real ones.Source: S (xS) and EDSN · pp. 73–74

6.3 Domain adaptation for control and robotics

• Sim-to-real control transfers include progressive-network reinforcement learning, image refinement, weakly paired distribution alignment, and jointly learned image translation and control.Progressive networks trained first in MuJoCo improved over simple fine-tuning on a real Jaco arm, while Bewley et al. report excellent end-to-end driving results and real-world policy experiments.Source: §6.3 · pp. 74–75, S and Dwve · p. 76
• Real-robot studies demonstrate transfer beyond image quality: Zhang et al. deploy simulated navigation policies on robots, while James et al. achieve better grasping with 5,000 real episodes than with 580,000.The latter represents a more than 99% reduction in required real-world input.Source: S and Dwve · p. 76

6.4 Case study: GAN-based domain adaptation for medical imaging

• Medical imaging motivates GAN-based adaptation because expert annotation, privacy constraints, and scarce pathology data limit labeled datasets; synthetic generation and refinement improve downstream results.Reported gains include better classification from synthetic liver-lesion images than standard augmentation, improved segmentation after adding synthetic image-mask pairs, and improved endoscopy depth estimation by making real images synthetic-like.Source: §6.4 · pp. 77–78
• Bailo et al. [31] generated realistic blood-cell images from randomized segmentation masks, improving FCN segmentation and Faster R-CNN detection when combining synthetic with real data.Their real dataset contained 60 annotated 1920 × 1200 RGB images, plus 40 test images, averaging 669 annotated cells per image.Source: §6.4 · pp. 78–79
• Zhao et al. used FILA-sGAN to generate highly realistic filamentary images from segmentation maps and one style image, improving state-of-the-art segmentation across multiple styles.The approach combines content, style, and total-variation losses to preserve filamentary structure while controlling texture.Source: §6.4 · p. 79
• Other studies refined or generated histopathology, CT, and chest-X-ray images, reporting improved nucleus, glioma, lymph-node, and chest-X-ray segmentation, while 3D GAN adaptation supported lung-nodule detection.The cited methods include GAN refinement, pix2pix, MUNIT, and 3D GAN-based data synthesis.Source: §6.4 · pp. 79–80

A and Svol

• Zhang et al. learned unpaired CycleGAN-based 3D CT-to-MR volume translation and augmented it with segmentors to preserve segmentation-mask consistency.The model translates between CT and MR domains using volume-to-volume mapping for unpaired datasets.Source: §6.4 · p. 80, A and Svol · p. 80

A and Dvol

• A and B use discriminators trained with the standard GAN loss Lvol to distinguish real images from synthetic images generated by Gvol.The discriminators are denoted by A and B for their respective domains.Source: A and Dvol · p. 80
• Lvol cyc is the cycle-consistency loss.Source: A and Dvol · p. 80

• B : B → Y denotes segmentors that produce 3D segmentation masks.Source: A , Gvol · p. 80
• L_vol includes a shape consistency loss, denoted L_shape.Source: A , Gvol · p. 80

B log Svol

• 3D segmentation improves over both real-only training and standard fine-tuning of Svol B in Zhang et al.’s architecture.Source: B log Svol · p. 80

A and Svol

• Medical imaging uses cross-modal synthesis and synthetic-to-real adaptation for lesion detection, localization, segmentation, and domain transfer across imaging modalities [51].Methods include image-to-image translation and feature-level domain adaptation with instance weighting or domain-discriminating representations.Source: A and Svol · pp. 80–81
• GAN-based medical-imaging methods are promising, but differing architectures and limited cross-method comparisons leave their relative effectiveness unresolved.The survey calls for large-scale experiments across domain-adaptation problems.Source: A and Svol · p. 81

7.1 Differential privacy in deep learning

• Differential privacy protects individuals by limiting what outputs reveal about any single training record, while offering composability, group privacy, and robustness to auxiliary information.DP-SGD clips per-example gradients and adds Gaussian noise, providing architecture-agnostic privacy under suitable parameters.Source: §7.1 · pp. 82–83

7.2 Differential privacy guarantees for synthetic data generation

• Private synthetic-data release lets third parties access, pool, trade, or publicly release data without transferring sensitive records, supporting collaboration and reproducibility.Classical approaches include query perturbation, PrivBayes, and DataSynthetizer, which preserve statistics or structure while providing differential privacy.Source: §7.1 · pp. 82–83, §7.2 · p. 84
• Differentially private GANs protect the discriminator, commonly through noisy gradients or layers, but privacy noise generally reduces downstream utility and makes high-resolution training difficult.DPGAN approaches reach near-real classifier accuracy on electronic health records, whereas performance can degrade substantially on harder or higher-dimensional datasets.Source: §7.2 · pp. 84–85
• Empirical-DP GAN modifications can achieve practical privacy–accuracy tradeoffs, with a randomizing discriminator layer preserving generation quality through datasets up to CelebA [597].The layer may also act as a regularizer and improve generation quality.Source: §7.2 · p. 85

7.3 Case study: synthetic data in economics, healthcare, and social sciences

• Applications span multiply imputed finance and social-science datasets, synthetic healthcare populations and records, clinical text, and robust financial-strategy evaluation, but forecasting gains remain unsupported.Synthetic financial data is used to test strategies under altered abnormalities for regulatory robustness, while medical time-series generation shows useful synthetic classification but significant synthetic-to-real quality drops [239].Source: §7.3 · pp. 87–88

8.1 Procedural generation of synthetic data

• Learning probabilistic procedural generators could scale synthetic data beyond manually created scenes and support structured indoor environments and fully procedural cities.SUNCG’s manually created scenes cannot scale to millions of 3D environments, while procedural models can learn from real or manually prepared data and generate new samples.Source: §8.1 · p. 89

8.2 From domain randomization to the generation feedback loop

• The generation feedback loop can be extended across augmentation and simulation levels, including automated transformation search, task-driven scene generation, and reinforcement-learned simulator policies.MCTS-based augmentation improves sim-to-real robotic manipulation, Meta-Sim tunes scene structure for detection, and related methods learn simulator parameters to maximize validation accuracy.Source: §8.2 · pp. 90–91
• Automatically learning synthetic-data generation policies from real feedback can outperform regular domain randomization for synthetic-to-real transfer.VADRA learns rendering parameters using downstream performance and domain-classifier errors, while [97] learns simulation-parameter distributions from real observations; adversarial deception networks also show comparable state-of-the-art performance and better generalization.Source: §8.2 · pp. 90–91

8.3 Improving domain adaptation with domain knowledge

• Combining domain-informed probabilistic models with GANs enables bidirectional synthesis and inference while improving gaze estimation and the usefulness of generated training data.Wang et al. combine an anatomical 27-landmark eye-shape model with a conditional BiGAN, improving the model over for sufficiently large training sets and improving standard gaze estimators with generated data.Source: §8.3 · p. 92

8.4 Additional modalities for domain adaptation architectures

• Using synthetic-only depth and 3D modalities can strengthen domain adaptation by jointly refining images and aligning segmentation and depth outputs.GIO-Ada [109] takes synthetic images, segmentation maps, and depth maps as input, adds output-level adaptation, and reports promising Virtual KITTI-to-KITTI and SYNTHIA-to-Cityscapes results, though this remains an initial step.Source: §8.4 · pp. 92–94

• Synthetic data provides virtually limitless, perfectly labeled data and has been explored across many deep learning problems, especially computer vision.The survey identifies substantial potential for additional use cases beyond those already studied.Source: §9 · p. 95
• The survey covers applications from basic vision tasks to simulated autonomous-driving, aerial, and robotic environments, alongside synthetic-to-real adaptation, improved generation, and differential privacy.It reviews both generative models for synthetic-to-real refinement and feature-level domain adaptation, as well as privacy-preserving data generation.Source: §9 · p. 95
• Synthetic data works across diverse tasks because it can address costly or impossible labeling, incomplete real-data coverage, and missing modalities, with emerging applications including sound and speech processing.The survey expects its importance to grow as synthetic data becomes more realistic and encompasses more use cases and modalities.Source: §9 · p. 95