PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Datasets for Autonomous Driving: A Survey

Zhihang Song, Zimin He, Xingyu Li, Qiming Ma, Ruibo Ming, Zhiqi Mao, Huaxin Pei, Lihui Peng, Jianming Hu, Danya Yao, Yi Zhang

arXiv:2304.12205v2eess.IVcs.CV

TL;DR

Real-world datasets are costly to collect and label, and synthetic data can complement them with controllable, changeable data. This survey reviews autonomous-driving synthetic datasets and proposes evaluation and generation approaches; it reports their value as supplements that can improve algorithm performance.

  • Problem

    Real-world datasets require expensive experiments and labeling, while changing requirements make it difficult for them to keep pace.

    Source: 2) Interactive elements · p. 9

  • Method

    The survey reviews single-task and multi-task synthetic datasets and proposes a dataset-evaluation framework and a generation process with evaluation feedback.

    Source: §I · pp. 2–3, §VIII · p. 14

  • Results

    The survey reports that synthetic datasets can effectively supplement real-world data and improve algorithm performance.

    Source: 2) Interactive elements · p. 9

  • Takeaways & Limitations

    The paper emphasizes synthetic datasets’ relevance to trustworthiness and safety in autonomous driving and discusses promising directions for generation.

    Source: §VIII · p. 14

    Synthetic datasets may not adequately represent real-world complexity, and the domain has few datasets for driver behavior analysis and decision learning.

    Source: §VII · p. 13

Abstract

from arXiv · show

Autonomous driving techniques have been flourishing in recent years while thirsting for huge amounts of high-quality data. However, it is difficult for real-world datasets to keep up with the pace of changing requirements due to their expensive and time-consuming experimental and labeling costs. Therefore, more and more researchers are turning to synthetic datasets to easily generate rich and changeable data as an effective complement to the real world and to improve the performance of algorithms. In this paper, we summarize the evolution of synthetic dataset generation methods and review the work to date in synthetic datasets related to single and multi-task categories for to autonomous driving study. We also discuss the role that synthetic dataset plays the evaluation, gap test, and positive effect in autonomous driving related algorithm testing, especially on trustworthiness and safety aspects. Finally, we discuss general trends and possible development directions. To the best of our knowledge, this is the first survey focusing on the application of synthetic datasets in autonomous driving. This survey also raises awareness of the problems of real-world deployment of autonomous driving technology and provides researchers with a possible solution.

I. INTRODUCTION

Real-world data collection is costly and offers limited control over conditions, motivating synthetic data as a complement. The survey reviews synthetic-dataset methods and categories for autonomous-driving perception, and proposes evaluation and generation approaches focused on trustworthiness.

I. INTRODUCTION

  • Real-world datasets require costly experiments and labeling, while weather and corner cases are difficult to control during collection.Synthetic data can control environmental factors, scenarios, and data distributions that may be difficult or impossible to achieve in real-world collection.Source: §I · p. 1
  • The survey traces early synthetic-data uses and notes that access to movie-based source data is limited by licensing permissions.FRIDA was introduced in 2010 for foggy-weather depth estimation, and MPI Sintel followed in 2012 for optical flow estimation.Source: §I · pp. 1–2
  • Synthetic dataset methods progressed from splicing 3D models into images to game-engine data capture, GAN-based image generation, and simulation platforms such as Carla [34].Early splicing had low fidelity and simple scenes, while early GAN image generation could have deformed or abnormal details that hindered automated annotation.Source: §I · p. 2
  • The survey selects datasets for recognition, adoption, relevance to autonomous-driving perception, ground-truth availability, and diversity of driving scenarios.It aims to describe their tasks and generation challenges and provide guidance for evaluation and generation.Source: §I · p. 2
  • The survey covers single-task and multi-task synthetic datasets for autonomous-driving perception, and proposes an evaluation framework and a generation process with evaluation feedback.It is presented as the first survey focused on synthetic datasets used in autonomous driving.Source: §I · pp. 2–3

II. MOTIVATION

Real-world datasets are expensive to build and update, offer limited control over changing conditions, and underrepresent safety-critical long-tail scenarios. Synthetic data can provide controllable scenarios and labels, with cost-effectiveness improving as data volume grows.

II. MOTIVATION

  • One image averages 15 to 30 labeled segments, and labeling one segment by an individual worker averages 0.86 USD.The paper notes that high labeling costs contribute to long update and iteration cycles for real datasets.Source: §II · p. 3
  • Real-world collection makes it difficult to obtain controllable changes in weather, lighting, and other domain-shift conditions for model studies.Synthetic data is described as a resource for conducting comprehensive studies under such conditions.Source: §II · p. 3
  • Routine driving dominates real-world datasets, leaving challenging safety-critical scenarios such as accidents and illegal driving too rare for system testing.Synthetic datasets can formulate safety-relevant scenarios to specific requirements; scenario engineering is used to generate challenging synthetic scenarios.Source: §II · p. 3
  • Synthetic data can generate accurate labels automatically and support efficient updates, while its cost-effectiveness tends to improve as its volume increases.The paper notes that the initial cost may exceed capturing and annotating a single real image.Source: §II · p. 3

III. SINGLE-TASK SYNTHETIC DATASETS

The section surveys and compares single-task synthetic datasets, including early datasets for foggy-weather depth estimation and optical flow. It also describes their generation characteristics and use in perception research.

III. SINGLE-TASK SYNTHETIC DATASETS

  • The section overviews and evaluates single-task datasets, with samples in Fig. 2 and a comparison in Table I.The datasets discussed include FRIDA and MPI Sintel, among others.Source: §III · p. 3, Figure 2 · p. 3, Table I · p. 4

A. FRIDA

  • FRIDA uses a 3D urban environment and simulated fog, pairing each clear image with four foggy versions and a depth map.FRIDA2 expands the collection to 330 synthetic images across 66 road scenes; clear and foggy pairs also support visibility-enhancement and dehazing evaluation [44],.Source: A. FRIDA · pp. 3–4

B. MPI Sintel

  • MPI Sintel provides synthetic stereo images for optical-flow estimation, but it often ignores physical constraints, warranting caution for algorithms that rely heavily on real-world physical laws.The dataset contains 1,064 synthetic stereo images and real-world data from 23 scenes; it is widely used for optical-flow tasks because it represents natural scenes and motion well.Source: B. MPI Sintel · p. 4

C. Flying Things

These datasets use rendering and image synthesis to provide driving-task annotations, with reported benefits for model training, segmentation in fog, and domain adaptation.

C. Flying Things

  • Flying Things contains over 25,000 stereo image pairs with disparity, optical-flow, and scene-flow ground truth, and can train large convolutional networks for challenging tasks.The related Driving dataset uses the same vehicle models in a naturalistic street scene with a KITTI-like viewpoint.Source: C. Flying Things · p. 4

D. GTA-V dataset

  • Adding GTA-V data to CamVid or KITTI can greatly improve model training accuracy while reducing expensive manual annotation.GTA-V generates pixel-level labels by hashing rendering resources, without boundary tracking.Source: D. GTA-V dataset · p. 4

E. SYNTHIA

  • SYNTHIA provides over 213,400 images with pixel-level annotations for 13 categories, and models trained on it then fine-tuned on real data achieve improved semantic accuracy.SYNTHIA-SF adds depth and 19 semantic classes, while SYNTHIA-AL supports active-learning evaluation in road scenes,.Source: E. SYNTHIA · p. 5

F. VEIS

  • VEIS renders instance-level segmentation maps in real time, and its framework combines separately processed foreground and background segmentation results.The dataset contains 61,305 frames of multi-class and single-class scenes.Source: F. VEIS · p. 5

G. Foggy Cityscapes

  • Foggy Cityscapes adds optical-model fog to Cityscapes images with fine-grained labels; it has improved segmentation and detection in challenging fog in fully and semi-supervised settings.It can also be applied to real-time rapid defogging systems.Source: G. Foggy Cityscapes · p. 5

H. IDDA

The section describes datasets that broaden synthetic driving data across tasks and scenarios, and reports their use in domain-transfer assessment and odometry testing.

H. IDDA

  • IDDA provides an evaluation benchmark for domain-transfer methods, while tests on real-world data show a remaining performance gap and some proximity to real-world scenarios.Feature embeddings were used to quantify the disparity between IDDA and real-world datasets.Source: H. IDDA · p. 5

I. CarlaScenes

  • CarlaScenes covers seven difficult-to-measure driving scenarios with camera, LiDAR, IMU, and GNSS data for odometry; three monocular methods were found unsuitable for these scenarios,,.The authors suggest that deep-learning odometry methods may measure mileage accurately with proper training and validation.Source: §I · p. 5

IV. MULTI-TASK SYNTHETIC DATASETS

  • Multi-task synthetic datasets use virtual engines such as Unity and Unreal Engine to provide simulated sensors and annotations across tasks, sensor suites, and domain shifts.The section presents dataset samples in Fig. 3 and a comparison in Table II.Source: §IV · pp. 5–6, Figure 3 · p. 6

A. Virtual KITTI

  • Virtual KITTI created 35 synthetic videos from five KITTI sequences, with multi-task annotations and shifts in camera angle, lighting, and weather; it also proposed a practical definition and experiments for cross-domain effectiveness and transferability.The videos contain about 17,000 high-resolution frames.Source: A. Virtual KITTI · p. 6

B. VIPER

VIPER, PreSIL, and ParallelEye extend synthetic driving data across sensors and annotations, with ParallelEye also tested for domain shift against real datasets.

B. VIPER

  • VIPER provides 254,064 high-resolution frames with camera, GPS, and IMU data and annotations spanning detection, tracking, optical flow, segmentation, odometry, and pose; it has also been used with real data in training and domain adaptation,.Its domain shifts include five environmental conditions and varied scene types.Source: B. VIPER · p. 6

C. PreSIL

  • PreSIL combines GTA-V-based LiDAR point clouds with RGB imagery and annotations for image and point-cloud tasks, but lacks reflectance values, which can negatively affect point-cloud segmentation and detection.It also supports domain-adaptation benchmark work alongside synthetic and real datasets.Source: C. PreSIL · p. 6

D. ParallelEye

  • ParallelEye domain-shift tests against KITTI and Cityscapes were used to demonstrate its usefulness, and the work underpinned a later image-generation pipeline and ParallelEye-CS visual-intelligence testing,.Source: D. ParallelEye · p. 6

E. Virtual KITTI2

The section surveys multi-task datasets spanning richer sensor suites, varied domain shifts, and collaborative perception, including benchmark uses on synthetic and real data.

E. Virtual KITTI2

  • Virtual KITTI2 reuses five real KITTI sequences with more realistic image details, adds a second-view camera, and provides binocular RGB and depth images for multiple perception tasks.Later work has used Virtual KITTI1&2 for 3D detection, depth completion, and 3D segmentation,, [80],.Source: E. Virtual KITTI2 · p. 7

F. SHIFT

  • SHIFT supplies 4,800+ sequences and about 2.5M labeled images across 13 perception tasks, with scene, weather, time, traffic, and crowd-density shifts uniformly distributed across variables.It provides a benchmark and experiment basis for domain-shift algorithms.Source: F. SHIFT · p. 7

G. V2X-Sim

  • V2X-Sim provides an open-source testbed for collaborative perception algorithms, with synthetic data and benchmarks for detection, tracking, and segmentation intended to stimulate V2X algorithm improvement.Source: G. V2X-Sim · p. 7

H. AIODrive

  • AIODrive targets high-density, long-distance point clouds with eight sensor types, including the first SPAD-LiDAR in a public autonomous-driving dataset, and includes adverse weather and special scenes.It provides about 100K labeled images and annotations for detection, tracking, prediction, instance segmentation, and depth estimation.Source: H. AIODrive · p. 7

I. OPV2V

  • OPV2V supports collaborative 3D detection, BEV semantic segmentation, tracking, and prediction, with a benchmark of up to 16 models for fusion strategies and radar target detection.It contains 11,464 frames and 232,913 annotated 3D vehicle boxes; collaborative-perception studies also use it with V2X-Sim and real-world DAIR-V2X,,,.Source: §I · p. 8

V. COMPREHENSIVE EVALUATION ON SYNTHETIC DATASETS

The section proposes evaluating synthetic datasets through static dataset qualities and interactive effects on algorithms, while noting that no generally accepted evaluation theory exists. Static quality is organized around form, content, and utility, with acquisition and processing as ways to assess or improve it.

A. Evaluation and Comparison

  • The proposed evaluation groups dataset assessment into static and interactive elements, but the paper notes that no generally accepted theory defines evaluation criteria for synthetic autonomous-driving datasets.Interactive elements concern how datasets affect training, testing, model improvement and comparison, and system evaluation.Source: A. Evaluation and Comparison · p. 8, §V · p. 8

1) Static elements:

  • Static evaluation covers formal quality, content quality, and utility quality, including data validity and coherence, scenario coverage and balance, and relevance and adaptability to real-world objectives.Adaptability includes editability, scalability, and reconstructibility.Source: 1) Static elements · pp. 8–9
  • Data acquisition assessment examines task-relevant completeness, label richness, and scene balance using dataset descriptions, sensor suites, and domain shifts.Some properties, including accuracy and coverage, require manual collation or may be reflected through application to specific machine-learning tasks.Source: 1) Static elements · p. 9
  • Data cleaning can remove harmful samples such as label errors, while augmentation modifies or fuses samples to improve data quality in data-constrained settings.The section notes that static-element quality can also be improved at the data-processing level.Source: 1) Static elements · p. 9

2) Interactive elements:

Interactive evaluation examines how synthetic datasets affect algorithm performance, transferability, and domain-shift research. The section also frames the synthetic-to-real gap as differences in appearance and content.

2) Interactive elements:

  • Synthetic-data training followed by testing in a real environment is a direct effectiveness test, and Virtual KITTI comparisons show synthetic data can supplement real data and improve algorithm performance.Virtual KITTI compares real-data training, synthetic-data training, and synthetic pretraining followed by real-data fine-tuning.Source: 2) Interactive elements · p. 9
  • Virtual KITTI shows limited multi-object-tracking performance gaps between real and cloned sequences, while SHIFT compares model-performance trends with BDD100K under matched discrete domain shifts.Source: 2) Interactive elements · p. 9
  • SHIFT tests perception algorithms and calibration under discrete and continuous domain shifts, illustrating synthetic datasets’ use as platforms for domain-adaptation research and the need for fine-grained domain-shift coverage.The calibration-error estimates use Softmax, MCDO, and Deep Ensembles.Source: 2) Interactive elements · p. 9

B. Gap Discussion

  • The synthetic-to-real gap has appearance differences such as color and texture, and content differences such as task-label distributions and scene layout.Source: B. Gap Discussion · pp. 9–10

1) Appearance gap:

The section discusses ways to bridge synthetic-to-real differences and generate intermediate data from real-scene imagery and point clouds. It also describes migrating reconstructed objects into outdoor datasets to enrich their coverage.

1) Appearance gap:

  • Higher-quality renderers, realistic sensor models, and detailed 3D objects can reduce appearance differences; proposed approaches also include domain randomization, image translation, and unsupervised domain adaptation.DRIVE Sim is described as using RTX path tracing to generate physically accurate camera, radar, LiDAR, and ultrasonic data.Source: 1) Appearance gap · p. 10

2) Content gap:

  • Content-gap methods include self-training with optimized pseudo-labels and automatic scene generation, with pseudo-label iteration used to annotate target-domain data.Meta-Sim parameterizes a dataset generator and modifies scene-graph properties to reduce the distribution gap between generated and real data.Source: 2) Content gap · p. 10

3) Intermediary:

  • The proposed intermediary is high-fidelity virtual reconstruction or rendering from acquired data, intended to produce virtual data with a distribution close enough for models to transfer directly to real scenes.The section notes that synthetic pretraining followed by real-data fine-tuning can waste computation and produce less-than-optimal results.Source: 3) Intermediary · p. 10
  • RGB-D images can support new-view synthesis with NeRF, while outdoor LiDAR point clouds are sparse and lack semantic information, motivating semantic scene completion.NeRF represents a scene as a continuous 3D function queried for color and opacity; point-cloud methods include S3CNet for sparse large-scene inputs.Source: 3) Intermediary · p. 10
  • Reconstructed desktop, indoor, pedestrian, and other objects can be migrated into outdoor synthetic scenes to enrich edge-case distributions and dataset-generation flexibility.The section cites ShapeNet objects as a way to add objects that do not normally appear in autonomous-driving scenes.Source: 3) Intermediary · p. 11

C. Trustworthiness and Safety

In the static-element evaluation, SHIFT scores higher overall than Virtual KITTI, though the datasets differ in availability, content, and utility. The section also discusses data-based trustworthiness and safety evaluation, then applies its quality framework to two datasets.

C. Trustworthiness and Safety

  • Data design and generation matter for trustworthiness because dataset biases can reduce model credibility; the section discusses ablation and Shapley valuation, uncertainty-based cleaning, and label aggregation.The paper also describes synthetic labels as reducing manual annotation cost and human bias error.Source: C. Trustworthiness and Safety · p. 11
  • Naturally collected data makes rare safety-critical scenarios costly to test, so the section discusses synthesizing targeted scenarios, including SafeBench’s 2,352 generated scenarios evaluated with ten metrics.Source: C. Trustworthiness and Safety · p. 11

VI. EXPERIMENTS

  • The experiments evaluate Virtual KITTI and SHIFT using ANP-derived indicator weights and criteria scores, combining formal, content, and utility quality into an overall dataset assessment.The described criteria include image-quality measures, scene-element distributions, redundancy, practical-task relevance, and performance degradation when training on one dataset and applying to another.Source: A. Static Element Evaluation · pp. 11–12

A. Static Element Evaluation

  • SHIFT’s overall dataset quality is higher than Virtual KITTI’s, while Virtual KITTI lacks pedestrians, cyclists, and nighttime data and synthetic datasets remain less scalable and relevant to real-world use.SHIFT is larger and may require user-directed weather changes, lowering its availability relative to Virtual KITTI.Source: A. Static Element Evaluation · p. 12

B. Interactive Element Evaluation and Safety Test

The evaluation tests Virtual KITTI’s transfer across datasets and conditions, then uses Cleanlab to identify label issues in synthetic and real data. Results show a domain gap, especially for BDD100K night scenes, while corrupted synthetic labels are detected in the test.

B. Interactive Element Evaluation and Safety Test

C. Trustworthiness Test

  • The trustworthiness test used separate Virtual KITTI- and KITTI-trained models to assess label quality in the two datasets.Cleanlab was adapted by using original labels to crop object patches before class-probability inference.Source: C. Trustworthiness Test · p. 12
  • Cleanlab reported 84 of 100 corrupted Virtual KITTI labels among 459 issues with a KITTI-trained model and all 96 corrupted labels with a Virtual KITTI-trained model.It also reported 709 label issues in 3,769 KITTI test images; many were misreported, and some were blurred, obstructed, misplaced, or wrongly labeled.Source: C. Trustworthiness Test · pp. 12–13

VII. DISSCUSIONS AND FUTURE DIRECTIONS

The discussion identifies realism and task coverage as limitations, and proposes a systematic generation process with evaluation feedback for iterative improvement. The conclusion surveys synthetic datasets and their evaluation, including trustworthiness and safety, and outlines future directions.

VII. DISSCUSIONS AND FUTURE DIRECTIONS

  • Synthetic datasets may not capture real-world complexity, including changing weather and dynamic traffic, and most focus on environmental perception rather than driver behavior or decision learning.The discussion calls for greater realism and diversity, including improved simulation and real-world calibration.Source: §VII · p. 13
  • The proposed generation process uses evaluation and model-use results, including on-road test data, as feedback for iterative dataset improvement.The process combines multi-evaluation, trustworthiness and safety tests with training and evaluation on targeted tasks.Source: §VII · p. 13
  • Future work should examine the synthetic-to-real gap, evaluate generated data’s usefulness, and expand coverage of corner cases and long-tail problems.Source: §VII · p. 14

VIII. CONCLUSION

  • The survey reviews synthetic dataset generation and single- and multi-task datasets, proposes two-aspect evaluation, and discusses usefulness in model training and bridging the synthetic-to-real gap.Source: §VIII · p. 14
  • The conclusion emphasizes synthetic datasets’ relevance to trustworthiness and safety and introduces possible directions for future generation work.Source: §VIII · p. 14

Related papers

Loading 2304.12205v2…