A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook

I. INTRODUCTION

• Autonomous-driving datasets span synthetic and real-world collection, multiple sensor modalities and task annotations, sensing domains, and varied geometry and weather.Source: §I · p. 1

A. Research Gap & Motivation

• Earlier surveys either covered relatively few datasets or collected many without detailed attribute analysis, while other reviews focused on particular dataset types or tasks.Source: A. Research Gap & Motivation · p. 2
• The survey systematically covers datasets across autonomous-driving tasks from perception to control, real-world and synthetic data, and the modalities and quality of key datasets.Source: A. Research Gap & Motivation · p. 2

B. Main Contributions

• It comprehensively records public datasets’ publication year, size, modalities, sensing domains, geometric and environmental conditions, and supported tasks.Source: B. Main Contributions · p. 2
• The survey explains sensing domains, sensors, and modalities, and describes autonomous-driving tasks’ goals, required data modalities, and evaluation metrics.Source: B. Main Contributions · p. 2
• It categorizes datasets by sensing domain and relevant task to help researchers identify and compile datasets for targeted research and development.Source: B. Main Contributions · p. 2
• It introduces an impact score for published perception datasets as a guide to future dataset development and analyzes high-impact datasets’ advantages and utility.Source: B. Main Contributions · p. 2
• The survey examines annotation quality and labeling procedures, details data distributions and their limitations and use cases, and discusses future directions including language integration, vision-language-model generation, standardization, and open data.Source: B. Main Contributions · p. 2

C. Scope & Limitations

• The survey covers datasets for perception, prediction, planning, control, and end-to-end driving, but discusses the most impactful datasets in depth.Versatile datasets are described only within the main scope they support, while a larger set is presented with its primary characteristics in tables.Source: C. Scope & Limitations · p. 3

D. Survey Structure

• The survey proceeds from dataset sourcing and sensors/tasks to dataset studies, annotation quality, distribution statistics, future directions, and conclusion.Its taxonomy is shown in Fig. 3.Source: D. Survey Structure · p. 3

II. METHODOLOGY

• The methodology has two parts: collecting and filtering datasets, then evaluating each dataset’s impact on autonomous driving.The paper details these parts in Sections II-A and II-B, respectively.Source: §II · p. 3

A. Datasets Collection

• The authors conducted a systematic review, searching multiple languages and sources and verifying candidate datasets through keyword searches and manual review.Searches covered Google, Google Scholar, Baidu, IEEE Xplore, Paperswithcode, relevant conferences, and GitHub repositories; queries included English, Chinese, and German terms.Source: A. Datasets Collection · p. 3

B. Dataset Evaluation Metrics

• The citation score combines total citations, ct, with average annual citations, ca, using earliest-version publication dates and counts collected as of March 05, 2024.Both measures undergo logarithmic transformation followed by Min-Max normalization to reduce citation-scale imbalance.Source: B. Dataset Evaluation Metrics · pp. 3–4
• The data-dimension score covers dataset size, temporal information, task number, and labeled categories, with task number spanning six perception tasks.Dataset size uses frame count; temporal information is binary, task number ranges from 1 to 6, and category levels are based on quintiles.Source: B. Dataset Evaluation Metrics · p. 4
• Environmental diversity scores weather, time of day, driving-scenario types, and geographic scope, assigning quintile levels and summing the four factors.Synthetic-dataset geography is undefined; missing values receive the median when diversity conditions are declared, and otherwise receive one.Source: B. Dataset Evaluation Metrics · p. 4
• The impact score assigns 60% to citation score and 40% to data-dimension and environmental-diversity scores.The metric is intended to assess dataset significance and guide preparation of new datasets.Source: B. Dataset Evaluation Metrics · pp. 3–4

• The section covers autonomous-driving sensors and their modalities, followed by data acquisition methods and cooperative perception technologies.Source: §III · p. 4

A. Sensor Modalities

• RGB cameras provide visual detail but are vulnerable to adverse conditions; LiDAR supplies precise 3D information but weakens with distance and fog, while coarse radar is weather-robust.These complementary strengths and limitations explain why radar often assists other sensors, while LiDAR suits tasks requiring concise 3D information.Source: A. Sensor Modalities · p. 5
• Event cameras capture fast motion at high temporal resolution; thermal cameras work in darkness but provide lower-resolution, non-color imagery, and IMUs track vehicle motion and orientation.Event cameras trigger on brightness changes, while thermal images are based on temperature differences.Source: A. Sensor Modalities · p. 5
• 52.79% of sensors are monocular cameras, 25.98% of datasets include LiDAR, and ten datasets use event cameras (2.79%).Stereo cameras appear in 29 datasets; radar, thermal, and fisheye cameras occur in 5.31%, 3.35%, and 1.68% of datasets, respectively.Source: A. Sensor Modalities · p. 5, Figure 6 · p. 5

B. Sensing Domains and Cooperative Perception Systems

• Autonomous-driving sensing domains include onboard, V2X, drone-based, and other sensing; onboard sensors provide direct vehicle feedback but have limited scope for blind spots and sharp bends.Onboard sensor suites typically include cameras, LiDARs, radars, and IMUs.Source: B. Sensing Domains and Cooperative… · p. 5
• V2X cooperation extends perception through vehicle, infrastructure, and network links, sharing sensor data and context to improve situational awareness and support cross-area updates.V2V shares information such as position, velocity, and sensor data; V2I incorporates roadside sensors and traffic infrastructure, while V2N can provide cloud data and traffic updates.Source: B. Sensing Domains and Cooperative… · pp. 5–6
• Drones provide aerial data for trajectory prediction and route planning, while other sensing covers data collected outside onboard, V2X, and drone domains.Drone data can also support traffic management and alert vehicles to accidents ahead.Source: B. Sensing Domains and Cooperative… · p. 6

IV. TASKS IN AUTONOMOUS DRIVING

• The survey frames perception and localization, prediction, and planning and control as pivotal tasks, detailing their objectives, data needs, and inherent challenges.The section situates these tasks within an autonomous-driving pipeline.Source: §IV · p. 6

2D/3D Object Detection

• 2D/3D object detection locates and classifies entities despite occlusion, varying light, and diverse appearances; Average Precision (AP) is usually used for evaluation.The passage also attributes an AP formulation to.Source: A. Perception and Localization · p. 6

2D/3D Semantic Segmentation

• 2D/3D semantic segmentation assigns semantic categories to image pixels or point-cloud points, with fine boundaries and extensive labeling posing dataset challenges; its main metrics are mPA and mIoU.Source: A. Perception and Localization · p. 6

Object Tracking

• Object tracking follows one or multiple objects over time using RGB, LiDAR, or radar sequences; MOTA combines false negatives, false positives, and mismatches, while AMOTA spans confidence thresholds.Source: A. Perception and Localization · p. 6

HD Map

• HD mapping constructs detailed representations of roads, signs, and landmarks; datasets need LiDAR for spatial information and camera data for visual detail, and map quality is usually assessed by accuracy [28].HD map automation and change detection have received increasing attention [66].Source: A. Perception and Localization · pp. 6–7

SLAM

• SLAM builds a map while localizing the vehicle, requiring camera data, IMUs, and real-time LiDAR point clouds; RPE and ATE evaluate estimated trajectories from RGB-D images.Source: A. Perception and Localization · p. 7

B. Prediction

• Prediction evaluation uses RMSE and Negative Log Likelihood (NLL), with NLL supporting comparisons of models’ trajectory uncertainty,,.NLL focuses on trajectory correctness.Source: B. Prediction · p. 7
• Trajectory prediction uses sensor time series to forecast the future paths of pedestrians, cyclists, and other vehicles.The passage identifies cameras and LiDARs as example sensors.Source: B. Prediction · p. 7
• Behavior prediction anticipates road users’ potential actions, such as lane changes, and requires extensive annotated data across varied scenarios.Source: B. Prediction · p. 7
• Intention prediction infers objects’ higher-level goals and requires perception-sensor data such as camera images, traffic signals, and hand gestures.Source: B. Prediction · p. 7

1) Planning:

• Hierarchical planning comprises path planning for long-term destination routes and behavior planning for decisions such as lane changes, overtaking, merging, and intersection crossing.Behavior planning depends on understanding and interacting with other agents’ behavior.Source: 1) Planning · p. 7
• Motion planning generates real-time trajectories for local objectives while accounting for obstacles, road conditions, and predicted behavior of other agents.Source: 1) Planning · p. 7

2) Control:

• Control executes the path or behavior decided by motion planning, corrects tracking errors, and translates high-level commands into throttle, brake, and steering commands.Source: 2) Control · p. 7

D. End-to-end Autonomous Driving

• End-to-end driving uses one learned model for perception through control, promising simplicity and adaptability but requiring extensive data and sacrificing interpretability and flexible modular tuning.Benchmarking is divided into closed-loop evaluation in simulation and open-loop comparison with expert behavior in real-world datasets.Source: D. End-to-end Autonomous Driving · p. 8

V. HIGH-INFLUENCE DATASETS

• The survey presents milestone datasets for perception, prediction, planning, control, and end-to-end autonomous driving.Source: §V · p. 8

A. Perception Datasets

• Perception datasets support reliability and robustness; the survey ranks collected datasets by impact, charts the top 50 chronologically, and presents 50 across sensing categories in Tab. II.The table is sorted by impact within each category, not as an overall top 50.Source: A. Perception Datasets · p. 8, Table II · p. 9

1) Onboard:

• KITTI, Cityscapes, and SemanticKITTI support varied perception tasks, but their geographic or environmental limits constrain real-world generalization.KITTI covers multiple sensors and tasks; Cityscapes provides pixel-level labels for 30 classes; SemanticKITTI contains over 43,000 LiDAR frames labeled across 28 categories.Source: 1) Onboard · p. 8
• VIPER offers over 250K synthetic video frames across weather, lighting, and urban scenarios, but transfer to real-world conditions remains challenging.Source: 1) Onboard · p. 8
• nuScenes uses six cameras and data from Boston and Singapore to capture varied driving behavior and urban layouts, though rare cases such as accidents need broader coverage.Source: 1) Onboard · pp. 8–9
• Waymo Open Dataset offers extensive, high-quality multimodal data across conditions and geographies, while BDD100K spans 100,000 videos, varied times and weather, and multiple tasks but has variable annotation quality.Source: 1) Onboard · p. 9
• Argoverse 2 [28], building on Argoverse 1, spans six cities and varied conditions with a broad taxonomy and supports detection, segmentation, and tracking; more adversarial cases and edge cases could extend its applications.Source: 1) Onboard · pp. 9–10

2) V2X:

• TUMTraf, –[86] supports cooperative perception with infrastructure and vehicle views, while DAIR-V2X addresses sensor asynchrony and transmission costs in vehicle–infrastructure cooperation.TUMTraf spans five releases and includes camera, event-camera, LiDAR, GPS, and IMU data; UAVDT adds 80,000 annotated frames for UAV object detection and tracking in challenging dense scenes.Source: 2) V2X · p. 10

3) Drone:

• DroneVehicle supplies 28,439 RGB–infrared image pairs for object detection under low illumination, across urban roads, residential areas, and parking lots.Source: 3) Drone · p. 10

4) Others:

• Pascal3D+ [97] adds 3D poses for 12 rigid-object categories, while Mapillary Vistas provides 25,000 street images labeled across 66 categories and varied conditions.Pascal3D+ may not represent dynamic scenes with pedestrians and other non-rigid objects; Mapillary includes instance annotations for 37 classes and diverse weather, times, and locations.Source: 4) Others · p. 10

1) Task-Specific Datasets:

• highD records trajectories of 110K vehicles in ideal-weather highway conditions, while exiD captures diverse interactions, especially lane changes at highway entries and exits.The highD weather restriction limits adversarial-weather use; extending exiD data to adverse weather and nighttime is an improvement direction.Source: 1) Task-Specific Datasets · pp. 10–11
• PIE provides over six hours of Toronto driving footage across lighting conditions, with pedestrian boxes, occlusion flags, crossing-intention confidence, and action labels.Source: 1) Task-Specific Datasets · p. 10
• Argoverse combines data from seven cameras, stereo imagery, LiDAR, and semantic maps with over 300K trajectories, though performance may be geographically constrained.Its recordings cover 290 km of mapped lanes and support 3D tracking and motion forecasting.Source: 1) Task-Specific Datasets · p. 10
• Around 1,500 hours from four cities make nuPlan the first closed-loop machine-learning planning benchmark, covering interactive traffic scenarios for realistic evaluation.Its scenarios include merges, lane changes, interactions with cyclists and pedestrians, and construction zones.Source: 1) Task-Specific Datasets · p. 10
• MONA contains 702K trajectories from 130 hours of video, with 0.51-meter average overall position accuracy, but its single-city coverage may limit generalizability.The data span urban roads, an inner-city highway stretch, and their transitions.Source: 1) Task-Specific Datasets · p. 11

2) Multi-Task Datasets:

• INTERACTION [20] combines complex driving scenes with a semantic map for motion prediction, imitation learning, and decision-and-planning validation across countries and continents.Environmental-condition impacts are not explicitly addressed.Source: 2) Multi-Task Datasets · p. 11
• rounD provides over six hours of 4K drone video recording more than 13K road users at roundabouts for scenario classification, behavior prediction, and driver modeling.Its clear-weather-only collection may restrict development of models for challenging situations.Source: 2) Multi-Task Datasets · p. 11
• Lyft Level 5 offers over 1,000 hours of data across 17,000 25-second scenes, supporting motion forecasting, planning, and simulation.Its detailed annotations and maps support prediction and planning, but the dataset may underrepresent uncommon traffic conditions and rare pedestrian behaviors.Source: 2) Multi-Task Datasets · p. 11
• LOKI supports multi-agent trajectory and intention prediction with diverse pedestrian and vehicle data, pairing camera images with LiDAR point clouds.Source: 2) Multi-Task Datasets · p. 11
• DeepAccident [12] is a synthetic dataset with 57K annotated frames and 285K samples that provides direct, explainable safety-evaluation metrics.It supports end-to-end motion and accident prediction and V2X perception tasks; varied environmental conditions aid generalizability, but real-driving performance needs further study because of domain adaptation.Source: 2) Multi-Task Datasets · pp. 11–12

C. End-to-end Datasets

• End-to-end driving is a growing alternative to modular architectures, with datasets such as nuScenes and Waymo and simulators such as CARLA enabling development.Some works introduce datasets specifically for end-to-end driving.Source: C. End-to-end Datasets · p. 12
• DDD17 [129] pairs APS images with DVS temporal-contrast events and records highway and city driving in varied weather for end-to-end algorithm training and testing.Source: C. End-to-end Datasets · p. 12

A. Annotation Generation

• Annotation needs vary by task, while pipelines range from professional manual labeling to semi-automatic refinement and fully automatic labeling.Detection uses boxes, segmentation uses pixel- or point-level labels, and trajectory prediction needs continuous trajectories.Source: A. Annotation Generation · p. 12, Figure 11 · p. 12
• Three-dimensional boxes encode object location, dimensions, and orientation, motivating sophisticated frameworks such as human-in-the-loop, weakly supervised, and dual-branch labeling,,.3D BAT supports semi-automatic 2D and 3D labeling.Source: A. Annotation Generation · p. 12
• Manual segmentation requires drawing object boundaries and filling or painting regions, making pixel- and point-level labels costly and inefficient to produce.Proposed alternatives include weakly supervised path segmentation, objectness-prior masks, interactive tools [142], and image-assisted or active-learning approaches,.Source: A. Annotation Generation · pp. 12–13
• Trajectory labeling maps object movement over time and usually relies on detection and tracking; proposed approaches include active learning and scalable machine annotation for pedestrian paths.Other methods add maneuver actions or aggregate crowdsourced labels with expert review [148].Source: A. Annotation Generation · p. 13
• Synthetic driving data offers an alternative to expensive real-world labeling because simulators know scene-object attributes and can automatically annotate generated scenarios.Open-source platforms include CARLA, SUMO, and AirSim; game engines such as GTA5 and Unity [156] are also used.Source: A. Annotation Generation · p. 13

VI. ANNOTATIONS PROCESS

• High-quality annotations underpin reliable autonomous-driving systems, and data quality plus active-learning curation can matter more for robust test performance than model-architecture tuning.Poor dataset quality can cause system errors and safety risks, undermining user confidence and trustworthiness.Source: B. Quality of Annotations · p. 13

B. Quality of Annotations

• Annotation quality depends on consistency, precision, correctness, and validation; expert or algorithmic review can catch errors before they affect vehicle performance [163],.Consistency is described as the foremost criterion, while correctness concerns alignment with dataset purpose and guidelines.Source: B. Quality of Annotations · pp. 13–14
• IPS300+ contains 319.84 labels per frame despite poor annotation quality, while nuScenes labeling took 7,937 hours and cost 100k USD [165],.The KITTI example also shows an imprecise car box and two clearly captured but unlabeled cars; several large datasets are described as having high-quality annotations.Source: B. Quality of Annotations · p. 14

VII. DATA ANALYSIS

• The analysis frames dataset evidence through worldwide collection locations, chronology, and data distributions, and reports growing dataset variety over time.The chronological overview covers perception datasets with the top 50 impact scores from 2009 to 2024, until the paper was written.Source: §VII · p. 14, B. Chronological Trends in Perception… · p. 14

A. Worldwide Distribution

• Among 191 datasets mapped worldwide, the USA leads with 40 datasets (21% share), followed by Germany with 24 and China with 16.The mapped collection locations are shown in Fig. 13.Source: Figure 13 · p. 14, A. Worldwide Distribution · p. 14
• Dataset collection is concentrated in high-income areas, and reliance on one region can leave systems unprepared for local differences such as China’s greater diversity and quantity of electric scooters.The survey describes KITTI as collected around Karlsruhe, Waymo and Argoverse 2 [28] across six U.S. cities each, and nuScenes in Boston and Singapore; it notes that, have been used to test generalizability through transfer learning.Source: A. Worldwide Distribution · p. 14
• Thirty-five simulator-generated synthetic datasets account for 18.32%, helping address limits of real-world data collection, though synthetic-to-real domain adaptation remains challenging.CARLA is given as an example simulator.Source: A. Worldwide Distribution · p. 14

B. Chronological Trends in Perception Datasets

• Perception datasets increased in number and variety from 2009 to 2024, alongside a shift toward diverse data-collection strategies.The timeline includes onboard datasets, synthetic datasets such as VirtualKITTI and newer works [7], V2X datasets DAIR-V2X and TUMTraf (, –[86]), and UAV-based UAVDT.Source: B. Chronological Trends in Perception… · pp. 14–15

C. Data Distribution

• Waymo has many frames with fewer than 50 objects and a broad density range, while Argoverse 2 peaks around 70 objects per frame.KITTI has a more constrained distribution and limited data size; ONCE distributes object density evenly across its supported perception range, while nuScenes and ZOD show similar curves.Source: Figure 14 · p. 15, C. Data Distribution · p. 15
• Waymo has numerous near- to mid-field objects, while Argoverse 2 and ZOD include detections beyond 200 meters.nuScenes is richer in short-range objects and tapers off quickly with distance; ONCE distributes objects more evenly across distances, while KITTI focuses on close-range objects.Source: Figure 15 · p. 15, C. Data Distribution · p. 15

D. Influence of Adversarial Environmental Conditions.

• Detection accuracy declines significantly under low illumination and slightly under nuScenes non-heavy rain, with heavy rain expected to be worse; restoration and broader weather coverage are proposed.The experiments use pretrained VoxelNext, UTVR, and Transfusion checkpoints; image restoration is proposed for camera or sensor-fusion approaches,.Source: Table IV · p. 15, D. Influence of Adversarial… · pp. 15–16

VIII. DISCUSSION AND FUTURE WORKS

• End-to-end driving datasets remain limited [129] despite potential validated by UniAD; future datasets should also support infrastructure interactions and applications such as urban planning.Automatic labeling pipelines can facilitate development of end-to-end frameworks and datasets [74].Source: §VIII · p. 16
• Language labels can improve explainability and reliability, while VLMs and world models can generate realistic data, including rare-event examples such as traffic accidents.DriveDreamer [204] derives from real-world scenarios, and Sora can generate realistic synthetic driving scenes.Source: §VIII · pp. 16–17
• Domain adaptation requires heterogeneous datasets spanning weather, illumination, geography, and sensor types, with balanced synthetic and real-world data as another route to generalization.The challenge includes transfer across driving conditions, sensor settings, and synthetic-to-real domains.Source: §VIII · p. 17
• Because incomplete training data contributes to uncertainty and weak handling of rare cases, datasets should be more diverse and include edge cases to support safer uncertainty estimation.The paper distinguishes data-driven aleatoric uncertainty from model-awareness-related epistemic uncertainty.Source: §VIII · p. 17
• Standardized formats, labeling guidelines, and storage and access protocols can improve annotation consistency, integrated processing, dataset sharing, and collaboration.The paper frames data standardization as covering data attributes, terminology, structure, and storage.Source: §VIII · p. 17
• Early datasets such as KITTI lacked anonymization, while later datasets adopt it,,; open data ecosystems broaden collaboration but require privacy and security safeguards.Data can also reveal indirect information about collection-area infrastructure and characteristics through vehicles and pedestrian attire.Source: §VIII · p. 17

IX. CONCLUSION

• The survey reviews 265 datasets, introduces an impact score, analyzes annotation quality and data distributions, and identifies dataset diversity and future challenges as priorities.Its analyses span perception, prediction, planning, control, and end-to-end driving, alongside chronological and geographical trends.Source: §IX · p. 17

APPENDIX SUPPLEMENTARY TABLES OF AD DATASETS

• Tables VI and VII present Parts I and II of the autonomous driving dataset tables; Table VI defines SS, OD, OT, and MOT.SS means semantic segmentation, OD object detection, OT object tracking, and MOT multi-object tracking.Source: Table VI · p. 18, Table VII · p. 19