Source-linked AI summary
Scalability in Perception for Autonomous Driving: Waymo Open Dataset
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng Zhao, Shuyang Cheng, Yu Zhang, Jonathon Shlens, Zhifeng Chen, Dragomir Anguelov
TL;DR
Autonomous-driving perception research lacks datasets with broad geographic coverage and multimodal scale. This paper introduces a large camera–LiDAR dataset with detection and tracking benchmarks, revealing geographic domain gaps in 3D detection.
Problem
Existing autonomous-driving datasets provide limited geographic coverage and variation for evaluating perception across operating regions.
Method
The paper constructs a large multimodal camera–LiDAR dataset with synchronized sensors, 2D and 3D tracked annotations, and detection and tracking benchmarks.
Results
Benchmarking 2D and 3D detection and tracking reveals a pronounced geographic domain gap in 3D object detection.
Takeaways & Limitations
The dataset opens research opportunities in domain adaptation and supports continued comparison through a public perception benchmark.
Abstract
from arXiv · showhide
The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the overall viability of the technology. In an effort to help align the research community's contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.
1. Introduction
The paper introduces a large, diverse multimodal autonomous-driving dataset with synchronized high-resolution camera and LiDAR data, extensive 2D/3D annotations, and support for tracking and sensor-fusion research. It also provides a geographic holdout test set and benchmark results for 2D and 3D detection and tracking.
- Dataset contribution: The dataset combines images from multiple high-resolution cameras with readings from multiple high-quality LiDAR scanners across a substantially larger geographical area than comparable datasets.It is presented as the largest and most diverse multimodal autonomous-driving dataset to date.
- Annotations and tasks: High-quality manually annotated 3D LiDAR boxes and tightly fitting 2D camera boxes include track identifiers for object tracking.A rolling-shutter-aware projection library also enables extracting 2D amodal camera boxes from 3D LiDAR boxes.
- Sensor data: Synchronized camera and LiDAR readings, range-image LiDAR release, elongation features, and accurate vehicle poses support cross-domain learning and alternative LiDAR input representations.The dataset provides low-level synchronized information beyond the popular 3D point-set format.
- Evaluation design: 1000 scenes support training and validation, while 150 test scenes span 20 s each and come from a geographic holdout area for evaluating generalization to unseen areas.The split is designed to assess how well models trained on the dataset generalize across geography.
- Benchmarks: The paper reports benchmark results for several state-of-the-art 2D and 3D object detection and tracking methods.These benchmarks are evaluated on the introduced dataset.
2. Related Work
Prior autonomous-driving datasets increasingly combine multiple sensors and, in some cases, semantic or geometric map information. Representative datasets support tasks including 3D detection, tracking, visual odometry, scene flow, and trajectory prediction, but differ in sensing modalities and environmental coverage.
- Multi-sensor datasets: Multi-sensor datasets combine cameras, LiDAR, radar, GPS, wheel odometry, or IMUs to support autonomous-driving research.KITTI provides synchronized stereo-camera and LiDAR data for 22 sequences, enabling 3D object detection and tracking, visual odometry, and scene flow estimation.
- Image and multispectral datasets: ApolloScape provides per-pixel semantic annotations for 140k camera images across traffic conditions and pose information relative to static background point clouds.Its scenes range from simple environments to challenging scenes containing many objects.
- Image and multispectral datasets: The KAIST Multi-Spectral Dataset organizes multisensor scenes by time slots including daytime, nighttime, dusk, and dawn.The dataset includes a thermal imaging camera alongside other sensors.
- Datasets with map information: nuScenes combines cameras, LiDAR, and radar with rasterized top-down semantic maps for 1k scenes, encoding driveable areas and sidewalks.The passage notes limited LiDAR quality, with 34K points per frame, and limited geographical diversity covering an effective area of 5km2.
- Datasets with map information: Argoverse contributes geometric and semantic maps containing ground height and vectorized road-lane connectivity, and studies map context for 3D tracking and trajectory prediction.Its maps support analysis of how environmental context influences autonomous-driving tasks.
3. Waymo Open Dataset
The Waymo Open Dataset provides synchronized, calibrated LiDAR and camera data with comprehensive 3D and 2D annotations, standardized coordinate transformations, and broad geographic coverage. Its scale includes millions of labeled objects and extensive tracking identities across urban and suburban scenes.
- Sensors and data format: Five LiDAR sensors and five high-resolution pinhole cameras provide the dataset’s multimodal sensing setup, with LiDAR data restricted to the first two returns.Camera images use rolling-shutter scanning and are downsampled and cropped from raw images.
- Coordinate systems: The dataset supplies right-handed coordinate systems and information to transform data between any two frames within a run segment.Vehicle poses map vehicle coordinates to the global frame, while sensor extrinsics map sensor coordinates to the vehicle frame.
- Annotations: 7-DOF 3D upright bounding boxes annotate vehicles, pedestrians, signs, and cyclists in LiDAR readings, with unique tracking IDs.Separate 4-DOF axis-aligned 2D boxes annotate vehicles, pedestrians, and cyclists in every camera image.
- Multimodal alignment: LiDAR range images encode each return’s range, intensity, and elongation, while accurate rolling-shutter-compensated projections map LiDAR points to camera pixels.The synchronization error is bounded in [-6ms, 7ms] with 99.7% confidence and [-6ms, 8ms] with 99.9995% confidence.
- Coverage and scale: 40km2 in Phoenix and 36km2 combined in San Francisco and other areas are covered across suburban and urban scenes, different times, and varied city locations.The dataset contains around 12M labeled 3D LiDAR objects, around 113k unique LiDAR tracking IDs, around 12M labeled 2D image objects, and around 254k unique image tracking IDs.
4. Tasks
The dataset defines 2D and 3D object detection and tracking tasks, with evaluation protocols tailored to sensor modality, heading accuracy, and temporal identity tracking. The 3D task supports multimodal and temporal inputs, while the 2D task uses only single camera images.
- Task Definitions: The dataset defines 2D and 3D object detection and tracking tasks, with additional tasks such as segmentation and behavior prediction planned for future work.The stated initial tasks are detection and tracking; the dataset may later expand to segmentation, domain adaptation, behavior prediction, and imitative planning.
- 3D Detection: 3D detection predicts upright boxes for vehicles, pedestrians, signs, and cyclists using any LiDAR or camera sensors, optionally including preceding frames.LiDAR and camera inputs are both permitted, and methods may incorporate temporal context from earlier frames.
- 3D Detection: APH extends average precision by weighting true positives according to heading accuracy, addressing heading information absent from conventional AP.Heading accuracy is computed from the angular difference between predicted and ground-truth headings, normalized over π.
- 2D Detection: 2D detection uses a single camera image to predict axis-aligned boxes for vehicles, pedestrians, and cyclists, excluding LiDAR and matching with 2D IoU.The task evaluates the AP metric for the three listed object classes.
- Multi-Object Tracking: Tracking is evaluated with MOT metrics over 20-second, 10Hz sequences using consistent object identifiers, with results supported in 2D image view and 3D vehicle-centric coordinates.MOT summarizes misses, false positives, mismatches, and localization performance; the final MOTA is the highest value across sampled score cutoffs.
5. Experiments
The experiments establish baseline methods and metrics for 2D and 3D detection and tracking, then examine geographic domain shift and the effect of increasing dataset size. Results include camera-image detection performance, tracking behavior, and asymmetric cross-domain effects for 3D detection.
- 3D LiDAR Detection: The 3D LiDAR detection baseline reimplemented PointPillars, trained on single-frame data with all LiDARs, and evaluated 7-degree-of-freedom 3D boxes and 5-degree-of-freedom BEV boxes.Difficulty ignores labels without LiDAR points, assigns LEVEL 2 to hard examples or examples with ≤5 LiDAR points, and assigns the remainder to LEVEL 1.
- 2D Object Detection in Camera Images: Faster R-CNN with ResNet-101 achieved vehicle AP of 63.7 at LEVEL 1 and 53.3 at LEVEL 2, and pedestrian AP of 55.8 and 52.7.The model was pre-trained on COCO, fine-tuned on the dataset, and evaluated by aggregating detections from all five camera images.
- 3D Tracking: The 3D tracking baseline uses PointPillars detections, 1 −IOU costs, Hungarian assignment, and a Kalman Filter, with low mismatch percentages for vehicles and pedestrians.Tracks use a constant-velocity 10-parameter state and a birth-and-death score updated by detections and unmatched-track penalties.
- 2D Tracking: Tracktor achieved vehicle MOTA of 34.8 at LEVEL 1 and 28.3 at LEVEL 2 for 2D tracking.Tracktor is based on a Faster R-CNN detector pre-trained on COCO and fine-tuned on the dataset.
- Domain Shift: 3D vehicle APH fell by 8.0 for SF-to-SUB evaluation and by 7.6 for SUB-to-SF evaluation relative to same-domain comparisons.For pedestrians evaluated on SUB, training on all data produced a 7+ APH improvement, whereas SF-only training on SF improved APH by 2.4 over the combined dataset.
- Dataset Size: Larger cumulative training subsets improved PointPillars results without requiring data augmentation, demonstrating the dataset’s utility for data-intensive experiments.The same PointPillars model was trained on cumulative subsets of the training sequences and evaluated on the test set.
6. Conclusion
The paper presents a large-scale multimodal camera-LiDAR dataset that is larger, higher quality, and more geographically diverse than existing similar datasets. It also demonstrates geographic domain diversity and evaluates 2D and 3D detection and tracking.
- 76km2 of coverage is provided when considering diluted ego poses at a visibility of 150 meters.The coverage calculation uses diluted ego poses and a 150-meter visibility range.
- The dataset is larger, higher quality, and more geographically diverse than any existing similar dataset.
- Domain diversity is demonstrated across Phoenix, Mountain View, and San Francisco, opening research opportunities for domain adaptation.
- The study evaluates 2D and 3D object detectors and trackers on the dataset.