Source-linked AI summary
A2D2: Audi Autonomous Driving Dataset
Jakob Geyer, Yohannes Kassahun, Mentar Mahmudi, Xavier Ricou, Rupesh Durgesh, Andrew S. Chung, Lorenz Hauswald, Viet Hoang Pham, Maximilian Mühlegg, Sebastian Dorn, Tiffany Fernandez, Martin Jänicke, Sudesh Mirashi, Chiragkumar Savani, Martin Sturm, Oleksandr Vorobiov, Martin Oelker, Sebastian Garreis, Peter Schuberth
TL;DR
Autonomous-driving research needs high-quality annotated data, but public datasets have limited multimodal coverage and vehicle-state information. A2D2 addresses this by releasing registered camera, LiDAR, vehicle-bus, and annotation data, together with unannotated sequences. The paper reports a 71.01% mean IoU baseline on 18 foreground semantic-segmentation classes and notes evolving dataset coverage.
Problem
Autonomous-driving research requires high-quality annotated data, while the available public datasets provide limited coverage of multimodal sensing and vehicle bus information.
Method
The authors release A2D2, a commercial-grade dataset combining camera, LiDAR, vehicle bus, perception annotations, and an evaluation of semantic segmentation.
Results
71.01% mean IoU over the 18 foreground classes is achieved by the evaluated semantic-segmentation network on the test set.
Takeaways & Limitations
A2D2 provides registered multimodal driving data intended for perception research and includes vehicle-state information supporting broader research use.
Takeaways & Limitations
The initial public release omitted instance-segmentation annotations, although they were later made available.
Abstract
from arXiv · showhide
Research in machine learning, mobile robotics, and autonomous driving is accelerated by the availability of high quality annotated data. To this end, we release the Audi Autonomous Driving Dataset (A2D2). Our dataset consists of simultaneously recorded images and 3D point clouds, together with 3D bounding boxes, semantic segmentation, instance segmentation, and data extracted from the automotive bus. Our sensor suite consists of six cameras and five LiDAR units, providing full 360 degree coverage. The recorded data is time synchronized and mutually registered. Annotations are for non-sequential frames: 41,277 frames with semantic segmentation image and point cloud labels, of which 12,497 frames also have 3D bounding box annotations for objects within the field of view of the front camera. In addition, we provide 392,556 sequential frames of unannotated sensor data for recordings in three cities in the south of Germany. These sequences contain several loops. Faces and vehicle number plates are blurred due to GDPR legislation and to preserve anonymity. A2D2 is made available under the CC BY-ND 4.0 license, permitting commercial use subject to the terms of the license. Data and further information are available at https://a2d2-dataset.github.io/.
1. Introduction
A2D2 is introduced as a multimodal autonomous-driving dataset designed to support research through broad sensor coverage, vehicle bus data, commercial usability, and public release.
- A2D2 provides camera, LiDAR, and vehicle bus data for exploring multimodal sensor fusion in autonomous driving.
- Full-surround coverage combines registered camera and LiDAR data with vehicle-state information such as speed, acceleration, steering, throttle, and braking.
- A2D2 is released under CC BY-ND 4.0, allowing commercial use subject to the license terms.
- The authors release A2D2 to catalyse autonomous-driving research, provide a public dataset from realistic driving, engage the research community, and contribute to commercial entities.
- The dataset is commercial grade, suitable for many perception tasks, and includes vehicle bus data that public datasets had lacked.
- The paper evaluates a semantic-segmentation convolutional network on A2D2.
2. Related Work
Related work surveys multimodal autonomous-driving datasets and positions A2D2 around semantic segmentation, vehicle bus data, broad sensor coverage, and long unannotated sequences.
- Datasets: Recent autonomous-driving datasets increasingly emphasize multimodal sensors, including cameras, LiDAR, radar, and 3D bounding boxes.
- Datasets: Prior datasets span semantic segmentation, point-cloud annotation, object detection, tracking, and varied urban, rural, and off-road driving scenes.
- Comparison: Table 1 compares multimodal datasets by sensor approach and records setup-specific LiDAR channel and additional-sensor details.
- A2D2 relative to prior work: A2D2 adds extensive vehicle bus data, including steering, throttle, and braking, supporting end-to-end and reinforcement-learning research.
- A2D2 relative to prior work: Its five-LiDAR configuration uses 80 channels optimized for uniform distribution and overlap with camera frames, also covering areas above the vehicle.
- A2D2 relative to prior work: A2D2 complements existing multimodal datasets with stronger emphasis on semantic segmentation, vehicle bus data, and long consecutive unannotated sensor sequences for self-supervised approaches.
3. Dataset
A2D2 contains annotated multimodal driving data and additional unannotated sequences collected across highways, country roads, and cities under varied weather conditions.
- The dataset covers highways, country roads, and cities in southern Germany under cloudy, rainy, and sunny weather.
- 41,277 non-sequential images have semantic and instance segmentation labels covering 38 categories.
- All annotated images have corresponding LiDAR point clouds, while 12,497 also have 3D bounding boxes within the front-center camera field of view.
- The release also includes unannotated sequence data.
3.1. Data Collection Platform
A2D2 was collected with six cameras, five LiDAR sensors, and vehicle bus recording on an Audi Q7 e-tron. The roof-mounted configuration targets 360° coverage, while calibration aligns the multimodal sensors and bus data supports motion-aware registration.
- Platform: The platform uses an Audi Q7 e-tron equipped with six cameras, five Velodyne VLP-16 sensors, and vehicle bus recording.
- Sensor Setup: Three fisheye cameras cover the sides and rear, while three front cameras provide greater front-view detail with reduced distortion.
- Sensor Setup: The roof-mounted sensors aim for 360° environmental coverage, with camera and LiDAR fields of view overlapping by over 90% outside LiDAR blind spots.
- Calibration and Registration: Calibration combines a fixed front-center LiDAR reference, static-scene LiDAR-to-LiDAR ICP registration, checkerboard camera calibration, and motion-based LiDAR-to-camera alignment.
- Vehicle Bus Data: Vehicle bus files contain timestamped signals such as acceleration, angular velocity, GPS, braking, and vehicle attitude, supporting imitation and reinforcement learning research.
3.2. Anonymization
A2D2 anonymizes annotated and unannotated imagery by automatically blurring license plates and pedestrian heads.
- More than 400,000 images were anonymized by blurring license plates and pedestrian heads with a semantic segmentation network.
3.3. Unlabelled Sequence Data
A2D2 provides long unlabelled multimodal sequences from three German cities, including closed loops and synchronized camera, LiDAR, and vehicle bus data.
- The dataset contains 392,556 unlabelled images from sequences recorded in Gaimersheim, Munich, and Ingolstadt, with corresponding LiDAR and bus data.
- The sequences contain closed loops and support end-to-end driving, depth prediction, and SLAM research.
3.4. Data Labels
A2D2 supplies dense semantic labels across 38 classes, derived 3D semantic labels, traffic-participant instance annotations, and front-camera 3D bounding boxes. Its LiDAR-derived boxes are limited by point sparsity and visibility, especially for distant or small objects.
- Semantic Segmentation: 41,277 camera images are semantically labelled across highway, country, and urban scenarios, with every pixel assigned a semantic class.
- Semantic Segmentation: The semantic segmentation schema contains 38 classes, with traffic-participant counts dominated by cars, trucks, and pedestrians.
- 3D Semantic Labels: LiDAR point clouds accompany 38,481 semantically labelled images, with 3D labels obtained by mapping points into semantic segmentation images.
- Instance Segmentation: Instance annotations are generated for semantic labels representing traffic participants, including pedestrians and cars.
- 3D Bounding Boxes: 12,497 front-camera frames additionally contain 3D bounding boxes for vehicles, pedestrians, and other relevant objects.
- 3D Bounding Boxes: Because boxes derive from sparse LiDAR points, distant or small objects may appear in images without corresponding 3D bounding boxes.
- Bounding-Box Distributions: Annotated trucks are visible farther away than cars and pedestrians, while car annotations decline more quickly with distance, plausibly due to occlusion patterns.
3.5. Tutorial
A2D2 includes a Jupyter Notebook tutorial that explains how to access and use the dataset, making the resource easier to use.
- A Jupyter Notebook tutorial is provided with the dataset download.It details how to access and use A2D2.
4. Experiment: Semantic Segmentation
The authors evaluate a fully convolutional semantic segmentation network on A2D2 using a ResNet-101 encoder and pyramid scene parsing decoder. The baseline reaches 71.01% mean IoU, while pre-training improves results and anonymization yields 70.94% mean IoU.
- Baseline Results: A fully convolutional network classifies each image pixel to establish baseline semantic segmentation results on A2D2.The experiment follows state-of-the-art methods.
- Experimental Setup: 40,030 RGB images are split into 28,015 training, 4,118 validation, and 7,897 test images across 19 classes.The images have resolution 1920 × 1208 pixels, with 18 foreground classes and background.
- Experimental Setup: The model uses a ResNet-101 encoder, pyramid scene parsing decoder, ImageNet initialization, and momentum-based stochastic gradient descent.The initial learning rate is 0.01 with momentum 0.9 and polynomial learning-rate decay.
- Baseline Results: 71.01% mean IoU is achieved over 18 foreground classes on the test set.Figure 8 provides visual examples of the network output.
- Pre-training: The model with ImageNet pre-trained weights achieves a better mean IoU than the corresponding randomly initialized model.This comparison uses a ResNet-50 encoder with otherwise matching training setup, except for the specified learning-rate change.
- Anonymization: 70.94% mean IoU is achieved when the network is trained on anonymized images with blurred faces and vehicle number plates.The anonymized-image setup otherwise matches the baseline architecture and experimental configuration.
5. Conclusions and Outlook
A2D2 provides commercially usable, synchronized multimodal sensor data with full 360° coverage and aims to advance autonomous-driving research. The dataset is intended to evolve through added annotations and future benchmarking resources.
- Dataset Scope: A2D2 combines six cameras, five LiDAR sensors, and vehicle-bus data registered to a global reference frame with precise timestamps.The dataset provides full 360° coverage of the vehicle environment and was recorded from an Audi Q7 e-tron.
- Accessibility: The dataset is commercially usable and designed to be accessible through its license, privacy measures, and interactive tutorial.Its stated goal is to advance commercial and academic research in computer vision, machine learning, and autonomous driving.
- Outlook: Instance segmentation annotations were not included in the initial public release, although they are now available for download.The authors expect to continuously update A2D2 in line with current research frontiers.
- Outlook: The authors plan to define benchmarks and challenges and are exploring how to enable community comparison against semantic-segmentation ground truth.A test set of approximately 10K images has been labeled for this purpose.