Source-linked AI summary
DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection
Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, Zaiqing Nie
TL;DR
The paper addresses the lack of real-world datasets for vehicle-infrastructure cooperative autonomous driving and releases DAIR-V2X for this setting. It formulates VIC3D, introduces fusion benchmarks including Time Compensation Late Fusion, and reports higher average precision for fusion methods than single-view detectors.
Problem
Real-world datasets for vehicle-infrastructure cooperation were unavailable, despite the need for broader and longer-range perception in autonomous driving.
Method
The paper releases a large-scale multimodal, multi-view dataset, formulates VIC3D, and benchmarks fusion methods while addressing temporal asynchrony and transmission cost.
Results
Fusion methods achieve average precision 10 to 20 points higher than detectors using only single-view information.
Takeaways & Limitations
DAIR-V2X provides a real-world benchmark for studying cooperative 3D detection and performance-bandwidth trade-offs.
Takeaways & Limitations
Ground-truth estimation for unsynchronized frames requires future tracking IDs and kinematic modeling when temporal differences cannot be ignored.
Abstract
from arXiv · showhide
Autonomous driving faces great safety challenges for a lack of global perspective and the limitation of long-range perception capabilities. It has been widely agreed that vehicle-infrastructure cooperation is required to achieve Level 5 autonomy. However, there is still NO dataset from real scenarios available for computer vision researchers to work on vehicle-infrastructure cooperation-related problems. To accelerate computer vision research and innovation for Vehicle-Infrastructure Cooperative Autonomous Driving (VICAD), we release DAIR-V2X Dataset, which is the first large-scale, multi-modality, multi-view dataset from real scenarios for VICAD. DAIR-V2X comprises 71254 LiDAR frames and 71254 Camera frames, and all frames are captured from real scenes with 3D annotations. The Vehicle-Infrastructure Cooperative 3D Object Detection problem (VIC3D) is introduced, formulating the problem of collaboratively locating and identifying 3D objects using sensory inputs from both vehicle and infrastructure. In addition to solving traditional 3D object detection problems, the solution of VIC3D needs to consider the temporal asynchrony problem between vehicle and infrastructure sensors and the data transmission cost between them. Furthermore, we propose Time Compensation Late Fusion (TCLF), a late fusion framework for the VIC3D task as a benchmark based on DAIR-V2X. Find data, code, and more up-to-date information at https://thudair.baai.ac.cn/index and https://github.com/AIR-THU/DAIR-V2X.
1. Introduction
Autonomous driving needs broader and longer-range perception, but real-world vehicle-infrastructure cooperation lacked a dedicated dataset. DAIR-V2X addresses this gap with a large-scale multimodal dataset, the VIC3D task, and fusion benchmarks that account for temporal asynchrony and transmission cost.
- Motivation: Vehicle-infrastructure cooperation can provide a global perspective beyond the current horizon and cover blind spots, but no real-world dataset supported related research.
- Dataset: DAIR-V2X provides 71,254 LiDAR frames and 71,254 camera frames from real intersection scenes with expert 3D annotations.The dataset includes vehicle and infrastructure sensing across diverse roads, weather, and lighting conditions.
- Task: VIC3D formulates cooperative 3D object localization and identification using sensory inputs from both vehicle and infrastructure.Unlike traditional 3D detection, VIC3D must consider temporal asynchrony and data transmission cost.
- Benchmarks: Fusion methods achieve average precision 10 to 20 points higher than single-view detectors, while early fusion performs better than late fusion but requires more transmission.For severe temporal asynchrony, the proposed Time Compensation Late Fusion framework is designed to alleviate the problem.
- Dataset: The dataset is the first large-scale, multimodal, multi-view resource for vehicle-infrastructure cooperative autonomous driving.
2. Relative Work
Prior autonomous-driving datasets and cooperative-perception methods largely lack real-world vehicle-infrastructure data, motivating DAIR-V2X as a real-scenario benchmark for VICAD.
- Autonomous Driving Datasets: DAIR-V2X is presented as the first large-scale, multi-modality, multi-view vehicle-infrastructure dataset captured from real scenarios.Earlier datasets generally provide single-vehicle views, while V2X-SIM is simulated.
- 3D Detection: Existing 3D detection methods use images, point clouds, or both, but blind spots and weak long-distance perception remain challenges.The paper groups prior methods into image-based, pointcloud-based, and multimodality-based detection.
- Multi-Sensor Fusion: Multi-sensor fusion combines heterogeneous sensor information through early, intermediate, or late fusion stages.VIC3D extends this setting by incorporating infrastructure information.
- V2X Cooperative Perception: Prior V2X approaches use vehicle-to-vehicle or vehicle-to-infrastructure communication, but none were verified on datasets captured from real scenarios.The paper identifies this as a gap between theory and practice.
3. The DAIR-V2X Dataset
DAIR-V2X combines infrastructure and vehicle sensing in real intersection scenes, with multi-view sensor configurations, broad scene collection, and expert-refined annotations.
- Setup: The dataset uses infrastructure and vehicle sensors to collect paired data for vehicle-infrastructure cooperative autonomous driving.The setup includes infrastructure deployments at selected intersections and sensors mounted on autonomous vehicles.
- Setup: Infrastructure sites use four pairs of 300-beam LiDARs and high-resolution cameras, while vehicles carry one 40-beam LiDAR and one forward-facing camera.DAIR-V2X selects one infrastructure sensor pair at each intersection.
- Setup: DAIR-V2X defines LiDAR, camera, image, world, and positioning coordinate systems for representing sensor data and poses.The LiDAR coordinate system is centered at the LiDAR, with axes oriented forward, left, and upward.
- Data Acquisition: The collection includes representative paired scenes plus additional vehicle-only and infrastructure-only segments to enlarge scene diversity.The paired scenes last 20 seconds, while additional segments provide 22K vehicle frames and 10K infrastructure frames.
- Data Acquisition: Cooperative annotations are created by transforming infrastructure boxes into vehicle coordinates, merging unmatched boxes, and manually supervising the results.The VIC-Sync benchmark uses 9331 infrastructure-vehicle frame pairs with cooperative annotations.
4. Task & Metrics
VIC3D integrates vehicle and infrastructure observations to detect surrounding 3D objects while addressing transmission cost and temporal asynchrony, evaluated with detection and communication metrics.
- VIC3D Object Detection: VIC3D uses vehicle-infrastructure sensory inputs to localize and recognize 3D objects surrounding the vehicle.Outputs include object location, category, and orientation.
- VIC3D Object Detection: VIC3D must balance detection performance against transmission cost because communication constraints limit infrastructure data transfer.The task considers raw data, intermediate representations, and object-level outputs as possible transmitted information.
- VIC3D Object Detection: Ground truth combines vehicle and infrastructure annotations, with both aligned to the vehicle timestamp tv for egocentric perception.For asynchronous data, object states may need estimation at tv using tracking and kinematic information.
- Evaluation Metrics: Average Precision measures cooperative 3D detection, while Average Byte measures transmission cost under a defined egocentric evaluation area.Encoder and decoder time is ignored when estimating transmission cost.
5. Benchmark
The benchmark evaluates cooperative 3D detection across modalities, fusion strategies, and temporal conditions, including the proposed TCLF framework. Results show substantial gains from cooperative views, a time-compensation benefit under asynchrony, and a performance–transmission-cost trade-off between early and late fusion.
- VIC3D benchmark: The VIC3D benchmark uses cooperative infrastructure–vehicle annotations and evaluates temporal asynchrony caused by sampling-rate differences and transmission delay.The VIC-Sync split contains 9311 infrastructure–vehicle frame pairs with cooperative ground truth, while VIC-Async-k simulates temporal asynchrony.
- Baselines: The benchmark includes LiDAR and image late-fusion baselines, LiDAR early fusion, and single-view infrastructure-only and vehicle-only comparisons.PointPillars supports LiDAR baselines, ImvoxelNet supports image baselines, and single-view results are reported as Inf.-Only and Veh.-Only.
- Cooperative-view vs. Single-view: 41.90 AP for 3D detection and 47.96 AP for BEV detection are achieved by LiDAR late fusion on VIC-Sync, versus 31.33% and 35.06% for vehicle-only detection.Infrastructure-only detection reaches 17.62% AP for 3D detection and 24.40% AP for BEV detection, showing the cooperative-view advantage over either single view.
- Temporal Asynchrony vs Time Compensation: 0.5% AP and 1.5% AP are recovered by TCLF on VIC-Async-1 and VIC-Async-2, respectively, after fusion performance drops under temporal asynchrony.TCLF estimates object velocities and infrastructure states at the vehicle timestamp before late fusion; the benefit is larger when time delay is larger.
- Early Fusion vs. Late Fusion: Up to 8% AP separates early from late fusion under BEV and 3D benchmarks, while early fusion incurs about 4000 times more transmission cost.Early fusion transmits the whole point cloud, whereas late fusion is more bandwidth-efficient; the benchmark identifies reducing this trade-off as a future research direction.
- SV3D Detection Benchmark: The paper also provides extensive single-view 3D detection benchmarks on the more diverse and challenging DAIR-V2X-V and DAIR-V2X-I datasets.Baselines include ImvoxelNet, PointPillars, SECOND, and MVXNet, evaluated using KITTI-style PASCAL criteria.
6. Conclusion
The paper introduces DAIR-V2X, a real-world, large-scale multi-view and multi-modality dataset, and defines VIC3D for cooperative vehicle–infrastructure detection. Its benchmarks show that infrastructure integration improves detection over single-vehicle sensing, while TCLF addresses temporal asynchrony.
- Dataset and task: VIC3D formulates cooperative 3D object localization and identification using vehicle and infrastructure sensory inputs.The task must account for temporal asynchrony and data transmission cost between the sensors.
- Results: Integrating infrastructure sensors achieves an average of 15% AP higher than single-vehicle 3D detection.The paper also provides VIC3D, vehicle-view, and infrastructure-view detection benchmarks, including TCLF for temporal compensation.