Source-linked AI summary
V2X-Real: a Large-Scale Dataset for Vehicle-to-Everything Cooperative Perception
Hao Xiang, Zhaoliang Zheng, Xin Xia, Runsheng Xu, Letian Gao, Zewei Zhou, Xu Han, Xinkai Ji, Mingxi Li, Zonglin Meng, Li Jin, Mingyue Lei, Zhaoyang Ma, Zihang He, Haoxuan Ma, Yunshuang Yuan, Yingqian Zhao, Jiaqi Ma
TL;DR
Real-world V2X cooperative perception lacks a dataset that spans collaboration modes and infrastructure perspectives. V2X-Real contributes multimodal urban data, four dataset variants, and multi-class cooperative detection benchmarks; its benchmarks report gains from cooperative methods over No Fusion across all four datasets.
Problem
Real-world V2X datasets have been limited to single collaboration modes, leaving a gap in data for multi-mode cooperative perception.
Source: §1 · p. 2Method
V2X-Real collects multimodal data from two connected vehicles and two smart infrastructure units and organizes it into four ego- and collaboration-mode-specific datasets.
Source: §1 · p. 2Results
Cooperative methods outperform the No Fusion baseline across all four datasets, and intermediate fusion typically gives more precise predictions than late or early fusion.
Source: §5.2 · pp. 13–14Takeaways & Limitations
The dataset and benchmark framework support multi-class cooperative 3D detection research across vehicle- and infrastructure-centric settings.
Source: §6 · p. 14
Abstract
from arXiv · showhide
Recent advancements in Vehicle-to-Everything (V2X) technologies have enabled autonomous vehicles to share sensing information to see through occlusions, greatly boosting the perception capability. However, there are no real-world datasets to facilitate the real V2X cooperative perception research -- existing datasets either only support Vehicle-to-Infrastructure cooperation or Vehicle-to-Vehicle cooperation. In this paper, we present V2X-Real, a large-scale dataset that includes a mixture of multiple vehicles and smart infrastructure to facilitate the V2X cooperative perception development with multi-modality sensing data. Our V2X-Real is collected using two connected automated vehicles and two smart infrastructure, which are all equipped with multi-modal sensors including LiDAR sensors and multi-view cameras. The whole dataset contains 33K LiDAR frames and 171K camera data with over 1.2M annotated bounding boxes of 10 categories in very challenging urban scenarios. According to the collaboration mode and ego perspective, we derive four types of datasets for Vehicle-Centric, Infrastructure-Centric, Vehicle-to-Vehicle, and Infrastructure-to-Infrastructure cooperative perception. Comprehensive multi-class multi-agent benchmarks of SOTA cooperative perception methods are provided. The V2X-Real dataset and codebase are available at https://mobility-lab.seas.ucla.edu/v2x-real.
1 Introduction
Single-view vehicles have limited perception range and suffer occlusions, motivating V2X agents to share sensing information. V2X-Real addresses the lack of a real-world, multi-mode V2X dataset with urban, multimodal data and four collaboration subsets.
1 Introduction
- Single-vehicle systems have limited perception range and suffer from occlusions, while connected agents can share sensing information to see through occlusions.Shared information can include raw point clouds, detection results, or intermediate neural features.Source: §1 · pp. 1–2
- Existing real-world V2X datasets cover only V2I or V2V, while many other V2X datasets are simulated; existing real datasets also have limited agent and traffic-density scope.The paper notes that DAIR-V2X and V2V4Real involve at most two agents and are mostly collected in suburban or rural areas, averaging 11–12 objects per scene.Source: §1 · p. 2
- V2X-Real provides 33K LiDAR frames, 171K camera images, and over 1.2M 3D bounding-box annotations from urban scenarios.The data were collected using two smart infrastructure units and two autonomous vehicles; the dataset includes 68 selected scenarios.Source: §1 · pp. 2–3
- The dataset provides four subsets: vehicle-centric, infrastructure-centric, V2V, and I2I cooperative perception.The subsets vary by ego-agent type and collaboration mode.Source: §1 · p. 3
2 Related Work
Prior work spans single-vehicle datasets, simulated cooperative datasets, and real-world V2I or V2V datasets. The paper situates V2X-Real as a multi-mode, multi-class dataset and discusses fusion strategies with different bandwidth and information trade-offs.
2.1 Self-driving datasets
- Single-vehicle datasets such as KITTI, NuScenes, and Waymo provide driving data, while OPV2V, V2XSet, and V2X-Sim use simulation and DAIR-V2X and V2V4Real provide real-world V2I and V2V data.The section also names AttFuse, V2VNet, F-Cooper, V2X-ViT, CoBEVT, and HM-ViT as cooperative perception methods.Source: §2.1 · pp. 3–4, §2.2 · p. 6
- Earlier real-world cooperative datasets focus on V2I or V2V, whereas V2X-Real covers V2V, V2I, and I2I and benchmarks both vehicle- and infrastructure-centric perception.The section also notes its congested urban collection and benchmarks for all annotation types, including vulnerable road users.Source: §2.1 · pp. 4–5
2.2 V2X Cooperative Perception
- Early fusion shares raw point clouds, late fusion shares detection outputs, and intermediate fusion shares neural features.Early fusion preserves raw information but requires high bandwidth; late fusion uses minimal bandwidth but loses context and may incur compound errors.Source: §2.2 · p. 5
3 V2X-Real datasets
V2X-Real combines multi-agent, multi-modal urban data with a reviewed annotation pipeline and analyses of object density and sensing coverage. Its collection spans intersection and corridor scenarios, supporting diverse V2X research.
3.1 Data Acquisition
- Two automated vehicles and two smart infrastructure units collected intersection and corridor interactions using LiDAR, cameras, and GPS-based positioning.The vehicles have four cameras for 360-degree coverage; intersection scenarios include 50 interactions and corridor scenarios include 10 V2V interaction types.Source: §3.1 · p. 6
- The dataset contains 33K LiDAR frames and 171K camera images selected from 68 scenarios and over 75 hours of driving logs.Collection spanned three months and diverse illumination conditions.Source: §3.1 · p. 6
3.2 Data Annotation and Processing
- Expert-reviewed annotations cover 10 object categories using 9-dimensional cuboids; combining two infrastructures’ LiDAR point clouds creates a holistic view that reduces annotation workload and increases annotation efficiency and accuracy.Each scenario undergoes an average of 5–6 review and revision rounds. A box is added to an infrastructure’s annotation result only when more than 5 LiDAR points fall inside it; otherwise, it is ignored for that agent.Source: §3.2 · pp. 8–9
- Annotations from different agents are matched in map coordinates using box overlap to assign consistent object IDs.Boxes with overlap above a threshold are treated as the same object.Source: §3.2 · p. 9
3.3 Dataset Analysis
- Scenes average 36 objects (range 9–115), with pedestrians and cars dominant; annotations show diverse spatial distributions, denser in the front, left, right, and rare sides of the ego vehicle.Source: Figure 4 · p. 8, §3.3 · p. 9
- LiDAR points per box decrease with radial distance, while varied box sizes and point coverage provide detection-task priors and motivate collaborative sensing.The analysis presents point-density and box-size distributions for cars, pedestrians, and trucks.Source: §3.3 · p. 10, Figure 6 · p. 10
4 Tasks
The paper frames cooperative 3D detection across four ego-agent and collaboration settings, with multi-class evaluation and benchmarks spanning fusion strategies. These choices extend the framework beyond vehicle-centric, single-category evaluations.
4.1 V2X Cooperative 3D object detection
- Four derived datasets cover vehicle-centric, infrastructure-centric, V2V, and I2I cooperation, with infrastructure as the evaluation ego in the infrastructure-centric setting.V2V corridor scenarios supplement intersection data, and I2I is framed as relevant to intelligent transportation systems.Source: §4.1 · pp. 10–11
4.2 Metrics
- Evaluation groups 10 annotated categories into car, pedestrian, and truck classes and reports AP and mAP at IoU thresholds 0.3 and 0.5.The smaller thresholds are adopted because object sizes range from small pedestrians to buses over 20 meters long.Source: §4.2 · p. 11
4.3 Benchmark methods
- Benchmarks compare No Fusion, Late Fusion, Early Fusion, and Intermediate Fusion, including AttFuse, F-Cooper, and V2X-ViT.The strategies range from ego-only LiDAR and shared detections to shared raw point clouds or intermediate BEV features.Source: §4.3 · pp. 11, 13
- The benchmark covers pedestrian, car, and truck detection, broadening prior V2X benchmarks that evaluated only single vehicle categories.It is presented as an open framework for multi-class cooperative 3D object detection.Source: §4.3 · p. 13
5 Experiments
Across all four datasets, cooperative methods outperform No Fusion, with Intermediate Fusion generally ranking ahead of Early Fusion, which in turn exceeds Late Fusion. The experiments use a shared LiDAR detection setup and multi-class models.
5.1 Implementation details
- Experiments split the data into 23,379 training, 2,770 validation, and 6,850 test frames, training models for 80 epochs with batch size 2.The cooperative framework is built on OpenCOOD, with Adam optimization and a multi-step learning-rate schedule.Source: §5.1 · p. 13
5.2 Benchmark results
- Cooperative methods outperform No Fusion on all datasets; Intermediate Fusion generally ranks first, followed by Early Fusion and then Late Fusion.Intermediate Fusion methods typically yield more precise predictions than Late Fusion methods.Source: §5.2 · pp. 13–14
- Models perform best on cars and less well on pedestrians and trucks, which the authors associate with sparse pedestrian LiDAR and diverse truck dimensions.The benchmark uses PointPillar as the LiDAR backbone and extends a single-class head to multi-class detection.Source: §5.1 · p. 13, §5.2 · p. 14
6 Conclusion
V2X-Real contributes a large-scale real-world cooperative perception dataset, four collaboration-mode subsets, and a multi-class benchmark framework. The current release uses LiDAR and cameras, with radar identified as a future extension.
6 Conclusion
- V2X-Real provides 33K LiDAR frames, 171K RGB images, over 1.2 million 3D boxes, four collaboration-mode subsets, and a multi-class cooperative detection framework.The authors identify the current lack of radar and plan to extend the sensor suite toward low-cost V2X research.Source: §6 · p. 14
Appendix
- The appendix previews additional dataset visualizations, model configurations, benchmark results, and qualitative examples, including V2V corridor scenarios.The corridor figure shows aggregated LiDAR point clouds above front-camera views from two collection vehicles.Source: Appendix · p. 15, Figure 7 · p. 15
A Dataset Visualization
The dataset visualizations show stereo and multi-view vehicle imagery with projected 3D boxes, and the following section introduces model configurations for LiDAR- and camera-based cooperative perception.
A Dataset Visualization
- Each vehicle has four stereo cameras facing perpendicular directions, with 120-degree horizontal fields of view that together cover 360 degrees.Ground-truth boxes are projected onto the camera image plane using intrinsics and extrinsics, with categories shown in different colors.Source: A Dataset Visualization · p. 16
- The supplementary material includes stereo image pairs from the front camera and visualizations of the vehicle’s four camera views.The views show projected 3D bounding boxes colored by category.Source: A Dataset Visualization · p. 16, Figure 8 · p. 16
B Model Details
- The model-details section covers configurations for LiDAR-based and camera-based cooperative perception.Source: B Model Details · p. 16
B.1 LiDAR-based Cooperative Perception
The LiDAR pipeline converts each agent’s point cloud into BEV features for fusion, then predicts boxes and class scores using category-specific anchors.
B.1 LiDAR-based Cooperative Perception
- Each agent first projects its LiDAR into the ego agent’s coordinate frame; raw points become pillar features, then scatter into a 2D BEV feature map shared with neighboring connected agents.The voxel size is 0.4 meters in x and y; the number of points per voxel is set to 32.Source: §B.1 · p. 16
- The ego agent fuses received collaborator feature maps with its own features using F-Cooper, AttFuse, or V2X-ViT.F-Cooper uses max pooling, AttFuse uses per-location attention, and V2X-ViT uses multi-scale window and heterogeneous self-attention.Source: §B.1 · p. 17
- Two 1x1 convolution layers predict box regression and classification, with category-specific fixed-size anchors and learned displacements.Regression predicts position, dimensions, and yaw relative to the anchor; classification outputs confidence scores for pedestrians, cars, and trucks.Source: §B.1 · p. 17
B.2 Camera-based Cooperative Perception
The camera-based setup extracts image features and benchmarks single-agent No Fusion against proposal-sharing Late Fusion, evaluating detections within 50 meters of the ego agent.
B.2 Camera-based Cooperative Perception
- For computational savings, the camera pipeline uses only the left image from each stereo pair, yielding four images per vehicle.Source: §B.2 · p. 17
- The qualitative figure shows cooperative perception in a smart intersection and a V2V corridor, with ground-truth and detection boxes colored blue and red.Source: Figure 10 · p. 18
- The camera model uses a BEVFormer-like architecture without temporal information, ResNet50 image features, and an anchor-based detection head.The benchmarks compare No Fusion with Late Fusion.Source: §B.2 · p. 19
- No Fusion predicts 3D boxes from one agent’s camera data, while Late Fusion sends agents’ box proposals and confidence scores to the ego agent for NMS.Source: §B.2 · p. 19
- Camera perception is evaluated within a 50-meter range along both x and y directions in the ego coordinate frame.Source: §B.2 · p. 19
C More experimental results
The qualitative results show fusion outperforming No Fusion, with intermediate fusion producing more precise detections than Late Fusion.
C More experimental results
- Intermediate Fusion methods show superior precision to No Fusion and Late Fusion, and produce fewer false positives.Late Fusion identifies more true positives than No Fusion but also has more false negatives; the authors attribute this to lost context when sharing only detections.Source: §C.1 · p. 19
C.2 Camera-based Cooperative Perception
The benchmarks cover vehicle-side multi-view camera cooperative perception, leaving infrastructure-side camera solutions for future work. Late Fusion outperforms No Fusion on both V2X-Real-VC and V2X-Real-V2V, while both models perform higher on VC than V2V.
C.2 Camera-based Cooperative Perception
- The benchmarks cover vehicle-side multi-view camera cooperative perception; infrastructure-side camera solutions are left for future work.Infrastructure cameras have fixed road topology that could serve as a detection prior, motivating a dedicated algorithm for efficient Infrastructure-to-Everything perception.Source: §C.2 · p. 19
- Late Fusion outperforms No Fusion on both V2X-Real-VC and V2X-Real-V2V, while both models perform higher on V2X-Real-VC than V2X-Real-V2V.The authors associate the lower V2V performance with corridor scenarios' diverse roads and steep slopes, which challenge BEV-based camera 3D pose estimation.Source: §C.2 · p. 20, Table 4 · p. 20