PaperlayerView exact revision on arXiv

Source-linked AI summary

V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving

Yiming Li, Dekun Ma, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, Chen Feng

arXiv:2202.08449v2cs.CV

TL;DR

Collaborative perception lacks well-designed public datasets because realistic multi-agent data collection is expensive and laborious. V2X-Sim provides simulated, synchronized multi-agent and multimodal data, annotations, an open-source testbed, and benchmarks for detection, tracking, and segmentation, with over 60% of annotations observed by at least two ego vehicles.

  • Problem

    Collaborative perception lacks well-designed, organized public datasets, while collecting realistic multi-agent data is expensive and laborious.

  • Method

    V2X-Sim uses CARLA-SUMO co-simulation to provide synchronized vehicle and RSU sensor streams, multimodal data, diverse annotations, an open-source testbed, and three-task benchmarks.

  • Results

    Over 60% of annotations are observed by at least two ego vehicles, and the benchmark covers collaborative detection, tracking, and semantic segmentation.

  • Takeaways & Limitations

    V2X-Sim supports collaborative perception research before realistic multi-agent datasets become widely available.

Abstract

from arXiv · show

Vehicle-to-everything (V2X) communication techniques enable the collaboration between vehicles and many other entities in the neighboring environment, which could fundamentally improve the perception system for autonomous driving. However, the lack of a public dataset significantly restricts the research progress of collaborative perception. To fill this gap, we present V2X-Sim, a comprehensive simulated multi-agent perception dataset for V2X-aided autonomous driving. V2X-Sim provides: (1) \hl{multi-agent} sensor recordings from the road-side unit (RSU) and multiple vehicles that enable collaborative perception, (2) multi-modality sensor streams that facilitate multi-modality perception, and (3) diverse ground truths that support various perception tasks. Meanwhile, we build an open-source testbed and provide a benchmark for the state-of-the-art collaborative perception algorithms on three tasks, including detection, tracking and segmentation. V2X-Sim seeks to stimulate collaborative perception research for autonomous driving before realistic datasets become widely available. Our dataset and code are available at \url{https://ai4ce.github.io/V2X-Sim/}.

I. INTRODUCTION

Single-vehicle viewpoints can degrade perception in long-range or occluded areas, motivating collaborative perception through V2X. V2X-Sim addresses the dataset gap with multi-agent, multimodal recordings, diverse annotations, an open-source testbed, and benchmarks for three tasks.

  • V2X communication adds viewpoints from vehicles and other entities to address long-range and occluded perception areas.
  • The dataset records synchronized sensor streams from multiple vehicles and an RSU using CARLA-SUMO co-simulation.
  • V2X-Sim provides diverse annotations and benchmarks collaborative detection, tracking, and segmentation.
  • The authors position V2X-Sim as the first public multi-agent, multi-modality dataset supporting different collaborative perception tasks.
  • V2X-Sim is a comprehensive dataset for multi-agent, multi-modality, and multi-task perception research.

II. RELATED WORK

Related work expands autonomous-driving datasets across scenarios and sensor modalities, while collaborative perception datasets remain limited in realistic multi-agent viewpoints and broad V2X support.

  • Autonomous-driving datasets have expanded coverage across urban, adverse-weather, night, and multi-city scenarios and across Radar, cameras, LiDAR, and fisheye cameras.
  • V2X research uses vehicle, infrastructure, or multimodal information sharing to address limited sensing range and occlusion.
  • Existing simulated V2V datasets may reuse single-agent frames rather than capture measurements from different viewpoints.

III. V2X-SIM DATASET

V2X-Sim combines vehicle and RSU sensing, broad sensor modalities, coordinate-system design, and diverse annotations to support collaborative perception research.

  • V2X-Sim aims to enable research on collaboration strategies among vehicles for more robust perception.
  • Each vehicle carries RGB cameras, LiDAR, GPS, and IMU, while the RSU carries RGB cameras and LiDAR.
  • Vehicle cameras and LiDAR provide 360° horizontal coverage, while the RSU uses four cameras directed toward the crossroad.
  • The sensor layout aligns the BEV semantic camera's x,y position with LiDAR while placing it higher for field of view.
  • The dataset includes 3D bounding boxes, pixel-wise and point-wise semantic labels, 23 categories, and precise depth values.

B. CARLA-SUMO co-simulation

The dataset models realistic V2X scenes by placing multiple routed vehicles in one intersection with an RSU and using open-source CARLA-SUMO co-simulation for traffic and recording.

  • A realistic V2X scenario contains multiple vehicles with their own routes simultaneously located in the same intersection.
  • Each intersection includes one sensing-capable RSU.
  • The authors use open-source CARLA-SUMO co-simulation to generate traffic flow and record the data.

C. Downstream tasks

V2X-Sim supports both single-agent and collaborative perception across detection, tracking, segmentation, and depth-related tasks. Its annotations and scene statistics provide varied multi-agent supervision for urban driving.

  • V2X-Sim supports single-agent detection, tracking, semantic segmentation, and depth estimation alongside collaborative 3D detection, tracking, and BEV semantic segmentation.
  • Scenes span crowded and less crowded conditions, with up to 100 annotations within 50–70m and as few as 10 within 30m.
  • 183.83 LiDAR points per annotation for a single agent increase to 875.59 points with multiple agents observing the same object.
  • Over 60% of annotations are observed by at least two ego vehicles, while vehicle speeds range from below 10 km/h for many cars to above 90 km/h.

IV. COLLABORATIVE PERCEPTION BENCHMARK

The benchmark evaluates collaborative perception under standardized BEV representations and compares no-collaboration, fusion, raw-data, and intermediate-feature strategies. It covers detection, tracking, and semantic segmentation with V2V and V2I information.

  • The benchmark covers collaborative detection, tracking, and semantic segmentation, evaluating V2V-only and V2V-plus-V2I settings against single-agent perception.
  • BEV inputs are represented as 3D voxel grids and processed as pseudo-images with efficient 2D convolution; outputs are generated in 2D BEV.
  • The benchmark includes lower-bound, co-lower-bound, upper-bound, and intermediate collaboration models based on different information-sharing stages.
  • DiscoNet adaptively highlights informative spatial regions and rejects noisy message regions using a directed collaboration graph.
  • V2VNet propagates agent information with pose-aware graph message passing, while When2com and Who2com select collaborators using attention or handshake mechanisms.
  • All intermediate collaboration methods use the same architecture and collaborate at the same intermediate feature layer for comparison.

A. Collaborative object detection in BEV

The BEV detection benchmark measures object localization from voxelized point clouds and compares collaborative strategies using AP. Collaboration improves detection, especially when additional RSU information is available, while late fusion can introduce false positives.

  • 3D object detection recognizes and localizes objects from a single frame, consuming voxelized point clouds and outputting BEV bounding boxes.
  • Average Precision is evaluated at IoU thresholds of 0.5 and 0.7 using an anchor-based detector with classification and regression losses.
  • The upper-bound improves lower-bound by 41.1% at AP@0.5 and 51.6% at AP@0.7 in the V2V-only scenario.
  • Adding V2I improves AP@0.5 by 9.4% for upper-bound and 5.5% for V2VNet compared with V2V-only collaboration.
  • DiscoNet performs best among intermediate models, whereas When2com and Who2com remain comparable to lower-bound in point-cloud collaboration.
  • Late collaboration hurts detection by introducing extra false positives from other vehicles.
  • Qualitative BEV results show collaboration mitigating long-range perception and occlusion problems.

B. Collaborative multi-object tracking in BEV

BEV tracking evaluates temporally consistent object identities using unified detection, association, and localization metrics. Collaboration improves detection and tracking accuracy, but localization remains comparatively limited and calls for stronger trackers.

  • Evaluation metrics: HOTA, MOTA, and MOTP evaluate unified tracking quality, detection and association errors, and localization accuracy, respectively.HOTA combines detection, association, and localization in one metric; MOTP measures localization alone.
  • Baseline tracker: SORT combines a Kalman Filter with the Hungarian algorithm to track objects from detection results.
  • Quantitative results: Upper-bound achieves the best MOTA and HOTA, while adding V2I substantially improves MOTA but provides limited MOTP gains.Co-lower-bound performs well on localization accuracy, measured by MOTP.
  • Quantitative results: A more advanced tracker is required to exploit collaboration and close the remaining performance gap.

C. Collaborative semantic segmentation in BEV

Collaborative BEV semantic segmentation uses geometry point clouds from multiple viewpoints to improve scene understanding. Results show benefits from complementary collaboration, especially for safety-critical vehicle and pedestrian categories.

  • Baseline segmentation method and evaluation metrics: The segmentation task predicts seven BEV semantic categories from a voxelized geometry point cloud and evaluates performance with mIoU.The baseline follows U-Net’s backbone architecture and loss function.
  • Quantitative results: V2VNet, DiscoNet, and upper-bound achieve comparable performance for terrain and road categories.
  • Quantitative results: Attention-based when2com and who2com perform worse because they seek highly correlated collaborators instead of complementary information.Nonalignment between attention selection and complementary 3D information makes the attention model difficult to train.
  • Quantitative results: Lower-bound versus upper-bound performance differs substantially for vehicle (45.93% v.s. 64.09%) and pedestrian (20.59% v.s. 31.54%).The reported gaps concern two safety-critical categories.
  • Quantitative results: V2V and V2I jointly generally enhance vehicle segmentation over V2V alone, while co-lower-bound outperforms lower-bound.
  • Qualitative results: Qualitative results show rich semantic restoration for upper-bound, satisfactory performance for V2VNet and DiscoNet, and degradation for When2com and Who2com.

D. Discussions on pose noise and compression ratio

The benchmark tests collaborative intermediate models under pose noise and feature compression. These models remain stable across the tested conditions, supported by low-resolution feature maps and jointly learned compression.

  • Pose noise: Intermediate models perform comparably with or without training-time pose-noise augmentation and remain stable against pose noise.The tested Gaussian noise has a mean of 0.05m−0.25m and a standard deviation of 0.02cm.
  • Pose noise: The feature map’s 2m × 2m grid coverage reduces vulnerability to noisy pose.
  • Compression ratio: A jointly learned 1 × 1 autoencoder can slightly improve performance while compressing transmitted feature channels.
  • Compression ratio: Most intermediate models achieve comparable performance across different compression ratios.

V. CONCLUSION

V2X-Sim is a CARLA-SUMO dataset and benchmark for multi-agent collaborative perception. It combines multi-agent, multi-modality sensing and diverse annotations across detection, tracking, and semantic segmentation.

  • V. CONCLUSION: V2X-Sim uses CARLA-SUMO co-simulation to provide multi-agent multi-modality streams from vehicles and an RSU in realistic traffic flows.
  • V. CONCLUSION: Diverse annotations support a variety of 3D perception tasks.
  • V. CONCLUSION: The benchmark evaluates collaborative BEV detection, tracking, and semantic segmentation with state-of-the-art collaborative perception methods.
  • V. CONCLUSION: Future work includes simulating latency issues and developing novel evaluation metrics for collaborative perception.
Loading 2202.08449v2…