PaperlayerView exact revision on arXiv

Source-linked AI summary

V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception

Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang, Zhengzhong Tu, Zonglin Meng, Hao Xiang, Xiaoyu Dong, Rui Song, Hongkai Yu, Bolei Zhou, Jiaqi Ma

arXiv:2303.07601v2cs.CV

TL;DR

Single-vehicle perception struggles with occlusions and limited range, and real-world V2V benchmarks are scarce. V2V4Real provides a large multimodal real-world dataset with three cooperative perception tasks and benchmarks, showing effectiveness across tasks while exposing practical deployment constraints.

  • Problem

    Single-vehicle systems face occlusions and short sensing range, while the lack of public real-world V2V benchmarks limits validation of cooperative perception.

  • Method

    V2V4Real collects multimodal data from two vehicles across diverse real-world scenarios and provides cooperative detection, tracking, and Sim2Real benchmarks.

  • Results

    V2V cooperation improves performance across multiple tasks; in detection, cooperative methods improve overall AP@0.5 by at least 15.2% over single-vehicle perception.

  • Takeaways & Limitations

    V2V4Real supplies real-world data and benchmarks for developing and evaluating cooperative perception beyond synthetic datasets.

  • Takeaways & Limitations

    Cooperative detection must account for relative-pose error, sensor asynchrony, and restricted communication bandwidth.

Abstract

from arXiv · show

Modern perception systems of autonomous vehicles are known to be sensitive to occlusions and lack the capability of long perceiving range. It has been one of the key bottlenecks that prevents Level 5 autonomy. Recent research has demonstrated that the Vehicle-to-Vehicle (V2V) cooperative perception system has great potential to revolutionize the autonomous driving industry. However, the lack of a real-world dataset hinders the progress of this field. To facilitate the development of cooperative perception, we present V2V4Real, the first large-scale real-world multi-modal dataset for V2V perception. The data is collected by two vehicles equipped with multi-modal sensors driving together through diverse scenarios. Our V2V4Real dataset covers a driving area of 410 km, comprising 20K LiDAR frames, 40K RGB frames, 240K annotated 3D bounding boxes for 5 classes, and HDMaps that cover all the driving routes. V2V4Real introduces three perception tasks, including cooperative 3D object detection, cooperative 3D object tracking, and Sim2Real domain adaptation for cooperative perception. We provide comprehensive benchmarks of recent cooperative perception algorithms on three tasks. The V2V4Real dataset can be found at https://research.seas.ucla.edu/mobility-lab/v2v4real/.

1. Introduction

Single-vehicle perception is limited by occlusion and short sensing range, while real-world V2V research lacks public benchmarks. V2V4Real addresses this gap with a large multimodal dataset and benchmarks for cooperative perception.

  • Single-vehicle systems suffer from occlusions and short-range perception because each vehicle has a limited field of view.
  • Synthetic V2V datasets leave a domain gap because simulator traffic behavior and sensor rendering are not realistic enough.
  • V2V4Real covers 410 km and contains 20K LiDAR frames with more than 240K annotated 3D bounding boxes.
  • The dataset uses real multimodal sensor readings collected across diverse scenarios and provides HD maps along the driving routes.
  • V2V4Real benchmarks cooperative 3D detection, object tracking, and Sim2Real domain adaptation, with results showing V2V effectiveness across multiple tasks.

2. Relaed Work

Autonomous-driving datasets have progressed from 2D imagery toward multimodal 3D sensing, but major benchmarks still target single-vehicle perception. Cooperative datasets extend sensing through communication, while real-world V2V coverage remains limited.

  • Modern autonomous-driving datasets increasingly combine camera images with LiDAR or radar to support 3D and map-view perception.
  • KITTI provides stereo camera and LiDAR data for 22 sequences with 200k annotated 3D boxes and multiple perception tasks.
  • NuScenes and Waymo Open scale multimodal 3D datasets to over 1K scenes, with 1.4M and 993K annotated 3D boxes, respectively.
  • OPV2V, V2XSet, and V2X-Sim study cooperative perception with synthesized CARLA scenes, whereas DAIR-V2X provides real-world V2I data.
  • V2V cooperation broadens perception beyond single-view limits, but requires real-world datasets covering diverse traffic scenarios, occlusions, and road types.

3. V2V4Real Dataset

V2V4Real is a real-world, large-scale, multimodal V2V dataset collected by two sensor-equipped vehicles across diverse driving scenarios. It provides aligned annotations, maps, and varied vehicle configurations for cooperative-perception research.

  • Dataset overview: V2V4Real is a real-world, large-scale, multimodal dataset with 3D bounding boxes and HDMaps for multi-vehicle cooperative perception.The dataset covers diverse driving scenarios and supports research on cooperative perception.
  • Data collection: Two connected vehicles—a Tesla and a Ford Fusion—each carry a Velodyne VLP-32 LiDAR, two mono cameras, and GPS/IMU integration systems.The vehicles collect synchronized multimodal observations while driving simultaneously in Columbus, Ohio.
  • Data collection: 20K LiDAR frames and 40K RGB frames are sampled at 10Hz from 67 manually selected scenarios lasting 10–20 seconds.The scenarios maintain less than 50 ms of sensor asynchronization between the vehicles and are aligned with maps.
  • Data annotation: The dataset uses four coordinate systems and initializes inter-vehicle relative poses from GPS to support unified spatial annotation.Separate LiDAR-coordinate annotations also allow each vehicle’s sensor data to be used for single-agent detection.
  • Data annotation: Cross-vehicle object identities are unified by transforming objects into a common coordinate system and matching them with BEV IoU.The map pipeline fuses point-cloud frames after removing dynamic objects and applies scan matching to estimate relative transformations.
  • Data analysis: Relative poses vary across scenarios, while Car is the most common vehicle class, Pickup Truck is second, and Bus is least frequent.V2V4Real therefore contains varied view combinations and vehicle-type distributions.
  • Data analysis: With cooperative visual information, object LiDAR point density remains high at 100 m, unlike single-vehicle scanning where density drops sharply with distance.This comparison uses Tesla-only scanning versus shared information from Ford Fusion.

4. Tasks

V2V4Real defines cooperative detection, tracking, and Sim2Real adaptation tasks, with evaluation protocols addressing accuracy, tracking quality, communication cost, and real-world transfer. The benchmark includes no-fusion and multiple fusion strategies under cooperative sensing constraints.

  • Task scope: The paper focuses on cooperative 3D object detection, cooperative object tracking, and Sim2Real transfer learning.The dataset also supports other cooperative-perception tasks, but these three are evaluated in the paper.
  • Cooperative 3D object detection: Cooperative detection uses multiple vehicles’ LiDAR views to detect 3D objects for the ego vehicle despite GPS error, asynchrony, and bandwidth limits.Ground truth is transformed into the ego vehicle’s unified coordinate system.
  • Cooperative 3D object detection: Detection is evaluated with AP at IoU 0.5 and 0.7, while Average MegaByte measures transmitted data size.The evaluation range is [−100, 100] m in x and [−40, 40] m in y relative to the ego vehicle.
  • Fusion strategies: The benchmark compares no fusion, late fusion, early fusion, and intermediate fusion strategies.Early fusion transmits raw point clouds, late fusion combines object predictions, and intermediate fusion exchanges compressed neural features.
  • Cooperative 3D object tracking: Tracking investigates whether cooperation benefits tracking-by-detection models using metrics including MOTA, MT, and ML.The task focuses on tracking by detection rather than joint detection and tracking.
  • Sim2Real domain adaptation: Sim2Real adaptation uses annotated simulated OPV2V data as the source and unlabeled V2V4Real LiDAR as the target for cooperative 3D detection.Methods are evaluated on the V2V4Real test set under the Sync setting.

5. Experiments

Experiments benchmark cooperative detection, tracking, and Sim2Real adaptation on V2V4Real. Cooperative methods generally outperform single-vehicle baselines, while communication delay and simulation-to-reality gaps remain important constraints.

  • 3D LiDAR Object Detection: At least 15.2% overall AP@0.5 gains over single-vehicle perception were observed across cooperative detection methods.
  • 3D LiDAR Object Detection: Long-range detection improved by at least 28.0% for AP@0.5 and 11.8% for AP@0.7 compared with the single-vehicle baseline.
  • 3D LiDAR Object Detection: CoBEVT achieved the best intermediate-fusion AP@0.5, exceeding V2X-ViT by 1.6% and outperforming Early Fusion and Late Fusion by 6.8% and 11.5%.These comparisons are reported in the Sync setting.
  • 3D LiDAR Object Detection: Communication delay reduced AP for most cooperative methods, including CoBEVT, V2X-ViT, and V2VNet by 6.3%, 7.6%, and 5.8% at AP@0.7.No Fusion was the exception reported in this comparison.
  • 3D Object Tracking: CoBEVT with cooperative detection improved tracking over the single-vehicle baseline by 16.04% in AMOTA, 23.81% in sAMOTA, and 14.01% in AMOTP.CoBEVT achieved the best performance in most reported tracking metrics.
  • Sim2Real Domain Adaptation: Without domain adaptation, simulated-data training caused detection accuracy drops ranging from 33.9% to 42.2% across the evaluated methods.Domain adaptation reduced the performance drop by an average of 7.46%.

6. Conclusion

The paper concludes by presenting V2V4Real as a public real-world benchmark for cooperative perception. It also identifies challenging unseen scenarios and out-of-distribution detection as directions for future robust and safe perception research.

  • Conclusion: V2V4Real covers 410 km, 20K LiDAR frames, 40K RGB images, 240K bounding boxes, and HDMaps for V2V cooperative perception research.
  • Conclusion: The benchmark includes cooperative 3D object detection, object tracking, and Sim2Real domain adaptation tasks.
  • Conclusion: The dataset, benchmarks, and baseline models are intended to be made fully public to support future research and task development.
  • Broader Impact: Some extremely challenging driving scenarios may not appear in the training set, requiring careful training to preserve generalization abilities.
  • Broader Impact: Out-of-distribution detection is outside the paper’s scope and is identified as an important topic for future robust and safe autonomous perception research.

B. Dataset Visualization

The visualization presents LiDAR appearances for each object class and illustrates multimodal observations across three driving scenes. Each scene combines aggregated LiDAR, HDMap annotations, and front-camera views from both vehicles.

  • Scene Visualization: Figure 9 visualizes three scenes—a city road, a highway, and another city road—using four views per scene.
  • Scene Visualization: The four views comprise aggregated 3D LiDAR points, an annotated HDMap, and front-camera views from the two vehicles.Green and red LiDAR correspond to the lower-left and lower-right views, respectively.
  • Object-Class LiDAR Appearance: Figure 8 shows the LiDAR appearance associated with each object class in V2V4Real.

C. Implementation Details

The benchmark implements cooperative detection and tracking with shared multimodal information, then applies standard tracking-by-detection components for prediction, association, updating, and trajectory management.

  • Fusion models: The implementation evaluates five fusion methods: F-Cooper, AttFuse, V2VNet, V2X-ViT, and CoBEVT.V2X-ViT treats the two vehicles as the same object type because V2V4Real contains no infrastructure.
  • Detection head: The detection head predicts anchor-box position, size, and yaw while classifying each anchor as object or background.The regression output is (x, y, z, w, l, h, θ), and the classification head produces confidence scores.
  • Cooperative tracking: Cooperative tracking follows tracking-by-detection, using detections from shared visual information rather than individual vehicles.Its inputs are bounding boxes generated by the cooperative 3D detection task.
  • Trajectory prediction and association: A constant-velocity Kalman filter predicts trajectories, while 3D IoU and the Hungarian algorithm associate predictions with current detections.The affinity matrix contains 3D IoU values between predicted trajectories and detected bounding boxes.
  • State update and trajectory management: The Kalman filter updates associated trajectories using current detections, and unmatched objects are retained across additional frames before creation or removal.This delays new-trajectory creation and dead-trajectory removal to reduce false positives and preserve true positives.
  • Domain adaptation: The benchmark’s domain adaptation uses feature-level and object-level discriminators trained with binary crossentropy and a gradient reversal layer.The feature discriminator receives fused features, while the object discriminator receives the detection classification score map.

D. Ablation Studies

The ablation study evaluates how point-cloud rotation, flipping, and scaling augmentation affect cooperative 3D object detection.

  • Effects of Data Augmentation: Removing point-cloud rotation, flipping, and scaling augmentations causes a significant performance decrease across all evaluated methods.The comparison is reported in Table 6, whose bracketed values indicate drops relative to the same models with augmentation.

E. Detection Results

Qualitative Sync-setting results show that cooperation improves detection in crowded urban scenes, while all cooperative methods successfully detect surrounding vehicles on highways.

  • Urban scenarios: In crowded urban scenes, cooperative solutions detect vehicles better than No Fusion, with intermediate fusion more accurate than early or late fusion within 50 m.CoBEVT’s predictions best align with the ground-truth boxes among the compared approaches.
  • Highway scenarios: In highway scenes, all cooperative methods successfully predict surrounding vehicles, while V2X-ViT and CoBEVT are slightly more accurate than other approaches.The highway setting has faster but less crowded traffic than the urban setting.

F. Domain Adaptation Results

Qualitative domain-adaptation results indicate benefits across highway and intersection scenarios, with method rankings and false-positive behavior varying by scene.

  • Highway scenario: In the highway scenario, all models benefit from domain adaptation, with AttFuse and F-Cooper showing the largest performance differences.The comparison is between detection results without and with domain adaptation.
  • Intersection scenario: In the crowded intersection, F-Cooper, V2X-ViT, and CoBEVT are top performers after domain adaptation.F-Cooper produces more false positives, whereas V2X-ViT produces fewer.
Loading 2303.07601v2…