PaperlayerView exact revision on arXiv

Source-linked AI summary

OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication

Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, Jiaqi Ma

arXiv:2109.07644v5cs.CVcs.RO

TL;DR

V2V perception lacks a large open dataset for developing and benchmarking cooperative methods, especially under occlusion. The paper introduces OPV2V, benchmarks fusion strategies, and proposes Attentive Intermediate Fusion; experiments report strong fusion gains and favorable accuracy-bandwidth trade-offs under compression.

  • Problem

    V2V perception lacks a large open dataset suitable for benchmarking cooperative algorithms, while occlusion and sparse observations challenge single-vehicle detection.

  • Method

    The paper constructs OPV2V with diverse simulated scenes, benchmarks early, late, and intermediate fusion, and proposes an Attentive Intermediate Fusion pipeline for connected vehicles.

  • Results

    All fusion methods achieve ≥10% AP gains at IoU 0.7 over no-fusion counterparts, while Attentive Intermediate Fusion drops around 3% at 4096x compression and surpasses early and late fusion.

  • Takeaways & Limitations

    OPV2V provides an open benchmark for V2V perception, and the proposed pipeline achieves state-of-the-art performance under large compression rates.

Abstract

from arXiv · show

Employing Vehicle-to-Vehicle communication to enhance perception performance in self-driving technology has attracted considerable attention recently; however, the absence of a suitable open dataset for benchmarking algorithms has made it difficult to develop and assess cooperative perception technologies. To this end, we present the first large-scale open simulated dataset for Vehicle-to-Vehicle perception. It contains over 70 interesting scenes, 11,464 frames, and 232,913 annotated 3D vehicle bounding boxes, collected from 8 towns in CARLA and a digital town of Culver City, Los Angeles. We then construct a comprehensive benchmark with a total of 16 implemented models to evaluate several information fusion strategies~(i.e. early, late, and intermediate fusion) with state-of-the-art LiDAR detection algorithms. Moreover, we propose a new Attentive Intermediate Fusion pipeline to aggregate information from multiple connected vehicles. Our experiments show that the proposed pipeline can be easily integrated with existing 3D LiDAR detectors and achieve outstanding performance even with large compression rates. To encourage more researchers to investigate Vehicle-to-Vehicle perception, we will release the dataset, benchmark methods, and all related codes in https://mobility-lab.seas.ucla.edu/opv2v/.

I. INTRODUCTION

V2V communication addresses blind spots that single vehicles struggle to perceive and enables cooperative perception through shared sensing information. OPV2V provides an open large-scale dataset and benchmark, including an attentive intermediate-fusion pipeline.

  • Heavily occluded or small objects cause detection performance to drop because sensor observations become too sparse.
  • Connected automated vehicles can share raw data, features, detections, and metadata to provide multiple viewpoints of the same obstacle.
  • Existing V2V studies lack a large open dataset suitable for benchmarking, because available datasets are small, low-variance, customized, or unreleased.
  • OPV2V collects 73 divergent scenes using OpenCDA and CARLA, adds a digital Culver City town, and benchmarks detectors with multiple fusion strategies.
  • The proposed Attentive Intermediate Fusion pipeline captures interactions between connected agents and reduces bandwidth requirements while achieving state-of-the-art performance.

II. RELATED WORK

Prior V2V perception methods exchange raw data, detections, or intermediate features, while the field lacks a large, public, reproducible dataset. OPV2V is positioned to address this gap with a multi-sensor cooperative benchmark.

  • Vehicle-to-Vehicle Perception: Early fusion shares raw data and preserves complete measurements but requires large bandwidth and is difficult to operate in real time.
  • Vehicle-to-Vehicle Perception: Late fusion transmits detection outputs, whereas intermediate fusion shares features to balance bandwidth and detection accuracy.
  • Vehicle-to-Vehicle Dataset: OPV2V includes sensor and scenario examples spanning CARLA simulations and a digital town designed to resemble Culver City.
  • Vehicle-to-Vehicle Perception: Intermediate-fusion examples include max pooling of shared Voxel features and joint reasoning over bounding boxes and trajectories.
  • Vehicle-to-Vehicle Dataset: Existing V2V datasets are often unrealistic, unavailable, limited in size or area, fixed in vehicle count, or difficult to reproduce.

A. Data Collection

OPV2V uses OpenCDA with CARLA and SUMO to generate cooperative driving data across multiple connected vehicles and varied sensor configurations. The collection combines eight CARLA towns with realistic Culver City scenarios.

  • Simulator Selection: OpenCDA supplies multi-CAV control, vehicle-network communication protocols, and traffic management integrated with CARLA and SUMO.
  • Culver City Digital Town: Most data comes from eight default CARLA towns, while Culver City scenes imitate realistic configurations and challenging driving environments.
  • Sensor Configuration: Each frame contains a minimum of 2 and maximum of 7 connected vehicles, with four cameras, a 64-channel LiDAR, and GPS/IMU sensors per CAV.
  • Sensor Configuration: Sensor data streams at 20 Hz and is recorded at 10 Hz.

B. Data Analysis

The dataset uses short, varied scenes and diverse traffic configurations, with annotations distributed across orientations, sizes, and long-range objects. Aggregated observations from multiple CAVs increase point density and compensate for sparse or occluded measurements.

  • 73 scenes use six road categories, short 16.4-second average travels, varied locations and maneuvers, and diverse traffic and CAV configurations.
  • Ground-truth vehicle boxes are distributed across divergent orientations and bounding-box sizes.
  • A large portion of objects lies at distances ≥ 100m because ground-truth boxes use aggregated LiDAR points from all CAVs.
  • Other CAVs compensate for sparse LiDAR points on distant objects, increasing point density and perception range while addressing occlusions.

IV. ATTENTIVE INTERMEDIATE FUSION PIPELINE

The Attentive Intermediate Fusion pipeline coordinates connected vehicles’ metadata, extracted features, optional compression, feature sharing, self-attention fusion, and prediction. It is designed to emphasize important observations while accommodating bandwidth constraints and existing LiDAR detector backbones.

  • Metadata Sharing and Feature Extraction: The pipeline uses metadata sharing to build a spatial communication graph, then projects neighboring CAV point clouds into the ego vehicle’s LiDAR frame.Each graph node is a CAV within communication range, and relative poses and extrinsics support spatial alignment.
  • Attentive Fusion: Self-attention fuses decompressed features so the network can emphasize important observations and ignore disrupted ones.The motivation is that connected vehicles may provide observations with different noise levels, for example because of distance.
  • Prediction: The pipeline ends with prediction and can be integrated with existing deep-learning LiDAR detectors.
  • Metadata Sharing and Feature Extraction: Feature extraction uses the backbones of existing 3D object detectors after neighboring point clouds are aligned to the ego vehicle.
  • Compression and Feature Sharing: An Encoder-Decoder optionally compresses high-dimensional feature maps before transmission and reconstructs them at the ego vehicle.The encoder uses 2D convolutions and max pooling, while the decoder uses deconvolution layers.

A. Benchmark models

The benchmark evaluates four LiDAR-based 3D detectors under no-fusion, early-fusion, late-fusion, and intermediate-fusion settings. It uses a unified implementation framework and fixed evaluation conditions for comparing the resulting 16 models.

  • Benchmark Models: Four state-of-the-art LiDAR 3D detectors are combined with four settings, producing 16 benchmark models.The settings are no fusion, early fusion, late fusion, and intermediate fusion.
  • Evaluation Setting: Table IV reports object detection results on Default CARLA Towns and the digital Culver City environment.
  • Late Fusion Baseline: Late fusion independently predicts bounding boxes on each CAV, broadcasts scored outputs, and applies non-maximum suppression to produce final predictions.
  • Intermediate Fusion: Intermediate fusion adds Compression, Sharing, and Attention modules to generalize the Attentive Fusion pipeline across object detection networks.
  • Evaluation Setting: Evaluation fixes one ego vehicle per test and validation scenario, uses a 70-meter broadcast range, and reports AP at IoU thresholds 0.5 and 0.7.Detection is evaluated near the ego vehicle within x ∈[−140, 140]m and y ∈[−40, 40]m.

C. Experiment Details

The experiments use fixed dataset splits, two-to-seven CAVs per frame, and standardized training procedures across models. Additional analyses vary CAV quantity and feature-data compression while measuring AP at IoU 0.7.

  • Dataset and Training: The dataset is split into 6764 training, 1981 validation, and 2719 testing frames.Testing includes all road types and is divided between CARLA default maps and the Culver City digital town.
  • Dataset and Training: Each frame contains a minimum of 2 and maximum of 7 CAVs, while models are trained with Adam and early stopping.
  • CAV Quantity and Compression: The compression study reports Average Precision at IoU=0.7 against data size in log scale, with number× indicating compression rate.
  • Training Cost: All training takes 14 days on 4 RTX 3090 GPUs.

D. Benchmark Analysis

Fusion improves cooperative detection across the benchmark, with intermediate fusion generally strongest and performance varying between CARLA and Culver City. Adding CAVs improves AP, but gains taper after four vehicles.

  • Fusion Strategy Comparison: ≥10% AP gains at IoU 0.7 occur for all fusion methods over no fusion across both CARLA and Culver City test sets.Early fusion generally outperforms late fusion, while intermediate fusion is usually best except for PIXOR in Culver City.
  • Fusion Strategy Comparison: Intermediate fusion generally achieves the best performance because self-attention captures correlations among different CAV perception features.PIXOR at Culver City is the stated exception.
  • Domain and Scenario Effects: Culver City results are generally inferior to CARLA results because its more realistic traffic pattern creates a training-data domain gap.Busy-hour congestion also produces severe occlusions, making detection challenging.
  • Effect of CAV Quantity: AP positively correlates with CAV number, but the increase slows after four vehicles.Four vehicles can cover most intersection blind spots; additional vehicles mainly provide denser measurements on the same object.

F. Effect of Compression Rates

Attentive Intermediate Fusion maintains strong perception accuracy under severe communication compression while offering a favorable bandwidth–accuracy trade-off. The broader benchmark positions this pipeline alongside an open V2V dataset and fusion-strategy evaluation, with future extensions planned for additional tasks and sensor suites.

  • 4096x compression reduces Attentive Intermediate Fusion performance by only around 3% while surpassing early and late fusion.The comparison uses VoxelNet on the CARLA-town testing set.
  • Attentive Intermediate Fusion outperforms other fusion methods and achieves state-of-the-art performance even under large compression rates.
  • Future work will extend the dataset with more tasks and sensor suites and investigate multimodal fusion in V2V and V2I settings.
Loading 2109.07644v5…