PaperlayerView exact revision on arXiv

Source-linked AI summary

V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer

Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, Jiaqi Ma

arXiv:2203.10638v3cs.CV

TL;DR

Autonomous vehicles can suffer from occlusion and sparse distant observations, while V2X systems must fuse heterogeneous agents under localization errors and communication delays. V2X-ViT uses unified heterogeneous and multi-scale attention with V2XSet for cooperative perception, achieving state-of-the-art 3D detection and robust performance in noisy settings. The paper reports a 21.2% AP gain over a single-agent baseline and at least 7.3% over leading intermediate fusion methods.

  • Problem

    Single-agent perception is limited by occlusion and sparse far-distance observations, while V2X fusion must handle heterogeneous vehicles and infrastructure, pose errors, and time delays.

  • Method

    V2X-ViT is a unified Transformer that alternates heterogeneous multi-agent self-attention with multi-scale spatial attention to fuse intermediate features from vehicles and infrastructure.

  • Results

    21.2% gain of AP over a single-agent baseline and at least 7.3% over leading intermediate fusion methods are reported for V2X LiDAR-based 3D object detection.

  • Takeaways & Limitations

    V2X-ViT significantly boosts cooperative 3D vehicle detection under both perfect and noisy settings, with robustness to common V2X challenges.

  • Takeaways & Limitations

    The study is limited to simulated data, LiDAR-only sensing, and vehicle detection; its initial LiDAR projection communication design also has limited scalability.

Abstract

from arXiv · show

In this paper, we investigate the application of Vehicle-to-Everything (V2X) communication to improve the perception performance of autonomous vehicles. We present a robust cooperative perception framework with V2X communication using a novel vision Transformer. Specifically, we build a holistic attention model, namely V2X-ViT, to effectively fuse information across on-road agents (i.e., vehicles and infrastructure). V2X-ViT consists of alternating layers of heterogeneous multi-agent self-attention and multi-scale window self-attention, which captures inter-agent interaction and per-agent spatial relationships. These key modules are designed in a unified Transformer architecture to handle common V2X challenges, including asynchronous information sharing, pose errors, and heterogeneity of V2X components. To validate our approach, we create a large-scale V2X perception dataset using CARLA and OpenCDA. Extensive experimental results demonstrate that V2X-ViT sets new state-of-the-art performance for 3D object detection and achieves robust performance even under harsh, noisy environments. The code is available at https://github.com/DerrickXuNu/v2x-vit.

1 Introduction

V2X-ViT addresses the limitations of single-agent and vehicle-only perception by fusing heterogeneous vehicle and infrastructure features under realistic communication conditions. Its unified Transformer and V2XSet dataset support robust cooperative 3D detection.

  • Motivation: Single-agent perception is vulnerable to occlusion and sparse far-distance observations, motivating collaboration across multiple viewpoints.V2V methods share detections, raw sensory information, or intermediate features to improve scene understanding.
  • Motivation: Infrastructure complements vehicles with persistent availability, elevated sensing, broader sight-of-view, and potentially less occlusion, but creates heterogeneous V2X graphs.Vehicle and infrastructure sensors can differ in configuration, types, and noise levels.
  • Method: V2X-ViT fuses intermediate features using heterogeneous multi-agent self-attention and multi-scale window attention to model agent interactions and spatial relationships.The modules are incorporated into a unified Transformer and address heterogeneity, localization errors, and time delay.
  • Dataset: V2XSet is a large-scale open dataset built with CARLA and OpenCDA that explicitly accounts for imperfect real-world V2X conditions.The dataset includes simulated scenarios with vehicles and infrastructure and communication-related noise conditions.
  • Results: 21.2% gain of AP over a single-agent baseline and at least 7.3% over leading intermediate fusion methods demonstrate improved V2X LiDAR-based 3D detection.The reported gains are from experiments on the proposed cooperative perception setting.

2 Related work

Prior cooperative perception methods differ by where information is fused, while vision Transformers provide long-range modeling but remain less studied for heterogeneous graphs. V2X-ViT adapts heterogeneous attention to multi-agent 3D feature fusion.

  • Cooperative perception: Cooperative perception uses early, intermediate, or late fusion depending on whether raw data, intermediate features, or detection outputs are transmitted.Intermediate fusion is presented as balancing accuracy and transmission bandwidth relative to early and late fusion.
  • LiDAR-based 3D object detection: LiDAR-based 3D detection methods extract features from raw points, voxels, bird-eye-view images, or mixtures of these representations.Prior approaches include two-stage raw-point processing and voxel-based feature extraction.
  • Transformers in vision: Vision Transformers use self-attention for long-range visual interactions, but full attention has heavy computational complexity on long sequences or high-resolution images.This limitation motivates more scalable attention designs.
  • Transformers in vision: Vision Transformers have been less studied for heterogeneous graphs than for homogeneous structured data.This gap is relevant to V2X systems containing infrastructure and vehicles.
  • Heterogeneous graph fusion: V2X-ViT builds a customized heterogeneous multi-head self-attention module for graph attribute-aware multi-agent 3D visual feature fusion.The design is inspired by heterogeneous graph transformers and captures V2X system heterogeneity.

3 Methodology

V2X-ViT is a unified fusion framework for heterogeneous V2X perception that addresses imperfect localization, transmission delays, and differing agent types. It alternates inter-agent HMSA with per-agent multi-scale spatial attention to produce robust features for object detection.

  • Problem setting: V2X perception is modeled as a heterogeneous multi-agent system in which vehicles and infrastructure communicate under imperfect localization and transmission delays.The ego vehicle constructs a V2X graph from connected vehicles and infrastructure using shared poses, extrinsics, and agent types.
  • Alignment and fusion: The architecture integrates spatial-temporal correction and delay-aware positional encoding so received features can be aligned before end-to-end Transformer fusion.Spatial-temporal correction warps feature maps to the current ego pose, while the projected time-delay embedding is added to agent features.
  • Overall architecture: V2X-ViT aggregates connected-agent features and applies iterative inter-agent and intra-agent self-attention before box regression and classification.The detection head predicts anchor-box position, size, and yaw together with object-or-background confidence.
  • Heterogeneous multi-agent self-attention: HMSA encodes heterogeneous relationships by attaching types to graph nodes and edges and using type-conditioned attention and message aggregation.Its operators include a node-type-indexed linear aggregator, an attention-weight estimator conditioned on node and edge types, and a message aggregator.
  • Multi-scale window attention: MSwin uses multiple window sizes to capture local and long-range spatial interactions, improving robustness against localization errors while preserving linear complexity in image size HW.Larger windows provide long-range cues for larger misalignments, while parallel branches are fused with a Split-Attention module.

4 Experiments

Experiments evaluate V2X-ViT on V2XSet under perfect and noisy conditions, comparing detection performance, robustness, component contributions, communication effects, and qualitative behavior. The results show strong performance under pose errors, delays, compression, and occlusion.

  • V2XSet: V2XSet contains 11,447 frames and explicitly models V2X cooperation with localization error and transmission time delay.The train/validation/test splits contain 6,694/1,920/2,833 frames.
  • Main performance comparison: V2X-ViT outperforms state-of-the-art intermediate fusion methods by 3.8%/1.7% for AP@0.5/0.7 under the Perfect Setting.It also exceeds ideal Early Fusion by 0.2% AP@0.7.
  • Main performance comparison: Under noisy conditions, V2X-ViT achieves 23% and 21.2% higher AP@0.5 and AP@0.7 than No Fusion.Its performance drops by less than 5% and 10% in AP@0.5 and AP@0.7 relative to the Perfect Setting.
  • Communication and qualitative evaluation: With 32× compression, V2X-ViT substantially outperforms other intermediate fusion methods and maintains high performance at 128× compression.Early Fusion drops to 28% under large bandwidth requirements.
  • Communication and qualitative evaluation: Attention visualizations show that V2X-ViT gives more attention to infrastructure in areas occluded from both autonomous vehicles.Qualitative examples report more accurate bounding boxes and identification of more dynamic objects.
  • Ablation studies: MSwin and HMSA provide the largest ablation gains, increasing AP@0.7 by 4.1% and 6.6%, respectively.The ablation progressively adds MSwin, split attention, HMSA, and delay-aware positional encoding.
  • Ablation studies: Multi-scale windows improve robustness to localization error, while delay-aware positional encoding improves performance under varied time delays.Larger windows capture long-range cues that compensate for larger localization errors.

5 Conclusion

The paper presents V2X-ViT and V2XSet for robust cooperative perception using heterogeneous-agent and multi-scale attention. Experiments show improved cooperative 3D vehicle detection under perfect and noisy settings, while the study remains limited in modality, task scope, scalability, and real-world generalization.

  • Conclusion: V2X-ViT combines heterogeneous multi-agent self-attention and multi-scale window attention for V2X perception.The modules capture inter-agent interactions and intra-agent spatial relationships.
  • Conclusion: V2XSet is a large-scale V2X perception dataset constructed to evaluate cooperative perception under realistic noise.The dataset is built using CARLA and OpenCDA.
  • Conclusion: V2X-ViT significantly improves cooperative 3D object detection under both perfect and noisy settings.The reported scope is LiDAR-based vehicle detection.
  • Conclusion: The work is limited to LiDAR-based cooperative 3D vehicle detection and a single sensor modality.Future work targets multi-sensor fusion for joint V2X perception and prediction.
  • Broader impacts and limitations: Simulated training data raises concerns about data bias and real-world generalization, while the communication design has limited scalability.The paper also identifies privacy and adversarial-robustness concerns in data sharing.

A Discussion of design choice

The design discusses scalable collaboration and contrasts the proposed feature-sharing strategy with broadcasting. It limits collaboration overhead through decentralized or centralized ego-vehicle aggregation.

  • Scalability of ego vehicles: Collaboration scales through decentralized sharing with selected agents or centralized aggregation by one ego agent within a communication range.The decentralized option limits feature sharing, while the centralized option uses one computation node for a group of agents.
  • Scalability of ego vehicles: When collaborators exceed four, performance gains become marginal while computation continues increasing linearly.This observation is reported from ablation studies and OPV2V.
  • Design choices for communication: The comparison between the proposed design and broadcasting is summarized in Table T0.The table is identified as a comparison of the two communication designs.
  • Design choices for communication: The proposed feature-sharing approach has higher detection accuracy than broadcasting, which transforms each CAV’s feature maps directly into the ego space.Broadcasting crops LiDAR data according to each CAV’s own evaluation range, limiting the shared information.

B Model Details and Analysis

The spatial-temporal correction module addresses feature misalignment caused by localization error and communication delay. It warps features using a differential 2D transformation and masks padded regions during attention.

  • Spatial-temporal correction: Received agent features are misaligned because the ego vehicle observes the scene at a different time than the agents’ transmitted observations.The correction uses the ego vehicle’s transformation from its past pose to its current pose.
  • Spatial-temporal correction: A differential 2D transformation Γξ(·) spatially warps intermediate features to correct pose and time-delay misalignment.The feature positions are transformed with an affine transformation.
  • Spatial-temporal correction: Bilinear interpolation samples feature vectors at non-integer transformed coordinates.An ROI mask prevents padded zeros introduced by spatial warping from receiving attention.
  • Spatial-temporal correction: The ROI mask sets attention weights for padded values to zero in heterogeneous multi-agent self-attention.This prevents spatially warped padding from influencing feature fusion.

B.2 Multi-Scale Window Attention (MSwin)

MSwin combines parallel window branches to capture multi-scale spatial interactions while keeping complexity linear in image size. Within V2X-ViT, it alternates with heterogeneous multi-agent attention in a unified feature-fusion architecture.

  • Multi-Scale Window Attention: MSwin applies self-attention within non-overlapping windows across multiple branches with different window sizes and head counts.Each branch independently processes local windows before its outputs are aggregated.
  • Multi-Scale Window Attention: Each MSwin branch computes relative self-attention with query, key, and value projections, then concatenates its attention-head outputs.Relative positional encoding is added as a bias to the attention map.
  • Multi-Scale Window Attention: Split-attention adaptively fuses the outputs from all MSwin branches into a combined feature representation.The branch outputs are aggregated after their separate multi-head computations.
  • Complexity analysis: MSwin has complexity O(HW), linear in image size, while providing multi-scale spatial interactions.The stated complexity is O((k^3P^2C/3 + 2k^2C^2/h)HW) ~ O(HW).
  • Architecture and complexity comparisons: The computational comparison in Table T1 covers full ViT, Axial, Swin, CSwin, and MSwin attention.Table T2 provides detailed architectural specifications for V2X-ViT.
  • Effective receptive field: Compared with Swin, Axial, and CSwin, MSwin aggregates multi-scale receptive fields in parallel with more symmetrical coverage and linear complexity.CSwin is described as requiring polynomial complexity O(N^1.5), whereas MSwin remains linear with respect to N.
  • V2X-ViT architecture: V2X-ViT processes corrected, delay-aware features through repeated HSMA, MSwin, and MLP blocks before prediction.The architecture maintains high-resolution feature maps throughout the Transformer.

C V2XSet Dataset

V2XSet is a simulated cooperative-perception dataset spanning 55 scenes, five roadway types, and eight towns. Its scenarios include multiple communicating agents equipped with 32-channel LiDAR.

  • Dataset statistics: V2XSet contains 55 representative CARLA scenes covering five roadway types and eight towns.Each scene lasts 25 seconds.
  • Dataset statistics: Each scene includes at least two and at most seven intelligent agents that can communicate with one another.The dataset models multi-agent cooperative driving scenarios.
  • Sensor configuration: Every agent uses a 32-channel LiDAR sensor with a 120-meter data range.Sensors are mounted on top of each autonomous vehicle, while infrastructure sensors are deployed in selected roadway settings.
  • Dataset distribution: Figure 9 presents the distribution of data across the five roadway types.The figure summarizes the dataset’s roadway-type coverage.

D.1 Performance for identifying dynamic objects

Under noisy conditions, V2X-ViT maintains strong detection performance across object-speed ranges and all five road categories. Higher-speed objects are harder because a fixed time delay creates greater positional misalignment.

  • Object speed: V2X-ViT outperforms all other intermediate fusion methods across object-speed ranges under noisy conditions.Performance is measured using AP@IoU=0.7.
  • Object speed: Higher-speed objects generally receive lower AP because the same time delay produces greater positional misalignment.
  • Object speed: The speed evaluation reports perception performance for objects at different speeds using AP@0.7 under the noisy setting.
  • Road type: V2X-ViT ranks first across all five road categories under noisy conditions.The evaluation uses AP@IoU=0.7 and covers different road types.

D.3 Qualitative results

Qualitative evaluations show that V2X-ViT produces more robust detections under noisy, occluded, and high-density traffic scenarios. Its attention maps indicate greater use of infrastructure features in regions occluded from the ego vehicle, while the delay model represents transmission and synchronization effects.

  • Qualitative detection: V2X-ViT produces more robust detections than comparison methods in noisy scenarios, with fewer regression displacements and undetected objects.The comparison includes V2VNet, OPV2V, F-Cooper, and DiscoNet.
  • Qualitative detection: In high-density traffic with more occlusions, V2X-ViT still identifies most objects accurately.Scene 7 is given as an example of a challenging scenario.
  • Attention visualization: V2X-ViT assigns brighter attention to infrastructure features, especially in regions occluded from other agents’ perspectives.Blue, green, and red indicate ego-vehicle, connected-vehicle, and infrastructure LiDAR points, respectively.
  • Transmission and delay modeling: The delay model accounts for transmission time and synchronization-induced idle time, with transmission time depending on data size and transmission rate.The simulated delay captures transmission-size effects and asynchrony-caused uncertainty.
  • Qualitative detection: The qualitative figures compare ground-truth and predicted 3D bounding boxes across roadway samples and scenarios.Green denotes ground truth and red denotes predictions.
Loading 2203.10638v3…