PaperlayerView exact revision on arXiv

Source-linked AI summary

V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer

Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, Jiaqi Ma

arXiv:2203.10638v3cs.CV

TL;DR

Autonomous-vehicle perception can be impaired by occlusion and sparse far-distance observations, while integrating heterogeneous vehicles and infrastructure into V2X perception is challenging. V2X-ViT uses a unified Transformer for cooperative fusion and reports a 21.2% AP gain over the single-agent baseline in V2X LiDAR-based 3D detection.

  • Problem

    Single-vehicle perception can suffer from occlusion and sparse far-distance observations, while infrastructure integration is challenging because V2X agents are heterogeneous.

    Source: §1 · pp. 1–2

  • Method

    V2X-ViT is a unified fusion Transformer with heterogeneous multi-agent and multi-scale window attention to address agent differences and localization errors.

    Source: §1 · p. 2

  • Results

    21.2% AP gain over the single-agent baseline, with V2X-ViT also performing favorably against leading intermediate-fusion methods by at least 7.3%, on V2X LiDAR-based 3D object detection.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The model may improve autonomous-driving performance and robustness by incorporating V2X communication.

    Source: §5 · p. 14

    The study is limited to LiDAR-based cooperative 3D vehicle detection, and simulated-data training may face data bias and limited real-world generalization.

    Source: §5 · p. 14

Abstract

from arXiv · show

In this paper, we investigate the application of Vehicle-to-Everything (V2X) communication to improve the perception performance of autonomous vehicles. We present a robust cooperative perception framework with V2X communication using a novel vision Transformer. Specifically, we build a holistic attention model, namely V2X-ViT, to effectively fuse information across on-road agents (i.e., vehicles and infrastructure). V2X-ViT consists of alternating layers of heterogeneous multi-agent self-attention and multi-scale window self-attention, which captures inter-agent interaction and per-agent spatial relationships. These key modules are designed in a unified Transformer architecture to handle common V2X challenges, including asynchronous information sharing, pose errors, and heterogeneity of V2X components. To validate our approach, we create a large-scale V2X perception dataset using CARLA and OpenCDA. Extensive experimental results demonstrate that V2X-ViT sets new state-of-the-art performance for 3D object detection and achieves robust performance even under harsh, noisy environments. The code is available at https://github.com/DerrickXuNu/v2x-vit.

1 Introduction

The introduction motivates infrastructure-inclusive V2X fusion to overcome single-view perception limits and challenges from heterogeneous agents, localization error, and sensing delay. It presents V2X-ViT and V2XSet as a unified transformer framework and noisy-condition cooperative-perception dataset.

1 Introduction

  • Single-view perception suffers from occlusion and sparse far-distance observations; adding infrastructure brings broader views but introduces agent heterogeneity, localization noise, and asynchronous sensing.The introduction notes that infrastructure can provide support at key scenes such as intersections and crosswalks, while elevated sensors may have less occlusion.Source: §1 · pp. 1–2
  • V2X-ViT unifies heterogeneous multi-agent attention and multi-scale window attention, with delay-aware positional encoding, to fuse agent interactions and spatial features while addressing pose errors and sensing delays.The modules account for agent types and connections, use parallel multi-resolution windows to capture local and global spatial interactions, and iteratively fuse features end-to-end.Source: §1 · pp. 2–3
  • The authors construct V2XSet, a large-scale open V2X perception dataset built with CARLA and OpenCDA that explicitly accounts for imperfect real-world communication conditions.The dataset is designed to include real-world noise during V2X communication.Source: §1 · p. 3
  • The passage reports that the proposed model achieves state-of-the-art performance on the challenging cooperative detection task, without stating a numerical gain or baseline in this excerpt.Source: §1 · p. 3

2 Related work

Related work frames V2X perception around communication tradeoffs and the limited maturity of intermediate fusion, while drawing on LiDAR detection, efficient vision Transformers, and heterogeneous graph attention.

V2X perception

  • Intermediate fusion balances accuracy and bandwidth, but V2X intermediate methods remain nascent; early fusion is bandwidth-intensive, while late fusion can lack scenario context.Prior examples include OPV2V’s self-attention [50], F-Cooper’s maxout, V2VNet’s spatial-aware message passing, localization-error regression [41], and DiscoNet’s distillation; V2X late fusion includes a Kalman-filter approach and Dempster-Shafer lane-mark fusion.Source: §2 · pp. 3–4

LiDAR-based 3D object detection

  • LiDAR 3D detection uses point clouds, voxels, BEV representations, or combined voxel-and-point methods [59], with 3D convolutions challenging inference speed and memory.PointRCNN uses a two-stage point-cloud strategy, while BEV methods avoid costly 3D convolutions.Source: §2 · p. 4

Transformers in vision

  • Vision Transformers apply self-attention to image patches [9], but global attention is computationally heavy, motivating local, window, and sparse attention and hierarchical receptive-field growth [30] [8] [7] [46] [40] [49].The Transformer stacks multi-head self-attention and feed-forward layers to capture long-range interactions.Source: §2 · p. 4
  • Because conventional vision Transformers are less studied for heterogeneous graphs, V2X-ViT adapts HGT into heterogeneous multi-head attention for attribute-aware multi-agent 3D feature fusion.HGT was originally designed for the web-scale Open Academic Graph, whose nodes contain text and attributes.Source: §2 · pp. 4–5

3 Methodology

V2X-ViT is an end-to-end heterogeneous multi-agent perception framework designed to address imperfect localization and communication delays through a five-component pipeline. Its Transformer alternates inter-agent fusion with per-agent spatial attention.

3.1 Main architecture design

  • The pipeline shares synchronized metadata, extracts PointPillar features, compresses and transmits them, warps delayed features to the ego pose, fuses them at high resolution, and predicts boxes and classes.An ROI mask excludes padded zeros after warping; the detection head uses smooth ℓ1 regression loss and focal classification loss.Source: §3.1 · pp. 5–6

3.2 V2X-Vision Transformer

  • V2X-ViT integrates heterogeneous multi-agent self-attention with multi-scale window attention in stacked blocks to iteratively fuse inter-agent interactions and per-agent spatial features while maintaining high resolution.The design targets V2X heterogeneity and localization errors; multi-scale windows capture long-range interactions at varied spatial ranges.Source: §3.2 · pp. 6–7

Heterogeneous multi-agent self-attention

  • HMSA models vehicle and infrastructure nodes and their four directed edge types, conditioning attention and message aggregation on agent and edge types while reasoning over corresponding spatial positions.Type-specific attention weights and feature projections encode differences in sensor characteristics across agent categories.Source: §3.2 · p. 7, Heterogeneous multi-agent self-attention · p. 7

Multi-scale window attention

  • MSwin applies parallel spatial-attention branches with different window sizes and adaptively fuses them, combining local context with long-range cues to improve robustness to localization errors.Its complexity is linear in image size HW, and unlike Swin Transformer it requires no masking, padding, or cyclic shifting.Source: Multi-scale window attention · p. 8

Delay-aware positional encoding

  • Delay-aware positional encoding adds a learned, linearly projected embedding conditioned on each agent’s time delay to its features before Transformer processing, promoting temporal alignment and generalization to unseen delays.The embedding is initialized with sinusoidal functions of delay and channel.Source: Delay-aware positional encoding · pp. 8–9

4 Experiments

V2X-ViT outperforms the compared methods on V2XSet and remains robust to localization error, time delay, and message compression. The experiments also assess the dataset, infrastructure’s contribution, qualitative detections, and individual model components.

4.1 V2XSet: An open dataset for V2X cooperative perception

  • V2XSet contains 11,447 frames (33,081 per-agent samples) and combines V2X cooperation with realistic noise simulation, unlike the cited datasets’ stated limitations.Its train/validation/test splits are 6,694/1,920/2,833 frames; it was built using CARLA and OpenCDA.Source: §4.1 · p. 9

4.2 Experimental setup

  • The evaluation compares perfect synchronization and pose accuracy against a noisy setting with 0.2 m positional noise, 0.2° heading noise, and 100 ms delay.Detection is evaluated using AP at IoU thresholds 0.5 and 0.7; the study focuses on LiDAR-based vehicle detection.Source: §4.2 · pp. 9–10

4.3 Quantitative evaluation

  • With positional and heading errors within σxyz ≤0.2 m and σheading ≤0.4°, V2X-ViT drops by less than 3%, while other intermediate fusion methods drop at least 6%.Even at 0.5 m and 1° noise, V2X-ViT stays around 60% detection accuracy; under 400 ms delay, it exceeds No Fusion by 6.8% in AP@0.7.Source: Figure 4 · p. 10, §4.3 · p. 11
  • V2X-ViT outperforms state-of-the-art intermediate fusion methods by 3.8%/1.7% in AP@0.5/0.7 and exceeds No Fusion by 23%/21.2% in AP@0.5/0.7 under noisy conditions.Under the Perfect Setting, it also exceeds Early Fusion by 0.2% in AP@0.7; under the Noisy Setting, its AP@0.5/AP@0.7 drops by less than 5%/10% versus Perfect. Inference takes 57 ms, while the one-layer V2X-ViTS takes 28 ms and beats DiscoNet.Source: Table 1 · p. 10, §4.3 · p. 11, Table 4 · p. 12
  • At the default 32x compression rate, V2X-ViT substantially outperforms other intermediate fusion methods and maintains high performance even at 128x compression.Large bandwidth demand can quickly erase cooperative perception’s benefit: Early Fusion drops to 28%.Source: Figure 5 · p. 12, §4.3 · p. 12
  • In collected scenes, V2X has better AP than V2V, and both improve as the number of sharing agents increases.The paper attributes V2X’s advantage to infrastructure sensors’ better sightlines and lower occlusion, which provide more informative environmental features.Source: §4.3 · pp. 11–12, Figure 5 · p. 12

4.5 Ablation studies

  • Ablations show that all added modules help, with MSwin and HMSA increasing AP@0.7 by 4.1% and 6.6%, respectively.MSwin’s large-and-small window combination improves localization-error robustness, and DPE improves performance across time delays, with gains increasing as delay grows.Source: Table 2 · p. 12, Figure 5 · p. 12, Table 3 · p. 12

4.4 Qualitative evaluation

  • Under noisy conditions, V2X-ViT produces more accurate bounding boxes and matches more ground-truth dynamic objects than the compared methods.Attention maps show greater attention to infrastructure in occluded areas, where infrastructure captures point clouds unavailable from the ego vehicle and AV2 perspectives.Source: §4.4 · pp. 12–13, Figure 6 · p. 13, Figure 7 · p. 13

5 Conclusion · Appendix

V2X-ViT combines heterogeneous inter-agent and multi-scale intra-agent attention for cooperative LiDAR-based vehicle detection, and experiments show improved detection in perfect and noisy settings. The paper also identifies limits in task scope, scalability, and real-world generalization, while the appendix provides further model and experiment details.

5 Conclusion

  • V2X-ViT uses HMSA and MSwin to capture heterogeneous inter-agent interactions and multi-scale intra-agent spatial relationships.The work focuses on LiDAR-based cooperative 3D vehicle detection with a single sensor modality.Source: §5 · p. 14
  • Experiments show that V2X-ViT significantly boosts cooperative 3D object detection under both perfect and noisy settings.The evaluation uses V2XSet, a newly constructed large-scale V2X perception dataset.Source: §5 · p. 14
  • The work is limited to single-modality LiDAR vehicle detection, with multi-sensor fusion for joint V2X perception and prediction left for future work.Source: §5 · p. 14
  • The paper notes that simulated-data training raises concerns about data bias and real-world generalization, and that projecting LiDAR to other agents at the start limits scalability despite an accuracy advantage.Source: §5 · p. 14
  • Data capture and sharing raise privacy and adversarial-robustness concerns that have received little attention, motivating further research on fairness, privacy, and robustness.Source: §5 · p. 14

Appendix

  • Appendix B provides model details and analysis, including spatial-temporal correction, MSwin, and overall architecture specifications.Appendix A discusses the design choice and scalability issue.Source: Appendix · p. 15
  • Appendices C and D provide V2XSet information and visualizations, additional quantitative and qualitative results, attention maps, and transmission-size experiment details.Source: Appendix · p. 15

A Discussion of design choice

The discussion argues that ego-range feature cropping improves detection over broadcasting, while decentralized or centralized aggregation can make the system scalable.

Scalability of ego vehicles

  • When collaborators exceed four, performance gains become marginal while computation grows linearly, so decentralized agents need share features with only a limited set of collaborators.Who2Com is cited as an approach for selecting which agent to request or transmit data, reducing computation.Source: A Discussion of design choice · p. 15
  • Centralized aggregation is also scalable: one ego agent within communication range can aggregate neighbors' features, predict bounding boxes, and share results using one computation node per group.The selected PointPillar backbone takes around 4 ms for four agents with full parallelization or 16 ms in sequence computing on an RTX3090.Source: A Discussion of design choice · p. 15

Design choices for communication

  • 8.5% for DiscoNet and 8.9% for V2X-ViT: the ego-range cropping design outperforms broadcasting.Broadcasting crops each CAV's LiDAR data to its own evaluation range, creating redundant data; this design crops to the ego's evaluation range for more effective feature transmission.Source: Table T0 · p. 15, A Discussion of design choice · p. 15

B Model Details and Analysis · B.1 Spatial-Temporal Correction Module

The spatial-temporal correction module addresses delay-induced misalignment between connected-agent features centered on the ego vehicle’s past pose and ego features centered on its current pose. It warps intermediate features and masks padded values during attention.

B.1 Spatial-Temporal Correction Module

  • Delayed communication leaves connected-agent features centered on the ego vehicle’s past pose, while ego features use its current pose.This difference creates spatial misalignment between received and ego-vehicle features.Source: §B.1 · p. 16
  • A global transformation from the ego vehicle’s past pose to its current pose is required to correct the delay-induced misalignment.Source: §B.1 · p. 16
  • The module uses a differential 2D transformation to spatially warp intermediate features.Source: §B.1 · p. 16
  • Feature positions are transformed with an affine mapping using rotation terms and x- and y-translation offsets.The mapping is given in Equation 11.Source: §B.1 · p. 16
  • Because transformed source coordinates may be non-integer, the module uses bilinear interpolation to sample input feature vectors.Source: §B.1 · p. 16
  • An ROI mask prevents attention from using padded zeros introduced by spatial warping.In heterogeneous multi-agent self-attention, the mask sets padded values’ attention weights to zero.Source: §B.1 · p. 16

B.2 Multi-Scale Window Attention (MSwin) · B.3 Architectural Configurations

MSwin combines attention across parallel window scales to capture multi-scale spatial interactions with linear complexity in image size. V2X-ViT processes agents’ features through delay-aware correction and alternating attention and MLP layers.

B.2 Multi-Scale Window Attention (MSwin)

  • Each MSwin branch applies multi-head relative self-attention within non-overlapping P_j × P_j windows, with head dimension d_hj = C/h_j.Branch outputs are concatenated across heads before the branches are fused.Source: §B.2 · pp. 16–17
  • A parameterized relative-position bias is added to the attention map, and split attention adaptively fuses features from the k branches.The bias is indexed over relative positions within each branch’s window.Source: §B.2 · p. 17
  • With branch j using window size P_j = jP and h_j = h/j heads, the MSwin operation has FLOPs O((k^3P^2C/3 + 2k^2C^2/h)HW).The paper characterizes this as linear with respect to image size N = HW.Source: §B.2 · pp. 17–18
  • MSwin provides multi-scale spatial interactions with linear complexity in image size, while ViT, Axial, and CSwin require more than linear complexity.The paper presents this as more scalable for high-resolution dense prediction tasks such as object detection and segmentation.Source: §B.2 · p. 18, Table T1 · p. 18
  • MSwin aggregates features from parallel branches to expand receptive fields symmetrically, unlike approaches that accumulate or divide attention across directions.Swin uses sequential shifted-window blocks, while Axial and CSwin attend along rows and columns or horizontal and vertical stripes, respectively.Source: §B.2 · p. 18, Figure 8 · p. 18

B.3 Architectural Configurations

  • V2X-ViT encodes each agent’s LiDAR point cloud into 2D features, then compresses, shares, decompresses, and spatially warps gathered features before adding delay-aware positional encodings.The spatial-temporal correction module performs the warping, and the delay-aware positional encoding is conditioned on each agent’s time delay.Source: §B.3 · p. 19
  • The gathered features pass through L V2X-ViT blocks, each comprising HSMA, MSwin, and an MLP, with Layer Normalization before the Transformer and MLP modules.The model’s detailed architectural specifications are listed in Table T2.Source: §B.3 · p. 19, Table T2 · p. 19

C V2XSet Dataset

V2XSet is a CARLA-based cooperative-perception dataset spanning varied road environments and vehicle–infrastructure configurations. Its sensor setup captures differences between vehicle and infrastructure measurements, while rural curvy roads feature V2V collaboration only.

C V2XSet Dataset

  • V2XSet contains 55 scenes across 5 roadway types and 8 CARLA towns, with 2–7 communicating agents per scene.Scenes last up to 25 seconds; each agent has a 32-channel LiDAR with 120-meter range, recorded at 10 Hz with positional data and timestamps.Source: C V2XSet Dataset · p. 20
  • Infrastructure sensors are deployed at intersections, mid-blocks, and entrance ramps, while rural curvy roads use V2V collaboration only.Infrastructure sensors are mounted 14 feet high on traffic-light or street-light poles; vehicle sensors are mounted on top of AVs.Source: C V2XSet Dataset · p. 20
  • Infrastructure and vehicle LiDAR show different measurement patterns near infrastructure installations, reflecting their differing sensor heights.The dataset includes straight segments, curvy segments, mid-blocks, entrance ramps, and intersections; more intersection scenes are collected because of high traffic volume and severe occlusions.Source: C V2XSet Dataset · p. 20, Figure 9 · p. 20

D More Experimental Results · D.1 Performance for identifying dynamic objects · D.2 Performance for different road types

Under noisy conditions, V2X-ViT leads intermediate-fusion methods across object-speed ranges and ranks first across all five road categories; higher-speed objects generally have lower AP.

D.1 Performance for identifying dynamic objects

  • Higher-speed objects generally have lower AP scores in the noisy-setting evaluation.The test set is grouped by object speed in km/h, and performance is measured as AP@IoU=0.7.Source: §D.1 · p. 21, Table T3 · p. 21
  • The paper attributes lower AP at higher speeds to greater positional misalignment from the same time delay.The passage specifically links this effect to high-speed vehicles.Source: §D.1 · p. 21
  • The dynamic-object comparison groups the test set by object speed and evaluates intermediate-fusion models under noisy conditions.The metric is AP@IoU=0.7.Source: §D.1 · p. 21, Table T3 · p. 21
  • V2X-ViT outperforms all other intermediate-fusion methods across the evaluated speed ranges under noisy conditions.The comparison measures AP@IoU=0.7 for objects grouped by speed.Source: Table T3 · p. 21, §D.1 · p. 21

D.2 Performance for different road types

  • The road-type evaluation groups test scenes by road category and compares AP@IoU=0.7 under noisy conditions.It covers five road categories.Source: §D.2 · p. 21, Table T4 · p. 21
  • V2X-ViT ranks first in all five road categories, demonstrating detection robustness across different scenes.Scores are AP@IoU=0.7 under the noisy setting.Source: Table T4 · p. 21, §D.2 · p. 21

D.3 Qualitative results · D.4 Attention visualization · D.5 Explanation on effects of transmission size

The qualitative results show robust detections under noisy conditions, while attention visualizations highlight infrastructure features in occluded regions. The transmission-size analysis models delay from communication volume and asynchrony using sender- and receiver-side idle times.

D.3 Qualitative results

  • V2X-ViT shows more robust noisy-setting detections than V2VNet, OPV2V, F-Cooper, and DiscoNet, with fewer regression displacements and undetected objects.The visual comparisons cover scenarios 1–8.Source: §D.3 · pp. 21–22, Figure 11 · p. 24, Figure 12 · p. 25
  • In high-density, occluded Scene 7, V2X-ViT still identifies most objects accurately.The paper identifies Scene 7 in Fig. 13 as a challenging case.Source: §D.3 · p. 22, Figure 13 · p. 26

D.4 Attention visualization

  • Under noisy conditions, V2X-ViT’s attention maps are brighter for infrastructure, especially in regions occluded from other agents.The paper attributes this pattern to infrastructure observations having fewer occlusions and therefore better feature representations.Source: §D.4 · p. 22, Figure 14 · p. 27

D.5 Explanation on effects of transmission size

  • Transmission volume motivates realistic delay modeling because fusion strategies require different bandwidths, and communication volume significantly affects delay.Early fusion transmits raw data and requires large bandwidth, whereas late fusion delivers a minimal data size.Source: §D.5 · p. 22
  • Following, total delay is decomposed into broadcasting transmission time and idle time caused by lack of synchronization between perception and communication.The passage identifies data size as fs and transmission rate as v.Source: §D.5 · p. 22
  • With both systems at 10Hz, sender- and receiver-side cycle misses yield 0–200 ms of idle time, modeled as U (0, 200) and discretized to observed timestamps.The sampled delay is discretized for a 10Hz update system.Source: §D.5 · p. 22

Related papers

Loading 2203.10638v3…