PaperlayerView exact revision on arXiv

Source-linked AI summary

HM-ViT: Hetero-modal Vehicle-to-Vehicle Cooperative perception with vision transformer

Hao Xiang, Runsheng Xu, Jiaqi Ma

arXiv:2304.10628v1cs.CV

TL;DR

Prior cooperative-perception work focused on homogeneous sensor setups, leaving multi-agent cooperation across different sensor modalities underexplored. HM-ViT fuses information from heterogeneous agents with a 3D graph-attention transformer, and the reported experiments show it outperforming state-of-the-art cooperative methods.

  • Problem

    Prior cooperative-perception work primarily studied homogeneous agents, leaving the benefits and design of multi-agent hetero-modal V2V cooperation unstudied.

    Source: §I · p. 1

  • Method

    HM-ViT uses modality-specific feature extraction and heterogeneous 3D graph attention to fuse information across agents with different sensor types.

    Source: §I · p. 2

  • Results

    HM-ViT exceeds other state-of-the-art intermediate-fusion methods by at least 1.7%, 8.4%, and 8.8% in AP@0.7 under V2V-C, V2V-L, and V2V-H, respectively.

    Source: C. Quantitative evaluation · p. 6

  • Takeaways & Limitations

    Experiments demonstrate the potential of hetero-modal multi-agent collaboration to increase V2V-system scalability and robustness.

    Source: §V · p. 8

Abstract

from arXiv · show

Vehicle-to-Vehicle technologies have enabled autonomous vehicles to share information to see through occlusions, greatly enhancing perception performance. Nevertheless, existing works all focused on homogeneous traffic where vehicles are equipped with the same type of sensors, which significantly hampers the scale of collaboration and benefit of cross-modality interactions. In this paper, we investigate the multi-agent hetero-modal cooperative perception problem where agents may have distinct sensor modalities. We present HM-ViT, the first unified multi-agent hetero-modal cooperative perception framework that can collaboratively predict 3D objects for highly dynamic vehicle-to-vehicle (V2V) collaborations with varying numbers and types of agents. To effectively fuse features from multi-view images and LiDAR point clouds, we design a novel heterogeneous 3D graph transformer to jointly reason inter-agent and intra-agent interactions. The extensive experiments on the V2V perception dataset OPV2V demonstrate that the HM-ViT outperforms SOTA cooperative perception methods for V2V hetero-modal cooperative perception. We will release codes to facilitate future research.

I. INTRODUCTION

The paper addresses hetero-modal V2V cooperation, where vehicles have different sensor types, a setting not covered by prior homogeneous-agent work. It introduces HM-ViT and reports improved cooperative perception, including a large gain for camera agents collaborating with LiDAR agents.

I. INTRODUCTION

  • Prior cooperative-perception research focused on homogeneous sensor setups, while heterogeneous V2V agents can contribute complementary LiDAR geometry and camera semantics.The paper notes that whether, when, and how heterogeneous cooperation benefits perception had not yet been studied.Source: §I · p. 1
  • Hetero-modal V2V systems form dynamic, heterogeneous sensor graphs with varying agent types and poses, creating spatial misalignment and semantic discrepancies that challenge fixed-geometry fusion methods.The paper distinguishes this setting from single-agent multi-modal systems, whose sensor types, counts, and relative positions are fixed.Source: §I · pp. 1–2
  • HM-ViT outperforms other state-of-the-art intermediate-fusion methods by at least 1.7%, 8.4%, and 8.8% in AP@0.7 under V2V-C, V2V-L, and V2V-H, respectively.For camera agents collaborating with LiDAR agents, AP@0.7 rises from 2.1% to 53.2%.Source: §I · p. 2
  • HM-ViT is a unified framework that uses heterogeneous 3D graph attention to reason over inter-agent and intra-agent interactions in highly dynamic traffic.Its local and global attention variants target local and global visual cues; the paper also reports benchmark experiments varying sensor modalities and plans to release code and baselines.Source: §I · p. 2

II. RELATED WORK

Related work spans cooperative perception, multi-camera 3D detection, and single-agent multi-modal fusion. The paper positions HM-ViT as the first sparse heterogeneous local-and-global attention approach for multi-agent hetero-modal perception.

II. RELATED WORK

  • Earlier cooperative-perception work focused primarily on LiDAR, and prior work had not explored multi-agent multi-camera 3D detection or multi-agent hetero-modal perception.Existing approaches include raw-sensor or output sharing and intermediate-feature fusion.Source: §II · p. 2
  • HM-ViT uses sparse heterogeneous local and global attention to capture 3D inter-agent and intra-agent interactions efficiently.This distinguishes its approach from the prior work reviewed in cooperative perception and camera-based 3D detection.Source: §II · p. 2
  • Single-agent multi-modal fusion methods assume camera and LiDAR coexist with fixed geometric relationships, unlike the dynamic heterogeneous collaboration graph in V2V settings.The paper describes point-, proposal-, and BEV-level fusion approaches.Source: Figure 2 · p. 2, §II · p. 3

III. METHODOLOGY

The methodology builds BEV features with modality-specific encoders, then compresses and shares them among agents for cooperative detection. LiDAR and multi-view camera agents use different feature-extraction paths.

III. METHODOLOGY

  • Each agent extracts BEV features, shares compressed features with connected agents, and uses HM-ViT fusion and a hetero-modal detection head to predict 3D boxes.This pipeline is designed for collaboration among any number of agents with varying sensor types.Source: §III · p. 3

A. Modality-specific feature extraction

  • LiDAR agents convert point clouds into 2D pseudo-images and process them with 2D convolutional networks to produce BEV feature maps for sharing.The method uses PointPillar for this LiDAR feature extraction.Source: A. Modality-specific feature extraction · p. 3
  • Camera agents use ResNet50 image features and learnable BEV queries with spatial cross-attention and projection matrices to produce shared BEV features.The camera feature extractor follows a BEVFormer-like architecture without temporal information.Source: A. Modality-specific feature extraction · p. 4

B. Heterogeneous 3D Graph Attention (H3GAT)

H3GAT uses type-dependent graph nodes and edges to jointly model spatial interactions and cross-agent relations. Its local and global attention variants capture complementary cues while preserving modality-specific processing.

B. Heterogeneous 3D Graph Attention (H3GAT)

  • H3GAT uses type-dependent nodes and edges to jointly reason about spatial interactions and cross-agent relations in BEV features.Each graph node represents an agent’s feature at a spatial location.Source: B. Heterogeneous 3D Graph Attention… · p. 4
  • Queries and keys use node-type-specific projectors, while values use edge-type-specific projectors to reflect heterogeneous interactions.The feature vectors are projected into query, key, and value spaces before attention.Source: B. Heterogeneous 3D Graph Attention… · p. 4
  • H3GAT-L uses local window attention to preserve object details, while H3GAT-G uses sparse global attention for context such as road topology and traffic density.The two variants capture local and global visual cues, respectively.Source: B. Heterogeneous 3D Graph Attention… · p. 4
  • HM-LN and HM-MLP use type-dependent parameters around attention blocks to maintain modality-specific characteristics during fusion.The transformer block applies modality-specific statistics in normalization before and after attention.Source: B. Heterogeneous 3D Graph Attention… · pp. 4–5

C. Hetero-modal Vision Transformer

The fusion process aligns and exchanges feature messages between agent nodes over repeated iterations. Cascaded local and global attention updates refine each agent’s state while retaining modality-specific statistics.

C. Hetero-modal Vision Transformer

  • Each node keeps an agent state in its own coordinate frame, while spatially warped messages are shared and aggregated over a fixed number of iterations.Intermediate features are compressed for transmission and decompressed by the receiving ego agent; camera and LiDAR use distinct compression and decompression parameters.Source: C. Hetero-modal Vision Transformer · p. 5
  • Each iteration aligns neighboring features, then applies H3GAT-L followed by H3GAT-G to update agent states with local and global interactions.The blocks mask non-overlapping fields of view, and agent updates can run in parallel.Source: C. Hetero-modal Vision Transformer · p. 5
  • After L iterations, a hetero-modal MLP refines the features while modality-specific statistics are maintained throughout fusion.Source: C. Hetero-modal Vision Transformer · p. 5

D. Hetero-modal Head

The hetero-modal head uses separate parameters for camera and LiDAR ego vehicles to produce final predictions from fused feature maps.

D. Hetero-modal Head

A. Datasets and Evaluation

The experiments use OPV2V under camera-only, LiDAR-only, and hetero-modal V2V configurations, evaluating detection around a selected ego vehicle with AP metrics.

A. Datasets and Evaluation

  • OPV2V contains over 70 CARLA/OpenCDA driving scenarios with 2–7 connected vehicles, each originally equipped with one LiDAR and four monocular cameras.For hetero-modal experiments, each vehicle uses only one sensor type: multiple cameras or LiDAR.Source: A. Datasets and Evaluation · p. 5
  • The dataset is split into 6,764/1,981/2,719 training, validation, and test examples, and evaluation covers a 100 m × 100 m area around a fixed ego agent.Detection performance is measured by average precision at IoU 0.5 and 0.7.Source: A. Datasets and Evaluation · p. 5
  • The three main configurations are camera-only V2V-C, LiDAR-only V2V-L, and V2V-H, with half the agents camera-only and half LiDAR-only.The evaluation also varies collaborator sensor types while holding ego modality fixed.Source: A. Datasets and Evaluation · p. 5
  • The modality-ratio experiment varies the fraction of LiDAR collaborators, while the agent-count ablation distinguishes camera and LiDAR collaborators for camera-ego and LiDAR-ego cases.The corresponding figures separate the ego modalities into panels.Source: Figure 6 · p. 6, Figure 7 · p. 6

B. Experimental Setups

The experiments use modality-specific backbones and a staged training procedure, then compare HM-ViT with baseline and intermediate-fusion methods.

B. Experimental Setups

  • PointPillar serves as the 3D backbone, while the camera stem uses a modified BEVFormer variant without temporal information and ResNet50 image backbones.The setup uses 0.4 m grid resolution, 128×128×256 intermediate BEV features, two graph-fusion iterations, and window size 8 for both attentions.Source: B. Experimental Setups · p. 6
  • Direct end-to-end training under V2V-H did not converge reliably, so the model is first trained on V2V-C and V2V-L for 40 epochs, then fine-tuned on V2V-H for 10 epochs.Modality-specific backbone parameters are fixed during hetero-modal fine-tuning.Source: B. Experimental Setups · p. 6
  • No Fusion is the baseline, Late Fusion transmits detection proposals, and five intermediate-fusion methods are benchmarked with a hetero-modal head for fairness.The intermediate methods are V2VNet, DiscoNet, AttFuse, CoBEVT, and V2X-ViT.Source: B. Experimental Setups · p. 6

C. Quantitative evaluation

HM-ViT is compared across homogeneous and hetero-modal settings, varying collaborator modalities and counts, with component and compression analyses also reported.

C. Quantitative evaluation

  • HM-ViT exceeds other SOTA intermediate-fusion methods by at least 1.7%, 8.4%, and 8.8% in AP@0.7 on V2V-C, V2V-L, and V2V-H, respectively.Across all three settings, cooperative methods outperform No Fusion, and intermediate fusion beats Late Fusion.Source: C. Quantitative evaluation · p. 6
  • Figure 8 contrasts no-fusion, camera-based, and hetero-modal detections for camera-ego and LiDAR-ego vehicles, with red boxes for detections and green boxes for ground truth.The panels include collaborator configurations using camera or LiDAR agents.Source: Figure 8 · p. 7
  • HM-ViT ranks first across all tested collaborator modality ratios for both camera-ego and LiDAR-ego vehicles; other intermediate-fusion methods perform well only over certain ratio ranges.Late Fusion usually beats No Fusion, except for LiDAR ego vehicles when most collaborators are camera agents.Source: C. Quantitative evaluation · p. 7
  • Perception performance rises as more agents participate, with LiDAR collaborators bringing larger AP gains than camera collaborators for both ego modalities.The gain rate generally decreases as agent count increases, while camera ego vehicles can improve substantially with only a small number of LiDAR collaborators.Source: C. Quantitative evaluation · p. 7
  • Local and global attention improve AP@0.7 by 4% and 5.5%, respectively, and all proposed components improve performance.The study attributes the attention gains to jointly reasoning about inter-agent and intra-agent heterogeneous interactions locally and globally.Source: C. Quantitative evaluation · pp. 7–8
  • HM-ViT still outperforms other methods under large compression rates.The compression-rate evaluation is reported for V2V-H.Source: Table III · p. 7, C. Quantitative evaluation · p. 8

D. Qualitative results

Qualitative examples show improved detections with collaboration, while the conclusion presents HM-ViT as a promising approach for scalable and robust hetero-modal V2V systems.

D. Qualitative results

  • Collaboration with homogeneous or heterogeneous agents enhances detections for both camera-ego and LiDAR-ego vehicles, with camera egos benefiting dramatically from LiDAR collaborators.Figure 8 compares these settings with no-collaborator baselines.Source: D. Qualitative results · p. 8

V. CONCLUSION

  • The authors conclude that HM-ViT and heterogeneous 3D graph attention support hetero-modal multi-agent perception, with experiments indicating potential to increase V2V-system scalability and robustness.They identify the problem as under-explored and express hope that the findings and open-source efforts will encourage further research.Source: §V · p. 8
Loading 2304.10628v1…