PaperlayerView exact revision on arXiv

Source-linked AI summary

An Extensible Framework for Open Heterogeneous Collaborative Perception

Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Yanfeng Wang, Siheng Chen

arXiv:2401.13964v3cs.CV

TL;DR

Open heterogeneous collaborative perception must accommodate continually emerging agent types while balancing perception performance and integration training cost. HEAL establishes a unified feature space, aligns new agents through individual backward alignment, and outperforms existing methods while reducing training parameters by 91.5% when integrating three heterogeneous agent types.

  • Problem

    Open heterogeneous collaborative perception seeks to integrate continually emerging agents with unknown modalities or models while minimizing training overhead and maintaining perception performance.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    HEAL creates a unified feature space with initial agents and uses locally trained backward alignment to adapt each new agent type to that space.

    Source: §1 · p. 2, §4 · p. 4, §4.2 · p. 6

  • Results

    HEAL outperforms existing collaborative perception methods and reduces training parameters by 91.5% when successively integrating three heterogeneous agent types.

    Source: §1 · p. 2, §5.3 · p. 7

  • Takeaways & Limitations

    HEAL provides an extensible approach for heterogeneous collaborative perception with high performance and low training costs across OPV2V-H and DAIR-V2X.

    Source: §1 · p. 3, §6 · p. 9

    Fully collective training requires retraining the entire model whenever a new agent type is introduced, making repeated integration increasingly training-expensive.

    Source: §1 · p. 2

Abstract

from arXiv · show

Collaborative perception aims to mitigate the limitations of single-agent perception, such as occlusions, by facilitating data exchange among multiple agents. However, most current works consider a homogeneous scenario where all agents use identity sensors and perception models. In reality, heterogeneous agent types may continually emerge and inevitably face a domain gap when collaborating with existing agents. In this paper, we introduce a new open heterogeneous problem: how to accommodate continually emerging new heterogeneous agent types into collaborative perception, while ensuring high perception performance and low integration cost? To address this problem, we propose HEterogeneous ALliance (HEAL), a novel extensible collaborative perception framework. HEAL first establishes a unified feature space with initial agents via a novel multi-scale foreground-aware Pyramid Fusion network. When heterogeneous new agents emerge with previously unseen modalities or models, we align them to the established unified space with an innovative backward alignment. This step only involves individual training on the new agent type, thus presenting extremely low training costs and high extensibility. To enrich agents' data heterogeneity, we bring OPV2V-H, a new large-scale dataset with more diverse sensor types. Extensive experiments on OPV2V-H and DAIR-V2X datasets show that HEAL surpasses SOTA methods in performance while reducing the training parameters by 91.5% when integrating 3 new agent types. We further implement a comprehensive codebase at: https://github.com/yifanlu0227/HEAL

1 INTRODUCTION

Collaborative perception addresses single-agent limitations by sharing complementary information, but existing methods largely assume homogeneous agents. HEAL targets the open heterogeneous setting with a unified feature space, backward alignment, and low-cost integration of new agent types.

Problem

  • Open heterogeneous collaborative perception must accommodate continually emerging agents with unseen modalities or models while preserving performance and limiting integration cost.New agents face a domain gap that impedes feature fusion with existing collaborators.Source: §1 · p. 1

HEAL framework

  • HEAL establishes a unified feature space through collective base training and aligns new agent types to it through individual backward alignment.The framework uses Pyramid Fusion for base training and updates only new agents’ front-end encoders during integration.Source: §1 · pp. 2–3

Backward alignment

  • Backward alignment enables new agent types to train individually, reducing integration costs while preserving model details and data privacy.The approach avoids collective retraining and supports repeated incorporation of emerging agent types.Source: §1 · pp. 2–3

Evaluation

  • HEAL outperforms other methods while reducing training parameters by 91.5% when three heterogeneous agent types are successively added.Experiments use OPV2V-H and DAIR-V2X.Source: §1 · p. 2

Pyramid Fusion

  • HEAL’s Pyramid Fusion uses multi-scale and foreground-aware designs to sustain a potent unified space for collaboration.The design selects important features for fusion across scales.Source: §1 · p. 2

2 RELATED WORKS

Related work develops collaborative perception through information exchange and modality fusion, but commonly assumes fixed, homogeneous agent systems. BEV representations provide a spatially unified basis for heterogeneous modality and model fusion.

2.1 COLLABORATIVE PERCEPTION

  • Collaborative perception exchanges complementary information through raw data, outputs, or intermediate features, while communication-efficient methods select critical messages and address pose errors.Datasets include V2X-Sim, OPV2V, and DAIR-V2X; collaborative perception also supports planning and control.Source: §2.1 · p. 3
  • Most prior methods assume identical sensor modalities and models, while HM-ViT addresses modality heterogeneity but requires retraining the whole model when adding agent types.The related-work discussion identifies extensibility as an unresolved issue.Source: §2.1 · p. 3

2.2 MULTI-MODALITY FUSION

  • LiDAR-camera fusion methods project points to images or decorate points with image information, whereas BEV supplies a spatially unified representation for different modalities.BEV is especially relevant to multi-agent settings with random sensor positions and types.Source: §2.2 · p. 3
  • Heterogeneous multi-agent collaboration differs from single-vehicle multimodal fusion because agent sensor positions and types vary rather than remaining fixed.This variability makes BEV alignment useful for feature fusion across agents.Source: §2.2 · p. 3

3 OPEN HETEROGENEOUS COLLABORATIVE PERCEPTION

Open heterogeneous collaborative perception models a collaborative network that grows as agents with previously unseen modalities or models join over time. Its central design objective is to balance low integration overhead with strong post-integration perception performance.

Problem setting

  • Open heterogeneous collaboration begins with homogeneous agents and sequentially adds new agent types whose sensor modalities or perception models were unseen initially.Unlike traditional heterogeneous frameworks, the set of agent types is not fixed at deployment.Source: §3 · p. 3
  • Open heterogeneous collaborative perception differs from traditional heterogeneous collaboration because new agent types can continue emerging after the initial network is established.The number and identities of agent types are not fully determined at the beginning.Source: §3 · p. 3

Design goals

  • The defining challenge is to accommodate a growing alliance while maximizing perception performance and minimizing the training overhead of each integration.Collective retraining is computationally expensive when integrations repeat over time.Source: §3 · p. 4

4 HETEROGENEOUS ALLIANCE (HEAL)

HEAL establishes a unified feature space for initial agents, then aligns each emerging heterogeneous agent type to that space through fixed-back-end individual training. Its Pyramid Fusion combines multiscale and foreground-aware features, while backward alignment enables continued feature-level collaboration with low integration cost.

4 HETEROGENEOUS ALLIANCE (HEAL)

  • HEAL uses two phases: collaboration base training establishes a unified feature space, and new-agent training aligns each emerging type to it.Only the second phase is required for each new agent-type integration.Source: §4 · p. 4, Figure 2 · p. 4

4.1 COLLABORATION BASE TRAINING

  • The base network transmits spatially transformed agent features, fuses them with Pyramid Fusion, and decodes the result through a detection head.Training uses detection supervision together with foreground-map supervision at multiple BEV scales.Source: §4.1 · pp. 4–5
  • Pyramid Fusion creates the collaboration base space by combining multiscale features and foreground-aware weights across BEV positions.Foreground estimators produce maps that are normalized into fusion weights and receive supervision at multiple scales.Source: §4.1 · pp. 5–6, Figure 3 · p. 5

4.2 NEW AGENT TYPE TRAINING

  • Backward alignment fixes the pretrained Pyramid Fusion and detection head while training only the new agent’s encoder on single-agent data.The encoder adapts new-sensor features to the established back-end and unified feature space.Source: §4.2 · p. 6
  • Backward alignment uses shared BEV coordinates, multiscale alignment, and retained foreground estimators to bridge modality and model differences.Foreground supervision emphasizes alignment with perceptually important foreground features.Source: §4.2 · p. 6
  • Training new agent types individually avoids synchronized multi-agent data collection and expensive retraining while supporting privacy-preserving integration.After an encoder is trained, agents of that type can collaborate with base agents, and the process can be repeated for additional types.Source: §4.2 · p. 6, §4.3 · p. 6

5 EXPERIMENTAL RESULTS

Experiments evaluate HEAL on heterogeneous simulated and real-world datasets under sequential agent integration, comparing perception performance, training cost, robustness, compression, and component contributions.

5.1 DATASETS

  • OPV2V-H broadens heterogeneous evaluation with extra 16- and 32-channel LiDAR and depth-camera data, while DAIR-V2X provides a real-world vehicle–RSU setting.OPV2V-H averages approximately three agents per frame; DAIR-V2X contains 9K frames.Source: §5.1 · p. 7

5.3 QUANTITATIVE RESULTS

  • HEAL surpasses existing collaborative perception methods while maintaining the lowest training cost during sequential heterogeneous-agent integration on OPV2V-H.Its backward alignment avoids collective retraining, reducing model size, FLOPs, training time, and memory usage.Source: §5.3 · p. 7, Table 2 · p. 8
  • HEAL outperforms previous SOTA HM-ViT by 7.6% in AP70 while using only 8.3% of HM-ViT’s parameters.Source: §5.3 · p. 8
  • HEAL maintains the best performance and lowest training cost across heterogeneous agent-type combinations on OPV2V-H and DAIR-V2X.The combinations include degraded LiDAR settings with lower channel numbers.Source: §5.3 · p. 8, Table 3 · p. 8
  • HEAL retains state-of-the-art performance under varied pose-error conditions and exceeds baseline methods even with 32-fold feature compression.Pose robustness is evaluated by adding Gaussian noise to accurate poses, while compression uses an autoencoder to reduce feature channels.Source: §5.3 · p. 8, Figure 4 · p. 9
  • Ablations show that multiscale pyramid encoding, foreground supervision, and backward alignment each benefit collaboration performance and unified-space construction.The designs support feature resolution diversity, foreground selection, and comprehensive alignment of new agent types.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 QUALITATIVE VISUALIZATIONS

  • The qualitative visualizations depict backward alignment from L2 to L1’s unified space and progressively add new agents while showing predicted and ground-truth boxes.Source: §5.4 · p. 9, Figure 5 · p. 9, Figure 6 · p. 9

6 CONCLUSION

HEAL offers an extensible framework for open heterogeneous collaborative perception with strong performance, low training costs, and model-detail protection. Its main boundary is the requirement for BEV features, which can require extra compatibility work for some detectors.

6 CONCLUSION

  • HEAL provides an extensible solution for heterogeneous collaborative perception with exceptional collaborative performance, minimal training costs, and model-detail protection.Source: §6 · p. 9
  • HEAL requires BEV features, creating extra compatibility effort for some models such as keypoint-based LiDAR detectors.Source: §6 · p. 9

A.1 REAL-WORLD SOLUTION DETAILS

HEAL is designed for practical deployment with limited multi-agent data, allowing new companies to join collaboration through local training while preserving privacy and reducing integration costs.

A.1 REAL-WORLD SOLUTION DETAILS

  • The real-world workflow jointly trains Pyramid Fusion on vehicle and RSU data, then distributes its weights for local adaptation by participating companies.The vehicle’s raw data and Pyramid Fusion weights are distributed so each company trains its model locally.Source: Figure 7 · p. 13, §A.1 · p. 13
  • This design addresses the scarcity of synchronized multi-vehicle training data, which makes real-world collaborative-perception dataset construction difficult.Existing real-world datasets typically contain no more than two agents because synchronized sensor collection is formidable.Source: §A.1 · p. 13
  • HEAL can expand collaboration to potentially unlimited heterogeneous agent types even when training data contains only two agents.Existing intermediate-fusion methods struggle to train three or more heterogeneous types together.Source: §A.1 · p. 13
  • Local data, models, and training protect participating companies’ technical privacy while reducing training costs.After backward alignment, new agents collaborate with the RSU at feature level.Source: §A.1 · p. 13

A.2 VISUALIZATION OF FEATURE ALIGNMENT

Pyramid Fusion produces aligned multiscale features whose foreground and background responses occupy a shared feature space, enabling feature fusion.

A.2 VISUALIZATION OF FEATURE ALIGNMENT

  • Features after ResNeXt Layers 1–3 occupy the same feature space with similar foreground and background responses, allowing them to fuse.The visualization covers all three ResNeXt stages.Source: Figure 8 · p. 14

A.3 PERFORMANCE OF PYRAMID FUSION ON HOMOGENEOUS SETTING

Pyramid Fusion is evaluated against existing collaborative-perception methods under homogeneous LiDAR and camera settings, with results showing superior performance across both modalities.

A.3 PERFORMANCE OF PYRAMID FUSION ON HOMOGENEOUS SETTING

  • The experiments compare homogeneous collaboration, where agents share the same sensor and detection model, to isolate fusion-module performance.Both LiDAR-based and camera-based homogeneous collaboration are evaluated.Source: §A.3 · p. 15
  • The LiDAR configuration uses PointPillars with 64-channel input and a three-level Pyramid Fusion feature dimension of [64,128,256].The LiDAR detection range is x,y ∈ [−102.4m,+102.4m], with 2× feature downsampling and 64-dimensional shared messages.Source: §A.3 · p. 15
  • The camera configuration uses Lift-Splat with EfficientNet, 384-pixel image height, 2× BEV downsampling, 64-dimensional shared messages, and depth supervision.The camera detection range is x,y ∈ [−51.2m,+51.2m].Source: §A.3 · p. 15
  • All compared methods are trained collectively end to end, and the results support the multiscale, foreground-aware fusion strategy.The reported superiority is attributed to Pyramid Fusion’s multiscale and foreground-aware design.Source: §A.3 · p. 15
  • Pyramid Fusion outperforms state-of-the-art methods in both LiDAR-based and camera-based homogeneous collaboration on OPV2V and DAIR-V2X.Table 5 reports the comparison across the two sensor settings and datasets.Source: Table 5 · p. 15, §A.3 · p. 15

A.4 OPV2V-H DATASET DETAILS

OPV2V-H extends collaborative-perception data with diverse synchronized sensors and annotations, while the paper uses PointPillars, SECOND, and Lift-Splat to encode modalities into BEV features.

A.4 OPV2V-H DATASET DETAILS

  • OPV2V-H is a heterogeneous dataset collected with OpenCDA and CARLA to represent agents with diverse sensor configurations.OpenCDA supplies driving scenarios, while CARLA supplies maps, weather, and movement control.Source: §A.4 · p. 15, Figure 9 · p. 16
  • Each autonomous agent has 11 sensors: four RGB cameras, four depth sensors, and three LiDARs with 16-, 32-, and 64-channel configurations.The cameras and depth sensors cover front, left, right, and rear views.Source: §A.4 · pp. 15–16, Figure 10 · p. 17
  • The dataset records synchronized sensor data, calibration and pose information, and fully annotated 3D vehicle bounding boxes.Camera images are 800×600, LiDARs operate at 20 Hz, and global and ego coordinates are recorded.Source: §A.4 · p. 16
  • OPV2V-H contains 10,524 samples split into 6,374 training, 1,980 validation, and 2,170 test samples, with 2–7 agents per sample.Each sample includes four RGB images, four depth images, and three LiDAR point clouds from an agent.Source: §A.4 · p. 16

A.5 FEATURE ENCODING DETAILS

  • PointPillars voxelizes point clouds into pillars, encodes them, and scatters the resulting features into a 2D BEV pseudo-image.The BEV feature is formed for convolutional processing.Source: §A.5 · p. 17
  • SECOND voxelizes point clouds into a 3D grid, applies sparse convolutions, and stacks voxel features along height to form a BEV map.The method processes sparse voxel representations before producing the BEV feature.Source: §A.5 · p. 17
  • Lift-Splat lifts image features into 3D using predicted depth and splats them onto a 2D BEV grid with different image encoders.The paper uses EfficientNet and ResNet50 variants for camera feature encoding.Source: §A.5 · p. 17

A.6 TRAINING DETAILS

The training details compare collective baseline training with HEAL’s two-phase procedure and describe how BEV features are spatially aligned across agents. Baselines train jointly across agent types, whereas HEAL trains new agent types individually after base training.

A.6 TRAINING DETAILS

  • Baseline intermediate-fusion methods collectively train across m agent types by randomly assigning types to agents and selecting an ego agent during each scenario.Each agent encodes features with its type-specific encoder before transmitting them to the ego agent for fusion.Source: §A.6 · p. 17
  • Validation fixes agent-type assignments and the ego agent across methods, while training randomization provides data augmentation and supports fair convergence comparisons.In OPV2V-H, the smallest agent ID is selected as the ego during validation.Source: §A.6 · p. 18
  • HEAL separates collaboration base training from new agent type training, using collective end-to-end training for the base and individual sensor-data training for each new type.During new-agent training, all agents are assigned the new type, and one randomly selected agent is used as the ego for individual training.Source: §A.6 · p. 18

B SPATIAL TRANSFORMATION

  • The implementation applies affine_grid and grid_sample continuously to transform feature maps between agent perspectives.This operation is treated as a 2D affine image transformation in BEV space.Source: B SPATIAL TRANSFORMATION · p. 18
  • Spatial transformation aligns each agent’s BEV feature map to the ego coordinate system using relative pose and the physical BEV dimensions.The relative pose is normalized into a unitless affine matrix after removing the z-axis component and normalizing by Dx and Dy.Source: B SPATIAL TRANSFORMATION · p. 18

Related papers

Loading 2401.13964v3…