PaperlayerView exact revision on arXiv

Source-linked AI summary

STAMP: Scalable Task And Model-agnostic Collaborative Perception

Xiangbo Gao, Runsheng Xu, Jiachen Li, Ziran Wang, Zhiwen Fan, Zhengzhong Tu

arXiv:2501.18616v1cs.CVcs.AIcs.RO

TL;DR

Collaborative perception must integrate heterogeneous agents despite differences in modalities, architectures, and tasks, while conventional fusion approaches face bandwidth, accuracy, compatibility, and security challenges. STAMP uses lightweight adapter-reverter pairs to align BEV features through a shared protocol domain, achieving comparable or superior accuracy with lower training-resource growth. The framework addresses all three heterogeneity types without retraining or model sharing, though weaker agents can bottleneck system performance.

  • Problem

    Heterogeneous collaborative perception must reconcile differences in modalities, models, architectures, tasks, and security assumptions across cooperating agents.

  • Method

    STAMP uses lightweight adapter-reverter pairs to map agent-specific BEV features into a shared protocol domain and back into local domains for collaboration.

  • Results

    2.36 GPU hours per additional agent versus 17.07 for existing heterogeneous pipelines, while STAMP achieves comparable or superior accuracy on simulated and real-world datasets.

  • Takeaways & Limitations

    STAMP supports collaboration across modalities, model architectures, and downstream tasks without retraining or sharing models, while preserving scalability and security.

  • Takeaways & Limitations

    Collaborative perception can exhibit a bottleneck effect in which the weakest agent constrains overall performance, motivating selective multi-group collaboration.

Abstract

from arXiv · show

Perception is crucial for autonomous driving, but single-agent perception is often constrained by sensors' physical limitations, leading to degraded performance under severe occlusion, adverse weather conditions, and when detecting distant objects. Multi-agent collaborative perception offers a solution, yet challenges arise when integrating heterogeneous agents with varying model architectures. To address these challenges, we propose STAMP, a scalable task- and model-agnostic, collaborative perception pipeline for heterogeneous agents. STAMP utilizes lightweight adapter-reverter pairs to transform Bird's Eye View (BEV) features between agent-specific and shared protocol domains, enabling efficient feature sharing and fusion. This approach minimizes computational overhead, enhances scalability, and preserves model security. Experiments on simulated and real-world datasets demonstrate STAMP's comparable or superior accuracy to state-of-the-art models with significantly reduced computational costs. As a first-of-its-kind task- and model-agnostic framework, STAMP aims to advance research in scalable and secure mobility systems towards Level 5 autonomy. Our project page is at https://xiangbogaobarry.github.io/STAMP and the code is available at https://github.com/taco-group/STAMP.

1 INTRODUCTION

Heterogeneous collaborative perception must reconcile differing agent modalities, models, and tasks while avoiding the bandwidth and compatibility limits of conventional fusion. STAMP addresses this with lightweight feature alignment and reports comparable or superior accuracy with substantially lower per-agent training growth.

  • Heterogeneous agents differ in input modalities, model parameters, architectures, and learning objectives, complicating collaborative perception.
  • Early fusion is bandwidth-intensive, while late fusion can reduce accuracy and cannot span models with different downstream tasks.
  • Black-box attacks are nearly ineffective without knowledge of other agents’ models, motivating task- and model-agnostic collaboration for system-level security.
  • STAMP transforms each agent’s BEV features into a shared protocol domain and maps them back through lightweight adapter-reverter pairs.The process, called collaborative feature alignment, supports collaboration within each agent’s source domain.
  • 2.36 GPU hours per additional agent, versus 17.07 GPU hours for existing heterogeneous pipelines, represents a 7.2x training-time saving.Experiments on OPV2V and V2V4Real also report comparable or superior accuracy as heterogeneous agents increase.

2 RELATED WORKS

Related work organizes collaborative perception around information-sharing schemes, heterogeneous-agent challenges, and evaluation datasets. STAMP’s evaluation uses complementary simulated and real-world datasets to cover diverse tasks and real-world performance.

  • Information-sharing schemes: Early fusion shares raw sensor data and maximizes information transfer but requires high communication bandwidth.
  • Information-sharing schemes: Late fusion and intermediate fusion form alternative information-sharing schemes alongside early fusion, motivating comparisons among communication and feature-sharing strategies.
  • Collaborative perception datasets: Collaborative perception datasets include simulated OPV2V and V2X-Sim with approximately 10k multi-agent scenes, plus real-world DAIR-V2X and V2V4Real datasets.The listed datasets provide combinations of RGB, LiDAR, and 3D annotation tasks.
  • Collaborative perception datasets: STAMP selects OPV2V for diverse downstream tasks and V2V4Real to validate performance in real-world scenarios.
  • Heterogeneous collaborative perception: Agent heterogeneity can involve modalities, model architectures or parameters, and downstream tasks, requiring different encoders for inputs such as RGB images or LiDAR point clouds.

3 METHODOLOGY

STAMP replaces conventional compression and decompression in intermediate collaborative perception with lightweight adapter-reverter pairs that translate heterogeneous agents’ BEV features through a shared protocol domain. Collaborative Feature Alignment trains the protocol network and local transformations so agents can exchange features while retaining their local representations.

  • Intermediate collaborative perception: The standard intermediate-fusion pipeline encodes inputs into BEV features, compresses and exchanges them within distance δ, fuses received features, and decodes the result.Each agent uses its own encoder, decompressor, fusion layer, and decoder; δ is the maximum collaboration range.
  • Framework overview: STAMP replaces compression and decompression with local adapters and reverters that transform features between agent-specific and unified protocol BEV domains.Adapted features are broadcast to nearby agents, then reverted into each receiving agent’s local feature representation.
  • Collaborative Feature Alignment: Adapters map local BEV features to protocol features, while reverters map received protocol features back to each agent’s local representation; an agent’s own local feature remains unchanged.The framework applies reversion only to features received from other agents and preserves the receiving agent’s native feature representation.
  • Framework properties: STAMP supports heterogeneous modalities, model architectures, and downstream tasks while keeping collaboration agnostic to other agents’ characteristics.The paper describes this simultaneous support as a first framework-level treatment of all three heterogeneity types, using lightweight adapters and reverters for scalability.
  • Collaborative Feature Alignment: Collaborative Feature Alignment trains a protocol network followed by local adapters and reverters to bridge heterogeneous feature domains.The protocol network establishes a unified BEV embedding space, while local transformations connect each agent’s features to and from that space.
  • Adapter and reverter architecture: The adapter and reverter architecture addresses spatial resolution, feature-pattern, and channel-dimension gaps using interpolation, ConvNeXt layers, and convolutional layers.The design uses linear interpolation for spatial alignment, three ConvNeXt layers for feature patterns, and two convolutions for channel alignment.

4 EXPERIMENTS

Experiments evaluate STAMP across simulated and real-world collaborative-perception settings, including heterogeneous 3D detection, model- and task-agnostic fusion, efficiency, and ablations. Results show competitive perception, lower scaling costs, and the importance of feature alignment and both loss components, while weaker agents can degrade stronger agents.

  • Experimental setup: STAMP is evaluated on OPV2V and V2V4Real for heterogeneous collaborative perception, with OPV2V additionally supporting model- and task-agnostic evaluation.The experiments use complementary simulated and real-world datasets and compare detection, segmentation, and efficiency settings.
  • 3D object detection: Under Gaussian position noise up to σ = 0.4, STAMP achieves superior or comparable performance across noise levels, whereas late fusion degrades as agents increase.Late fusion is especially vulnerable when camera agents contribute incorrect predictions.
  • 3D object detection: On V2V4Real, STAMP achieves the highest AP@30 for both agents, with scores of 0.523 and 0.633, while remaining competitive at AP@50.It outperforms late fusion and matches or exceeds HEAL, indicating effective CFA in real-world scenarios.
  • Model- and task-agnostic fusion: In model- and task-agnostic fusion, STAMP outperforms single-agent segmentation for agents 3 and 4 and improves agent 2’s AP@50 from 0.399 to 0.760 without noise.Collaboration without feature alignment can degrade segmentation and produces negligible gains for agent 2’s camera-based detection.
  • Model- and task-agnostic fusion: Both collaborative approaches degrade Agent 1 relative to its single-agent baseline when Agent 2 relies on less accurate camera sensing.The authors attribute this to a bottleneck effect in which a weaker agent constrains system performance.
  • Ablation studies: Ablations show that performance depends on feature alignment design and combining feature-space loss Lf with decision-space loss Ld.Using only Ld causes a significant OPV2V drop, using only Lf is generally lower and more fluctuating, and dropping either loss degrades V2V4Real performance.

5 CONCLUSION

STAMP is presented as a scalable, task- and model-agnostic collaborative perception framework that addresses modality, architecture, and downstream-task heterogeneity. Experiments support its performance and computational-efficiency benefits, while the authors identify weakest-agent bottlenecks and the need for further security analysis.

  • STAMP addresses heterogeneity in modalities, model architectures, and downstream learning tasks through lightweight adapter-reverter pairs.The framework is designed for scalable, secure, and flexible collaborative perception.
  • Experiments on OPV2V and V2V4Real demonstrate superior performance and computational efficiency over existing state-of-the-art methods.
  • The weakest agent can constrain overall collaborative-perception performance, motivating selective communication within multi-group systems.The authors discuss multi-group collaboration as a way to mitigate this bottleneck effect.
  • The codebase and detailed experimental information are released to support reproducibility.Implementation details include dataset descriptions, architectures, and training procedures.
  • The authors state that limiting model sharing improves security but requires further security analysis.They advocate collaboration with experts to evaluate and strengthen the framework.

A.1 IMPLEMENTATION DETAILS

The implementation uses standardized training settings and shared adapter-reverter architectures across collaborative-perception models. Experiments cover heterogeneous 3D object-detection models with varied modalities, encoders, and parameter counts.

  • Training setup: All models use an initial learning rate of 0.001, decayed by 0.1 at 50% and 83% of training, on one NVIDIA RTX A6000 GPU.Model training takes 7 to 30 GPU hours depending on architecture.
  • Training setup: Adapters and reverters use a 0.01 learning rate, decay it by 0.1 after one epoch, and are trained in pairs.
  • Adapter and reverter architecture: Adapters and reverters share the same architectures across collaborative-perception models, with a broadcasting feature map sized (128, 128, 64).Their input and output dimensions vary with each local model and the broadcasting feature map.
  • Adapter and reverter architecture: Figure A1 presents the architecture of the adapter and reverter components.
  • 3D object detection setting: The 3D object-detection experiments use 12 heterogeneous models differing in modality, encoder, and encoder parameter count.Some encoders are enlarged by increasing hidden-layer sizes, and all use pyramid fusion with three 1 × 1 convolutional classification layers.

A.2 ARCHITECTURAL COMPARISON BETWEEN EXISTING FRAMEWORKS

Existing heterogeneous collaborative-perception frameworks trade off scalability, security, task flexibility, communication cost, or accuracy. STAMP is presented as a contrasting framework in the architectural comparison, with fixed and trainable components distinguished visually.

  • Late fusion combines agent outputs after processing, while calibrators address domain gaps between heterogeneous outputs.
  • End-to-end training can be effective but lacks scalability because it requires retraining all agents’ models.It also compromises security and task flexibility through shared fusion models and decoders.
  • Figure A2 compares existing collaborative-perception frameworks with STAMP using blue boxes for fixed parameters and red boxes for parameters trained during collaboration.

A.3.1 DIFFERENT PROTOCOL MODELS

Protocol-model performance depends on alignment with agent objectives and remains compatible with different encoder architectures. Strong objective mismatches can instead cause severe degradation, making protocol selection important.

  • Impact of Model Objectives: Protocol-model alignment with an agent’s learning objective determines whether collaborative alignment improves performance.
  • Impact of Model Objectives: A camera protocol model raises camera-based Agent 2’s performance from 0.760 to 0.777.
  • Impact of Model Objectives: A dynamic-segmentation protocol model raises Agent 4’s performance from 0.690 to 0.723, while a static-segmentation protocol model raises Agent 3’s performance from 0.624 to 0.681.
  • Impact of Model Objectives: A static-segmentation protocol model drops Agent 4’s mAP from 0.690 to 0.235 under a significant objective mismatch.The result highlights the importance of careful protocol-model selection.
  • Encoder Architecture Variations: The evaluation compares CNN-based and Point-transformer encoder architectures for protocol models.
  • Encoder Architecture Variations: The Point-transformer protocol model achieves AP@50 = 0.991 versus 0.973 for the CNN-based counterpart, indicating compatibility with different encoder architectures.The passage emphasizes protocol-model performance over a specific architectural design.

A.3.2 ADVERSARIAL ROBUSTNESS EVALUATION

The evaluation compares end-to-end training, HEAL, and STAMP under collaborative white-box attacks on V2V4Real. STAMP shares no local models and communicates through protocol representations, while attacks substantially affect only the model-sharing setting.

  • The evaluation tests end-to-end training, HEAL, and STAMP with two agents on V2V4Real under collaborative white-box attacks.Agent 1 is the attacker and agent 2 is the victim.
  • STAMP shares no local models and uses protocol representations for communication, assuming no access to the victim model.
  • Adversarial attacks have minimal impact on HEAL and STAMP but significantly degrade performance in end-to-end training.The comparison attributes this difference to local model security in HEAL and STAMP.

A.3.3 MORE COMPARISON WITH THE STATE-OF-THE-ART METHODS

The comparisons examine heterogeneous agents, multi-group collaboration, feature alignment, and scalability. STAMP supports independent agent models and multiple collaboration groups with lightweight adapter-reverter pairs, while CFA aligns feature representations before fusion.

  • The heterogeneous comparison uses two LiDAR agents and two camera agents with different encoders, while accuracy is averaged across all samples.The comparison includes V2X-ViT, CoBEVT, HM-ViT, HEAL, and STAMP-related settings.
  • Multi-group collaboration: Multi-group multi-model systems let qualified agents join multiple collaboration groups, unlike multi-group single-model systems that restrict each agent to one group.
  • Multi-group collaboration: The multi-group structure mitigates bottlenecks by grouping high-performing agents with similarly capable peers while allowing assistance across other groups.It also supports dynamic group formation for task requirements or environmental conditions.
  • Scalability limitations: Existing heterogeneous pipelines face computational and memory constraints because end-to-end training must train all models, while group-specific encoders multiply with the number of groups.
  • STAMP scalability: Approximately 1MB per collaboration group, each STAMP adapter-reverter pair enables an agent to participate in multiple groups without significant computational overhead.
  • Collaborative feature alignment: Collaborative feature alignment maps disparate agent feature representations into the same domain, stabilizing feature fusion and improving post-alignment visualization consistency.Before CFA, some feature maps are at much lower scale; after CFA, features are aligned to a shared domain.
Loading 2501.18616v1…