PaperlayerView exact revision on arXiv

Source-linked AI summary

One is Plenty: A Polymorphic Feature Interpreter for Immutable Heterogeneous Collaborative Perception

Yuchen Xia, Quan Yuan, Guiyang Luo, Xiaoyuan Fu, Yang Li, Xuanhan Zhu, Tianyou Luo, Siheng Chen, Jinglin Li

arXiv:2411.16799v2cs.CV

TL;DR

Immutable heterogeneous collaboration must reconcile different fixed perception networks without retraining them, while existing interpreters trade extensibility against semantic loss. PolyInter uses one interpreter with agent-specific prompt tuning, and reports higher AP than SOTA interpreters at both evaluated IoU thresholds.

  • Problem

    Different fixed perception networks produce heterogeneous intermediate features that other agents cannot readily understand, while existing interpreters either require a new interpreter per agent or use two-stage mapping with cumulative semantic loss.

    Source: §1 · p. 1

  • Method

    PolyInter uses one interpreter network with general and neighbor-specific prompts, tuning new neighbors’ specific prompts while keeping the interpreter network and general prompt frozen.

    Source: §1 · p. 2

  • Results

    PolyInter improves AP by 7.9% and 11.1% at IoU thresholds of 0.5 and 0.7, respectively, compared with SOTA immutable heterogeneous feature interpreters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    New neighbor agents can be incorporated by tuning their specific prompts while retaining one interpreter for one-stage interpretation into the ego agent’s semantic space.

    Source: §1 · p. 2

Abstract

from arXiv · show

Collaborative perception in autonomous driving significantly enhances the perception capabilities of individual agents. Immutable heterogeneity, where agents have different and fixed perception networks, presents a major challenge due to the semantic gap in exchanged intermediate features without modifying the perception networks. Most existing methods bridge the semantic gap through interpreters. However, they either require training a new interpreter for each new agent type, limiting extensibility, or rely on a two-stage interpretation via an intermediate standardized semantic space, causing cumulative semantic loss. To achieve both extensibility in immutable heterogeneous scenarios and low-loss feature interpretation, we propose PolyInter, a polymorphic feature interpreter. It provides an extension point where new agents integrate by overriding only their specific prompts, which are learnable parameters that guide interpretation, while reusing PolyInter's remaining parameters. By leveraging polymorphism, our design enables a single interpreter to accommodate diverse agents and interpret their features into the ego agent's semantic space. Experiments on the OPV2V dataset demonstrate that PolyInter improves collaborative perception precision by up to 11.1% compared to SOTA interpreters, while comparable results can be achieved by training only 1.4% of PolyInter's parameters when adapting to new agents. Code is available at https://github.com/yuchen-xia/PolyInter.

1. Introduction

Immutable heterogeneous collaboration must bridge semantic differences without replacing fixed, safety-critical perception networks. PolyInter proposes a single interpreter that extends to new agents through prompt tuning, avoiding separate interpreters or two-stage translation.

2.1. Collaborative Perception

  • Intermediate fusion exchanges feature-level perceptual data aligned in semantics and granularity, while collaborative perception addresses occlusion and adverse weather through data exchange.Collaborative perception is used for object detection, obstacle recognition, and occupancy prediction.Source: §1 · p. 1

1. Introduction

  • Existing one-stage interpreters require a separate trained interpreter for each new neighbor, while two-stage methods use a standard semantic space but incur cumulative semantic loss.Both strategies preserve the original perception networks, but neither resolves the combined need for extensibility and low semantic loss.Source: §1 · pp. 1–2
  • PolyInter reuses one interpreter and a frozen general prompt, adapting to new neighbors by fine-tuning only their specific prompts for one-stage interpretation into the ego agent’s semantic space.The interpreter network interprets neighbor semantics; prompts are learnable parameters that guide interpretation.Source: §1 · p. 2
  • PolyInter improves AP by 7.9% and 11.1% at IoU thresholds of 0.5 and 0.7, respectively, compared with SOTA immutable heterogeneous feature interpreters.The approach combines channel and spatial alignment modules with learnable prompts.Source: §1 · p. 2

2.2. Heterogeneous Feature Fusion

  • Prior heterogeneous-fusion work includes modality-specific architectures and feature transformations, while MPDA and PnPDA use interpreters to align neighbor features with the ego agent’s semantic space.PolyInter builds on interpreter-based alignment with the goal of generalizing to new agents using a single interpreter network.Source: §2.2 · p. 3

2.3. Visual Prompt

  • Visual prompt tuning adapts vision Transformers with trainable prompts while keeping the backbone frozen, and other work has tailored prompts to tasks and domain adaptation.Related prompt research also targets generalization and adaptability across domains and tasks.Source: §2.3 · p. 3

3. Methodology

PolyInter uses a shared interpreter with agent-specific prompts to align heterogeneous features with the ego agent’s semantic space. Its two-phase training keeps the interpreter fixed when adapting to new agents, while its application uses the corresponding prompt and resizer for each collaborating agent type.

3.1. Overall Pipeline

  • Phase I trains the interpreter, shared general prompt, and specific prompts on features from collaborating agents; Phase II adapts to new agents by updating only their specific prompts and a necessary resizer.The core interpreter remains unchanged during Phase II.Source: §3.1 · p. 3

3. Methodology

  • PolyInter’s common interpreter structure combines channel selection and spatial attention with customizable prompts for interpreting heterogeneous agent features.The architecture provides an extension point for new agents while retaining the shared interpreter.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Polymorphic Feature Interpreter

  • The channel selection module uses cosine similarity to match and weight neighbor-feature channels, while spatial attention models spatial relationships and maps refined neighbor features into the ego feature’s semantic space.Prompts are initialized either by sampling training features or randomly; random initialization can use low-rank decomposition to reduce trainable parameters.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3. Phase I: Base Model Training

  • Phase I separates shared and agent-specific semantics with general and specific prompts, using adversarial and style losses alongside collaborative detection training.The specific feature is also trained for single-agent object detection, and its style loss aligns neighbor and ego feature distributions.Source: §3.3 · pp. 5–6

3.4. Phase II: Generalization

  • Phase II generalizes PolyInter to new agent types by optimizing their specific prompts rather than retraining the entire interpreter.The Phase II objective includes collaborative loss and a weighted specific-feature constraint.Source: §3.4 · p. 6

3.5. Application Paradigm

  • At inference, the ego agent uses one PolyInter for collaborative tasks, activating the corresponding specific prompt and resizer when a neighbor broadcasts features.Without collaboration, the ego agent processes its own features independently and bypasses PolyInter.Source: §3.5 · p. 6

4. Experiments

Across three new-neighbor scenarios, PolyInter matches or exceeds PnPDA and MPDA while adapting with fewer trainable parameters; ablations and visualizations examine its components and feature alignment.

4.1. Settings

  • Experiments train on OPV2V with frozen encoders and ego detection head, then adapt to pp4, sd1, and vn6 by training only each new neighbor’s specific prompt and resizer.The main experiments use two base encoder combinations and evaluate two-agent scenarios; three-agent results are in the supplementary material.Source: §4.1 · p. 6

4.2. Performance Comparison

  • PolyInter improves AP@0.7 by 9.2% for pp8-pp4 and averages 9.7%/14.3% gains for pp8-sd1 and 12.6%/9.8% for pp8-vn6 at AP@0.5/AP@0.7, respectively.For pp8-pp4 it achieves comparable AP@0.5; the reported gains are against the other two interpreters.Source: §4.2 · p. 7, Table 2 · p. 7
  • PolyInter needs only a specific prompt and resizer per new neighbor, whereas PnPDA and MPDA train interpreter and fusion parameters for each ego-neighbor pair.The parameter-growth comparison adds pp4, sd1, and vn6 incrementally; the difference is more pronounced with CoBEVT, where the baselines add a fusion network for each neighbor type.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3. Ablation Study

  • Removing any interpreter module or prompt reduces performance; channel selection has the largest reported impact with vn6, lowering AP by 8.1% and 11.7% at IoU 0.5 and 0.7.Spatial attention also matters after channel alignment; removing it lowers AP across all three scenarios, while removing the specific prompt causes a substantial drop and other prompt changes produce smaller declines.Source: Table 3 · p. 7, §4.3 · p. 8
  • With pp8 as ego agent, even at 0.16M trainable parameters for sd1 and 0.04M for vn6—1.4% and 0.4% of respective interpreters—precision surpasses MPDA and PnPDA.Collaborative perception performance improves as trainable parameters increase, but its improvement rate levels off beyond a certain threshold.Source: §4.3 · p. 8

4.4. Qualitative Evaluation

  • Feature visualizations show channel reorganization and the specific prompt bringing vn6 features closer in style to pp8, followed by spatial attention that highlights agent position more clearly.Spatial details remain partly blurred after channel reorganization and become more distinct after spatial attention.Source: §4.4 · p. 8, Figure 6 · p. 8

5. Conclusion

PolyInter addresses openness and improves perception precision in immutable heterogeneous collaboration by letting new neighbors reuse one interpreter and fine-tune their prompts.

5. Conclusion

  • PolyInter uses a single interpreter network with an extension point for new neighbor agents, which inherit it and fine-tune prompts for collaboration.The conclusion frames this as improving extensibility in settings where agents’ perception networks vary and remain fixed.Source: §5 · p. 8

1. Experimental Details

When adapting to a new neighbor, trainable parameters comprise its specific prompt and a feature-size resizer.

2. Experimental Details

  • The resizer combines max-pooling and a 1 × 1 convolution; prompt and resizer parameter counts are C2 × H1 × W1 and C1 × C2, respectively.Here ego and neighbor feature sizes are C1 × H1 × W1 and C2 × H2 × W2.Source: §1 · p. 11

2. Additional Ablation Study

The additional ablation evaluates three loss components in the paper’s main experimental setting.

2. Additional Ablation Study

  • Ablations test style loss for shared and agent-specific semantics, adversary loss for shared semantics, and single loss for agent-specific semantics.Results are reported in Table 2 following the main paper’s settings.Source: §2 · p. 11

3. Performance of Multi-Modal Fusion

PolyInter is evaluated in LiDAR-camera fusion using EfficientNet and ResNet encoders, and the authors report that it performs well in this setting.

  • The experiments integrate LiDAR and camera data using EfficientNet and ResNet image encoders.Source: §3 · p. 11
  • PolyInter performs well in the reported multi-modal fusion experiments.Source: §3 · p. 11
  • The experiments test two encoder combinations, switching which image encoder is used by new neighbors during generalization.Source: §3 · p. 11

4. New Datasets

The experiments compare PolyInter with PnPDA and MPDA on V2XSet and real-world DAIR-V2X, extending evaluation beyond OPV2V.

4.1. Performance Comparison on V2XSet

  • On V2XSet, the experiments compare PolyInter with PnPDA and MPDA.V2XSet includes vehicle-to-everything cooperation and realistic noise simulation, unlike OPV2V.Source: §4.1 · p. 11

4.2. Performance Comparison on DAIR-V2X

  • The section extends comparisons with PnPDA and MPDA from V2XSet to a real-world dataset, DAIR-V2X.Source: §4.1 · p. 11, §4.2 · p. 11
  • On real-world DAIR-V2X, the experiments compare PolyInter with PnPDA and MPDA using pp8-pp4-vn4 for base-model training.Source: §4.2 · p. 11

5. Additional Experiments

Additional experiments examine three-agent collaboration, PnPDA's two-stage interpretation, and PolyInter's Phase I performance across agent combinations.

5.1. Three-agent Collaborative Perception

  • In three-agent collaboration, the ego interpreter maps each of two heterogeneous neighbors' features into the ego semantic space before fusion and detection.Source: §5.1 · p. 11

5.2. Two-Stage Interpretation of PnPDA

  • The paper visualizes PnPDA's two-stage interpretation as mapping neighbor features first to a standard semantic space, then to the ego semantic space.Source: §5.2 · p. 12
  • Two-stage interpretation through a standard semantic space incurs two stages of semantic loss and considerably diminishes collaborative performance.Source: §5.2 · p. 12

5.3. Performance of PolyInter in Phase I

6. Additional Qualitative Evaluation

The qualitative visualization shows how neighbor features are processed and interpreted in the ego agent's semantic space.

  • Distinct specific prompts are matched to features from different heterogeneous neighbor agents.Source: §6 · p. 13
  • The Channel Selection Module reorganizes neighbor features to align them with ego features.Source: §6 · p. 13
  • The Spatial Attention Module establishes spatial connections before heterogeneous neighbor features are interpreted into the ego semantic space.Source: §6 · p. 13

Related papers

Loading 2411.16799v2…