Source-linked AI summary
CauseCollab: Causal Unified and Modality-Agnostic Network for Heterogeneous Collaborative Perception
Weize Li, Yang Li, Quan Yuan, Xiaoyuan Fu, Guiyang Luo, Jinglin Li
TL;DR
Heterogeneous sensors and independently trained protocol converters limit collaborative perception by producing inconsistent pseudo-protocol representations, especially across large modality gaps. CauseCollab uses causal metric learning and a context-guided Unified Converter to enforce semantic consistency, achieving state-of-the-art results while supporting new modalities through lightweight adapters.
Problem
Independently trained modality-specific converters can create inconsistent pseudo-protocol distributions, causing semantic inconsistency and error accumulation, especially with large modality gaps.
Method
CauseCollab uses causal metric learning to disentangle modality-specific confounders and a unified context-guided converter to learn semantically consistent protocol representations.
Results
CauseCollab achieves state-of-the-art performance on OPV2V and DAIR-V2X, with substantial improvements in scenarios involving larger modality gaps.
Takeaways & Limitations
New modalities can be added by training only lightweight adapters while existing modality components remain frozen and inaccessible.
Takeaways & Limitations
During adaptation, previously deployed modality encoders, fusion modules, and classification heads must remain frozen and inaccessible.
Abstract
from arXiv · showhide
Collaborative perception enhances environment understanding through multi-agent information sharing, but its performance in real-world scenarios is constrained by heterogeneous sensor modalities and model architectures. Recent protocol-based two-stage methods alleviate this problem by mapping heterogeneous features into a shared protocol space; however, independently trained modality-specific converters often generate modality-specific pseudo-protocol distributions, leading to semantic inconsistency and error accumulation, which is particularly pronounced in scenarios with large modality discrepancies. To address this issue, we propose CauseCollab, a causal unified and modality-agnostic network. CauseCollab formulates representation learning in the protocol space from a causal perspective, explicitly disentangling semantic factors from modality-specific statistical confounders via causal metric learning. Meanwhile, CauseCollab adopts context-guided Unified Converter for heterogeneous modalities to ensure cross-modal semantic consistency. In addition, integrating new modalities only requires training adapters with minimal parameters. Extensive experiments on the OPV2V and DAIR-V2X datasets demonstrate that CauseCollab achieves state-of-the-art performance, with more significant gains in scenarios involving large modality gaps.
1. Introduction
Real-world heterogeneous agents make direct feature fusion difficult, while independently trained protocol converters can preserve modality-specific inconsistencies. CauseCollab addresses this with causal semantic disentanglement, a unified converter, lightweight adapters, and strong results on heterogeneous benchmarks.
- Different sensor types, voxel sizes, and network architectures make intermediate features difficult to fuse in real-world collaborative perception.
- Two-stage protocol methods can accumulate errors because independently trained converters produce inconsistent pseudo-protocol distributions across modalities.
- CauseCollab uses causal modeling to disentangle shared semantic factors from modality-specific statistical confounders in protocol representations.
- The Unified Converter combines a semantic context extractor and context-guided dynamic refiner to constrain local refinement and improve cross-modal semantic consistency.
- New modalities can be integrated by training lightweight adapters while preserving the existing converter backbone.
- CauseCollab achieves state-of-the-art performance on OPV2V and DAIR-V2X, with larger improvements when modality gaps are substantial.
2. Related Work
Related work addresses heterogeneous collaboration through feature translation and introduces causal reasoning for modality-consistent representations. CauseCollab extends these directions by applying structural causal modeling and intervention to cross-modal semantic alignment.
- Collaborative perception research commonly shares intermediate features, while heterogeneous settings require translators between agents with different semantic spaces.
- One-stage translators map modality pairs directly but scale poorly as the number of modalities increases, whereas two-stage approaches use protocol spaces.
- Causal inference separates causal factors from spurious correlations and estimates intervention effects without confounding factors.
- CauseCollab introduces structural causal models and an intermediate-feature intervention module for more robust cross-modal semantic alignment.
3. Method
CauseCollab uses a two-stage heterogeneous perception pipeline that maps collaborative features into a shared protocol space, then reconstructs them for local fusion. Its causal Stage-1 representation learning separates semantic factors from modality-specific statistical confounders using context-guided conversion and intervention-based metric learning.
- Two-stage architecture: Collaborative features are encoded, projected into a shared protocol space, reconstructed into the ego agent’s local space, fused, and passed to a detection head.Stage-1 produces protocol representations, while Stage-2 uses modality-specific reconstruction before pyramid fusion and task prediction.
- Causal formulation: The structural causal model treats true semantics as causal factors and modality-specific statistical noises as confounders that can induce spurious protocol-space clusters.The model distinguishes causal paths from spurious correlation paths and targets reduced dependence on modality noise while preserving semantic sensitivity.
- Causal intervention: Mask-Guided Intervention via SPD Feature Geometry constructs positive and negative counterfactual protocol samples by suppressing or injecting modality statistics while preserving spatial semantic structure.Whitening–coloring transformations adjust channel means and correlations without altering spatial structure; the intervention is used only during Stage-1 training.
- Context-guided conversion: Stage-1 uses a Semantic Context Extractor to derive global semantic context from heterogeneous modality features through hierarchical downsampling and semantic extraction.The resulting context representation supports subsequent dynamic refinement.
- Context-guided conversion: The Context-Guided Dynamic Refiner performs layer-wise semantic correction on local modality features under aligned context constraints to generate semantically consistent protocol representations.Its transformations progressively refine local features while preserving discriminative details and anchoring them to consistent global semantics.
4. Experiments
Experiments on OPV2V and DAIR-V2X evaluate CauseCollab under heterogeneous modalities, large modality gaps, new-modality adaptation, ablations, and qualitative feature consistency. Across these settings, the method outperforms prior approaches, supports lightweight adaptation, and produces cleaner protocol semantics.
- Performance Comparison: CauseCollab outperforms prior heterogeneous collaborative perception methods on OPV2V and DAIR-V2X evaluations.The experiments include general collaboration, larger modality gaps, and real-world DAIR-V2X settings.
- Performance Comparison: 4.2% and 3.6% AP gains are achieved at IoU=0.3 and IoU=0.5, respectively, for CEff ego agents with CRes neighbors.For Lpp4 ego agents with CEff neighbors, the gains are 0.5% and 0.4% at the same thresholds.
- Scalability: Higher average precision with fewer trainable parameters is achieved when adapting the unified converter to novel modalities Lsd2 and CEffB1.The shared converter is adapted to new modalities through fine-tuning, supporting parameter reuse.
- Ablation Study: Removing any individual component consistently degrades performance, including the causal intervention module, intervention mask, semantic context guidance, SCE, CGDR, and adapter.The ablations support the contribution of these components to protocol-space consistency and adaptation.
- Qualitative Evaluation: CauseCollab produces cleaner protocol semantics than STAMP, which introduces semantically irrelevant noise in background regions during camera-to-protocol conversion.The comparison visualizes LiDAR and camera intermediate features and their converted protocol-space features.
5. Conclusion
CauseCollab addresses semantic inconsistency and modality confounding in heterogeneous collaborative perception through a causal, unified, modality-agnostic network. It learns semantically consistent protocol representations and supports new modalities through lightweight adapters.
- CauseCollab uses causal metric learning to achieve modality-agnostic semantic consistency while addressing modality confounding in the protocol space.
- New modalities can be extended to the semantically consistent protocol space by training lightweight adapters with minimal parameters.
Impact Statement
CauseCollab targets safer and more adaptable multi-agent perception by improving cross-modal semantic consistency and supporting integration of new sensing modalities. The paper evaluates this direction using both simulated and real-world collaborative-perception datasets.
- CauseCollab addresses semantic inconsistency and modality confounding in heterogeneous collaborative perception.
- The framework enables integration of new sensing modalities with minimal computational overhead.
- OPV2V contains over 70 scenarios and 11,464 frames generated through CARLA and OpenCDA co-simulation.
- DAIR-V2X represents vehicle-infrastructure collaboration between one vehicle and one roadside unit.
A.2. Training Details
The experiments use standardized encoder settings and multiple agent modalities, while the training details implement mask-guided weighted statistics and canonical feature transformations. These operations suppress modality-specific statistics or inject controlled styles while preserving spatial semantics.
- Training setup: The evaluation uses identical encoder weights across methods and trains heterogeneous networks within a shared spatial range.LiDAR encoders use [−102.4, −51.2, 102.4, 51.2], camera encoders use [−51.2, −51.2, 51.2, 51.2], and heterogeneous networks use the latter range.
- Agent configurations: The experiments cover eight agent modalities plus a protocol modality, with L denoting LiDAR and C denoting camera.
- Feature statistics: Weighted first- and second-order statistics model modality style, while eigendecomposition-based whitening–coloring transforms adjust channel statistics without changing spatial arrangement.
- Weighted statistics: The semantic mask defines spatially varying weights through w = η + ρm, guaranteeing positive weights while controlling contribution strength.
- Canonical suppression: Mask-aware Canonical Suppression maps protocol features to the canonical target (µ0, Σ0) = (0, I) and applies a mask-guided residual update.
- Controlled intervention: Canonical Suppress-Then-Inject first produces a modality-agnostic positive and then creates a hard negative by reintroducing controlled heterogeneous statistics while preserving semantics.
B.2. SCE Architecture
The Semantic Context Extractor builds global semantic context through progressive downsampling and ConvNeXt-based semantic extraction. Its detailed structure is presented in Figure 5.
- The Semantic Context Extractor progressively enhances global semantic modeling through two-step downsampling and ConvNeXt.
- The extractor combines downsampling with semantic extraction blocks and feature fusion to form the context representation.The fusion operation aligns channels and spatial dimensions before weighted concatenation and spatial mixing.
- Figure 5 presents the detailed structure of the Semantic Context Extractor.
B.3. CGDR Architecture
The Context-Guided Dynamic Refiner combines local modality features with semantic context, then modulates and filters the fused representation. On OPV2V, CauseCollab outperforms late fusion across tested pose-error levels and modality combinations.
- Architecture: The Context-Guided Dynamic Refiner receives local modality features and context features as inputs.
- Architecture: The refiner aligns and concatenates context with local features before joint fusion using depth-wise convolution and LayerNorm.
- Feature refinement: FiLM-style context modulation, channel recalibration, and gated filtering suppress noise in the fused representation.
- Feature refinement: Residual connections with Layer Scale stabilize gradients while preserving detailed features through subsequent MLP transformation.
- Architecture: Figure 6 depicts the detailed architecture of the Context-Guided Dynamic Refiner.
- Comparison with late fusion: CauseCollab significantly outperforms late fusion across all tested pose-error levels and collaboration combinations on OPV2V.
C.2. Analysis of Different Protocol Modalities
CauseCollab remains effective when either LiDAR or Camera serves as the protocol modality, indicating reduced protocol-modality asymmetry. It also outperforms NegoCollab in general and novel-modality scenarios.
- CauseCollab achieves comparable collaborative perception performance with either LiDAR-based or Camera-based protocol modalities.The comparison evaluates LiDAR (Lpp4) and Camera (CEff) as protocol modalities.
- The results indicate that causal intervention reduces protocol-modality asymmetry and improves robustness of the learned protocol space.
- CauseCollab explicitly disentangles semantic factors from modal entanglement to learn a modality-agnostic protocol space.
- CauseCollab outperforms NegoCollab in both general and novel-modality scenarios.The general setting is denoted ALL, while Lsd2new + CEffB1new evaluates novel-modality adaptation.
C.4. Robustness Evaluation on More Novel Modality Adaptation
CauseCollab is evaluated with four novel modalities, including newly introduced LiDAR and Camera configurations. Visualizations using CEff and CRes further examine robustness under larger modality gaps.
- The evaluation combines four novel modalities: Lvn4new, CRes34new, Lsd2new, and CEffB1new.The results are reported in Table 10 under the four-novel-modality collaborative perception scenario.
- Lvn4new is a LiDAR modality encoded by VoxelNet, while CRes34new is a Camera modality based on ResNet34 with 128 output channels.
- CEff and CRes are selected for visualization because they have larger modality gaps.
- The visualizations demonstrate CauseCollab’s robustness in scenarios with larger domain gaps.