A2DINOv3: Rethinking Multi-Modal Object Detection via Socialized Collaboration

Jiekang Feng; Zhihe Fan; Yunqi Zhu; Xinjie Yao; Yueying Zhang; Yike Gao; Ranxin Li; Guanzuo Chen

Source-linked AI summary

A2DINOv3: Rethinking Multi-Modal Object Detection via Socialized Collaboration

Jiekang Feng, Zhihe Fan, Yunqi Zhu, Xinjie Yao, Yueying Zhang, Yike Gao, Ranxin Li, Guanzuo Chen

arXiv:2608.21099v1Aug 21, 2026cs.CVcs.AI

TL;DR

A2DINOv3 addresses fusion degradation and pretrained-representation distortion in multimodal detection by regulating communication between RGB and infrared experts. Its Socialized Collaboration Protocol and zero-initialized pathways enable selective, gradual interaction, and experiments across four benchmarks show consistently state-of-the-art performance.

Problem
Multimodal adaptation is challenged by unrestricted cross-modal signals and abrupt joint optimization, which can degrade fusion and distort pretrained representations.
Method
A2DINOv3 models RGB and infrared as complementary experts and regulates their interaction through SCP with zero-initialized pathways.
Results
A2DINOv3 achieves state-of-the-art performance and consistently improves robustness across four multimodal detection benchmarks.
Takeaways & Limitations
Effective multimodal adaptation depends on regulating when, what, and how much information is communicated rather than maximizing feature exchange.
Takeaways & Limitations
Both experts share the same communication constraint, with transfer directions learned from data rather than manually defined reliability rules.

Abstract

from arXiv · show

Multi-modal object detection is essential for robust scene understanding in challenging conditions, including low-light and adverse environments. Recent vision foundation models (e.g., DINOv3) have exhibited strong representation capabilities, yet adapting them to multi-modal scenarios remains challenging. Existing dense cross-modal fusion strategies often force heterogeneous modalities to interact indiscriminately, which may introduce redundant information and disrupt the valuable pre-trained representations. To address this issue, we revisit multi-modal fusion from the perspective of socialized learning and propose adapter to DINOv3 (A2DINOv3), a multi-expert collaboration framework with a Socialized Collaboration Protocol (SCP). Specifically, RGB and infrared branches are modeled as heterogeneous experts that independently preserve their specialized knowledge while exchanging complementary information through selective and constrained interactions. This design mitigates harmful cross-modal interference and prevents degradation of pre-trained priors during adaptation. Furthermore, a zero-initialization strategy is introduced to gradually activate cross-modal collaboration, enabling a smooth transition from modality-specific learning to cooperative representation learning. Extensive experiments on four multi-modal benchmarks, including aerial detection (GAIIC), autonomous driving (FLIR), low-light surveillance (LLVIP), and diverse real-world scenarios (M3FD), demonstrate that A2DINOv3 consistently achieves state-of-the-art performance in multi-modal object detection.

Introduction

A2DINOv3 adapts DINOv3 to multi-modal detection by treating RGB and infrared as heterogeneous experts that exchange complementary information through selective, constrained collaboration. This design targets fusion degradation and distortion of pre-trained representations while achieving state-of-the-art performance across four benchmarks.

Motivation: Vision foundation models provide transferable visual priors, but their effectiveness is limited when input modalities suffer severe visual degradation.Nighttime conditions, illumination changes, and atmospheric interference can leave single-modal sensors without sufficient reliable cues.
Motivation: RGB and infrared offer complementary evidence: RGB captures fine-grained appearance under sufficient illumination, while infrared supplies structural and thermal cues under degraded visibility.
Challenges: Existing adaptation methods lack regulated cross-modal collaboration, allowing unreliable signals to cause fusion degradation and aggressive optimization to distort pre-trained visual priors.
Approach: A2DINOv3 models RGB and IR as heterogeneous experts and uses the Socialized Collaboration Protocol to preserve modality-specific knowledge while enabling selective information exchange.
Approach: Zero-initialized up-projections keep communication inactive initially, then task-driven gradients progressively activate cross-modal interaction during optimization.
Results: A2DINOv3 achieves state-of-the-art performance and consistently improves robustness across four multi-modal benchmarks and challenging scenarios.

Related Works

Prior visible-infrared detection methods emphasize feature-level fusion, while adapter research primarily addresses efficient single-stream adaptation. A2DINOv3 instead uses constrained bidirectional adapter pathways for selective cross-modal transfer while preserving modality-specific representations.

Multi-Modal Fusion: Visible-infrared detection methods predominantly use feature-level fusion, including spatial-focused dense attention and frequency-focused architectures.
Multi-Modal Fusion: Dense cross-modal interaction can propagate unreliable signals and distort pretrained representations, creating a trade-off between exploiting complementarity and preserving modality-specific competence.
Adapter-Based Transfer Learning: Adapter-based transfer learning introduces lightweight trainable modules while freezing most backbone parameters for parameter-efficient adaptation.
Adapter-Based Transfer Learning: Existing adapters mainly support efficient single-stream adaptation rather than heterogeneous cross-modal communication.
Adapter-Based Transfer Learning: A2DINOv3 repurposes low-rank adapters as bidirectional SCP pathways with zero-initialized up-projections for progressive, selective transfer.

Methodology

A2DINOv3 structures RGB–IR detection as collaboration between separate experts, using shared DINOv3 representations, constrained communication, and gradual pathway activation. Multi-level outputs are then aggregated and passed to a standard detection pipeline.

Parameter-Shared Dual Experts: A parameter-shared dual-stream backbone maintains separate RGB and infrared activations while transferring DINOv3’s semantic prior to both modalities.Separate propagation delays heterogeneous feature mixing until task learning can assess modality reliability.
Socialized Collaboration Protocol: SCP regulates bidirectional RGB-to-IR and IR-to-RGB communication through low-dimensional residual pathways rather than fully aligning the modalities.The constrained subspace limits how strongly one modality can perturb the other and favors compact, task-relevant exchange.
Sparse Interaction: Cross-modal exchange occurs only at selected layers, preserving the backbone as the primary representation learner and confining interaction to controllable interfaces.The predefined interaction set is I = {5, 8, 11}.
Progressive Collaboration: Zero-initialized up-projection matrices make communication residuals initially zero, preserving pretrained representations while leaving the pathway trainable through subsequent updates.The communication path receives updates from the first optimization step and progressively enables gradient flow through SCP.
Collaborative Evidence Aggregation: Selected interaction-layer outputs retain modality-specific representations, incorporate compact cross-modal evidence, and are combined by parameter-free mean fusion.Mean fusion separates SCP’s information-selection role from aggregation and preserves balanced contributions from both streams.
Detection Pipeline: The fused feature pyramid is forwarded to a hybrid encoder and DETR decoder, with Hungarian matching and classification, L1, and generalized IoU losses used for training.The reported table compares detection results on the GAIIC dataset across validation and test sets using percentage metrics.

Experiments

Experiments evaluate A2DINOv3 across four RGB–infrared benchmarks, showing strong cross-domain performance, improved knowledge retention, and benefits from bounded, gradually activated communication.

GAIIC2024: A2DINOv3 reaches 64.00% validation mAP and 43.05% test mAP on GAIIC2024, surpassing AFF-Net by 5.52% test mAP.It also surpasses AFF-Net by 5.03% in mAP50 and 7.18% in mAP75.
GAIIC2024: A2DINOv3 improves over the strongest IR-only baseline by 5.93% test mAP on GAIIC2024 while incorporating complementary RGB cues.SCP selectively incorporates informative RGB cues while preserving reliable infrared representations.
Knowledge Retention: A2DINOv3 preserves 67.27% of its validation performance on the GAIIC2024 test set while achieving the highest multimodal test mAP.The test set contains more distant views, smaller targets, and larger distribution variations than the validation set.
Cross-Domain Generalization: A2DINOv3 achieves 61.78%, 46.80%, and 71.56% mAP on M3FD, FLIR, and LLVIP, respectively, demonstrating consistent generalization across domains.On FLIR, it exceeds the 46.57% mAP of IR-only RF-DETR, while AFF-Net and ICAFusion score 41.84% and 41.40%.
Ablation Study: With the backbone frozen, SCP raises mAP from 38.10% to 40.27% compared with dense cross-attention.The 2.17-point gain is attributed to constraining exchanged residuals to a low-dimensional subspace under asymmetric modality quality.
Ablation Study: Zero-initialized SCP reaches 43.05% mAP, improving over random-initialized SCP at 42.43% by gradually activating cross-modal communication.Early gradient dynamics show approximately 10^-4 magnitude with zero initialization before increasing progressively.
Qualitative Analysis: Under severe low-light conditions, A2DINOv3 recovers difficult targets missed by M2D-LIF, while bounded communication suppresses cross-modal interference.Feature responses are more target-focused, with A2DINOv3 producing a correct prediction at 0.82 confidence versus a 0.74-confidence false positive from dense cross-attention.

Conclusion

The paper frames multimodal adaptation as regulated collaboration between heterogeneous experts rather than unrestricted feature exchange. A2DINOv3 uses SCP and zero-initialized pathways to support gradual coordination while addressing fusion degradation and pretrained-representation distortion.

Conclusion: A2DINOv3 addresses fusion degradation and pretrained-representation distortion by modeling RGB and IR as complementary experts with regulated interaction.Its collaboration protocol controls when, what, and how much information is communicated.
Conclusion: Zero-initialized pathways enable a gradual transition from independent modality learning to coordinated collaboration.The design targets unreliable cross-modal signals and abrupt joint optimization.

Appendix

The appendix provides implementation details, method details, full experimental results, supplementary theoretical analysis, and in-depth discussions.

Implementation Details: The appendix describes compared methods and datasets used in the main paper.
Method Details: It presents the formal background of Social Interdependence Theory and the complete A2DINOv3 algorithmic pipeline.
Full Experimental Results: It reports full experimental results with detailed performance analysis.

Implementation Details

The implementation evaluates A2DINOv3 on RGB-T benchmarks using shared architectural and optimization settings, with dataset-specific training schedules and augmentation procedures.

Model Configuration: A2DINOv3 uses dual DINOv3 ViT-S/16+ backbones and a 6-layer DEIM decoder with 300 object queries.The backbones use interaction indexes [5, 8, 11].
Optimization: Training uses AdamW, mixed precision, EMA, gradient clipping of 0.1, and a confidence threshold of 0.25 under a unified COCO-style protocol.The optimizer uses β = (0.9, 0.999) and weight decay 1.25×10^-4.
Dataset-Specific Settings: GAIIC2024, FLIR, and LLVIP receive dataset-specific epoch, batch-size, learning-rate, and augmentation settings.GAIIC2024 training uses 58 epochs and 640×640 images with Mosaic, mixup, photometric distortion, and CopyBlend.

Compared Methods

The paper compares object detectors spanning proposal-based, query-based, real-time, lightweight transformer, and multimodal fusion designs.

Faster R-CNN unifies a Region Proposal Network with shared full-image convolutional features for efficient region proposals.
DDQ-DETR filters similar proposals through Dense Distinct Query selection to mitigate optimization difficulties and accelerate convergence.
YOLO26-X uses dual-head NMS-free inference and removes Distribution Focal Loss for a lighter regression head.
RF-DETR uses weight-sharing neural architecture search to identify target-domain accuracy-latency Pareto curves without retraining.
CSAA is described as a lightweight multimodal fusion module using channel-related processing.

Datasets

The evaluation uses four RGB-T benchmarks spanning aerial detection, autonomous driving, and low-light vision, with differing environmental complexity and sensor alignment.

The four benchmarks represent different levels of environmental complexity and sensor heterogeneity.
GAIIC2024 is an RGB-TIR aerial benchmark from unmanned aerial vehicles, covering challenging lighting, frequent misalignment, and five vehicle categories.
FLIR contains 5,142 aligned visible-infrared paired images from day and night scenes across person, car, and bicycle categories.
FLIR also contains misaligned pairs requiring careful filtering for training, supporting robustness evaluation under varying illumination.
LLVIP contains 33,672 images, or 16,836 visible-infrared pairs, predominantly captured in extremely dark scenes with strict temporal and spatial alignment.

Theoretical Background

The paper frames multimodal collaboration through social interdependence, then implements bounded RGB-IR communication and evaluates it across datasets, localization criteria, and distribution shifts.

Social Interdependence Theory: Social interdependence theory classifies relationships through conditional goal-attainment probabilities for positive, negative, or absent dependence.
Social Interdependence Theory: Positive interdependence means each entity’s success facilitates the other’s, whereas negative interdependence means each entity’s success hinders the other’s.
Social Interdependence Theory: No interdependence models independent execution without interference or synergistic gains, motivating structured boundary conditions for collaboration.
A2DINOv3 Pipeline: A2DINOv3 shares DINOv3 initialization across RGB and infrared streams, uses zero-initialized SCP projections, and exchanges compressed residuals at interaction layers.
A2DINOv3 Pipeline: Refined modality features are mean-fused and passed to the DETR decoder for bounding-box and classification prediction before joint optimization.
Evaluation Context: On M3FD, A2DINOv3 reaches 61.78% mAP and 66.47% mAP75, while GAIIC2024 test distributions contain smaller targets and larger domain variation than validation.