PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Optimal Transport-Based Experts Collaboration Towards Modality-Aware Infrared-Visible Object Detection

Yue Zhao, Hua Yu, Yukun Zhao, Yuzhi Zhang, Maoguo Gong, Xin Mei, Zhuping Hu, Yanchi Li, A. K. Qin

arXiv:2609.03516v1cs.CV

TL;DR

IVOD methods commonly assume both sensor modalities are available, while spectral differences make semantic correspondence across modalities difficult to estimate. FlexibleFusion adapts expert pathways to modality availability and uses residual self-paced optimal transport; it is reported to achieve state-of-the-art performance and robustness across complete- and missing-modality conditions.

  • Problem

    Existing IVOD methods assume complete modalities, and reliably estimating semantic correlation across heterogeneous modalities under spectral distribution shift remains challenging.

    Source: §I · p. 1

  • Method

    FlexibleFusion combines availability-aware MAEC expert pathways with RSPEOT, which uses residual-driven self-paced refinement to align heterogeneous features while alleviating standard EOT's optimization burden.

    Source: §I · p. 2, C. Residual Self-Paced Entropic… · p. 5

  • Results

    FlexibleFusion is reported to achieve state-of-the-art accuracy and robustness on standard IVOD benchmarks in both complete- and missing-modality conditions.

    Source: §I · p. 2

  • Takeaways & Limitations

    The conclusion presents FlexibleFusion as a unified adaptive framework for robust and semantically consistent integration across complete- and missing-modality IVOD settings.

    Source: §V · p. 9

Abstract

from arXiv · show

Infrared-visible object detection (IVOD) integrates complementary evidence from visible and infrared sensors for reliable perception in challenging scenes. In practice, sensors may fail or drop frames, leaving one modality unavailable or intermittent. Existing methods for IVOD assume both modalities are always present, and fixed fusion collapses when one stream is missing. Furthermore, it remains a critical challenge to reliably estimate semantic correlation across heterogeneous modalities, especially under spectral distribution discrepancy. We present FlexibleFusion, a unified and adaptive method that flexibly allocates integration pathways and fusion strength, operating seamlessly across complete and missing-modality regimes. At its core, the Modality-Aware Experts Collaboration (MAEC) mechanism selectively activates and aggregates cross-modal or intra-modal expert pathways. It allows cross-modal fusion when full modalities are available and falls back to self-fusion under missing conditions. Additionally, we design Residual Self-Paced Entropic Optimal Transport (RSPEOT) to align heterogeneous feature distributions from a transport perspective. Instead of relying on the fixed sparsity coefficient in standard entropic optimal transport (EOT), RSPEOT introduces a residual-driven self-paced update that prioritizes reliable matches and progressively refines harder ones. This design alleviates the additional optimization burden of standard EOT while preserving reliable semantic alignment. Comprehensive experiments under complete and missing-modality protocols show consistent performance across arbitrary modality configurations. Code will be released upon publication.

I. INTRODUCTION

The introduction identifies two IVOD challenges: fixed fusion under modality absence and unreliable semantic correspondence across spectrally different modalities. FlexibleFusion addresses both with availability-aware expert pathways and residual self-paced optimal transport, and is reported to achieve state-of-the-art accuracy and robustness in complete- and missing-modality conditions.

  • IVOD is presented as a capability for safety-critical and all-weather applications including autonomous driving, surveillance, and night-time scene understanding –.Source: §I · p. 1
  • Existing IVOD methods largely assume both modalities are available, while fixed aggregation can incorporate invalid information when a sensor modality is missing.The stated goal is to keep missing-modality accuracy at least comparable to single-modality detectors.Source: §I · pp. 1–2
  • Heterogeneous modalities share semantic content but have different spectral distributions, and static weighting or attention struggles to handle this shift.Source: §I · p. 2
  • The method combines MAEC, which selects cross-modal or intra-modal pathways according to modality availability, with RSPEOT for semantic alignment under spectral shift.RSPEOT uses residual-driven dynamic optimization to alleviate standard EOT's iterative optimization burden while preserving reliable alignment.Source: §I · p. 2
  • FlexibleFusion is reported to achieve state-of-the-art accuracy and robustness on standard IVOD benchmarks in both complete- and missing-modality conditions.Source: §I · p. 2

II. RELATED WORKS

Prior IVOD work explores fixed and adaptive fusion, while missing-modality strategies are more established in other domains than in IVOD. FlexibleFusion combines availability-conditioned expert pathways with transport-based feature alignment for complete- and missing-modality settings.

A. Complete-Modality Object Detection

  • Earlier IVOD approaches use feature- or decision-level fusion and increasingly adaptive attention or transformer interactions, while FlexibleFusion configures inter- and intra-modal information for modality unavailability.Source: A. Complete-Modality Object Detection · p. 2

B. Learning under Missing Modality

  • Missing-modality methods in other domains use hallucination, imputation, robustness training, or representation learning, but unified availability-conditioned fusion remains underexplored in IVOD.Source: B. Learning under Missing Modality · p. 2

A. Overview of FlexibleFusion

  • FlexibleFusion uses a dual-input, dual-output design with shared pathway experts and modality-aware routing; each output selects its top two experts and fuses them with RSPEOT.DINO is the detector, and modality-aware dropout simulates missing-modality scenarios during training.Source: A. Overview of FlexibleFusion · p. 3, Figure 2 · p. 3

B. Modality-Aware Experts Collaboration Mechanism

MAEC adapts expert routing to modality availability, using cross-modal fusion when both streams are present and intra-modal self-fusion when one is missing. RSPEOT aligns selected expert features with normalized Gram geometry and residual-guided self-paced sparsity updates.

B. Modality-Aware Experts Collaboration Mechanism

  • MAEC switches from cross-modal fusion to intra-modal self-fusion when a modality is absent.This avoids fixed pathways incorporating invalid information from a missing stream.Source: B. Modality-Aware Experts… · pp. 3–4
  • Each of two output branches scores experts from both modality perspectives, then fuses its top two selected experts.The top-two choice supports directed pairwise fusion between two inputs.Source: B. Modality-Aware Experts… · pp. 3–4

C. Residual Self-Paced Entropic Optimal Transport

  • RSPEOT uses normalized Gram geometry on the channel manifold, rather than rigid Euclidean cost, to quantify structural divergence while preserving content affinity.The formulation encodes intra-channel dependencies through normalized Gram similarity.Source: C. Residual Self-Paced Entropic… · p. 4
  • Residual-driven self-paced regularization adjusts sparsity to optimize easier matches first, then progressively harder ones, alleviating standard EOT's iterative burden.Pairs closer to convergence receive smaller sparsity coefficients and are treated as easier to optimize.Source: C. Residual Self-Paced Entropic… · pp. 4–5

A. Dataset Description and Evaluation

FlexibleFusion is evaluated on FLIR-Aligned and LLVIP, two IVOD benchmarks with paired infrared and visible imagery. Evaluation uses mAP at three IoU-based settings.

A. Dataset Description and Evaluation

  • The evaluation reports FlexibleFusion's effectiveness and state-of-the-art performance on both FLIR-Aligned and LLVIP.Source: A. Dataset Description and Evaluation · p. 6
  • FLIR-Aligned contains approximately 5,142 paired images from traffic scenarios, while LLVIP contains 15,488 pairs for low-light, single-class pedestrian detection.FLIR-Aligned includes person, bicycle, and car categories; LLVIP was collected in real-world surveillance environments with illumination variations.Source: A. Dataset Description and Evaluation · p. 6
  • Performance is measured with mAP@50, mAP@75, and mAP@:95, covering IoU thresholds of 0.5, 0.75, and ten thresholds from 0.5 to 0.95.These metrics measure detection performance at different levels of localization precision.Source: A. Dataset Description and Evaluation · p. 6

B. Experimental Configuration

Experiments use a SwinLarge backbone initialized with COCO-pretrained weights and four experts on each dataset. The supplied configuration also specifies sparse-coefficient settings and a single-GPU setup.

B. Experimental Configuration

  • SwinLarge is the backbone, with modality-specific backbones and the detection head initialized from COCO-pretrained weights.The encoder-decoder detection head has six encoder and six decoder layers, eight attention heads, four sampling points, and 300 queries.Source: B. Experimental Configuration · p. 6
  • The included figure shows visualization examples on FLIR-Aligned, while Table II is scoped to complete-modality comparisons on LLVIP.Source: Figure 4 · p. 7, Table II · p. 7
  • The model uses four experts on both datasets, with ϵini set to 0.1 and ϵ0 set to 0.008 on FLIR-Aligned and 0.01 on LLVIP.The passage says grid search and cross-validation were used to select these hyperparameters.Source: B. Experimental Configuration · p. 7
  • Experiments run on a single NVIDIA RTX A6000 GPU with 48GB memory.Source: B. Experimental Configuration · p. 7

C. Comparison under Complete-Modality

The complete-modality comparison covers classical detection architectures and recent multimodal fusion methods, spanning several detector families.

C. Comparison under Complete-Modality

1) Performance on FLIR-Aligned Dataset:

On FLIR-Aligned under complete modalities, FlexibleFusion achieves the best overall performance among the compared baselines across all three reported mAP metrics.

1) Performance on FLIR-Aligned Dataset:

2) Performance on LLVIP Dataset:

On LLVIP, FlexibleFusion reports the highest mAP@75 and competitive mAP@50 and mAP@:95 results.

2) Performance on LLVIP Dataset:

D. Comparison under Missing-Modality

FlexibleFusion maintains strong reported performance as modalities go missing, across the evaluated rates and datasets.

D. Comparison under Missing-Modality

  • Across missing rates on FLIR-Aligned and LLVIP, FlexibleFusion consistently achieves the best performance; on LLVIP, mAP@50 falls from 98.4 to 95.6 as pd rises from 0 to 1.0.The comparison varies pd across 0, 0.2, 0.5, 0.8, and 1.0 while fixing pm = 0.5.Source: Table III · p. 8, D. Comparison under Missing-Modality · p. 8
  • With pm = 0.5, FlexibleFusion maintains 76.0 mAP@50 on FLIR-Aligned under complete modality absence (pd = 1.0), demonstrating robustness to missing modalities.Source: D. Comparison under Missing-Modality · p. 8
  • FlexibleFusion preserves strong robustness as pm increases, never falling strongly below single-modality baselines.Supplementary tests vary pm over 0, 0.5, and 1 with pd fixed at 1 during inference.Source: D. Comparison under Missing-Modality · pp. 8–9

E. Ablation Analysis

FlexibleFusion is reported to perform strongly across complete- and missing-modality benchmarks; its ablations and mechanism analyses describe component contributions and routing behavior.

E. Ablation Analysis

  • Adding MAEC raises FLIR-Aligned mAP@75 from 47.2 to 52.5, and adding RSPEOT achieves the best overall mAP@75 and mAP@:95.RSPEOT maintains solution accuracy with fewer optimization iterations.Source: E. Ablation Analysis · p. 9
  • Under missing-modality conditions, the dual-branch DINO baseline misses difficult objects, while FlexibleFusion performs comparably in complete and missing settings.The examples are from FLIR-Aligned with pd = 1 and pm = 0.5.Source: E. Ablation Analysis · p. 9

F. Mechanism Analysis of MAEC and RSPEOT

V. CONCLUSION

  • Extensive public-benchmark experiments show state-of-the-art performance and superior robustness to modality absence.The conclusion identifies extending to more modality combinations and relaxing the alignment assumption as future work.Source: §V · p. 9
Loading 2609.03516v1…