Source-linked AI summary
SuppreSensing: Expert-Guided Feature Recalibration and Discrepancy Augmentation for Multimodal Object Detection
Xin Wu, Zhenyu Gao, Qiankun Zhang, Shaoyong Guo
TL;DR
Remote-sensing multimodal detection must handle semantic heterogeneity, modality-specific noise, and the loss of discriminative cues caused by strong alignment. SuppreSensing uses expert-guided recalibration, discrepancy augmentation, and customized purification to selectively combine shared and modality-specific information. It achieves state-of-the-art performance across reported datasets and shows robustness and generalization across diverse conditions.
Problem
Remote-sensing multimodal detection must address semantic heterogeneity, modality-specific noise, and strong-alignment effects that suppress useful modality-specific cues.
Method
SuppreSensing reformulates fusion as selective collaboration using input-adaptive EMFR, bidirectional discrepancy augmentation, and iterative ECFP purification.
Results
SuppreSensing achieves state-of-the-art detection performance on DroneVehicle, VEDAI, LLVIP, and FLIR, with robustness across diverse illumination and environmental conditions.
Takeaways & Limitations
The reported evaluations support SuppreSensing as a robust multimodal detection approach that preserves discriminative modality-specific cues while mitigating destructive semantic homogenization.
Abstract
from arXiv · showhide
Multimodal object detection in remote sensing faces challenges due to semantic heterogeneity and modality-specific noise interference. To this end, we propose SuppreSensing, which reformulates multimodal fusion as a selective collaboration process that jointly models shared information and modality-specific cues. SuppreSensing first designs an Expert-driven Multimodal Feature Recalibration (EMFR) module, which reformulates shared-consensus extraction as an input-adaptive multi-expert selection process to alleviate the symmetry trap in multimodal fusion. Complementing this, a modality-specific attribute augmentation strategy is employed to enhance specific modality features by modeling bidirectional discrepancy patterns, mitigating cross-modal heterogeneity. Furthermore, we propose an Expert-driven Customized Feature Purification (ECFP) module based on a "specialized inspection-comprehensive analysis-diagnostic update" physical examination paradigm to iteratively filter redundancies and reinforce task-relevant semantics. Extensive experiments on the DroneVehicle and VEDAI datasets demonstrate that SuppreSensing achieves state-of-the-art detection performance. Cross-domain evaluations on natural scene datasets (FLIR and LLVIP) further validate its superior robustness and generalization capability across diverse environmental conditions.
1 Introduction
Remote-sensing multimodal detection must combine complementary visible and infrared information despite modality-specific degradation and semantic heterogeneity. SuppreSensing addresses this through selective collaboration, expert-driven recalibration, discrepancy enhancement, and progressive purification.
- Motivation: Visible imagery supplies spatial and texture detail, whereas infrared imagery better withstands illumination interference but provides lower resolution and weaker fine-grained semantics.Their complementarity motivates visible-infrared fusion for reliable perception in degraded conditions.
- Motivation: Strong-alignment fusion can fall into a symmetry trap that suppresses discriminative thermal signatures and fine-grained visible textures.Treating modality-specific cues as unreliable interference can weaken or eliminate useful evidence.
- Motivation: Effective multimodal detection therefore requires selective cross-modal collaboration that captures shared information while retaining modality-specific evidence under asymmetric degradation.Balanced routing can be unsuitable when one modality is degraded and noise propagates across modalities.
- Approach: SuppreSensing uses EMFR to adaptively select experts for consensus extraction, augment bidirectional modality discrepancies, and preserve fine-grained modality-specific differences.These components target noise suppression and cross-modal heterogeneity without relying on full semantic homogenization.
- Approach: ECFP progressively filters redundant information and strengthens task-relevant semantics through iterative expert-guided refinement.The module follows a specialized inspection-comprehensive analysis-diagnostic update paradigm.
- Results: SuppreSensing achieves state-of-the-art performance on DroneVehicle, VEDAI, LLVIP, and FLIR, with robustness under diverse illumination and environmental conditions.The reported evaluation spans remote-sensing and natural-scene datasets.
2 Related Work
Related work centers on feature-level fusion and increasingly dynamic expert mechanisms for heterogeneous multimodal data. SuppreSensing is positioned within this trajectory while addressing strong-alignment limitations through an EMFR–ECFP pipeline.
- Fusion paradigms: Pixel-level fusion preserves low-level cues but is vulnerable to spatial misalignment and noise propagation, while decision-level fusion aggregates independent modality-specific inferences.These represent two established multimodal fusion paradigms.
- Feature-level fusion: Feature-level fusion has become dominant because it offers a favorable trade-off between representation capacity and robustness to interference.Recent methods use attention-based calibration and spatial-semantic enhancement to improve fused representations.
- Feature-level fusion: Many feature-level methods remain grounded in strong alignment, forcing heterogeneous modalities toward a shared representation.This limitation motivates alternatives that preserve useful modality-specific information.
- Mixture of Experts: Mixture-of-Experts enables input-dependent sparse activation, dynamic computation allocation, and adaptive modeling for complex heterogeneous data.Its core mechanism combines multiple expert subnetworks with a gating router.
- Mixture of Experts: Prior multimodal MoE methods dynamically allocate experts using modal features, environmental context, or scene awareness for specialized cross-modal synthesis.Examples include dynamic aggregation, gating, and infrared-visible fusion under extreme illumination.
- SuppreSensing: SuppreSensing organizes its method around EMFR for recalibration and ECFP for customized feature purification.The pipeline applies these modules sequentially before detection outputs.
3 Proposed Method
SuppreSensing performs selective multimodal collaboration through expert-driven recalibration, discrepancy enhancement, and customized feature purification. Its EMFR and ECFP modules adapt feature processing across modalities and spatial locations to preserve useful cues while reducing interference.
- EMFR: EMFR uses expert-driven dynamic channel recalibration to enhance unimodal features through cross-modal recalibration and modality-specific attribute augmentation.The module is designed for modality conflicts such as visible overexposure and infrared low resolution.
- Cross-modal Feature Recalibration: A multimodal cue-driven router dynamically activates channel-subspace experts, replacing fixed consensus extraction with input-adaptive selection based on scene context.The resulting consensus is injected into visual and infrared branches through cross-gating.
- Modality-specific Attribute Augmentation: EMFR computes bidirectional modality discrepancies and nonlinearly refines them to preserve informative differences while suppressing irrelevant disturbances.The discrepancy pathway complements shared-consensus extraction before feature aggregation.
- ECFP: ECFP follows a specialized inspection-comprehensive analysis-diagnostic update paradigm using five customized pathways for spatially adaptive multimodal refinement.The pathways include evolved primary modality, global static prior, spatial customized prescription, auxiliary modal mapping, and stage-specific projection.
- ECFP: The five pathways are concatenated into a comprehensive report that residually updates primary features, iteratively suppressing redundancies and strengthening task-relevant semantics.The ECFP process is divided into symptom diagnosis and customized prescription, with spatially adaptive processing across heterogeneous regions.
- Customized Feature Paths: Auxiliary modal mapping projects heterogeneous auxiliary features into a latent space, while stage-specific projection adapts primary features across network depths.These paths calibrate representational deviation and semantic consistency across modalities and hierarchical stages.
4 Experiments
Experiments across aerial and natural-scene benchmarks evaluate SuppreSensing’s detection accuracy, robustness, generalization, and module contributions. The method achieves strong benchmark performance and benefits from combining recalibration, discrepancy enhancement, and purification.
- Experimental Setup: 28,439 Vis-Inf pairs in DroneVehicle and 1,210 in VEDAI provide aerial benchmarks with illumination variation, complex backgrounds, and small objects.LLVIP and FLIR extend evaluation to low-light surveillance and day-night autonomous-driving scenes.
- Comparison with SOTA Methods: SuppreSensing achieves the best overall mAP on both VEDAI and DroneVehicle benchmarks.The comparison evaluates the method against state-of-the-art multimodal object detection methods.
- Comparison with SOTA Methods: 0.840 mAP@0.5 is achieved on VEDAI, exceeding ADMPF’s 0.816 by 0.024.The method also reaches 0.933 AP for trucks and 0.900 AP for camping cars, compared with 0.744 and 0.847 for ADMPF.
- Comparison with SOTA Methods: 0.812 mAP@0.5 is achieved on DroneVehicle, exceeding MGMF’s 0.803 and NOC-YOLO’s 0.795.For trucks, SuppreSensing reaches 0.823 AP50, a 12.2% gain over MGMF’s 0.701.
- Evaluation Across Different Domains: 0.977 mAP@0.5 on LLVIP and 0.760 mAP@0.5 on FLIR exceed YOLO-Adaptor and BU-LTT by 1.2% and 2.8%, respectively.These results evaluate generalization beyond aerial remote sensing under low-light, occlusion, clutter, and varying modality quality.
- Ablation Study: 0.812 and 0.840 mAP@0.5 are obtained on DroneVehicle and VEDAI after adding ECFP to EMFR, while mAP@0.5:0.95 reaches 0.621 and 0.554.The full configuration improves over EMFR alone, supporting complementary refinement from ECFP.
- Ablation Study: 0.068 mAP@0.5 is the EMFR gain over baseline on VEDAI, compared with 0.028 on DroneVehicle.Adding ECFP further raises VEDAI mAP@0.5 from 0.831 to 0.840.
- Ablation Study: 0.9 percentage points is the largest drop when an ECFP pathway is removed, occurring after removing EPM.The complete ECFP configuration achieves 0.812 mAP@0.5 on DroneVehicle.
5 Conclusion
SuppreSensing addresses multimodal remote-sensing detection by combining adaptive recalibration, modality-specific discrepancy modeling, and iterative purification. Experiments report strong aerial detection performance and robustness across natural-scene domains.
- Conclusion: SuppreSensing integrates input-adaptive recalibration, modality-specific discrepancy modeling, and iterative purification for multimodal object detection.The method is designed to suppress modality-specific interference while preserving discriminative modality-specific cues.
- Conclusion: Experiments on DroneVehicle and VEDAI show strong detection performance, while FLIR and LLVIP evaluations confirm robustness and generalization across diverse conditions.The conclusion covers both aerial and cross-domain natural-scene evaluations.