Source-linked AI summary
FoCal: Frequency-Oriented Cross-Modal Interaction and Spectral Calibration for Aerial Visible-Infrared Object Detection
Ben Liang, Chao Sui, Junqi Bai, Yuan Liu, Chunlai Li, Xiubao Sui, Qian Chen
TL;DR
Aerial RGB–IR detection needs to exploit complementary modality information, but existing approaches leave frequency-dependent cross-modal interaction patterns underexplored. FoCal uses FADC and DGSM to calibrate interaction and spectral imbalance, and reports state-of-the-art results across three benchmarks alongside high throughput and a compact model.
Problem
Existing multimodal detectors emphasize spatial interaction or frequency-specific enhancement, while how cross-modal components at different frequencies should interact remains underexplored.
Source: §I · pp. 1–2Method
FoCal combines FADC, which forms low-frequency consensus and selectively exchanges high-frequency details to calibrate original streams, with DGSM, which uses confidence-weighted spectral discrepancy to modulate the joint spectrum.
Source: §I · p. 2Results
FoCal is reported to achieve state-of-the-art performance across DroneVehicle, ESCVehicle, and ATR-UMOD, with mAP50 values of 83.5%, 54.8%, and 64.6%, respectively.
Source: §I · p. 2Takeaways & Limitations
Across aerial RGB–IR benchmarks, FoCal's strong detection performance and compact architecture provide a favorable basis for practical multimodal detection.
Source: §V · pp. 10–11
Abstract
from arXiv · showhide
In aerial RGB--IR object detection, effectively exploiting complementary information across modalities is critical for robust perception under complex illumination and environmental conditions. Existing multimodal detectors mainly focus on spatial-domain interaction or frequency-specific feature enhancement, while the cross-modal interaction patterns of different frequency components remain insufficiently explored. Moreover, spectral discrepancy itself may contain both useful complementary cues and unreliable modality-specific responses, making indiscriminate frequency fusion suboptimal. To address these issues, we propose FoCal, a frequency-oriented framework for aerial RGB--IR object detection. First, a Frequency-Aware Dual-Domain Calibration (FADC) module is developed to explicitly model frequency-dependent cross-modal interaction. Low-frequency components are collaboratively consolidated into a shared structural consensus, whereas high-frequency components preserve modality-specific information through selective cross-modal exchange. The resulting frequency-aware cues are further transferred to the original feature domain to regulate cross-modal calibration. Second, we introduce a Discrepancy-Guided Spectral Modulation (DGSM) module, which characterizes cross-modal spectral imbalance using confidence-weighted relative amplitude discrepancy and transforms it into a bounded signed gate for adaptive enhancement, preservation, or attenuation of the joint multimodal spectrum. Extensive experiments on DroneVehicle, ESCVehicle, and ATR-UMOD demonstrate the effectiveness of FoCal, yielding $\mathrm{mAP}_{50}$ values of 83.5\%, 54.8\%, and 64.6\%, respectively. Meanwhile, with only 3.0M parameters, FoCal achieves 113.6 FPS while preserving leading detection accuracy, highlighting a favorable accuracy--efficiency trade-off. Code is available at {https://github.com/universeliang/FoCal.
I. INTRODUCTION
FoCal addresses the limited modeling of frequency-dependent cross-modal interaction in aerial RGB–IR detection with separate mechanisms for structural consensus, selective detail exchange, and spectral modulation. It reports strong results on three benchmarks with a compact model.
I. INTRODUCTION
- Existing multimodal detectors focus mainly on spatial interaction or post-decomposition frequency processing, leaving the appropriate cross-modal interaction pattern for each frequency band underexplored.RGB is illumination-sensitive, whereas IR is less sensitive to illumination and can provide complementary contours and structural cues.Source: §I · pp. 1–2
- 83.5%, 54.8%, and 64.6% mAP50 on DroneVehicle, ESCVehicle, and ATR-UMOD, respectively, with FoCal reported as state of the art across all three benchmarks.FoCal is also reported at 113.6 FPS with 3.0M parameters and 8.4G FLOPs.Source: §I · p. 2
- FADC forms low-frequency structural consensus, preserves high-frequency modality individuality through selective exchange, and transfers frequency-aware cues to calibrate original feature streams.This distinguishes both how frequency components interact and where counterpart information is introduced.Source: §I · p. 2
- DGSM uses confidence-weighted spectral discrepancy as a signed control signal to adaptively enhance, preserve, or attenuate the joint multimodal spectrum.The discrepancy is used to regulate the spectrum rather than being treated directly as informative content.Source: §I · p. 2
A. Multimodal Object Detection
- Prior RGB–IR detectors use illumination estimation, cross-modal attention, and Mamba-based fusion, while frequency disentanglement remains a route for more refined aerial fusion.The cited work identifies small-object detection in complex aerial scenarios as a particular motivation for frequency-domain disentanglement.Source: A. Multimodal Object Detection · pp. 2–3
B. Frequency in computer vision
Frequency-domain vision methods separate low- and high-frequency information, but multimodal detection has not fully explored frequency-specific cross-modal interaction or used frequency cues to regulate original feature streams. FoCal's FADC applies distinct interaction patterns and calibration cues.
B. Frequency in computer vision
- Existing multimodal methods often assume equal contributions from modalities at each frequency and use frequency-enhanced features as final outputs, leaving cross-modal interaction and spectral discrepancy underexplored.FFT supports global frequency interaction, while DWT supports local multi-scale decomposition; prior methods apply these tools in different vision and fusion settings.Source: B. Frequency in computer vision · p. 3
A. Frequency-Aware Dual-Domain Calibration
- FADC assigns low frequencies to shared structural consensus and high frequencies to recipient-conditioned selective exchange, then uses frequency-conditioned representations to calibrate original streams.This responds to the risk that collapsing high-frequency details can suppress modality-specific cues or propagate sensor-specific noise.Source: A. Frequency-Aware Dual-Domain… · p. 3
- FADC decomposes RGB and IR feature maps with Haar-based DWT into one low-frequency approximation and three directional high-frequency components.The directional responses represent horizontal, vertical, and diagonal details.Source: A. Frequency-Aware Dual-Domain… · p. 3
III. METHODS
- The depicted FoCal pipeline comprises a Backbone for feature extraction, a PAN for multi-scale feature fusion, and a Detection Head.Source: Figure 3 · p. 4
1) Frequency-Dependent Interaction Topology:
FADC gives low- and high-frequency components different cross-modal interaction topologies: the former form a shared structural anchor, while the latter retain modality-specific streams and exchange selected details.
1) Frequency-Dependent Interaction Topology:
- The distinct topologies follow frequency-specific information characteristics: consensus for corresponding coarse structures and selective exchange for details that may include modality-dependent noise.The resulting features share a low-frequency structural anchor while retaining individually calibrated high-frequency details.Source: 1) Frequency-Dependent Interaction… · pp. 4–5
- Low-frequency RGB and IR responses are adaptively weighted into one shared consensus rather than enhanced independently after aggregation.The weights are normalized to sum to one, allowing local modality contributions to vary with observation quality.Source: 1) Frequency-Dependent Interaction… · p. 4
- High-frequency components remain modality-specific and use recipient-conditioned masks for selective cross-modal exchange instead of direct aggregation.Each modality retains intrinsic responses while selectively absorbing complementary details, avoiding a single collapsed high-frequency representation.Source: 1) Frequency-Dependent Interaction… · pp. 4–5
2) Frequency-Guided Reliability Calibration:
Frequency-conditioned features are auxiliary cues for estimating reliable cross-modal transfer, not final fused representations. Their reliability estimates combine with a bimodal prior to calibrate injections into the original streams.
2) Frequency-Guided Reliability Calibration:
- Frequency-conditioned features estimate where and what information can be reliably transferred between the original RGB and IR streams rather than serving as the final fused representation.The frequency interaction determines transfer reliability without replacing the original modality representation.Source: 2) Frequency-Guided Reliability… · p. 5
- Spatial and channel reliability masks derived from frequency-conditioned features are combined with a joint bimodal contextual prior to obtain modality-specific calibration masks.The spatial weighting captures position-sensitive reliability, while channel weighting models channel dependencies from pooled bimodal statistics.Source: 2) Frequency-Guided Reliability… · p. 5
- The calibration masks selectively inject complementary information into each original modality stream, followed by reliability-conditioned reference and residual calibration.Residual calibration adjusts augmented responses according to the learned reliability reference.Source: 2) Frequency-Guided Reliability… · p. 5
- FADC uses frequency-conditioned reliability cues to regulate cross-modal transfer while retaining complementary information and avoiding premature modality collapse.Source: 2) Frequency-Guided Reliability… · p. 5
B. Discrepancy-Guided Spectral Modulation
DGSM uses confidence-weighted cross-modal spectral discrepancy as a control signal for regulating a separately constructed joint spectrum. Its modulation is bounded and can enhance, preserve, or attenuate spectral responses.
B. Discrepancy-Guided Spectral Modulation
- DGSM uses confidence-weighted relative spectral discrepancy to guide adaptive enhancement, preservation, or attenuation of an independently constructed joint spectrum.The discrepancy branch controls the joint branch rather than supplying content for fusion, reducing the risk of indiscriminately amplifying modality-specific noise.Source: B. Discrepancy-Guided Spectral… · pp. 5–6
- RGB and IR features are projected into a common comparison space, jointly normalized during training, then transformed with FFT for spectral comparison.The joint normalization is intended to reduce modality-dependent feature-scale bias.Source: B. Discrepancy-Guided Spectral… · p. 5
- The scale-normalized discrepancy measures relative spectral imbalance, while energy confidence lowers the weight of discrepancies at nearly inactive locations.Large relative differences at nearly inactive responses may be unreliable; confidence retains discrepancies supported by meaningful spectral responses.Source: B. Discrepancy-Guided Spectral… · pp. 5–6
- DGSM converts discrepancy into a learned signed gate rather than injecting it directly, letting the detection objective determine its effect on representation learning.The gate is used for bounded residual modulation; γ sets the maximum modulation magnitude and is empirically set to 0.5.Source: B. Discrepancy-Guided Spectral… · p. 6
- After modulation, DGSM recombines the complex spectrum, transforms it back to the spatial domain, and projects the reconstructed feature into a residual connection.The original joint spectrum remains the stable information basis, with discrepancy controlling an additional residual adjustment.Source: B. Discrepancy-Guided Spectral… · p. 6
A. Datasets and Evaluation Metrics
FoCal is evaluated on three drone-based visible–infrared detection datasets with varied vehicle categories and acquisition conditions. The section defines mAP50 and mAP as the evaluation metrics.
A. Datasets and Evaluation Metrics
- DroneVehicle contains 953,087 vehicle instances across 28,439 aerial RGB–IR pairs, spanning five vehicle categories and daytime and nighttime conditions.The official split uses 17,990 pairs for training, 1,469 for validation, and 8,980 for testing.Source: A. Datasets and Evaluation Metrics · p. 6
- ESCVehicle contains 10,727 aligned RGB–IR pairs, 369,714 instances, and seven vehicle categories across varied scenes and weather.The official split uses 6,983 pairs for training, 507 for validation, and 3,237 for testing; its diverse scenes support evaluation under challenging environmental conditions.Source: A. Datasets and Evaluation Metrics · pp. 6–7
- ATR-UMOD comprises 13,353 aligned RGB–IR pairs across 11 object categories and varied altitude, angle, time, weather, illumination, and scenario conditions.The official split uses 11,850 pairs for training and 1,503 for testing; mAP50 is measured at IoU 0.50, while mAP averages IoU thresholds from 0.50 to 0.95 in 0.05 steps.Source: A. Datasets and Evaluation Metrics · p. 7
B. Implementation Details
Experiments use a dual-stream extension of the Ultralytics single-modal framework, trained from scratch with a shared optimization setup across datasets.
B. Implementation Details
- Experiments run on one NVIDIA RTX 4090 with 24 GB memory, using CUDA 12.1 and PyTorch 2.5.1.The implementation extends the Ultralytics single-modal framework into a dual-stream multimodal architecture.Source: B. Implementation Details · p. 7
- All models are trained from scratch with SGD, momentum 0.937, weight decay 0.0005, and initial learning rate 0.01.No pretrained weights are used, to ensure a fair evaluation of the architectural designs.Source: B. Implementation Details · p. 7
- DroneVehicle, ESCVehicle, and ATR-UMOD training uses batch size 16 for 150 epochs.Unless otherwise specified, loss functions, other hyperparameters, and augmentation follow the default configuration.Source: B. Implementation Details · p. 7
C. Comparisons with State-of-the-Art Methods
FoCal leads the reported comparisons on DroneVehicle, ESCVehicle, and ATR-UMOD, with its strongest results paired with a compact model. It also reports category-level gains on DroneVehicle and ATR-UMOD.
C. Comparisons with State-of-the-Art Methods
- FoCal achieves 83.5% mAP50 on DroneVehicle, 54.8% mAP50 on ESCVehicle, and 64.6% mAP50 on ATR-UMOD, leading the reported overall comparisons.On DroneVehicle, it also records 68.6% mAP and exceeds the second-best mAP50 result by 1.5 percentage points; on ESCVehicle, it improves on C2-VeD by 2.4 percentage points in mAP50.Source: C. Comparisons with State-of-the-Art… · pp. 7–8, Table II · p. 8, Table III · p. 8
- On DroneVehicle, FoCal ranks first in four of five vehicle categories and ties for the best car AP.Its APs are 98.6% for car, 83.6% for truck, 96.2% for bus, 66.9% for van, and 72.0% for freight car.Source: C. Comparisons with State-of-the-Art… · p. 7
- On ESCVehicle, FoCal reaches 54.8% mAP50 and 41.7% mAP with 3.0M parameters and 8.4G FLOPs, both lowest among compared methods.Compared with C2-VeD, it reduces parameters from 16.5M to 3.0M and FLOPs from 13.1G to 8.4G while achieving higher accuracy.Source: C. Comparisons with State-of-the-Art… · pp. 7–8
- On ATR-UMOD, FoCal achieves the best AP for bus, trailer, crane, and excavator, at 91.1%, 52.1%, 78.3%, and 58.5%, respectively.The comparison uses oriented bounding box heads across 11 object categories.Source: C. Comparisons with State-of-the-Art… · p. 8, Table III · p. 8
D. Ablation Studies
Ablations show that redundancy reduction, FADC, and DGSM contribute to the DroneVehicle results, while the proposed frequency-to-interaction assignment outperforms uniform and reversed alternatives. Visualizations illustrate target responses and intermediate frequency representations in challenging scenes.
D. Ablation Studies
- Redundancy reduction raises mAP50, mAP75, and mAP from 80.8%, 76.6%, and 65.2% to 81.9%, 78.5%, and 67.3%, while reducing parameters from 4.01M to 2.63M.The design retains modality-specific branches through P4, constructs shared P5 features, and concentrates final predictions on P4 and P5 while preserving P3 cues in multi-scale aggregation.Source: D. Ablation Studies · p. 9
- With FADC and DGSM combined, the model reaches 83.5% mAP50, 79.9% mAP75, and 68.6% mAP on DroneVehicle.FADC alone on the redundancy-reduced model yields 82.9% mAP50, 79.5% mAP75, and 68.2% mAP; DGSM alone yields 82.6%, 79.1%, and 68.0%, respectively.Source: D. Ablation Studies · p. 9, Table IV · p. 9
- The proposed low-frequency consensus and high-frequency selective exchange reach 82.9% mAP50, 79.5% mAP75, and 68.2% mAP, exceeding the uniform and reversed assignments.The reverse assignment reaches 82.4%, 78.9%, and 67.7% on those metrics; uniform consensus reaches 82.7%, 79.2%, and 68.0%.Source: D. Ablation Studies · p. 9, Table V · p. 9
E. Visualization
- In two challenging scenes, FoCal detects the highlighted vehicle missed by the baseline and one competing method, while maintaining detections of surrounding vehicles.The examples involve low illumination, weak contrast, and a densely arranged parking area.Source: E. Visualization · p. 9, Figure 5 · p. 10
- FADC consolidates differing RGB and IR low-frequency responses into a shared structural representation while retaining two modality-specific high-frequency streams.The reported heatmaps show more concentrated activations around vehicles and weaker diffuse background responses than the baseline, particularly under low-contrast RGB conditions.Source: E. Visualization · pp. 9–10, Figure 6 · p. 10
V. DISCUSSION
FoCal reports strong detection results across three aerial RGB–IR benchmarks alongside compact model size and high inference throughput. The conclusion identifies hardware-aware optimization and broader temporal and multimodal settings as future work.
V. DISCUSSION
- On an NVIDIA RTX 4090, FoCal reaches 113.6 FPS with 8.8 ms latency per RGB–IR pair and 83.5% mAP50.COMO achieves 81.2% mAP50 at 64.5 FPS, while CrossWeaver achieves 80.6% mAP50 at 31.2 FPS.Source: §V · p. 11
- Future work includes end-to-end deployment optimization, including mixed-precision inference, TensorRT acceleration, and hardware-aware optimization on embedded and edge platforms.The conclusion also proposes extending frequency-dependent interaction to temporal and more diverse multimodal perception scenarios.Source: §V · p. 11, §VI · p. 11
VI. CONCLUSION
- FoCal reports mAP50 values of 83.5% on DroneVehicle, 54.8% on ESCVehicle, and 64.6% on ATR-UMOD, with 3.0M parameters.The discussion characterizes performance across the benchmarks as strong and relates the accuracy gains to frequency-dependent cross-modal interaction.Source: §V · pp. 10–11, §VI · p. 11