Source-linked AI summary
SAGE: Source-Anchored Guidance via Frequency Equalization for Hierarchical RGB-T Alignment and Fusion
Timing Li, Yiming Sun, Boan Tao, Xiyuan Gao, Haifang Cao, Pengfei Zhu
TL;DR
RGB-T fusion must reconcile geometric misregistration and cross-modal appearance differences that can produce ghosting, blur, and imbalanced content. SAGE propagates source-anchored frequency guidance through hierarchical alignment and subband fusion, and experiments report robust performance across diverse scenes and deformation conditions.
Problem
RGB-T fusion must address residual geometric misregistration and cross-modal appearance differences, which can cause ghosting, structural blur, and biased information aggregation.
Source: §1 · p. 1Method
SAGE preserves source-specific information and propagates structural and gain cues through low-to-high frequency alignment and guided subband fusion.
Source: §1 · p. 2, §3.1 · p. 4Results
Experiments demonstrate robust performance across diverse scenes and deformation conditions, with competitive overall performance across alignment and fusion metrics.
Source: §1 · p. 3, §5 · p. 10Takeaways & Limitations
SAGE coordinates source preservation, geometric alignment, and frequency-specific fusion while maintaining structural correspondence and task-relevant cross-modal information.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Spatial misregistration and cross-modal discrepancies often cause ghosting, structural blurring, and content imbalance in RGB-T fusion. Existing methods typically decouple appearance adaptation, geometric alignment, and information fusion, limiting dependency propagation across stages. We propose Source-Anchored Guidance via Frequency Equalization for Hierarchical RGB-T Alignment and Fusion (SAGE), a unified framework integrating frequency equalization, hierarchical alignment, and subband fusion. SAGE employs invertible joint encoding and source-specific low-frequency modulation to derive structural and gain guidance while preserving source information. Hierarchical frequency collaborative alignment estimates global affine geometry from low-frequency approximations and transfers geometric and contextual cues to high-frequency correlation reasoning for reliability-aware residual refinement. Guided subband fusion jointly aggregates the aligned frequency coefficients under propagated source and alignment guidance, coordinates complementary low- and high-frequency information, and reconstructs the fused image through the inverse wavelet transform. Extensive experiments on RGB-T datasets with real-world and synthetic misalignments demonstrate consistently competitive performance in alignment and fusion, validating the effectiveness of source-anchored guidance for weakly registered RGB-T images.
1 Introduction
Residual geometric misregistration and cross-modal appearance differences complicate RGB-T fusion, while existing approaches insufficiently coordinate representation, alignment, and fusion. SAGE uses source-anchored frequency guidance to connect source preservation, low-to-high alignment, and subband fusion.
1 Introduction
- RGB-T fusion combines visible textures and scene details with thermal targets, but misregistration and appearance differences can produce ghosting, structural blur, and biased aggregation.Existing methods coordinate cross-modal representation, alignment, and fusion insufficiently, leaving cross-stage and cross-frequency dependencies underexploited.Source: §1 · p. 1
- Frequency decomposition separates coarse global layout from local edges and textures, supporting global geometry estimation, local displacement refinement, and distinct fusion behavior.It provides a shared basis for discrepancy reduction, alignment, and adaptive fusion.Source: §1 · pp. 1–2
- SAFE jointly models paired wavelet coefficients while retaining source-specific streams, modulating low-frequency components and preserving high-frequency structures to derive guidance cues.Structural and gain cues propagate from discrepancy reduction to alignment and fusion.Source: §1 · p. 2
- HFCA assigns global geometry estimation to low-frequency affine alignment and local residual correction to high-frequency correlation refinement, transferring geometry and context between them.This models the global-to-local dependency between frequency bands.Source: §1 · p. 2
- Guided subband fusion combines source cues with correspondence and reliability for frequency-specific aggregation; experiments report competitive performance across alignment and fusion metrics.It also coordinates cross-frequency information.Source: §1 · p. 3
2 Related Work
Prior alignment and fusion work increasingly couples tasks or adapts information selection, but typically estimates deformation from spatial or mixed representations and derives fusion decisions from current features. SAGE instead uses frequency-aware alignment and source-anchored cues for fusion.
Image Alignment
- Alignment approaches combine registration with fusion, correspondence learning, or self-supervision, but generally use spatial images or mixed latent representations for deformation estimation.SAGE uses low-frequency affine estimation as a coarse prior for high-frequency correlation refinement.Source: §2 · p. 3
Image Fusion
- Fusion research explores adaptive interaction, task-aware gating, generation, domain adaptation, and region-dependent strategies, while decisions remain mainly derived from current source or feature representations.SAGE conditions frequency-dependent gating and cross-frequency interaction on source-anchored structural and gain cues.Source: §2 · p. 3
3 Method
SAGE connects frequency equalization, hierarchical alignment, and guided subband fusion through source-derived cues, preserving distinct low- and high-frequency roles while propagating guidance across stages.
3.1 Overview
- SAFE uses source-specific bounded gains to modulate low-frequency coefficients while preserving detail coefficients, then propagates structural and gain cues to alignment and fusion.Its gain predictor processes both sources, their elementwise absolute difference, and the joint feature.Source: §3.1 · p. 4
3.2 Hierarchical Frequency Collaborative Alignment (HFCA)
- HFCA estimates global affine geometry from low-frequency cues, then uses compensated high-frequency details and transferred context to refine local residual displacement.HCR derives a structural reliability map from coherent high-frequency responses and low-frequency gradients to guide residual recovery.Source: §3.2 · pp. 4–5
3.3 Guided Subband Fusion (GSF)
- GSF conditions low-frequency fusion gates on aligned coefficients and source cues, while high-frequency gates also use local correlation responses and reliability.Cross-frequency refinement uses reliability to regulate high-frequency updates before inverse-wavelet reconstruction.Source: §3.3 · p. 5
3.4 Loss Function
- Training combines equalization, alignment, and fusion losses to enforce source consistency, correspondence, and preservation constraints from aligned images.The terms use Charbonnier reconstruction and source-consistency errors, normalized cross-correlation, and maximum-intensity, maximum-gradient, and edge-preservation constraints, respectively.Source: §3.4 · p. 5
4 Experiments
Across DroneVehicle, MFNet, and RoadScene, SAGE produces coherent alignments and balanced RGB-T fusion, with fewer ghosting artifacts. Ablations and guidance-propagation tests support its coordinated frequency-specific design, while downstream detection and efficiency results show its broader evaluated performance.
4.1 Setup
- The evaluation covers three RGB-T datasets and complementary real-world, synthetic non-rigid, and rigid misalignment settings.DroneVehicle retains inherent aerial misalignment; MFNet uses translated and elastically deformed visible images, and RoadScene uses shifted thermal images.Source: §4.1 · p. 5
4.2 Comparing with SOTA Methods
- Across DroneVehicle, MFNet, and RoadScene, SAGE produces sharper, more spatially coherent alignment and balanced fusion with fewer ghosting artifacts and less source bias.The comparisons include local and spatially varying misalignment, with structures and thermal saliency retained alongside visible details.Source: §4.2 · p. 6, Figure 3 · p. 6
4.3 Ablation Studies
- On MFNet, removing source-aware guidance, either alignment stage, or guided subband fusion weakens structural preservation, alignment, detail retention, or source balance.The complete model preserves sharper boundaries and more coherent structures; Table 2 reports its competitive overall performance.Source: §4.3 · pp. 6–7, Figure 4 · p. 7, Table 2 · p. 8
4.4 Effect of Guidance Propagation
- Guidance across all stages yields the strongest positive high-frequency alignment response while keeping low-frequency alignment near the reference, and maintains positive mean effects on all three fusion indicators.Low-frequency-only guidance degrades high-frequency alignment and edge and gradient responses; high-frequency guidance benefits alignment and fusion measures.Source: §4.4 · p. 7
4.5 Frequency Specialized Hierarchical Collaboration
- Assigning global affine alignment to low frequencies and local correlation refinement to high frequencies outperforms applying either mechanism to both branches.SAGE establishes global correspondence before refining residual local errors, whereas LFA+LFA offers limited improvement and HCR+HCR must address both global and local misalignment.Source: §4.5 · p. 8
4.6 Downstream Task Evaluation
- On unaligned DroneVehicle, SAGE achieves the best recall, precision, and mAP@0.5 among the evaluated fusion methods.The evaluation trains and tests YOLO11m on fusion results from the unaligned dataset.Source: §4.6 · pp. 8–9, Table 3 · p. 8
4.7 Complexity Comparison
- SAGE uses 91.45G FLOPs and 3.53M parameters, with 0.02s runtime and 56.50 FPS, indicating competitive inference efficiency at moderate computational cost.The comparison includes methods that jointly perform frequency equalization, hierarchical alignment, and subband fusion.Source: §4.7 · p. 9
5 Conclusion
SAGE unifies source preservation, geometric alignment, and frequency-specific fusion for weakly registered RGB-T images. Experiments report robust performance across scenes and deformation conditions, with guidance propagation and downstream detection results confirming structural correspondence and task-relevant cross-modal information.
5 Conclusion
- SAGE is a unified frequency-domain framework for weakly registered RGB-T alignment and fusion, combining SAFE guidance, HFCA residual refinement, and GSF subband fusion.SAFE preserves source content and derives structural and gain guidance; HFCA couples global geometry with local residual refinement; GSF uses propagated cues and alignment reliability.Source: §5 · pp. 9–10
- Experiments demonstrate robust performance across diverse scenes and deformation conditions, while guidance propagation and downstream detection results confirm structural correspondence and task-relevant cross-modal information.The conclusion frames these results as evidence that SAGE coordinates source preservation, geometric alignment, and frequency-specific fusion.Source: §5 · p. 10