Source-linked AI summary
RoES: Rotational Equivariant Selective-frequency Fusion for Multimodal Images
Jiabao Wang, Wenjian Liu, Yaoming Cai, Gengyu Zhang, Boyan Zhao, Zijia Zhang, Yao Ding, Xiaobo Liu
TL;DR
Infrared-visible fusion is ill-posed and existing rotation-equivariant fusion can apply the same constraint across frequencies despite their different roles. RoES adaptively decomposes features and selectively fuses them; the paper reports strong fusion benchmark performance and downstream detection results.
Problem
Fusion lacks a definitive target image preserving all complementary content, and uniform rotation-equivariant modeling overlooks differences between low-frequency structure and high-frequency detail.
Source: §1 · p. 2Method
RoES learns low/high-frequency decomposition, applies rotation-equivariant Mamba to low-frequency structure, and guides high-frequency Dual-Fourier refinement with low-frequency cues.
Source: §1 · p. 2Results
RoES achieves the best TOPSIS score on M3FD, RoadScene, and TNO, and obtains 0.679 mAP@0.5 on M3FD object detection.
Source: §4.2.1 · p. 7, §4.4 · p. 8Takeaways & Limitations
Results across fusion benchmarks and downstream detection support frequency-selective equivariance as an effective principle for robust multimodal fusion.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Infrared-visible image fusion facilitates robust multimodal perception by integrating complementary textural nuances from visible sensors with thermal signatures from infrared systems. Due to the task's inherently ill-posed nature, existing methods heavily rely on structural priors but typically enforce rotation equivariance uniformly across all features. Such a holistic approach overlooks a critical distinction where low-frequency shared structures strictly adhere to equivariant constraints while high-frequency modality-specific details require greater flexibility to preserve unique information. To bridge this gap, we propose RoES, a Rotational Equivariant Selective-frequency fusion network. Instead of employing static decomposition, we introduce a trainable rotation-enhanced updater/predictor module to dynamically decouple low- and high-frequency components. The resulting representations are then processed through a dual-branch fusion module tailored for spectral consistency. Specifically, a rotation-equivariant Mamba is employed to capture long-range structural dependencies in the low-frequency domain, while a polar spectral attention-based Dual-Fourier block refines high-frequency details under explicit low-frequency guidance. Extensive experiments demonstrate that RoES consistently achieves state-of-the-art performance in both fusion quality and downstream object detection, establishing a robust solution for multimodal fusion by reconciling frequency-selective features with equivariant constraints. The source code is available at https://github.com/BryceLosky/RoES-Fusion.
1 Introduction
Infrared-visible fusion combines complementary visible textures and infrared thermal cues, but its ill-posed nature requires structural priors. RoES addresses the mismatch of uniform rotation equivariance and fixed frequency decomposition with an adaptive, frequency-selective fusion design.
1 Introduction
- Visible sensors provide spatial texture and fine structure, while infrared sensors provide illumination-invariant thermal detection but less texture resolution and possible thermal crossover artifacts.Their complementary cues support fused imagery for downstream detection and semantic segmentation.Source: §1 · p. 1
- Fusion is ill-posed because no ground-truth image preserves all complementary content from a registered infrared-visible pair, so models rely on structural priors.The choice of which prior to impose is therefore central to the task.Source: §1 · p. 2
- Uniform rotation equivariance can stabilize shared structure while overregularizing high-frequency details, which are less compatible with the same geometric prior.The introduction also notes approximation errors in practical equivariant modules [53] and accuracy limits when the assumed symmetry does not hold precisely [1].Source: §1 · p. 2
- RoES combines adaptive low/high-frequency decomposition with equivariant low-frequency structure modeling and low-frequency-guided high-frequency refinement.Its components include the learnable RoUP module, rotation-equivariant Mamba, and Frequency-guided Dual-Fourier block with Polar Spectral Attention.Source: §1 · p. 2
- RoES’s contributions include a frequency-selective prior, learnable lifting-based decomposition with direction-aware supervision, selective fusion branches, and polar spectral refinement.The Polar Spectral Attention design injects rotation-consistent low-frequency cues into the high-frequency amplitude spectrum while preserving phase-sensitive localization.Source: §1 · p. 2
2 Related Work
Prior fusion work has used decomposition and spectral interaction, while rotation-equivariant modeling motivates a more frequency-selective treatment. RoES’s overview applies different fusion branches to low- and high-frequency representations.
2.1 Infrared-visible Image Fusion
- Earlier fusion methods decompose features by functional role, but rarely account for the different geometric properties of low-frequency structure and high-frequency detail.The overview describes RoES as using a shared Restormer encoder, trainable RoUP decomposition, an equivariant Mamba for low frequencies, and a frequency-guided Dual-Fourier block for high frequencies.Source: §2.1 · p. 2, Figure 2 · p. 3
- Many spectral fusion methods use fixed transforms, predefined bands, or whole-spectrum processing, limiting adaptive partitioning and low-to-high structural guidance.The lifting scheme supports task-adaptive decomposition through split, predict, and update steps, with explicit reconstruction under invertible designs.Source: §2.1 · p. 3
2.2 Rotation Equivariance Constraint
- Rotation equivariance makes feature transformations predictable under input rotations, but strict implementations can limit accuracy on complex local patterns.Prior fusion studies have paid limited attention to whether low- and high-frequency branches differ in compatibility with strict equivariant modeling.Source: §2.2 · p. 3
3 Methodology
RoES’s methodology follows a selective-equivarience principle: decompose features by frequency, then apply different treatment to the resulting branches.
3 Methodology
- RoES imposes rotation equivariance selectively rather than uniformly across low- and high-frequency components.The overview identifies frequency decomposition and selectivity as the framework’s organizing design.Source: §3 · p. 3
- The design uses a frequency decomposition and selectivity framework, with detailed architecture and components elaborated in subsequent subsections.Source: §3 · pp. 3–4
3.1 Frequency-Selective Equivariance Prior
RoES treats rotation equivariance as a branch-selective prior, constraining low-frequency structure while leaving high-frequency detail fusion flexible. A controlled frequency-threshold experiment supports this choice.
3.1 Frequency-Selective Equivariance Prior
- Low-frequency fusion models shared spatial layout, whereas high-frequency fusion selects reliable modality-specific details and enhances complementary edges, textures, or thermal cues under low-frequency guidance.The paper therefore applies explicit equivariant modeling only to the low-frequency branch.Source: §3.1 · p. 4
- The frequency-selective equivariance constraint requires equivariance in the low-frequency fusion mapping but imposes no strict equivariance constraint on the high-frequency mapping.Source: §3.1 · p. 4
- VIF peaks and Nabf reaches its minimum when equivariance is restricted to low frequencies; extending it toward the full spectrum reverses both gains.The controlled U-Net experiment applies C4 equivariance below a frequency threshold τ and a standard branch above it, with τ=0 meaning none and τ=1 the full spectrum.Source: §3.1 · p. 4, Figure 3 · p. 4
- The implementation uses modality-specific convolution stems, a parameter-shared Restormer backbone, and bidirectional cross-attention before RoUP decomposes features.The stated group is C4, or N=4.Source: §3.1 · p. 4
3.2 Trainable Frequency Decoupling via RoUP
RoUP learns a task-adaptive low/high-frequency decomposition through asymmetric lifting, pairing flexible detail prediction with rotation-consistent structural updates. Its cross-modal supervision encourages agreement in low-frequency structure while suppressing redundant high-frequency correlation.
3.2 Trainable Frequency Decoupling via RoUP
- RoUP uses a CNN predictor for flexible high-frequency residuals and an F-Conv updater for rotation-consistent low-frequency corrections in a task-adaptive lifting decomposition.Independent RoUP modules process the infrared and visible streams; learned refinement is compared with a fixed wavelet split and retains an exact inverse path before fusion.Source: §3.2 · p. 4
3.2.1 Lifting-based decomposition.
- Each lifting step predicts detail from the coarse state and updates the low-frequency state, yielding modality-specific outputs through an exactly reversible decomposition.The predictor preserves modality-specific components, while the updater emphasizes shared structure; the inverse path helps preserve sparse high-frequency information.Source: §3.2.1 · pp. 4–5
3.2.2 Cross-modal decomposition supervision.
- Cross-modal supervision encourages low-frequency agreement within matched directional subspaces while suppressing redundant correlation between final high-frequency states.The low-frequency comparison is made before directional-subspace aggregation; high-frequency decorrelation promotes complementary detail decomposition.Source: §3.2.2 · p. 5
3.3 Frequency-Selective Fusion
RoES concatenates the modalities’ low- and high-frequency components, then fuses them in separate branches. The low-frequency branch models stable shared structure, while the high-frequency branch refines complementary details under low-frequency guidance.
3.3 Frequency-Selective Fusion
- RoES concatenates the modalities’ low- and high-frequency components and applies branch-specific fusion before decoding the fused features into an image.The low-frequency branch extracts stable shared structure; the high-frequency branch enhances complementary details under its guidance.Source: §3.3 · p. 5, §3.3.2 · pp. 5–6
3.3.1 Equivariant Mamba (EM) Block.
- The low-frequency encoder applies rotation-aware patchification and aggregation before a spectral state-space stack models global structure.It rotates features through the four quarterturns, applies a shared CNN patchification stem, rotates responses back, and performs pointwise max aggregation.Source: §3.3.1 · p. 5
3.3.2 Frequency-guided Dual-Fourier (FDF) Block.
- The Frequency-guided Dual-Fourier block uses fused low-frequency features to guide high-frequency amplitude refinement, while lightly refining phase.It recombines the refined amplitude and phase paths in spatial space, then fuses them with the encoded high-frequency feature.Source: §3.3.2 · p. 5
3.4 Polar Spectral Attention (PSA)
Polar spectral attention (PSA) injects low-frequency rotation-consistent structure into high-frequency amplitude spectra without rigidly constraining high-frequency features. Polar coordinates make rotation-related spectral variation easier to align along the angular dimension.
3.4 Polar Spectral Attention (PSA)
- PSA injects low-frequency rotation-consistent structure into high-frequency amplitude spectra without rigidly constraining high-frequency features.Its two stages share a formulation but use independent parameters; low-frequency representations provide a guidance signal for detail restoration.Source: §3.4 · p. 6, §3.4.2 · p. 6
3.4.1 Polar Representation.
- In polar coordinates, rotation of a feature map becomes an angular shift in its amplitude spectrum, turning geometric rotation into translation along the angular axis.This representation supports convolution along that axis and similarity-based cross-frequency attention.Source: §3.4.1 · p. 6
3.4.2 Cross-Frequency Matching and Angular Gating.
- Each high-frequency radial ring queries a global low-frequency descriptor, and the resulting guidance helps recover details at that ring’s spatial-frequency scale.A gated angular convolution rescales the high-frequency response, which is then mapped back to Cartesian coordinates to produce refined amplitude.Source: §3.4.2 · p. 6
3.5 Learning Objectives
RoES trains with a primary fusion loss and an auxiliary RoUP decomposition loss. The objectives preserve complementary input information while encouraging accurate frequency decoupling and feature reconstruction.
3.5 Learning Objectives
- RoES combines the primary fusion loss with an auxiliary RoUP decomposition loss, using empirical trade-off weights.The fusion objective integrates infrared and visible information; the RoUP objective supports frequency decoupling and reconstruction.Source: §3.5 · p. 6, §3.5.1 · p. 6, §3.5.2 · p. 6
3.5.1 Fusion objective.
- The fusion loss combines intensity and gradient L1 penalties with structural similarity, total variation, and VGG perceptual terms.These terms address pixel and gradient fidelity, structural similarity, local artifacts, and perceptual fidelity.Source: §3.5.1 · p. 6
3.5.2 RoUP objective.
- The RoUP objective combines feature reconstruction with cross-modal losses supervising shared low-frequency and modality-specific high-frequency components.The reconstruction loss compares encoder features with features reconstructed by the exact inverse RoUP transform.Source: §3.5.2 · pp. 6–7
4 Experiments
RoES is evaluated against other fusion methods on three datasets, with no fine-tuning on the test datasets. It leads the reported overall rankings and shows qualitative detail preservation and deployment-scale memory use.
4.1.2 Comparison Methods.
- RoES trains on 1,602 MSRS pairs and transfers to M3FD, TNO, and RoadScene without fine-tuning, compared with 17 methods under matched protocols.The comparison uses official implementations and released weights unless otherwise stated.Source: §4.1.2 · p. 7
4.1.3 Implementation Details.
- RoES uses end-to-end MSRS training with 128 × 128 inputs, four RoUP lifting steps, a width-64 four-layer spectral SSM, eight attention heads, and two PSA stages.The model is implemented in PyTorch and trained on one NVIDIA RTX 5090 GPU.Source: §4.1.3 · p. 7
4.1.4 Metrics.
- The evaluation reports nine metrics spanning retained information, detail, fidelity, and structural preservation, with higher scores better for all.TOPSIS rankings summarize normalized metric scores.Source: §4.1.4 · p. 7
4.2.1 Quantitative Comparison.
- RoES achieves the best TOPSIS score on M3FD, RoadScene, and TNO, leading six metrics on M3FD, seven on RoadScene, and six on TNO.Table 1 reports cross-dataset results.Source: Table 1 · p. 7, §4.2.1 · p. 7
- RoES ranks first on MI, FMI, VIF, and Qabf throughout; these gains support adaptive frequency separation and a balance of fidelity and detail.The authors associate MI/FMI gains with separating shared structure from complementary detail, and VIF/Qabf gains with selective equivariance and spectral refinement.Source: §4.2.1 · pp. 7–8
4.2.2 Qualitative Comparison.
- Under severe smoke occlusion, RoES preserves thermal targets, background consistency, and fine textures, while comparison methods show distortion, blur, or detail loss.Figure 5 compares M3FD fusion results.Source: §4.2.2 · p. 8
4.2.3 Efficiency.
- At 1024 × 768 resolution, RoES has 2.28M parameters, requires 1,989G FLOPs, and uses approximately 6.3GB peak memory.The authors state that memory fits within a single consumer GPU, though RoES is not the lightest in FLOPs.Source: §4.2.3 · p. 8
4.3 Ablation Studies
The ablations report complementary contributions from low-frequency equivariance, adaptive decomposition, and spectral refinement. On M3FD detection, RoES achieves 0.679 mAP@0.5 after transfer from MSRS without fine-tuning.
4.3 Ablation Studies
- The ablations compare built-in low-frequency equivariance with rotation augmentation and C4-TTA, test adaptive decomposition and PSA, and replace polar projection with Cartesian attention.The variants are evaluated on M3FD under the stated experimental protocol.Source: §4.3 · p. 8
4.4 Downstream Object Detection Task
- RoES achieves 0.679 mAP@0.5 on M3FD object detection after training the fusion model on MSRS and transferring it without fine-tuning.The evaluation uses YOLOv5 v7.0 and a separate detector trained for each fusion method under the same protocol.Source: Table 3 · p. 8, §4.4 · p. 8
- Figure 6 shows sharp contours, salient thermal targets, and precise boxes for small or distant objects; most competitors except LRRNet miss or misalign pedestrians.The authors connect these observations with stable low-frequency structure and preserved high-frequency texture.Source: Figure 6 · p. 8, §4.4 · p. 8
4.3 Ablation Studies
- Removing low-frequency equivariance raises SF/AG but lowers fidelity; fixed wavelets, absent decomposition supervision, and removing PSA also degrade results.Cartesian attention trades minor EN/SF/AG gains for substantial fidelity loss; the ablations support complementary contributions from the components.Source: §4.3 · p. 8
5 Conclusion
RoES applies rotation equivariance selectively: RoUP learns adaptive frequency decomposition, an equivariant Mamba models low-frequency structure, and a guided Dual-Fourier branch refines high-frequency details. Fusion benchmarks and downstream detection show state-of-the-art performance and support frequency-selective equivariance for multimodal fusion.
5 Conclusion
- RoES combines adaptive RoUP decomposition, an equivariant Mamba branch for low-frequency structure, and a low-frequency-guided Dual-Fourier branch for high-frequency detail refinement.Results across fusion benchmarks and downstream detection support frequency-selective equivariance as an effective principle for robust multimodal fusion.Source: §5 · p. 8