Source-linked AI summary
Equivariant Multi-Modality Image Fusion
Zixiang Zhao, Haowen Bai, Jiangshe Zhang, Yulun Zhang, Kai Zhang, Shuang Xu, Dongdong Chen, Radu Timofte, Luc Van Gool
TL;DR
Image fusion lacks definitive ground truth because different sensors observe modality-specific information, limiting supervised training. EMMA uses pseudo-sensing and equivariant imaging priors for self-supervised fusion, and experiments report high-quality infrared-visible and medical fusion with downstream segmentation and detection benefits.
Problem
Definitive fused-image ground truth is unavailable because individual sensors capture modality-specific features, making supervised deep-learning training difficult.
Method
EMMA combines a fusion module, pseudo-sensing module, and equivariant fusion module to model sensing-imaging processes and enforce transformation equivariance.
Results
EMMA produces high-quality infrared-visible and medical fusion results while facilitating downstream multi-modal segmentation and detection tasks.
Takeaways & Limitations
The paradigm supports end-to-end self-supervised multi-modality fusion without requiring ground-truth fusion images.
Takeaways & Limitations
EMMA assumes natural images and fused images form invariant sets under transformations including translations, rotations, and reflections.
Abstract
from arXiv · showhide
Multi-modality image fusion is a technique that combines information from different sensors or modalities, enabling the fused image to retain complementary features from each modality, such as functional highlights and texture details. However, effective training of such fusion models is challenging due to the scarcity of ground truth fusion data. To tackle this issue, we propose the Equivariant Multi-Modality imAge fusion (EMMA) paradigm for end-to-end self-supervised learning. Our approach is rooted in the prior knowledge that natural imaging responses are equivariant to certain transformations. Consequently, we introduce a novel training paradigm that encompasses a fusion module, a pseudo-sensing module, and an equivariant fusion module. These components enable the net training to follow the principles of the natural sensing-imaging process while satisfying the equivariant imaging prior. Extensive experiments confirm that EMMA yields high-quality fusion results for infrared-visible and medical images, concurrently facilitating downstream multi-modal segmentation and detection tasks. The code is available at https://github.com/Zhaozixiang1228/MMIF-EMMA.
1. Introduction
Multi-modality image fusion combines complementary information from multiple sensors, but inaccessible fused-image ground truth makes supervised training difficult. EMMA addresses this with self-supervision built around sensing processes and equivariant imaging responses.
- Motivation: Multi-modality fusion combines complementary sensor information for applications including image registration, enhancement, object detection, and semantic segmentation.Infrared-visible fusion combines thermal radiation and visible texture, while medical fusion integrates multiple imaging techniques for comprehensive abnormality representation.
- Problem formulation: Fusion is a nonlinear blind inverse problem because a rich fused image is observed only through different, difficult-to-model sensing processes.The formulation separates data fidelity, requiring source images to be sensed from the fused image, from a prior on the fused image.
- Training challenge: No single real sensor captures all modalities, so definitive fused-image ground truth is unavailable for supervised deep learning.This limitation motivates alternatives to direct supervised training for image fusion.
- Training challenge: Direct source-to-fused distance losses ignore feature-manifold differences, while handcrafted fused-image priors rely on speculative assumptions without accessible ground truth.Generative methods also face interpretability, controllability, and training challenges according to the introduction.
- Proposed direction: EMMA learns fusion through pseudo-sensing back to source modalities and uses a non-domain-specific equivariance prior for shifts, rotations, and reflections.Its framework includes a U-Fuser fusion module, a learnable pseudo-sensing module, and an equivariant fusion module.
2. Related Work
Related work spans generative, autoencoder-based, algorithm-unrolling, and unified fusion models, alongside equivariant imaging. EMMA distinguishes itself by using frozen pseudo-sensing and imaging-system priors rather than direct fusion-to-source matching or domain-specific fusion priors.
- Multi-modality image fusion: Existing fusion methods are grouped into generative, autoencoder-based, algorithm-unrolling, and unified models.These families differ in whether they model distributions, encoder-decoder mappings, iterative computational graphs, or cross-task meta-knowledge.
- Equivariant imaging: Equivariant imaging learns reconstruction from partial measurements by exploiting symmetry properties of natural signals, such as translation invariance.This provides the conceptual basis for EMMA’s imaging prior.
- Comparison with existing approaches: EMMA’s pseudo-sensing loss compares reconstructed sources with their originals in the same modality domain instead of directly comparing fused and source images.This addresses the stated manifold difference between fused images and source images.
- Comparison with existing approaches: Unlike jointly updated decomposition modules, EMMA keeps its pseudo-sensing module frozen during fusion training, making the fused-to-source mapping explicit and determinate.The paper connects this design to improved rationality and interpretability of the sensing module.
- Comparison with existing approaches: EMMA relies on imaging-system priors rather than domain knowledge about fused images to support self-supervised learning.This contrasts with prior-based optimization methods that impose assumptions on fusion-image structure.
3. Method
EMMA formulates image fusion as self-supervised learning through pseudo-sensing and an equivariant imaging prior. Its workflow fuses source images, transforms the fused result, reconstructs pseudo-sensor images, and enforces consistency through re-fusion.
- 3.1. Problem Overview: EMMA models fusion and sensing jointly because the ground-truth fused image is information-rich but unavailable from any complete perception device.The formulation represents fusion as F(i, v) and sensing through modality-specific operators Ai and Av.
- 3.2. Model hypothesis: Natural-image transformations such as shifts, rotations, and reflections motivate treating the composite sensing-and-fusion system as equivariant.The required property applies to F ◦ A rather than requiring F or A individually to be equivariant.
- 3.3. Equivariant image fusion paradigm: U-Fuser is a U-Net-like fusion module combining Restormer and CNN blocks to model global and local cross-modal features.Restormer blocks model global features through channel self-attention, while CNN blocks provide complementary local processing.
- 3.3. Equivariant image fusion paradigm: EMMA transforms the fused image, applies parameter-frozen pseudo-sensing modules to obtain modality-specific pseudo-images, then re-fuses them.The training objective aligns pseudo-sensing outputs with the original inputs and encourages the transformed fused image to match the re-fused result.
- 3.3. Equivariant image fusion paradigm: During early training, inaccurate pseudo-sensing models can make reconstruction unsatisfactory, while equivariant learning calibrates fusion and recovers missing null-space components.At deployment, only the fine-tuned U-Fuser is retained; the sensing modules are discarded.
4. Experiment
Experiments evaluate EMMA on infrared-visible and medical image fusion, ablate its sensing and equivariant losses, and test fused outputs on detection and segmentation. Results show strong fusion quality, generalization to medical images without fine-tuning, and benefits for downstream vision tasks.
- Infrared and visible image fusion: EMMA produces informative infrared-visible fused images that combine thermal radiation with visible-image texture, sharpen edges, reduce artifacts, and perform strongly across six fusion metrics.The six metrics are EN, SD, SF, AG, SCD, and VIF; higher values indicate superior fusion effects.
- Ablation studies: Removing the equivariant loss weakens results because the fusion module loses the equivariant prior’s constraint on the solution space.This is reported in ablation Exp. I on the MSRS test set.
- Ablation studies: Replacing sensing loss with traditional ℓ1(f, i) + ℓ1(f, v) loss demonstrates that sensing loss is necessary for an ideal fusion network.Sensing loss compares pseudo-sensing results with source images, whereas the traditional loss directly forces the fused image toward source images.
- Ablation studies: An incomplete feature extraction module degrades performance by weakening local texture modeling or long-range dependency capture.The ablation separately removes Restormer or Res blocks while keeping parameter counts consistent.
- Downstream IVF applications: EMMA achieves the strongest reported multi-modal detection performance, while its fused images combine thermal and RGB information to emphasize hard-to-detect objects.Detection is evaluated with mAP@0.5.
- Downstream IVF applications: EMMA’s fused edge and contour details support more precise semantic segmentation by improving object-boundary recognition.Segmentation performance is evaluated with IoU on the MSRS dataset.
- Medical image fusion: Models trained for infrared-visible fusion generalize to MRI-CT, MRI-PET, and MRI-SPECT fusion without fine-tuning, achieving strong structural-highlight and texture integration.Medical experiments use 50 image pairs from the Harvard Medical dataset and the same quantitative metrics as IVF.
5. Conclusion
EMMA addresses missing fusion ground truth with an equivariant self-supervised paradigm that simulates sensing-imaging processes and uses global-local feature extraction. Experiments support its effectiveness for multi-modality fusion and downstream segmentation and detection.
- Conclusion: EMMA uses translation, shift, rotation, and reflection equivariance as a prior for self-supervised multi-modality image fusion.The paradigm adjusts the loss to reflect natural imaging principles and simulate the sensing-imaging process.
- Conclusion: Its U-Net-like fusion module uses Restormer-CNN blocks to extract global and local features and efficiently fuse information.The conclusion identifies this module as part of the proposed paradigm.
- Conclusion: Experiments corroborate EMMA’s effectiveness for multi-modality image fusion and its usefulness for downstream multi-modality segmentation and detection.The conclusion states these outcomes without reporting specific numerical values.