Source-linked AI summary
DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion
Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, Luc Van Gool
TL;DR
Multi-modality image fusion needs to preserve complementary information across infrared-visible and medical modalities, while existing approaches face limitations from manually designed losses and GAN-based generation. DDFM formulates fusion as DDPM posterior sampling, combining an unconditional generative prior with EM-based hierarchical Bayesian likelihood rectification. Experiments report promising fusion results while using only a pre-trained unconditional model and no fine-tuning.
Problem
Multi-modality image fusion must preserve cross-modality information from source images, but explicit likelihood modeling for image fusion is not feasible.
Method
DDFM decomposes conditional DDPM fusion into unconditional generation and maximum likelihood rectification, modeling the latter hierarchically with latent variables and EM inference.
Results
Experiments on infrared-visible and medical image fusion demonstrate promising fusion results, with DDFM preserving source structure and detail information while satisfying visual fidelity requirements.
Takeaways & Limitations
A pre-trained unconditional DDPM can perform conditional image fusion without fine-tuning by integrating likelihood rectification into diffusion sampling.
Abstract
from arXiv · showhide
Multi-modality image fusion aims to combine different modalities to produce fused images that retain the complementary features of each modality, such as functional highlights and texture details. To leverage strong generative priors and address challenges such as unstable training and lack of interpretability for GAN-based generative methods, we propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM). The fusion task is formulated as a conditional generation problem under the DDPM sampling framework, which is further divided into an unconditional generation subproblem and a maximum likelihood subproblem. The latter is modeled in a hierarchical Bayesian manner with latent variables and inferred by the expectation-maximization (EM) algorithm. By integrating the inference solution into the diffusion sampling iteration, our method can generate high-quality fused images with natural image generative priors and cross-modality information from source images. Note that all we required is an unconditional pre-trained generative model, and no fine-tuning is needed. Our extensive experiments indicate that our approach yields promising fusion results in infrared-visible image fusion and medical image fusion. The code is available at \url{https://github.com/Zhaozixiang1228/MMIF-DDFM}.
1. Introduction
Multi-modality image fusion must preserve complementary cross-modality information, but existing GAN-based approaches face unstable training and limited interpretability. DDFM addresses this by combining unconditional DDPM generation with likelihood rectification, requiring only a pre-trained unconditional model and no fine-tuning.
- Motivation: IVF and MIF must model cross-modality features while preserving critical information from all sensors and modalities.IVF combines infrared thermal radiation with visible texture, while MIF fuses medical modalities to support abnormality localization.
- Motivation: GAN-based fusion uses a generator and discriminator to create images containing source information and match the source-image manifold.The discriminator-based workflow is associated with the training and interpretability challenges motivating DDFM.
- DDFM: DDFM formulates fusion as DDPM posterior sampling, decomposing it into unconditional generation for natural image priors and likelihood rectification for source-image similarity.The method combines generative priors with conditional information from the source images.
- DDFM: Likelihood rectification is formulated as hierarchical Bayesian inference with latent variables and solved by the EM algorithm within the diffusion sampling loop.This replaces explicit likelihood calculation with an inference procedure integrated into conditional image generation.
- Results: DDFM requires only a pre-trained unconditional DDPM and no fine-tuning, while experiments report favorable results for infrared-visible and medical image fusion.The evaluation emphasizes preservation of structure and detail information together with visual fidelity.
2. Background
Diffusion models generate images by reversing a noise-adding process through learned score functions and iterative denoising. In image fusion research, they complement generative and discriminative approaches by providing natural-image priors alongside optimization and feature-fusion methods.
- Score-based diffusion models: A forward diffusion process gradually transforms a clean image into nearly Gaussian noise, and its time reversal generates samples.The reverse process is represented as a stochastic differential equation whose unknown score is learned.
- Score-based diffusion models: The reverse-time score ∇x_t log p_t(x_t) is modeled by a learned score function trained with denoising score matching.The training pairs consist of clean and noise-corrupted states sampled across diffusion time.
- Sampling with diffusion models: Unconditional diffusion sampling starts from x_T ∼ N(0, I), predicts a denoised estimate of x_0, and iteratively updates states until the final image is produced.The predicted denoised image and current state determine the next sampling iterate.
- Diffusion model applications: Diffusion models have been applied as generative priors for conditional generation and low-level vision tasks including image restoration and posterior sampling.Examples include stable diffusion, DDRM, DDNM, and DPS.
- Multi-modality image fusion: Multi-modality fusion methods include generative GAN-based models, discriminative autoencoders, algorithm-unfolding models, unified models, and cross-modality Transformer-CNN architectures.These branches differ in their use of neural feature extraction, traditional optimization, training-data requirements, and modality-specific or shared features.
- Multi-modality image fusion: Optimization-based fusion can be limited by manually designed losses, while natural image priors provide information beyond what generation loss functions alone model.This motivates combining diffusion-based priors with source-image constraints.
3. Method
DDFM formulates image fusion as conditional DDPM posterior sampling, combining an unconditional natural-image prior with likelihood rectification based on source images. The rectification is expressed as hierarchical Bayesian inference and integrated into diffusion sampling through EM updates.
- DDPM posterior sampling: DDFM models the fused image distribution p(f|i,v) through conditional DDPM posterior sampling using infrared, visible, and fused images.The conditional score combines unconditional diffusion sampling with a likelihood term involving the source images.
- Likelihood model: DDFM converts an image-fusion optimization loss into a maximum-likelihood problem by constructing a hierarchical Bayesian model with latent variables.Variable substitutions and probabilistic modeling connect the fusion loss to likelihood inference; total variation can be added to better preserve visible-image texture.
- Likelihood model: A Laplace-distributed variable is represented through normally and exponentially distributed latent variables, transforming ℓ1-norm optimization into differentiable ℓ2-norm optimization.The latent-variable formulation avoids potential non-differentiable points in the ℓ1 objective.
- EM inference: The EM algorithm estimates latent-variable expectations in the E-step and updates the fused-image estimate by minimizing the negative Q-function in the M-step.The M-step uses half-quadratic splitting with iterative coordinate updates for auxiliary variables and the image estimate.
- DDFM architecture: The method decomposes fusion into unconditional diffusion generation for natural image priors and likelihood rectification for source-image fidelity.The unconditional module improves visual plausibility, while the EM module preserves information from the source images.
- Integrated sampling: DDFM embeds one EM iteration into each DDPM sampling step: the preliminary estimate ˜f 0|t is rectified to ˆf 0|t, and one-step EM is theoretically equivalent to one-step conditional diffusion sampling.This embedding avoids the numerous iterations required by conventional EM while preserving the conditional sampling formulation.
4. Infrared and visible image fusion
The IVF experiments evaluate DDFM on four datasets using six fusion metrics without task-specific fine-tuning. DDFM combines thermal and texture information, performs strongly across metrics, and outperforms its ablation variants.
- Setup: IVF experiments use TNO, RoadScene, MSRS, and M3FD with an ImageNet-trained DDPM and no task-specific fine-tuning.The evaluation uses six metrics: EN, SD, MI, VIF, QAB/F, and SSIM.
- Qualitative comparison: DDFM combines infrared thermal radiation with visible-image texture, accentuating dimly lit objects and clarifying background edges and contours.
- Quantitative comparison: DDFM performs strongly across almost all quantitative metrics and datasets, with notably high MI, VIF, and QAB/F values.These metrics indicate preservation of source information and alignment with human visual perception according to the reported analysis.
- Ablation studies: None of the ablation groups achieves fusion results comparable to DDFM, supporting the effectiveness of its diffusion and EM components.The ablations evaluate the unconditional diffusion sampling module, total-variation penalty, Bayesian inference, and alternative parameter settings.
5. Medical image fusion
Medical image fusion experiments use 50 Harvard Dataset image pairs spanning MRI-CT, MRI-PET, and MRI-SPECT. DDFM retains intricate textures while emphasizing structural information and performs strongly visually and numerically.
- Setup: MIF experiments use 50 Harvard Medical Image Dataset pairs covering MRI-CT, MRI-PET, and MRI-SPECT.Generation and evaluation metrics match those used for IVF.
- Comparison with SOTA methods: DDFM retains intricate textures while emphasizing structural information, achieving strong visual and nearly all numerical metrics.Qualitative and quantitative results are reported in Fig. 6 and Tab. 3.
6. Conclusion
DDFM formulates multi-modality image fusion as an unconditional DDPM generation process paired with a maximum-likelihood subproblem. A hierarchical Bayesian EM solution is integrated into DDPM sampling to preserve source cross-modality information, with promising results in infrared-visible and medical image fusion.
- DDFM splits fusion into unconditional DDPM generation and a maximum-likelihood subproblem preserving cross-modality information.
- The maximum-likelihood component uses hierarchical Bayesian modeling and integrates an EM-based solution into the unconditional DDPM sampling loop.
- Experiments demonstrate promising DDFM fusion results in infrared-visible and medical image fusion.