PaperlayerView exact revision on arXiv

Source-linked AI summary

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Jun Yue, Leyuan Fang, Shaobo Xia, Yue Deng, Jiayi Ma

arXiv:2301.08072v1cs.CV

TL;DR

Infrared-visible fusion methods often underuse visible color and multi-channel complementary information. Dif-Fusion models the joint multi-channel distribution with forward and reverse diffusion, extracts multimodal diffusion features, and directly generates three-channel fused images. The paper reports superior fusion performance, including color fidelity measured with Delta E.

  • Problem

    Existing fusion methods often convert three-channel visible images to single-channel data and pay limited attention to color preservation and multi-channel complementary information.

  • Method

    Dif-Fusion constructs a latent multi-channel distribution with diffusion, extracts infrared-visible diffusion features, and directly generates three-channel fused images using multi-channel gradient and intensity losses.

  • Results

    Dif-Fusion outperforms existing state-of-the-art methods, with experiments reporting benefits across fusion metrics and Delta E for color fidelity.

  • Takeaways & Limitations

    The framework supports direct chromatic infrared-visible fusion while evaluating color fidelity alongside texture and intensity fidelity.

Abstract

from arXiv · show

Color plays an important role in human visual perception, reflecting the spectrum of objects. However, the existing infrared and visible image fusion methods rarely explore how to handle multi-spectral/channel data directly and achieve high color fidelity. This paper addresses the above issue by proposing a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data, which increases the ability of multi-source information aggregation and the fidelity of colors. In specific, instead of converting multi-channel images into single-channel data in existing fusion methods, we create the multi-channel data distribution with a denoising network in a latent space with forward and reverse diffusion process. Then, we use the the denoising network to extract the multi-channel diffusion features with both visible and infrared information. Finally, we feed the multi-channel diffusion features to the multi-channel fusion module to directly generate the three-channel fused image. To retain the texture and intensity information, we propose multi-channel gradient loss and intensity loss. Along with the current evaluation metrics for measuring texture and intensity fidelity, we introduce a new evaluation metric to quantify color fidelity. Extensive experiments indicate that our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity.

I. INTRODUCTION

Dif-Fusion addresses limited color preservation and multi-channel information use in infrared-visible fusion by modeling multi-channel data with diffusion and generating chromatic fused images directly.

  • Dif-Fusion constructs a multi-channel distribution in latent space using forward noise addition and reverse denoising.The input combines three-channel visible data with one-channel infrared data.
  • The denoising network extracts diffusion features containing infrared and visible information for multi-channel fusion.
  • The framework directly generates three-channel fused images and uses multi-channel gradient and intensity losses to preserve fidelity.
  • Existing methods often preserve visible texture and infrared intensity while overlooking visible color and multi-channel complementary information.
  • A new evaluation metric quantifies color fidelity alongside existing texture and intensity fidelity metrics.
  • Extensive experiments report that Dif-Fusion outperforms existing state-of-the-art fusion methods.

II. RELATED WORK

Prior infrared-visible fusion research includes traditional, autoencoder, CNN, and GAN approaches, while Dif-Fusion introduces diffusion-based multi-channel distribution construction to address single-channel processing limitations.

  • Traditional fusion methods include sparse representation, multi-scale transformation, subspace representation, saliency detection, and hybrid approaches.
  • Deep learning-based fusion methods mainly use autoencoders, convolutional neural networks, or generative adversarial networks.
  • Autoencoder methods encode source-image features and reconstruct fused images with decoders; DenseFuse, NestFuse, and SEDRFuse exemplify this family.
  • CNN-based methods preserve gradient and intensity information, while GAN-based methods estimate distributions through adversarial generator-discriminator training.
  • Existing generative fusion methods generally cannot construct infrared-visible distributions in latent space and often reduce three-channel visible images to one channel.
  • Dif-Fusion applies diffusion models to construct multi-channel distributions for infrared-visible image fusion.

B. diffusion models

Diffusion models gradually add Gaussian noise and learn to reverse that process, and Dif-Fusion uses this framework to extract multimodal features for chromatic image fusion.

  • Diffusion models are deep generative models with forward and reverse processes across multiple timesteps.
  • The forward process adds Gaussian noise until the data approaches pure noise, while the reverse process predicts and removes that noise.
  • Dif-Fusion concatenates visible and infrared images across channels and learns their multi-channel distribution in the diffusion framework.
  • Diffusion features from the denoising network feed a multi-channel fusion module that produces chromatic fused images.
  • The training process uses multi-channel intensity and gradient losses to guide fusion.

A. Joint Diffusion with Infrared and Visible Images

Dif-Fusion concatenates registered infrared and visible images into a four-channel input and models their joint latent distribution through forward diffusion. The process progressively adds Gaussian noise across timesteps to construct noisy multi-channel samples.

  • Registered one-channel infrared and three-channel visible images are concatenated into a four-channel image for joint latent-structure learning.
  • Fig. 3 presents visible and infrared image pairs generated by the diffusion models.
  • The DDPM-based forward process constructs the multi-channel data distribution by gradually adding Gaussian noise over T timesteps.
  • At timestep t, noisy four-channel images are generated from the preceding timestep by adding Gaussian noise controlled by a variance schedule.
  • Given the original input I0=I, the noisy sample at timestep t can be directly calculated from the original input, schedule values, and sampled noise.

2) Reverse Diffusion Process:

The reverse diffusion process uses a denoising network to recover the original multi-channel image through a sequence of timestep-wise denoising operations.

  • Neural networks iteratively denoise the noisy multi-channel image It to obtain the preceding image It−1 at each reverse-process timestep.
  • The reverse process models the conditional distribution of It−1 given It.
  • The conditional-distribution mean is parameterized using βt=1−αt and the denoising network ϵθ.

3) Loss Function of Diffusion Process:

Dif-Fusion trains its diffusion model by sampling registered image pairs, Gaussian noise, and a timestep, then learning to predict the added noise. The trained model generates plausible paired visible and infrared images resembling real data.

  • Training samples registered visible–infrared pairs, standard-normal noise, and a uniformly sampled timestep before applying the diffusion-model loss.
  • The denoising network uses an SR3 U-Net backbone with contracting and expansive paths plus a diffusion head that predicts the added noise.
  • Generated visible and infrared pairs visually resemble real pairs, with plausible targets appearing in the corresponding infrared images.

4) Structure of the Denoising Network:

The fusion stage extracts multi-channel diffusion features and directly generates a three-channel fused image using multi-channel gradient and intensity losses.

  • The denoising network extracts multi-channel features, while multi-channel gradient and intensity losses train the fusion stage.
  • Features from five denoising-network stages are summed and passed to a fusion head that produces a three-channel fused image.
  • Multi-channel gradient loss extends gradient fidelity to three-channel fused images to retain visible-image texture information.
  • Multi-channel intensity loss extends intensity fidelity to three-channel outputs using the intensity distributions of infrared and visible inputs.
  • The final loss combines multi-channel gradient and intensity losses, enabling direct three-channel generation without color-space transformation.

IV. EXPERIMENTS

The experiments evaluate Dif-Fusion on three public datasets against six state-of-the-art methods using quantitative and qualitative analyses, including a color-fidelity metric.

  • The study combines quantitative and qualitative analysis with ablation experiments to assess diffusion-based multi-channel information fusion.
  • Dif-Fusion is evaluated on MSRS, RoadScene, and M3FD against FusionGAN, SDDGAN, GANMcC, SDNet, U2Fusion, and TarDAL.Compared methods use publicly accessible codes and pretrained models.
  • The evaluation uses MI, VIF, SF, Qabf, and SD for information, fidelity, spatial-frequency, edge, and contrast-related assessment.MI measures source-information aggregation, VIF measures fused-image information fidelity, and SF measures spatial-frequency information.
  • Delta E, computed in CIELAB space, quantifies color distortion between the fused image and the original visible image.The metric is intended to better reflect human color perception than direct Euclidean color-space distance.
  • Lower Delta E indicates smaller color distortion and better fusion quality, whereas higher values indicate better results for the other five metrics.

3) Training Details:

The study presents qualitative comparisons across daytime and nighttime MSRS scenes, examining target preservation, complementary information, texture, brightness, and color retention.

  • Training Details: The MSRS dataset contains daytime and nighttime scenarios, and two image pairs from each are selected to compare fusion results.Infrared images emphasize thermal targets, while visible images provide texture and color information.
  • Training Details: On the first daytime pair, only Dif-Fusion clearly preserves distinctive window contours while retaining brightness differences beneath the eaves.Dif-Fusion also separates foreground greenery from background walls more clearly than the compared methods.
  • Training Details: On the second daytime pair, Dif-Fusion is the only method reported to preserve both infrared window structure and the visible-image sign characteristics simultaneously.Other methods preserve one highlighted region or struggle with both features.
  • Training Details: In one nighttime pair, Dif-Fusion avoids losing pedestrian structure and zebra-crossing information while better preserving visible-image brightness, color, and clarity.
  • Training Details: In a complex nighttime scene, only Dif-Fusion is reported to preserve weak-signal surface structure with clarity and brightness while avoiding visible-image color distortions.Other methods either blur or contaminate the structure with noise, or make a white vehicle appear green.

2) Quantitative Results:

Dif-Fusion shows notable quantitative benefits on five of six MSRS metrics, while performing marginally below SDDGAN and TarDAL on spatial frequency.

  • Dif-Fusion performs best on MI, VIF, Qabf, SD, and Delta E among the compared methods on MSRS.These metrics respectively reflect information aggregation, visual information fidelity, edge preservation, contrast, and color fidelity.
  • Dif-Fusion is marginally inferior to SDDGAN and TarDAL on SF.

1) Qualitative results:

Cross-dataset qualitative evaluations indicate that Dif-Fusion preserves complementary infrared information together with visible-image color, texture, and structure across RoadScene and M3FD scenes.

  • 1) Qualitative results: The MSRS, RoadScene, and M3FD evaluations use qualitative comparisons with six state-of-the-art methods, including quantitative MSRS metrics.
  • RoadScene: On RoadScene, Dif-Fusion most closely resembles the original visible image while preserving infrared saliency and visible-image color, texture, and structure.Compared outputs from U2Fusion and TarDAL have less color distortion but are blurry and lack significant structure information.
  • M3FD: On M3FD, Dif-Fusion and TarDAL are reported as similar to the original visible image in overall perception in an underground-garage scene.
  • Conclusion: The qualitative analysis concludes that Dif-Fusion can mine complementary information across multimodal data in different scenes while retaining texture and color.

2) Quantitative results:

Dif-Fusion performs strongly across quantitative fusion metrics and preserves color fidelity, while ablation results support the contribution of its diffusion process. The evaluation also introduces Delta E as a color-fidelity metric alongside existing texture and intensity measures.

  • Cross-dataset results: Dif-Fusion ranks first in all six reported metrics on M3FD.The fused images are reported to have rich texture details, highest contrast, and best visual quality.
  • Cross-dataset results: Dif-Fusion outperforms competing methods on VIF, Qabf, and Delta E on RoadScene.It ranks first in Delta E on both M3FD and RoadScene.
  • Method contribution: The framework uses diffusion models to extract multi-channel complementary information for improving color fidelity and visual quality.The method constructs the multi-channel input distribution in latent space through forward and reverse diffusion processes.
  • Ablation study: Removing the diffusion process decreases performance on five metrics in the MSRS ablation study.The ablation retains the original network structure while removing only the diffusion process.
  • Evaluation metrics: Delta E is introduced to quantify color fidelity in addition to existing texture and intensity fidelity metrics.This extends fusion evaluation beyond information, texture, and intensity preservation.
Loading 2301.08072v1…