Source-linked AI summary
DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion
Zixiang Zhao, Shuang Xu, Chunxia Zhang, Junmin Liu, Pengfei Li, Jiangshe Zhang
TL;DR
IVIF seeks to combine infrared thermal information with visible-image detail despite their complementary weaknesses. DIDFuse uses an auto-encoder whose encoder performs background-detail decomposition and whose decoder reconstructs fused images. Across TNO, FLIR, and NIR, the authors report robust fusion with highlighted targets and rich details that outperforms SOTA methods.
Problem
IVIF must retain infrared thermal information and visible-image gradients, while existing methods may use deep learning only during fusion and rely on manually designed decomposition.
Method
DIDFuse uses an auto-encoder whose encoder decomposes images into background and detail feature maps, while its decoder reconstructs images after separately fusing those maps.
Results
Across TNO, FLIR, and NIR datasets, the model produces fusion images with highlighted targets and rich details and outperforms other SOTA methods.
Takeaways & Limitations
DIDFuse provides a robust IVIF approach evaluated on indoor and outdoor scenes under daylight and nightlight illumination.
Abstract
from arXiv · showhide
Infrared and visible image fusion, a hot topic in the field of image processing, aims at obtaining fused images keeping the advantages of source images. This paper proposes a novel auto-encoder (AE) based fusion network. The core idea is that the encoder decomposes an image into background and detail feature maps with low- and high-frequency information, respectively, and that the decoder recovers the original image. To this end, the loss function makes the background/detail feature maps of source images similar/dissimilar. In the test phase, background and detail feature maps are respectively merged via a fusion module, and the fused image is recovered by the decoder. Qualitative and quantitative results illustrate that our method can generate fusion images containing highlighted targets and abundant detail texture information with strong robustness and meanwhile surpass state-of-the-art (SOTA) approaches.
1 Introduction
IVIF combines infrared thermal information with visible-image detail, addressing their complementary weaknesses. DIDFuse uses deep decomposition within an auto-encoder and is evaluated across three datasets against SOTA methods.
- 1 Introduction: Infrared images resist illumination changes but have low resolution and poor texture, whereas visible images provide detail but are sensitive to obstructions and reflections.Fusion seeks to retain thermal radiation and visible-image gradients for target recognition and tracking.
- 1 Introduction: Existing IVIF approaches include traditional transforms, sparse representation, subspace learning, saliency methods, GANs, and multi-scale decomposition.Deep-learning methods use learning in different stages, including generation, decomposition-related processing, and fusion.
- 1 Introduction: A key limitation is that second-group methods use deep learning only for fusion while relying on filters or optimization for decomposition.DIDFuse is proposed by combining principles from the second and third method groups.
- 1 Introduction: DIDFuse uses an auto-encoder in which the encoder decomposes source images and the decoder reconstructs them.Training encourages similar background features, dissimilar detail features, source-image intensity preservation, and visible-image gradient preservation; testing separately fuses background and detail maps.
- 1 Introduction: 132 test images from TNO, FLIR, and NIR cover indoor and outdoor scenes under daylight and nightlight conditions.Compared with SOTA methods, the model produces brighter targets and richer details and may support target recognition and tracking.
2 Related Work
Related IVIF work decomposes images into background and detail components before separately fusing and reconstructing them. U-Net supplies a related encoder-decoder design with skip connections for preserving texture.
- 2.1 U-Net and Skip Connection: U-Net uses contracting and expanding paths for feature extraction and precise localization, with channel-wise concatenation of corresponding feature maps.Compared with an auto-encoder, U-Net's skip connections help preserve texture details during downsampling and can accelerate convergence.
- 2.2 Two-Scale Decomposition: Two-scale decomposition separates an original image into a background image containing low-frequency information and a detail image containing high-frequency information.The background captures large-scale pixel-intensity changes, while the detail component captures small-scale changes.
- 2.2 Two-Scale Decomposition: Traditional decomposition obtains the background through an optimization problem or box filter, then computes the detail image as Id = I −Ib.The optimization formulation uses gradient kernels gx and gy, while later fusion applies different criteria to background and detail images before reconstruction.
3 Method
DIDFuse uses an encoder–decoder network to learn two-scale background/detail representations, reconstruct source images, and fuse the representations separately during testing.
- Network Architecture: The encoder performs two-scale decomposition, while the decoder recovers original images from concatenated background and detail feature maps.Background and detail feature maps represent different information types extracted from infrared or visible inputs.
- Network Architecture: The network uses four encoder convolutional layers and three decoder convolutional layers, with reflection padding in the first and last layers to reduce edge artifacts.The architecture also uses batch normalization and specified activation functions, including tanh for background/detail outputs and sigmoid for reconstruction.
- Loss Function: During training, the decomposition loss makes background feature maps similar and detail feature maps dissimilar across infrared and visible images.Background maps capture common features, whereas detail maps capture distinct infrared and visible characteristics.
- Loss Function: The reconstruction objective preserves source-image pixel intensities, structural similarity, and visible-image gradient texture information.L2 measures pixel-intensity agreement, SSIM measures brightness, contrast, and structure, and a gradient sparsity penalty regularizes visible-image reconstruction.
- Fusion Strategy: At test time, a fusion layer merges background and detail feature maps separately before the decoder reconstructs the fused image.The paper considers summation, weighted averaging, and L1-norm activity-based fusion strategies.
4 Experiment
Experiments evaluate DIDFuse across multiple datasets, fusion strategies, qualitative and quantitative comparisons, and repeated training runs. The method produces brighter targets, sharper contours, richer details, and consistently strong metric performance.
- Experiments compare DIDFuse with eight SOTA methods using six fused-image quality metrics.The metrics are entropy (EN), mutual information (MI), standard deviation (SD), spatial frequency (SF), visual information fidelity (VIF), and average gradient (AG).
- The evaluation uses TNO, NIR, and FLIR datasets with grayscale, center-cropped 128 × 128 pixel images.The FLIR training set contains 180 randomly selected image pairs.
- 4.1 Experiments on Fusion Strategy: Summation fusion achieves higher validation values, especially for SD, SF, VIF, and AG, so subsequent experiments adopt it.
- 4.2 Experiments on Image Decomposition: DIDFuse separates similar infrared and visible backgrounds while preserving distinct detail maps containing thermal, gradient, and texture information.The decomposition results support the proposed network structure and decomposition loss.
- 4.3 Comparison with Other Models: Qualitatively, DIDFuse produces brighter targets, sharper edge contours, and richer detailed information than competing methods.Other methods show weak highlighted objects, poor contrast, blurred boundaries, and insufficient sharpness in representative scenes.
- 4.3 Comparison with Other Models: DIDFuse is the best performer on all datasets and metrics in the reported quantitative comparison.Competitors can perform well on individual datasets or subsets of metrics, but DIDFuse leads across the evaluation.
- 4.4 Experiments on Reproducibility: Across 25 repeated training runs, DIDFuse basically remains first on the six metrics, indicating steady generation of high-quality fused images.Figure 5 compares DIDFuse with the greatest and second-greatest competitor values across the repeated experiments.
5 Conclusion
The proposed AE network decomposes images, fuses background and detail features, and reconstructs fusion images. Across TNO, FLIR, and NIR, it steadily produces highlighted targets and rich details.
- The AE encoder performs two-scale image decomposition, while the decoder reconstructs the fused image after feature fusion.The test pipeline inserts a fusion layer between encoder and decoder.
- The model is evaluated on TNO, FLIR, and NIR image-fusion datasets.
- Figure 5 reports reproducibility results using EN, MI, SD, SF, VIF, and AG across the three datasets.Rows correspond to TNO, FLIR, and NIR; columns correspond to EN, MI, SD, SF, VIF, and AG.
- The fusion images steadily contain highlighted targets and rich details.