Source-linked AI summary
Infrared and Visible Image Fusion using a Deep Learning Framework
Hui Li, Xiao-Jun Wu, Josef Kittler
TL;DR
Infrared–visible fusion must extract and combine salient information from two source images into one image. The paper decomposes images into base and detail content, averages the base parts, and uses multi-layer deep features to generate and select detail candidates. Author-reported evaluations indicate better visual and objective fusion performance, including less artificial noise.
Problem
Infrared–visible fusion requires extracting salient features from source images and combining them into a single image.
Method
The method weighted-averages decomposed base parts and uses multi-layer deep features, l_1-norm weighting, candidate generation, and max selection for detail fusion.
Results
The proposed method achieves better fusion performance than existing methods, with more natural images, preserved detail, and less artificial noise.
Takeaways & Limitations
The method provides a deep-learning-based framework for combining infrared and visible image information while preserving salient detail and reducing artifacts.
Abstract
from arXiv · showhide
In recent years, deep learning has become a very active research tool which is used in many image processing fields. In this paper, we propose an effective image fusion method using a deep learning framework to generate a single image which contains all the features from infrared and visible images. First, the source images are decomposed into base parts and detail content. Then the base parts are fused by weighted-averaging. For the detail content, we use a deep learning network to extract multi-layer features. Using these features, we use l_1-norm and weighted-average strategy to generate several candidates of the fused detail content. Once we get these candidates, the max selection strategy is used to get final fused detail content. Finally, the fused image will be reconstructed by combining the fused base part and detail content. The experimental results demonstrate that our proposed method achieves state-of-the-art performance in both objective assessment and visual quality. The Code of our fusion method is available at https://github.com/hli1221/imagefusion_deeplearning
I. INTRODUCTION
Infrared–visible image fusion seeks to extract salient features from two source images and combine them into one image. The paper proposes a deep-learning framework that fuses base information by averaging and detail information through multi-layer features and candidate selection.
- Infrared–visible fusion combines features from source images into a single image for applications including preprocessing, target recognition, and classification.
- The central challenge is extracting salient source features and combining them into the fused image.
- Earlier approaches use transforms, sparse representations, saliency detection, convolutional sparse representation, or CNN-derived decision maps.
- The proposed method decomposes source images into base parts and detail content, then weighted-averages the base parts.
- A deep network extracts multi-layer detail features; soft-max weighting generates candidate detail fusions at multiple layers, and max selection produces the final detail.
- The paper uses fixed VGG-19 features, following the multi-layer feature strategy common to deep-learning style-transfer methods.
III. THE PROPOSED FUSION METHOD
The proposed framework decomposes each preregistered source image into base and detail components, fuses the base components separately, and reconstructs the final image from both fused components.
- The method assumes K preregistered source images, using K = 2 while retaining the same fusion strategy for K > 2.
- Each source image is decomposed into a base part and detail content using an optimization-based image decomposition method.
- The detail content is obtained from each source image after extracting its base part.
- The framework fuses base parts by weighted averaging and reconstructs detail content through a deep-learning framework before combining both components.
A. Fusion of base parts
The base-part fusion stage treats extracted base components as containing common and redundant information, then combines them using weighted averaging with equal weights.
- Extracted base parts contain common features and redundant information.
- The fused base part is computed with a weighted-averaging strategy.
- At each image-intensity position, α1 and α2 specify the contributions of the two source base parts to the fused base part.
- The method sets α1 = 0.5 and α2 = 0.5 to preserve common features and reduce redundant information.
B. The fusion of the detail content
The method fuses detail content by extracting multi-layer VGG features, converting them into weight maps, generating candidate detail images, and selecting the maximum value at each position.
- Feature extraction: VGG-network features from multiple layers provide the basis for detail-content fusion.The strategy uses VGG-19 and extracts features from relu 1_1, relu 2_1, relu 3_1, and relu 4_1.
- Activity measurement: Feature maps are converted into activity-level maps using l1-norm and block-based averaging.The block size is controlled by r; r = 1 is selected to balance misregistration robustness and detail preservation.
- Weight-map generation: Soft-max generates initial weight maps for each layer, which are upsampled to the input detail-content size.The four selected layers produce weight-map pairs aligned with the original detail-content resolution.
- Candidate generation: Each pair of weight maps produces an initial fused detail content through weighted combination of the source detail contents.The procedure generates four candidate fused detail contents, one for each selected network layer.
- Candidate selection: The final fused detail content selects the maximum candidate value independently at every position.This max-selection step combines the candidate detail images into one fused detail image.
C. Reconstruction
The final fused image is reconstructed by combining the fused base part with the fused detail content.
- Reconstruction: The fused base part and fused detail content are added to reconstruct the final fused image.The reconstruction is represented as F(x, y) = Fb(x, y) + Fd(x, y).
D. Summary of the Proposed Fusion Method
The supplied passage introduces the summary of the proposed deep-learning-based fusion method but provides no procedural details.
- Summary: The proposed method is summarized as a deep-learning-based image-fusion approach.
1) Image decomposition:
The source images are decomposed into base parts and detail contents before fusion.
- Image decomposition: Each of the two source images is decomposed into a base part and detail content.The sources are denoted with k ∈ {1, 2}.
2) Fusion of base parts:
The method fuses decomposed base parts by weighted averaging and uses a multi-layer deep-feature strategy for detail fusion before reconstructing the image.
- 2) Fusion of base parts:: Base parts are fused using weighted averaging, with each base-part weight set to 0.5.
- 2) Fusion of base parts:: Detail content is fused through a multi-layer fusion strategy using deep features extracted from selected layers of a pre-trained VGG-19 network.
- 2) Fusion of base parts:: The final fused image is reconstructed by combining the fused base part and detail content.
- 2) Fusion of base parts:: Experiments use 21 source-image pairs and compare the proposed method with CBF, JSR, JSRSD, WLS, and ConvSR.
B. Subjective Evaluation
Subjective and objective evaluations compare the proposed fusion method with existing approaches on image quality, detail preservation, structural information, and added artifacts.
- B. Subjective Evaluation: The proposed method preserves more detail in highlighted regions and contains less artificial noise than the compared methods in the street and people examples.
- B. Subjective Evaluation: Compared with CBF, JSR, JSRSD, and WLS, the proposed method produces more natural fused images with clearer salient features and more preserved detail.
- C. Objective Evaluation: Four metrics evaluate mutual information, artifact and noise rate, and structural similarity: FMIdct, FMIw, Nabf, and SSIMa.FMIdct and FMIw measure mutual information for discrete cosine and wavelet features, respectively.
- C. Objective Evaluation: The proposed method achieves the best average values for FMIdct, FMIw, SSIMa, and Nabf across 21 fused images.Higher FMIdct, FMIw, and SSIMa are better, whereas lower Nabf indicates less artificial information and noise.
- C. Objective Evaluation: The proposed method's Nabf values are nearly two orders of magnitude better than those of CBF, JSR, and JSRSD.The comparison is reported using the 21-image Nabf results in Table II and Fig. 7.
V. CONCLUSION
The paper concludes with a VGG-network-based infrared-visible fusion method that separates base and detail content, applies multi-layer detail fusion, and combines the results.
- V. CONCLUSION: The proposed method uses a deep learning framework based on a VGG network for infrared and visible image fusion.
- V. CONCLUSION: Source images are decomposed into base parts and detail content, with base parts fused by weight averaging.
- V. CONCLUSION: A pre-trained VGG-19 network extracts deep features for a multi-layer detail-fusion strategy using l1-norm, block averaging, and soft-max weighting.
- V. CONCLUSION: The authors believe the method and multi-layer fusion strategy can also be applied to medical, multi-exposure, and multi-focus image fusion.