Source-linked AI summary
Infrared and Visible Image Fusion with ResNet and zero-phase component analysis
Hui Li, Xiao-Jun Wu, Tariq S. Durrani
TL;DR
The paper addresses limitations in image fusion methods that use deep features without sufficient processing and representation-learning methods with dictionary and efficiency drawbacks. It proposes a ResNet- and ZCA-based framework that processes deep features into weight maps before weighted-average reconstruction. Experiments report better fusion performance than existing methods in objective and visual evaluations, while FMIpixel is not the best although it remains close to the best value.
Problem
Representation-learning methods have dictionary-learning and time-efficiency drawbacks, while deep-learning methods may use deep features without sufficient processing, potentially degrading fusion performance.
Method
The framework extracts source-image features with ResNet50, projects them with ZCA, derives l1-norm and soft-max weight maps, and reconstructs the fused image by weighted averaging.
Results
The proposed method achieves better fusion performance than existing methods in subjective and objective evaluations, obtaining almost all best values for Nabf, SSIMa, and EPIa.
Takeaways & Limitations
The framework combines deep-feature extraction with explicit feature processing for infrared and visible image fusion.
Takeaways & Limitations
FMIpixel is not the best for the proposed fused images, although its values remain close to the best.
Abstract
from arXiv · showhide
Feature extraction and processing tasks play a key role in Image Fusion, and the fusion performance is directly affected by the different features and processing methods undertaken. By contrast, most of deep learning-based methods use deep features directly without feature extraction or processing. This leads to the fusion performance degradation in some cases. To solve these drawbacks, we propose a deep features and zero-phase component analysis (ZCA) based novel fusion framework is this paper. Firstly, the residual network (ResNet) is used to extract deep features from source images. Then ZCA is utilized to normalize the deep features and obtain initial weight maps. The final weight maps are obtained by employing a soft-max operation in association with the initial weight maps. Finally, the fused image is reconstructed using a weighted-averaging strategy. Compared with the existing fusion methods, experimental results demonstrate that the proposed framework achieves better performance in both objective assessment and visual quality. The code of our fusion algorithm is available at https://github.com/hli1221/imagefusion_resnet50
1. Introduction
Image fusion methods combine salient information from infrared and visible sources, but representation-learning and deep-learning approaches retain processing and efficiency limitations. The proposed framework addresses these issues by processing ResNet deep features with ZCA before constructing weight maps and reconstructing the fused image.
- Infrared and visible image fusion combines salient features from source images into a single image for widely used applications.
- Representation-learning methods face difficulty learning effective dictionaries offline and low time efficiency, especially with online dictionary learning.
- Deep-learning fusion methods commonly use deep features from fixed networks directly to reconstruct fused images, treating deep learning primarily as feature extraction.
- VGG-based fusion combines multi-layer deep features, but its simple multi-layer strategy loses useful information during feature extraction, increasingly so for deeper networks.
- The proposed framework uses fixed ResNet features, ZCA projection, l1-norm weight estimation, bicubic resizing, soft-max refinement, and weighted-average reconstruction.
2. Related work
The related work introduces residual networks for retaining multi-layer information and ZCA for whitening and processing features. These ideas motivate applying ZCA to image fusion, where projected features can support classification and reconstruction.
- 2.1. Deep residual network (ResNet): Residual blocks add the input X to a transformed representation Φ(X), using shortcut connections to utilize multi-layer information.
- 2.2. Zero-phase component analysis(ZCA): ZCA, also called whitening, projects a random vector into an irrelevant sub-space and is presented as a feature-processing tool for image processing.
- 2.2. Zero-phase component analysis(ZCA): The covariance matrix is decomposed with singular value decomposition, producing U, Σ, and V subject to Co = UΣV^T.
- 2.2. Zero-phase component analysis(ZCA): The ZCA formulation uses the identity matrix I and a small ε to avoid bad matrix inversion when calculating the transformed vector.
- 2.3. ZCA utilization in image style transfer: Prior style-transfer methods project style and content features into a common space with ZCA before coloring or reassembling features and decoding the image.
- 2.3. ZCA utilization in image style transfer: These style-transfer results motivate using ZCA in image fusion because the projection makes image features easier to classify and reconstruct.
3. The Proposed Fusion Method
The proposed framework extracts deep features from preregistered source images with fixed ResNet50, processes them with ZCA and local l1-norm operations, and reconstructs the fusion using learned weight maps. It uses bicubic resizing and soft-max normalization to produce final image-sized weights.
- The method assumes K preregistered source images and uses the same fusion strategy for K > 2, with K = 2 in this paper.
- Fixed ImageNet-trained ResNet50 extracts deep features from the source images through its convolutional blocks.The framework uses outputs from blocks i = 4 and i = 5 for evaluation.
- The fused image is reconstructed from the source images using a weighted-average strategy.
- ZCA projects the original deep features into a common sparse or whitened subspace before subsequent processing.The paper illustrates that channels can have different magnitudes before ZCA and become more significant after processing.
- A local average l1-norm over a window centered on each processed feature location produces the initial weight maps.The window parameter is t = 2.
- Bicubic interpolation resizes the initial weight maps to source-image size, and soft-max operation produces the final weight maps.The resize and soft-max stages are shown as the final weight-map construction steps.
4. Experiments and Analysis
Experiments evaluate the framework across networks, norms, ZCA usage, subjective comparisons, and objective metrics. ResNet50 with ZCA and l1-norm is selected, while the proposed method shows strong visual and quantitative performance across eight image pairs.
- Experimental design: Experiments compare VGG19, ResNet50, and ResNet101 with l1-norm, l2-norm, and nuclear-norm configurations.The study also analyzes ZCA, then evaluates subjective and objective performance against existing fusion methods.
- Networks and norms: ResNet achieves better fusion performance than VGG19 across the evaluated norms and metrics.The ResNet50 and ResNet101 configurations obtain all best and second-best values in the reported tables.
- Networks and norms: ResNet50 and ResNet101 have very close quality-metric values, so ResNet50 is used for better time efficiency.The framework therefore uses ResNet50 for subsequent experiments.
- Networks and norms: l1-norm has better performance than other norms with ResNet50 in the framework.It obtains four best values and one second-best value in Table 4.
- ZCA influence: With ZCA, l1-norm obtains most best values, whereas without ZCA, nuclear-norm achieves the best values for ResNet50.The authors associate this shift with ZCA projecting original data into a sparse space.
- Method comparison: Across eight image pairs, the proposed method obtains almost all best Nabf, SSIMa, and EPIa values and performs better subjectively and objectively.These metrics indicate less noise and better preservation of structure and edge information; FMIpixel is close to the best rather than highest.
5. Conclusions
The paper proposes an infrared–visible image fusion algorithm combining ResNet50 deep features with ZCA-based processing. The method reconstructs fused images through learned weight maps and achieves better objective and subjective fusion performance.
- The proposed fusion algorithm combines ResNet50 with ZCA operation for infrared and visible image fusion.
- ResNet50 extracts deep features, while ZCA projects them into a sparse subspace for subsequent weight-map generation.
- Local average l1-norm, bicubic interpolation, and soft-max processing produce final weight maps at the source-image resolution.
- The fused image is reconstructed by weighted averaging of the source images and their final weight maps.
- Experimental results show better fusion performance in both objective and subjective evaluation.