Source-linked AI summary
Infrared and visible image fusion using Latent Low-Rank Representation
Hui Li, Xiao-Jun Wu
TL;DR
Infrared and visible image fusion must combine complementary information while preserving useful structure from both source images. The paper uses LatLRR to separate global and local structure, fuses the components with different strategies, and reports better subjective and objective performance than compared methods.
Problem
Infrared and visible image fusion must generate a single image containing complementary information from multiple source images while extracting salient objects from both modalities.
Method
LatLRR decomposes infrared and visible images into low-rank global-structure parts and salient local-structure parts, which are fused separately before reconstruction.
Results
The proposed method achieves better subjective and objective fusion performance than the compared fusion methods.
Takeaways & Limitations
The method preserves more salient features and detail information while producing fused images described as more natural for human perception.
Abstract
from arXiv · showhide
Infrared and visible image fusion is an important problem in the field of image fusion which has been applied widely in many fields. To better preserve the useful information from source images, in this paper, we propose a novel image fusion method based on latent low-rank representation(LatLRR) which is simple and effective. Firstly, the source images are decomposed into low-rank parts(global structure) and salient parts(local structure) by LatLRR. Then, the low-rank parts are fused by weighted-average strategy to preserve more contour information. Then, the salient parts are simply fused by sum strategy which is a efficient operation in this fusion framework. Finally, the fused image is obtained by combining the fused low-rank part and the fused salient part. Compared with other fusion methods experimentally, the proposed method has better fusion performance than state-of-the-art fusion methods in both subjective and objective evaluation. The Code of our fusion method is available at https://github.com/hli1221/imagefusion\_Infrared\_visible\_latlrr
I. INTRODUCTION
Infrared and visible image fusion combines complementary information from two sensors, but existing representation and deep-learning methods have limitations in capturing structure or training with limited data. The paper proposes LatLRR-based fusion to separate global and local structure before reconstructing the fused image.
- Motivation: Infrared and visible image fusion seeks one image containing complementary information from multiple views of the same scene.The task is used in applications including surveillance, object detection, and target recognition.
- Related methods: Conventional transform methods have limited detail-preservation ability, motivating alternative fusion representations.
- Research gap: Sparse-representation methods can achieve good fusion performance but have limited ability to capture global structure.
- Research gap: Deep-learning fusion methods can be difficult to train when infrared and visible-image training data are insufficient.
- Contribution: The proposed method decomposes source images into low-rank global-structure parts and salient local-structure parts, then reconstructs the fused image from separately fused components.The paper reports better fusion performance than other fusion methods in experiments.
II. LATENT LOW-RANK REPRESENTATION THEROY
LatLRR extends low-rank representation by extracting both global and local structure from raw data. Its optimization separates observed data into low-rank, salient, and sparse-noise components.
- LatLRR motivation: Standard low-rank representation cannot preserve local structure information, whereas LatLRR extracts both global and local structure from raw data.
- Optimization formulation: LatLRR formulates an optimization problem whose balance coefficient λ controls the relative regularization terms.The nuclear norm is the sum of singular values, while the l1-norm is also used in the formulation.
- Decomposition: The observed data matrix X is decomposed into low-rank coefficients Z, salient coefficients L, and a sparse noisy part E.The resulting low-rank and salient parts are XZ and LX, respectively.
III. THE PROPOSED FUSION METHOD
The proposed fusion framework applies LatLRR separately to infrared and visible images, then fuses their low-rank and salient components before reconstructing the final image. Weighted averaging is used for low-rank components to retain global structure and brightness information.
- Decomposition: LatLRR decomposes each input image into a low-rank part and a salient part before component-wise fusion.The framework identifies the low-rank component with global structure and the salient component with local structure.
- Fusion framework: The infrared and visible source images are represented as I1 and I2, and their corresponding components are fused separately.
- Reconstruction: The final fused image is reconstructed by adding the fused low-rank part to the fused salient part.
- A. Fusion of low-rank parts: Low-rank parts are combined by weighted averaging because they contain global structure and brightness information.
- A. Fusion of low-rank parts: The low-rank fusion weights are set to w1 = 0.5 and w2 = 0.5 to preserve global structure and brightness while reducing redundant information.
B. Fusion of salient parts
The salient parts contain local structure and salient features, and this method fuses them with a sum strategy using unit weights for both source images.
- Salient parts contain local structure information and salient features.
- The salient parts from the two source images are fused using a sum strategy.
- The fusion uses s1 = 1 and s2 = 1 to preserve more local structure and salient features.
C. The reason of choose sum strategy
The source salient parts can emphasize different features at corresponding positions, so sum fusion retains both contributions and increases the fused salient coefficients.
- At corresponding positions, infrared or visible salient coefficients can dominate while the other source has very small values.
- The fused salient part and its same-row coefficients are obtained using the sum strategy.
- Weighted-average fusion would reduce salient features, whereas sum fusion retains them without loss and increases their values.
D. Reconstruction
The method decomposes each source image with LatLRR, fuses low-rank and salient parts using different strategies, and reconstructs the fused image by adding them.
- The fused image is reconstructed by adding the fused low-rank part and fused salient part.
- Each source image is decomposed by LatLRR into low-rank and salient parts.
- Low-rank parts are fused by weighted averaging, while salient parts are fused by summation.
IV. EXPERIMENTAL RESULTS
The experiments evaluate the proposed fusion method visually and quantitatively on 21 infrared-visible source-image pairs, comparing it with several classical and learned fusion methods.
- The experiments assess fusion performance using subjective and objective evaluation methods.
- The source data contain 21 pairs of infrared and visible images, with several examples shown in Fig. 6.
- Comparisons include CBF, DCHWT, JSR, JSRSD, GTF, CSR, and CNN fusion methods.
- The LatLRR parameter is λ = 0.8 and the low-rank fusion weighting value is 0.5.
- All experiments were implemented in MATLAB R2016a on a 3.2 GHz Intel Core CPU with 12 GB RAM.
B. Subjective Evaluation
Subjective comparisons on street and car image pairs show that the proposed method preserves more salient features and fine details while producing more natural fused images than the compared methods.
- The proposed method preserves more windows, chair details, and glass details in the street and car examples.
- CBF and DCHWT produce more artifacts and less-clear salient features than the proposed method.
- JSR, JSRSD, GTF, CSR, and CNN produce ringing artifacts around salient features and unclear detail information.
- Compared with the evaluated methods, the proposed fusion method provides better subjective fusion performance and appears more natural to human perception.
C. Objective Evaluation
Objective evaluation uses four quality metrics across 21 source-image pairs, with the proposed method achieving the best reported values for SCD, SSIMa, and Nabf and less artificial information in the fused images.
- Four metrics—Qabf, Nabf, SCD, and SSIMa—are used for quantitative comparison.Nabf measures noise or artifacts added during fusion, while SSIMa measures structural preservation.
- SSIMa averages structural similarity between the fused image and the two source images.Its value denotes the ability of structural preservation.
- Higher Qabf, SCD, and SSIMa indicate better fusion performance, whereas lower Nabf indicates better performance.
- The averages are computed for 21 pairs of source images, and the proposed method has the best values for SCD, SSIMa, and Nabf.
- The reported metric results indicate that the proposed fused images are more natural and contain less artificial information than those from the compared methods.
V. CONCLUSIONS
The paper concludes with a simple infrared-visible fusion method based on latent low-rank representation and reports better performance than the compared methods under subjective and objective evaluation.
- LatLRR decomposes each source image into low-rank global-structure parts and salient local-structure parts.
- The method fuses the decomposed parts using weighted-average strategies and reconstructs the fused image by adding the fused components.
- Subjective and objective experiments show better performance for the proposed method than for the compared methods.