PaperlayerView exact revision on arXiv

Source-linked AI summary

MDLatLRR: A novel decomposition method for infrared and visible image fusion

Hui Li, Xiao-Jun Wu, Josef Kittler

arXiv:1811.02291v5cs.CV

TL;DR

Infrared–visible fusion requires extracting complementary salient information, while existing approaches can involve computationally complex transforms, time-consuming dictionary learning, or data-hungry training. The paper proposes MDLatLRR and an associated fusion framework that separates detail and base parts, fuses them with nuclear-norm and averaging strategies, and reports better fusion performance in subjective and objective evaluations, including improved tracking performance in an RGB-T task.

  • Problem

    Infrared–visible fusion must extract complementary salient content, while existing transform, representation-learning, and deep-learning approaches have computational, training-complexity, or insufficient-data limitations.

  • Method

    MDLatLRR learns a LatLRR projection matrix, decomposes source images into multi-level detail and base parts, fuses detail parts with a nuclear-norm strategy, and averages base parts.

  • Results

    The proposed method achieves better subjective and objective fusion performance than existing methods and improves tracking performance when combined with different trackers in an RGB-T task.

  • Takeaways & Limitations

    MDLatLRR provides a general decomposition approach for extracting multi-level features from an arbitrary number of input images and supports infrared–visible fusion and RGB-T tracking.

Abstract

from arXiv · show

Image decomposition is crucial for many image processing tasks, as it allows to extract salient features from source images. A good image decomposition method could lead to a better performance, especially in image fusion tasks. We propose a multi-level image decomposition method based on latent low-rank representation(LatLRR), which is called MDLatLRR. This decomposition method is applicable to many image processing fields. In this paper, we focus on the image fusion task. We develop a novel image fusion framework based on MDLatLRR, which is used to decompose source images into detail parts(salient features) and base parts. A nuclear-norm based fusion strategy is used to fuse the detail parts, and the base parts are fused by an averaging strategy. Compared with other state-of-the-art fusion methods, the proposed algorithm exhibits better fusion performance in both subjective and objective evaluation.

I. INTRODUCTION

The paper addresses infrared–visible image fusion by proposing MDLatLRR, a multi-level decomposition and fusion framework designed to extract and combine complementary salient information. It uses LatLRR-based decomposition with separate strategies for detail and base parts, and reports better subjective and objective fusion performance than existing methods.

  • Infrared–visible fusion combines complementary information from multiple images to generate an enhanced image containing salient content from both modalities.
  • The method targets limitations of prior approaches, including transform-domain computational complexity, time-consuming dictionary learning, and insufficient training data for some deep fusion methods.
  • MDLatLRR applies LatLRR-based multi-level decomposition to extract salient detail parts and base parts for image fusion.
  • The framework fuses detail parts with an adaptive nuclear-norm strategy and base parts by averaging before reconstructing the fused image.
  • The learned projection matrix L is obtained once during training, allowing source images of arbitrary size to be processed without recalculating low-rank coefficients for each image.
  • Multi-level decomposition captures more detail information, while the base part’s contour features make weight averaging effective.

B. Projection Matrix L in LatLRR

The projection matrix L learned by LatLRR enables DLatLRR to separate an image into detail and base components. The method operates on overlapping patches, reconstructs detail images, and obtains base images by subtraction.

  • Once learned by LatLRR, projection matrix L can process images of arbitrary size because its size depends on patch size rather than image size.
  • DLatLRR divides the input image into overlapping n × n patches, reshuffles them into a source matrix, and determines detail and base parts.
  • The detail component is computed as Vd = L × P(I), reconstructed as Id = R(Vd), and the base component is Ib = I − Id.
  • During detail reconstruction, overlapping pixels are restored using their average pixel value.

B. MDLatLRR

MDLatLRR repeatedly applies DLatLRR to base parts, producing multiple detail-part matrices and one final base part. A projection matrix learned by LatLRR supports feature extraction for image patches of varying source-image sizes.

  • B. MDLatLRR: MDLatLRR applies DLatLRR recursively to each previous base part across r decomposition levels.This yields a multi-level version of DLatLRR.
  • B. MDLatLRR: The decomposition produces r detail-part matrices and one base part at the highest level.The input image can be reconstructed by adding the final base part and the r detail images.
  • A. Learning the Projection Matrix L: Five infrared-visible image pairs are used to learn the projection matrix L by LatLRR.The training images are divided into patches using a sliding-window technique.
  • A. Learning the Projection Matrix L: Training uses 2000 randomly selected image patches to form X, with N = n^2 and n ∈ {8, 16}.The resulting projection matrices have sizes 64 × 64 and 256 × 256 for the two patch sizes.
  • A. Learning the Projection Matrix L: The learned L extracts salient features from test images using the matching patch size.The projection matrix is learned by LatLRR and ALM, with λ set to 0.4.

B. Fusion Method

The fusion framework decomposes infrared and visible inputs into multi-level detail parts and base parts, fuses them with separate strategies, and reconstructs the fused image.

  • B. Fusion Method: Each input image is first decomposed by MDLatLRR into r detail-part pairs and one base-part pair.The same framework applies when the number of input images exceeds two.
  • B. Fusion Method: Corresponding detail parts are fused column by column using an adaptive fusion strategy.This produces r fused detail images after reconstruction.
  • B. Fusion Method: The base parts are fused with a weighted averaging strategy.The base part contains contour and brightness information.
  • B. Fusion Method: The fused image is reconstructed by combining the fused detail images and fused base part.The reconstruction follows the adaptive detail-part and base-part fusion stages.

1) Fusion of base parts:

Base parts contain common, redundant, contour, and brightness information, so they are fused by weighted averaging. Detail parts retain more structural and salient information and therefore use nuclear-norm-based adaptive weighting.

  • 1) Fusion of base parts:: Base parts contain more common, redundant, and brightness information than detail parts.They also contain contour information used by the base-part fusion strategy.
  • 1) Fusion of base parts:: The fused base part is obtained by weighted averaging corresponding base-part values.The weights of the two base parts are denoted by wb1 and wb2.
  • 2) Fusion of detail parts:: Detail parts preserve more structural information and salient features, requiring a more carefully designed fusion strategy.The method applies adaptive strategies to fuse these parts.
  • 2) Fusion of detail parts:: Nuclear norms calculate weights for corresponding image-patch columns in the detail parts.The nuclear norm is the sum of the singular values of a matrix.
  • 2) Fusion of detail parts:: Each fused detail vector is formed by weighting corresponding detail-part vectors, and the strategy is applied across all decomposition levels.The resulting fused detail images are reconstructed from the fused detail-part matrices.

3) Reconstruction:

The fused image is reconstructed from the fused detail images and fused base part, with experiments comparing the proposed framework against existing fusion methods.

  • 3) Reconstruction:: The fused image is generated by combining the fused detail images with the fused base part through Eq 9.The reconstruction follows the preceding fusion of detail and base components.
  • 3) Reconstruction:: The experiments evaluate fusion performance by comparing the proposed method with other existing fusion methods.The evaluation uses five pairs of infrared and visible source images shown in Figure 7.

A. Experimental Setting

The experiments use infrared–visible image pairs, patch-based projection-matrix learning, and systematic tests of patch composition, patch size, norms, decomposition levels, and implementation settings.

  • A. Experimental Setting: A sliding window divides source images into patches, with window sizes determined by projection matrices trained on 8 × 8 or 16 × 16 patches.The initial stride is set to 1.
  • A. Experimental Setting: The base parts use equal weights, wb1 = wb2 = 0.5, because they contain more common features and redundant information.
  • A. Experimental Setting: Training patches are classified as detail or smooth using standard deviation, with Th = 0.5 determining the category.For 8 × 8 patches, 2316 detail and 38338 smooth patches are obtained; for 16 × 16 patches, the counts are 2646 and 7444.
  • A. Experimental Setting: Projection matrices are learned from 2000 patches across five detail-to-smooth ratios and evaluated with six quality metrics.For L16, the [1000, 1000] ratio gives four best values, while En and MI remain comparable; the selected strategy uses 1000 patches of each type.

2) Decomposition Levels and Nuclear-norm:

The study finds that moderate decomposition depth and nuclear-norm detail fusion provide the strongest reported configuration, while excessive depth degrades performance.

  • 2) Decomposition Levels and Nuclear-norm:: Decomposition levels 1 to 4 achieve almost best values, whereas more than four levels cause fusion performance degradation.At deeper levels, detail parts contain more base information that the detail fusion strategy is not equipped to fuse.
  • 2) Decomposition Levels and Nuclear-norm:: Levels 1 to 4 achieve seven best values across nine quality metrics, with comparable En and MI results at levels 3 and 4.
  • 2) Decomposition Levels and Nuclear-norm:: Nuclear-norm fusion achieves six best values across nine quality metrics and comparable SCD, SSIMa, and MS-SSIM values.The nuclear norm represents the sum of singular values and captures structural information in image patches.
  • 2) Decomposition Levels and Nuclear-norm:: The selected MDLatLRR configuration uses L16, decomposition levels 1 to 4, and nuclear-norm fusion for detail parts.The decomposition levels correspond to r = 1, 2, 3, 4.

3) The Stride of Sliding Window Technique:

Smaller sliding-window strides improve information capture and fusion quality but increase computation, while the proposed method preserves salient infrared and visible details in visual comparisons.

  • 3) The Stride of Sliding Window Technique:: At decomposition level 1, most quality metrics remain stable across strides because detail information is not fully extracted.The base parts retain more source-image information at this shallow level.
  • 3) The Stride of Sliding Window Technique:: Increasing decomposition depth preserves more texture and structural information, making stride selection more consequential.
  • 3) The Stride of Sliding Window Technique:: Larger strides degrade all six metrics at levels 2 to 4 because less feature information is processed and detail parts contain less detail.A small stride extracts more useful information and improves overall fusion performance.
  • 3) The Stride of Sliding Window Technique:: The method uses stride 1 for best fusion performance, while stride 4 is acceptable for tasks where time efficiency matters.Small strides improve fusion but make the method time-consuming, so stride should be optimized for each task.
  • C. Subjective Evaluation: Compared with several existing methods, MDLatLRR preserves more salient infrared features and visible-image detail in subjective evaluation.The comparison uses L16, nuclear-norm fusion, levels 1 to 4, and stride 1; increasing levels enhances salient features and clarifies detail.

D. Objective Evaluation

The proposed fusion method is evaluated with ten quality metrics and in an RGBT tracking task. It outperforms existing methods on most image-fusion metrics and improves tracking performance after preprocessing with fused frames.

  • Image-fusion evaluation: Ten quality metrics are used to compare the proposed fusion method with thirteen existing fusion methods.Average metric values are reported for the test-image set.
  • Image-fusion evaluation: The method achieves the best values in seven metrics and second-best values in four, with a comparable SSIMa result.Best results occur for En, MI, SD, Qabf, FMIw, MS-SSIM and VIF; second-best results occur for FMIp, FMIw, VIF and EPI.
  • Image-fusion evaluation: The fused images are reported as more natural, with fewer artefacts, and the objective evaluation shows better fusion performance than existing methods.The method also achieves the best EN and MI values by enhancing salient features rather than amplifying noise and generating artefacts.
  • RGBT tracking evaluation: In RGBT tracking, RGB and infrared frames are fused before being fed to LADCF and GFSDCF trackers on the VOT-RGBT2019 benchmark.The benchmark contains 60 video sequences, and the experiment uses the fused frames as tracker input.
  • RGBT tracking evaluation: Both LADCF and GFSDCF achieve better performance in three measures with the fusion preprocessing, while higher decomposition levels make tracking more robust.Robustness is reflected by lower failure values; EAO increases with fused frames, though the increase is slight for GFSDCF.
  • RGBT tracking evaluation: The fused images consistently improve tracking performance, with the effect more limited for trackers relying more heavily on deep features.LADCF with level-2 decomposition obtains higher EAO, whereas GFSDCF shows an EAO close to its RGB-frame result.

VI. CONCLUSIONS

The paper introduces MDLatLRR for multi-level image decomposition and an associated infrared-visible fusion framework. Experiments show superior performance to existing methods and improved RGBT tracking, while future work will examine broader decomposition links and applications.

  • Contributions and findings: MDLatLRR learns a projection matrix through LatLRR to extract multi-level detail and base parts from input images.The framework reconstructs fused images using adaptive strategies for detail and base parts.
  • Contributions and findings: The proposed fusion framework delivers superior performance to existing methods in subjective and objective experiments.The framework is presented as a general decomposition approach for an arbitrary number of input images and other image-processing fields.
  • RGBT tracking application: Applying the fusion method to RGBT object tracking improves tracking performance even with the simplest combination of fusion and different trackers.This extends the evaluation beyond image fusion to a computer-vision tracking task.
  • Future directions: Future work will investigate links between MDLatLRR and other decomposition methods and explore additional image-processing applications.These directions define the stated boundary of the current investigation.
Loading 1811.02291v5…