PaperlayerView exact revision on arXiv

Source-linked AI summary

DenseFuse: A Fusion Approach to Infrared and Visible Images

Hui Li, Xiao-Jun Wu

arXiv:1804.08361v9cs.CV

TL;DR

Infrared and visible image fusion requires combining salient source-image features into one image, while CNN approaches may lose useful middle-layer information. The paper proposes DenseFuse, an encoder with convolutional layers and dense blocks, fusion strategies, and a decoder. The method achieves state-of-the-art fusion performance under subjective and objective assessment and also applies to RGB images.

  • Problem

    Infrared and visible image fusion must extract and integrate salient source-image features, but CNN-based methods may discard useful information from middle encoding layers.

  • Method

    DenseFuse uses an encoder combining convolutional layers and dense blocks, fusion strategies for encoded features, and a decoder to reconstruct the fused image.

  • Results

    The proposed method achieves state-of-the-art fusion performance under subjective and objective assessment.

  • Takeaways & Limitations

    The architecture can fuse both grayscale and RGB visible images with infrared images and may extend to other fusion problems using appropriate fusion layers.

Abstract

from arXiv · show

In this paper, we present a novel deep learning architecture for infrared and visible images fusion problem. In contrast to conventional convolutional networks, our encoding network is combined by convolutional layers, fusion layer and dense block in which the output of each layer is connected to every other layer. We attempt to use this architecture to get more useful features from source images in encoding process. And two fusion layers(fusion strategies) are designed to fuse these features. Finally, the fused image is reconstructed by decoder. Compared with existing fusion methods, the proposed fusion method achieves state-of-the-art performance in objective and subjective assessment. Code and pre-trained models are available at https://github.com/hli1221/imagefusion_densefuse

I. INTRODUCTION

Infrared and visible image fusion combines salient features from source images into one fused image, but prior CNN methods may discard useful middle-layer information. DenseFuse addresses this problem with an encoder using convolutional layers and dense blocks, followed by fusion and decoding.

  • Image fusion extracts salient features from infrared and visible sources and integrates them into a single image for applications including surveillance and military use.
  • Conventional fusion methods include multi-scale decomposition, sparse or low-rank representation, and deep learning-based approaches.
  • CNN-based fusion methods may lose useful middle-layer information because they use only the final encoding-layer results as image features.
  • DenseFuse combines convolutional layers and dense blocks so encoding-layer outputs are cascaded and used to construct feature maps.
  • The proposed architecture reconstructs the fused image through fusion strategies and a decoding network containing four CNN layers.
  • Dense connections are incorporated to preserve useful middle-layer information and facilitate training, motivating the name DenseFuse.

III. PROPOSED FUSION METHOD

The proposed method uses registered infrared and visible images with an encoder, fusion layer, and decoder. Its encoder extracts features through convolution and dense-block layers, while the decoder reconstructs the fused image after fusion-strategy selection.

  • For color images, the fusion strategy remains the same as for grayscale images, while the paper focuses its main discussion on grayscale fusion.
  • The framework accepts at least two pre-registered grayscale infrared or visible images and consists of an encoder, fusion layer, and decoder.
  • The encoder uses an initial convolutional layer and a three-layer dense block to extract deep features through cascaded layer outputs.
  • Different fusion strategies are selected in the fusion layer for combining encoded feature maps.
  • The decoder uses four convolutional layers to reconstruct the final fused image from the fusion-layer output.

A. Training

Training learns encoder–decoder weights to reconstruct input images before fusion, using a weighted pixel-and-SSIM loss. The strategy trains on grayscale MS-COCO images and then leaves the fusion layer available for task-specific design.

  • Training procedure: The training phase excludes the fusion layer and optimizes the encoder and decoder to reconstruct input images.After their weights are fixed, adaptive fusion is applied to encoder features.
  • Fusion design: Separating reconstruction training from fusion allows fusion layers to be designed for specific fusion tasks and leaves room for further development.The fixed encoder and decoder are paired with adaptive fusion during testing.
  • Network architecture: The encoder uses convolutional and dense blocks, while the decoder reconstructs the input through convolutional layers.The dense block connects each layer’s output to subsequent layers through cascade operations.
  • Training objective: The loss combines pixel loss and structural similarity loss as a weighted objective.Pixel loss measures Euclidean distance between output and input images, while SSIM loss measures structural similarity.
  • Training data: Grayscale MS-COCO images provide the training data because infrared and visible training data are insufficient.The training setup uses 80,000 resized 256 × 256 grayscale images, with learning rate 1 × 10^-4, batch size 2, and 4 epochs.

1) Addition Strategy:

The method combines encoder feature maps using addition or an l1-norm-based strategy before decoder reconstruction. The l1-norm strategy measures feature activity and applies block-based averaging to support salient feature selection.

  • Fusion strategies: Two fusion strategies—addition and l1-norm—combine salient feature maps produced by the encoder.
  • Feature maps: The network uses M = 64 feature maps, with k ≥2 indicating feature maps obtained from the input images.
  • Addition strategy: The addition strategy sums corresponding encoder feature maps across input images at each position.The resulting fused feature maps are passed to the decoder for image reconstruction.
  • l1-norm strategy: The l1-norm strategy replaces rough addition-based selection with l1-norm and soft-max operations.The paper presents this strategy as a new approach for salient feature selection.
  • l1-norm strategy: Feature activity is computed with an l1-norm, then refined using a block-based average operator to produce the final activity map.The refined activity map is used to calculate fused feature maps, which the decoder reconstructs into the final image.

IV. EXPERIMENTAL RESULTS AND ANALYSIS

The experiments evaluate the proposed fusion method with subjective and objective criteria and compare it with existing methods. The training and validation analyses examine loss behavior and reconstruction performance.

  • Experimental design: Training-phase loss and validation behavior are analyzed using the plots in Fig. 5 and Fig. 6.
  • Experimental design: The testing experiment evaluates the proposed fusion method using subjective and objective criteria and compares it with existing methods.
  • Experimental design: Additional fusion results for infrared and visible RGB images are reported in Section IV-D.

A. Training phase analysis

The training analysis uses MS-COCO images to assess reconstruction and examines how the SSIM loss weight λ affects convergence and validation losses. Larger λ values accelerate early convergence, while sufficiently long training reaches optimal weights regardless of λ.

  • Training data: About 79,000 MS-COCO images train the network, while 1,000 images validate reconstruction ability in each iteration.
  • Convergence: Increasing λ produces faster convergence during the first 2,000 iterations because SSIM loss receives greater training importance.The paper notes that pixel and SSIM losses differ by orders of magnitude.
  • Validation: At iteration 500, pixel loss and SSIM achieve better values with larger λ settings.
  • Convergence: After more than 40,000 iterations, the network obtains optimal weights regardless of the selected loss weight.

B. Experimental Settings

The evaluation uses 20 infrared-visible image pairs, compares DenseFuse with six established methods, and assesses fusion using seven quality metrics.

  • Dataset and baselines: The experiment uses 20 pairs of infrared and visible source images.A sample of the source images is shown in Fig. 7.
  • Dataset and baselines: DenseFuse is compared with CBF, JSR, GTF, JSRSD, CNN, and DeepFuse.DeepFuse uses a 3 × 3 filter size in this experiment.
  • Quality metrics: The evaluation reports entropy, Qabf, SCD, FMIw, FMIdct, SSIMa, and MS SSIM.These metrics measure information, correlation, mutual information, structural similarity, and no-reference fusion performance.
  • Quality metrics: SSIMa represents the fused image’s ability to preserve structural information.It is computed from structural similarity between the fused image and the two source images.
  • Quality metrics: Fusion performance improves as the numerical index of all seven metrics increases.The study evaluates this performance pattern across the reported quality measures.

C. Fusion methods Evaluation

DenseFuse is evaluated visually and quantitatively against existing fusion methods on car and street image pairs, with results favoring its detail preservation and reduced noise.

  • Visual evaluation: Because of space limits, relative visual performance is evaluated on two image pairs: “car” and “street”.Figures 8 and 9 show the fused outputs from six existing methods and the proposed method.
  • Visual evaluation: CBF, JSR, and JSRSD produce more artificial noise and less-clear salient features than the proposed method.Examples include sky, floor, and billboard regions in the visual comparisons.
  • Visual evaluation: The proposed method contains less noise in the highlighted red-box region across parameter choices.Compared with GTF, CNN, and DeepFuse, it also preserves more detailed information in that region.
  • Visual evaluation: The proposed fused images look more natural than CNN-based results, which appear darker on the street images.The authors attribute the darker CNN outputs probably to CNN-based fusion being unsuitable for infrared and visible images.
  • Quantitative evaluation: Human observers show almost no visual difference between DeepFuse and the proposed method.The authors therefore use objective metrics for the subsequent evaluation.
  • Quantitative evaluation: The proposed addition and l1-norm variants obtain five best average values among the seven quality metrics.Table II reports average values for 20 fused images, with best and second-best values marked separately.
  • Quantitative evaluation: The method achieves the best FMIdct and SSIMa values, indicating stronger preservation of structural information and features.The results also report second-best values for FMIw and MS SSIM.
  • Quantitative evaluation: DenseFuse has best or second-best values in seven quality metrics under both addition and l1-norm fusion strategies.The authors interpret this as evidence that the network is effective for infrared-visible image fusion.

D. Additional results for RGB images and infrared images

DenseFuse also fuses RGB visible images with infrared images by processing each RGB channel separately and recombining the three fused channels.

  • RGB-infrared fusion: The fusion algorithm extends beyond grayscale images to visible RGB and infrared image fusion.The RGB inputs were collected from a cited external dataset.
  • RGB-infrared fusion: Each RGB channel is treated as a grayscale image and paired with the infrared image for network processing.A fixed encoder, fusion layer, and decoder process each of the three channel pairs.
  • RGB-infrared fusion: The three separately fused channels are combined into one fused RGB image.The resulting RGB-infrared fusion outputs are shown in Fig. 11.

V. CONCLUSION

The paper concludes that DenseFuse is an effective CNN-and-dense-block architecture for infrared-visible fusion, with reported state-of-the-art performance and broader potential applications.

  • Conclusion: DenseFuse combines CNN layers and dense blocks for infrared and visible image fusion.The architecture is presented as a novel deep learning approach for this task.
  • Conclusion: The algorithm applies to both grayscale and RGB visible-image fusion with infrared images.RGB processing is illustrated through a dedicated fusion framework and fused-result figure.
  • Conclusion: The network contains an encoder, fusion layer, and decoder that extract, combine, and reconstruct image information.The encoder produces feature maps, fusion strategies integrate salient features, and the decoder reconstructs the fused image.
  • Conclusion: The experimental results show state-of-the-art fusion performance using subjective and objective quality metrics.The conclusion summarizes the reported evaluation rather than introducing a new metric.
  • Conclusion: Initial experiments suggest applicability to multi-focus, multi-exposure, and medical image fusion with an appropriate fusion layer.This is presented as an initial experimental observation rather than a fully established result.
Loading 1804.08361v9…