PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossFuse: A Novel Cross Attention Mechanism based Infrared and Visible Image Fusion Approach

Hui Li, Xiao-Jun Wu

arXiv:2406.10581v1cs.CV

TL;DR

Multimodal fusion must integrate infrared and visible information while preserving complementary features, yet transformer methods often emphasize correlation and overlook cross-modal complementarity. CrossFuse introduces a hybrid CNN-transformer network with a cross-attention mechanism and reports better fusion performance than state-of-the-art methods. The paper also acknowledges that the mechanism has limited ability to substantially improve performance within the transformer framework.

  • Problem

    Transformer-based fusion methods primarily rely on self-attention and overlook the complementary information and interplay between different modalities.

  • Method

    CrossFuse combines convolutional layers with self- and cross-attention, using a re-softmax operation to enhance complementary features and reduce redundant information.

  • Results

    CrossFuse demonstrates better fusion performance than state-of-the-art fusion methods in experiments on public datasets.

  • Takeaways & Limitations

    The proposed cross-attention mechanism provides a fusion strategy that emphasizes complementary information between modalities.

  • Takeaways & Limitations

    The cross-attention mechanism has limited ability to significantly enhance fusion performance within the transformer framework.

Abstract

from arXiv · show

Multimodal visual information fusion aims to integrate the multi-sensor data into a single image which contains more complementary information and less redundant features. However the complementary information is hard to extract, especially for infrared and visible images which contain big similarity gap between these two modalities. The common cross attention modules only consider the correlation, on the contrary, image fusion tasks need focus on complementarity (uncorrelation). Hence, in this paper, a novel cross attention mechanism (CAM) is proposed to enhance the complementary information. Furthermore, a two-stage training strategy based fusion scheme is presented to generate the fused images. For the first stage, two auto-encoder networks with same architecture are trained for each modality. Then, with the fixed encoders, the CAM and a decoder are trained in the second stage. With the trained CAM, features extracted from two modalities are integrated into one fused feature in which the complementary information is enhanced and the redundant features are reduced. Finally, the fused image can be generated by the trained decoder. The experimental results illustrate that our proposed fusion method obtains the SOTA fusion performance compared with the existing fusion networks. The codes are available at https://github.com/hli1221/CrossFuse

1. Introduction

Multimodal image fusion combines information from different imaging modalities, but transformer-based methods often emphasize correlation through self-attention while overlooking complementary information between modalities. CrossFuse addresses this gap with a hybrid CNN-attention fusion strategy and reports stronger results than existing fusion methods.

  • Multimodal image fusion combines information from modalities such as visible and infrared images to improve visual quality and information reliability.
  • Deep-learning fusion methods include multi-stage pipelines that separately process inputs and end-to-end models that directly generate fused images.Multi-stage methods are flexible but can be computationally intensive and lose information during processing.
  • Transformer-based fusion methods primarily use self-attention to capture global dependencies but often ignore interactions between modalities and their complementary information.Self-attention can improve correlation while reducing complementary information in multimodal fusion.
  • CrossFuse introduces a cross-attention mechanism that enhances inter-modal complementary features while self-attention strengthens intra-modal features.The method combines convolutional layers with self- and cross-attention mechanisms to extract deep features, preserve details, and enhance complementarity.
  • Compared with state-of-the-art fusion methods, CrossFuse is reported as a promising, robust, and efficient alternative for multimodal image fusion.

2. Related works

Transformer-based fusion methods use attention to model feature relationships, while cross-attention integrates information across modalities. However, existing approaches may neglect modality relations or complementary information, motivating mechanisms that preserve cross-modal complementarity.

  • Transformer based fusion methods: Transformers use self-attention to weigh input elements and capture global feature dependencies for image fusion.
  • Transformer based fusion methods: Existing transformer-based fusion methods may use transformers only for feature extraction or reconstruction, without considering relations between modalities.
  • Transformer based fusion methods: Self-attention-based fusion can reduce complementary information even when it is used during feature fusion.
  • Cross-attention based fusion methods: Cross-attention focuses on interactions between different modalities and is used to integrate multimodal information in fusion tasks.
  • Cross-attention based fusion methods: Existing cross-attention fusion methods primarily model correlation and therefore may ignore complementary information.
  • Cross-attention based fusion methods: Designing cross-attention that preserves complementary information remains an important unresolved issue for image fusion.

3. The proposed method

CrossFuse combines modality-specific encoders, a complementary-focused cross-attention mechanism, and a decoder in a two-stage fusion framework. Skip connections and intensity-aware feature processing preserve salient and detailed information from infrared and visible images.

  • Overall architecture: The framework uses two encoders to extract infrared and visible features, CAM to fuse them, and a decoder to generate the fused image.The encoders share an architecture but use different parameters for the two modalities.
  • Encoder architecture: Each encoder uses convolution, pooling, and DenseBlock components to extract shallow texture, multiscale, and deeper salient features.DenseBlock contains four densely connected convolutional layers.
  • Cross-attention mechanism: CAM applies self-attention, shift operations, and cross-attention with re-softmax to enhance inter-modality uncorrelation rather than redundant correlation.The architecture contains twice as many self-attention blocks as cross-attention blocks.
  • Decoder architecture: The decoder uses convolutional layers, up-sampling, skip connections, and feature-intensity-aware processing to reconstruct the fused image while preserving salient and detailed information.Deep connections target salient features, while shallow connections target detail information.
  • Two-stage training: Training first reconstructs each modality with separate auto-encoders, then fixes their encoders while training CAM and the decoder on multimodal data.The first-stage auto-encoders use pixel-level and structural similarity losses; the second-stage loss combines intensity and gradient terms.

4. Experimental validation

The experiments evaluate fusion performance through ablation studies and objective metrics. The network is implemented in PyTorch on an NVIDIA GTX 3090Ti GPU.

  • Evaluation design: Ablation studies investigate the effects of different elements in the proposed fusion network.The evaluation uses several performance metrics to assess fusion performance objectively.
  • Implementation: The network is implemented using PyTorch on an NVIDIA GPU (GTX 3090Ti).

4.1. Experimental settings

Training uses KAIST image pairs, while testing uses TNO and VOT-RGBT with different scene characteristics. Existing fusion methods and six objective metrics provide comparison baselines.

  • Training data: The first training stage uses 40000 KAIST infrared-visible pairs, and the second uses 20000 pairs.The two stages use 4 and 8 epochs, respectively, with batch sizes of 2 and 8.
  • Test datasets: Testing uses 21 image pairs from TNO and 40 pairs from VOT-RGBT.TNO contains more intricate scenarios, whereas VOT-RGBT focuses primarily on street scenes with smaller salient targets.
  • Test datasets: Examples of the TNO and VOT-RGBT datasets are presented in Figure 9.
  • Compared methods: The experiments compare the proposed method with GAN-, CNN-, dense-connection-, transformer-, downstream-task-, semantic-, and diffusion-based fusion methods.
  • Evaluation metrics: Objective evaluation uses Entropy, Standard Deviation, Mutual Information, feature-based Mutual Information, and SCD.The feature-based Mutual Information measures are FMIdct and FMIpixel.

4.2. Ablation study

Ablation studies show that the proposed CAM design, its re-softmax and shift operations, combined loss, and two-stage training improve complementary-feature preservation and fusion quality.

  • Attention-block settings: The one-block configuration s1-c1 obtains better EN, SD, and MI values than the two- and three-block configurations.Its visual result contains more detail and less artificially generated noise around the salient object.
  • Attention operations: Removing re-softmax and shift decreases detail and salient-object intensity, while the complete scheme achieves superior En, SD, and MI values.Re-softmax focuses cross attention on complementary information, whereas shift enhances intra-features.
  • Fusion module: Compared with CNN and dense fusion modules, CAM produces more salient features, fewer artifacts, and more natural fused images.The corresponding architectures and middle-feature visualizations are shown in Figures 11 and 12.
  • CAM feature behavior: Self attention retains salient regions and enriches modality-specific details, while cross attention amplifies complementary infrared and visible regions.The final addition operation produces enhanced features through CAM.
  • Loss function: Removing the gradient term reduces detail, while removing the intensity term decreases salient-object intensity.The combined loss preserves both detailed information and salient pixel intensity.
  • Training strategy: The two-stage strategy converges faster, reaches a smaller loss value, and attains the three highest metric values compared with one-stage training.The two-stage approach trains the encoders separately before training CAM and the decoder together.

4.3. Fusion results analysis

CrossFuse is evaluated visually and objectively on TNO and VOT-RGBT against existing fusion methods. It preserves complementary information, detail, and salient objects with strong or comparable metric performance.

  • TNO results: On TNO, CrossFuse produces more detail than FusionGAN, IFCNN, and U2Fusion, and clearer images than the compared transformer and downstream-task methods.Visual comparisons use the “yard” and “man” examples in Figure 15.
  • TNO results: On TNO, CrossFuse achieves four best values—EN, SD, MI, and FMIdct—and two second-best values—FMIpixel and SCD.The authors associate these results with preserving complementary information at pixel and feature levels.
  • VOT-RGBT results: On VOT-RGBT, CrossFuse enhances infrared objects compared with IFCNN, YDTR, U2Fusion, and DATFuse.The visual examples are “crossroad” and “two-man”; visible images are processed through luminance and chrominance components.
  • VOT-RGBT results: CrossFuse preserves more detail than FusionGAN, IRFS, SemLA, and DDFM while obtaining comparable salient objects on VOT-RGBT.
  • VOT-RGBT results: On VOT-RGBT, CrossFuse obtains the best SD and FMIdct values and second-best EN, MI, and FMIpixel values.The method does not achieve all best values but retains comparable metrics and better sharpness measured by SD.

5. Conclusions

CrossFuse addresses the need to emphasize complementary information in multimodal image fusion by introducing a cross attention mechanism that reduces redundancy. Experiments report better fusion performance than state-of-the-art methods, while the mechanism has limited ability to significantly improve transformer-based fusion.

  • CrossFuse introduces a hybrid CNN-transformer fusion network with a novel cross attention mechanism focused on complementary information.
  • CAM uses a re-softmax operation to enhance complementary features between modalities and reduce redundant information.
  • Experiments on public datasets show better fusion performance than state-of-the-art fusion methods.
  • The cross-attention mechanism has limited ability to significantly enhance fusion performance within the transformer framework.
Loading 2406.10581v1…