PaperlayerView exact revision on arXiv

Source-linked AI summary

TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

Dongyu Rao, Xiao-Jun Wu, Tianyang Xu

arXiv:2201.10147v2cs.CV

TL;DR

Existing CNN-based fusion methods aggregate local multi-modal appearance but neglect long-range dependencies, limiting global image-level perception in complex scenarios. The paper combines lightweight spatial and channel transformers with adversarial learning, and reports outstanding subjective and objective performance, including the best results on seven of nine indicators and third place on the other two.

  • Problem

    Existing end-to-end CNN fusion approaches neglect long-range dependencies while aggregating multi-modal local appearance, limiting global image-level perception in complex scenarios.

  • Method

    The method uses CNN-extracted features with lightweight spatial and cross-channel transformers for global fusion learning, plus two-modality adversarial training.

  • Results

    The method reports outstanding subjective and objective performance, ranking first on seven of nine evaluation indicators and third on the remaining two.

  • Takeaways & Limitations

    The paper presents transformer and adversarial learning as a combined paradigm for infrared and visible image fusion.

Abstract

from arXiv · show

The end-to-end image fusion framework has achieved promising performance, with dedicated convolutional networks aggregating the multi-modal local appearance. However, long-range dependencies are directly neglected in existing CNN fusion approaches, impeding balancing the entire image-level perception for complex scenario fusion. In this paper, therefore, we propose an infrared and visible image fusion algorithm based on a lightweight transformer module and adversarial learning. Inspired by the global interaction power, we use the transformer technique to learn the effective global fusion relations. In particular, shallow features extracted by CNN are interacted in the proposed transformer fusion module to refine the fusion relationship within the spatial scope and across channels simultaneously. Besides, adversarial learning is designed in the training process to improve the output discrimination via imposing competitive consistency from the inputs, reflecting the specific characteristics in infrared and visible images. The experimental performance demonstrates the effectiveness of the proposed modules, with superior improvement against the state-of-the-art, generalising a novel paradigm via transformer and adversarial learning in the fusion task.

I. INTRODUCTION

Infrared and visible image fusion supports information association across modalities, but CNN-based end-to-end methods face limitations in global dependence and effective integration. The paper proposes lightweight transformer modules and adversarial learning to learn spatial and channel fusion relationships and incorporate modality-specific characteristics.

  • Infrared and visible image fusion supports applications including military, security, detection, and visual tracking.
  • End-to-end fusion methods target specific tasks through network structures and loss functions but require strong representation and fusion capabilities.
  • The proposed algorithm combines a lightweight transformer with adversarial learning to address global dependence and effective integration.
  • A channel-token transformer explores channel relationships, while a transformer module learns global fusion relationships in complex scenarios.
  • Adversarial learning introduces characteristics from the two modalities into the fused image to improve the fusion effect.

II. RELATED WORK

Deep learning has become prominent in image fusion, with end-to-end CNN methods achieving strong single-task performance. The related work also identifies spatial limitations in CNN-based fusion and reviews GAN and transformer developments relevant to the proposed approach.

  • Deep learning fusion methods have shown excellent performance in infrared-visible, multi-focus, and medical image fusion.
  • End-to-end CNN fusion methods achieve better performance on single tasks but remain limited by the spatial scope of convolutional fusion.

B. Generative Adversarial Network

The paper introduces adversarial learning to improve infrared and visible image fusion by encouraging competitive consistency from the source images. Its transformer-based design also models global spatial and channel relationships during fusion.

  • A generative adversarial network trains a generator and discriminator against each other to produce high-quality generated images.
  • Adversarial learning improves infrared and visible characteristics in the fused result by obtaining competitive consistency from the inputs.
  • The proposed transformer fusion module focuses on global image correlations across both spatial and channel dimensions.
  • Compared with a general transformer, the proposed transformer fusion module is lightweight and explores channel-relationship modeling for image fusion.

III. PROPOSED METHOD

The proposed method combines a transformer-based generator with two discriminators to create and refine infrared-visible fused images. CNN features are transformed to learn global fusion relations, while adversarial training improves perceptual quality.

  • The model contains one transformer-based generator and two discriminators that refine the generator’s fused-image output through adversarial learning.
  • The generator concatenates source images, extracts mixed CNN features, and feeds them into a transformer fusion module for global fusion-relation learning.
  • Three downsampling operators precede the transformer fusion module to balance computational-resource consumption and feature representation.
  • The two discriminators compare the fused image with infrared and visible images to provide infrared significance and visible high-resolution details.
  • Training alternates generator and discriminator updates, feeding discriminator feedback through the loss function to obtain the final fusion generator.

B. The Transformer Fusion Module

The transformer fusion module combines spatial and cross-channel transformers to learn global relationships for infrared and visible image fusion. The channel transformer models channel correlations, while the spatial transformer models relationships between image patches.

  • The composite transformer fusion module combines spatial and cross-channel transformers to learn global spatial and channel relationships.
  • Spatial Transformer: The spatial transformer divides images into patches, converts them into vectors, and learns global relationships between image patches.Position embedding is removed, and the restored relationship map has one channel.
  • Channel Transformer: The channel transformer uses image channels as tokens to learn correlations across the channel dimension.Position embedding is removed, making the input image size more flexible.
  • Composite Transformer: Experiments find that applying the channel transformer before the spatial transformer achieves better results.This ordering learns coefficients more suitable for infrared and visible image fusion.

C. Loss Function

The method uses a variance-guided SSIM-based loss to coordinate structural consistency between the fused image and the more contrasted source image. This single loss avoids conflicts among multiple loss functions.

  • A single SSIM-based loss is used to avoid mutual conflict and entanglement among multiple loss functions.The authors report that this single loss achieves a good fusion effect.
  • SSIM measures structural similarity using image means, standard deviations, covariance, and stability coefficients.
  • The reference image is selected from the infrared and visible inputs according to which has higher variance.Variance is treated as an image-contrast measure.
  • The variance-guided SSIM loss divides images into 11×11 sliding-window blocks to coordinate consistency across image regions.

A. Setup

The experiments train on paired infrared and visible pedestrian images from KAIST and test on ten image pairs from a separate source. The setup compares the proposed method with 15 fusion methods using subjective and objective evaluation.

  • 40,000 paired infrared and visible images from KAIST are used for training.The images cover campus, street, and countryside scenes, with training patches sized 256×256 pixels.
  • Testing uses 10 image pairs from the test images of, with arbitrary image sizes generally not exceeding 2048×2048 pixels.
  • The model is trained with Adam at a constant learning rate of 0.0001, batch size 16, and 20 epochs.The spatial and channel transformer patch sizes are 4 and 16, respectively.
  • The proposed method is compared with 15 classic and recent methods in subjective and objective evaluations.

B. Results Analysis

Subjective evaluation reports that deep-learning fusion methods better preserve visible environmental details while retaining salient infrared information. The proposed method is specifically reported to preserve visible door details and high-resolution background scenes while highlighting an infrared person.

  • Subjective evaluation judges fused images by human visual perception, including clarity and salient information.
  • Deep-learning methods generally preserve visible environmental information and salient infrared information better than methods introducing artificial noise.
  • The experiments include evaluations of GAN use, transformer fusion variants, position embedding, transformer encoder layers, CNN layers, and channel counts.
  • The proposed method highlights the infrared person while maintaining visible details of the door.
  • The fused image retains high-resolution visible background scenery and is described as friendly for human visual information intake.
  • Objective evaluation uses nine indicators: SF, EN, Qabf, FMIw, MS-SSIM, FMIpixel, SD, VIF, and MI.

C. Ablation Study

The ablation study evaluates adversarial learning, transformer configurations, position embedding, transformer depth, and CNN feature extraction settings. Results support using GAN-based modality discrimination and a balanced transformer depth for fusion.

  • Adversarial Learning: Compared with no adversarial training, adding GAN improved seven objective indicators.The GAN uses a pre-trained VGG-16 discriminator to enhance different modality characteristics at the feature level.
  • Transformer Fusion Module: The transformer fusion module can use spatial and channel transformers separately or together, with channel-first processing performing better.The reported comparison evaluates the two modules alone and in combination.
  • Position Embedding: Removing position embedding had a positive effect on fusion results.The model was trained and compared with and without position embedding.
  • Transformer Module Layers: Three transformer layers produced a meaningless black image, while five layers worsened results, suggesting an intermediate depth was selected.The paper attributes the three-layer failure to insufficient fusion-relation learning and the five-layer degradation to redundant learned relationships.
  • CNN Layers: CNN depth affects feature granularity and depth because additional layers increase image downsampling, potentially preventing effective fusion-relation learning.The study varies the number of Res-Blocks to identify a suitable CNN configuration.
  • CNN Channels: The number of feature channels changes the distribution of channel-dimension information and is evaluated as an important performance factor.The ablation compares several typical channel counts across image features with matching dimensions but different scales.

V. CONCLUSION

The paper presents an infrared and visible image fusion method combining a lightweight transformer module with generative adversarial learning. It reports outstanding subjective and objective evaluation performance, supporting the effectiveness and advancement of the approach.

  • V. CONCLUSION: The method deeply integrates a lightweight transformer as a fusion-relation learning module with adversarial learning for feature-level modality characteristics.The paper describes this as the first deep combination of transformer and adversarial learning for image fusion.
  • V. CONCLUSION: The method achieved outstanding subjective and objective evaluation performance, supporting the reported effectiveness and advancement of the approach.
Loading 2201.10147v2…