PaperlayerView exact revision on arXiv

Source-linked AI summary

SwinFuse: A Residual Swin Transformer Fusion Network for Infrared and Visible Images

Zhishe Wang, Yanlin Chen, Wenyu Shao, Hui Li, Lei Zhang

arXiv:2204.11436v1cs.CV

TL;DR

Infrared and visible fusion must reconcile complementary thermal-target and visible-detail information, while existing convolutional methods may lose important context. SwinFuse uses a pure residual Swin Transformer backbone with row- and column-based L1-norm sequence fusion, and experiments report strong fusion performance, generalization, and computational efficiency.

  • Problem

    Existing deep-learning fusion methods mainly use convolutional networks, whose content-independent interactions and limited receptive fields may lose important contexts.

  • Method

    SwinFuse combines fully attentional residual Swin Transformer feature extraction, L1-norm sequence-matrix fusion across row and column dimensions, and feature reconstruction.

  • Results

    Experiments on TNO, Roadscene, and OTCBVS against nine traditional and deep-learning methods report remarkable fusion performance, strong generalization ability, and competitive computational efficiency.

  • Takeaways & Limitations

    SwinFuse retains competitive infrared brightness and distinct visible details while serving as a strong fusion baseline across evaluated datasets.

Abstract

from arXiv · show

The existing deep learning fusion methods mainly concentrate on the convolutional neural networks, and few attempts are made with transformer. Meanwhile, the convolutional operation is a content-independent interaction between the image and convolution kernel, which may lose some important contexts and further limit fusion performance. Towards this end, we present a simple and strong fusion baseline for infrared and visible images, namely\textit{ Residual Swin Transformer Fusion Network}, termed as SwinFuse. Our SwinFuse includes three parts: the global feature extraction, fusion layer and feature reconstruction. In particular, we build a fully attentional feature encoding backbone to model the long-range dependency, which is a pure transformer network and has a stronger representation ability compared with the convolutional neural networks. Moreover, we design a novel feature fusion strategy based on $L_{1}$-norm for sequence matrices, and measure the corresponding activity levels from row and column vector dimensions, which can well retain competitive infrared brightness and distinct visible details. Finally, we testify our SwinFuse with nine state-of-the-art traditional and deep learning methods on three different datasets through subjective observations and objective comparisons, and the experimental results manifest that the proposed SwinFuse obtains surprising fusion performance with strong generalization ability and competitive computational efficiency. The code will be available at https://github.com/Zhishe-Wang/SwinFuse.

I. INTRODUCTION

Infrared and visible sensors provide complementary strengths, motivating fusion methods that preserve thermal targets alongside visible texture. SwinFuse addresses limitations of convolutional approaches with a pure-transformer backbone and an L1-norm fusion strategy.

  • Infrared imaging detects camouflaged targets in varied conditions but lacks typical background details and structural texture.
  • Visible imaging captures scene details, color, and texture but is vulnerable to weather and lighting variations and may miss prominent targets.
  • Fusion combines complementary sensor features to produce images more consistent with human perception and support subsequent visual tasks.
  • Existing methods extract and combine features using fixed representations or learnable convolutional filters, but convolution is content-independent and limited in receptive field.
  • SwinFuse uses residual Swin Transformer blocks to model long-range dependencies with fully attentional feature interactions and residual information retention.
  • Its L1-norm sequence-matrix fusion measures activity across row and column vectors to retain infrared brightness and visible background details.
  • Experiments against nine traditional and deep-learning methods report strong subjective and objective performance, generalization, and computational efficiency.

II. RELATED WORK

Related fusion research progressed from handcrafted representations to convolutional and adversarial deep-learning models, while transformer use remained limited. SwinFuse differs by using a pure transformer backbone rather than treating transformers as a CNN supplement.

  • Transformers moved from machine translation to vision, with self-attention modeling long-range dependencies in image patch sequences.
  • Swin Transformer uses local windows and shifted operations to connect cross-window information while keeping computational complexity linear with image size.
  • Few studies had extended transformers to image fusion before this work.
  • Deep-learning fusion methods commonly use convolutional encoders, decoders, and manually designed or specialized fusion rules, but often omit long-range dependency modeling.
  • GAN-based methods achieved satisfactory results to some extent, but single-discriminator FusionGAN results were biased toward infrared images and lacked visible texture details.
  • Unlike CNN-dominant methods and hybrid TransMEF, SwinFuse introduces a pure transformer encoding backbone without convolutional neural networks.

III. METHOD

SwinFuse extracts global features, fuses them, and reconstructs an image. Its method combines residual Swin Transformer processing with row- and column-based L1-norm normalization.

  • Network Overview: A 1×1 convolution performs positional encoding and maps input channels from Cin to C, producing initial features Φl.
  • Network Overview: Initial features are reshaped into sequence vectors and processed by residual Swin Transformer blocks to extract global features.
  • Fusion Strategy: The fusion layer applies L1-norm operations across row and column vector dimensions to combine infrared and visible global features.
  • Network Overview: SwinFuse contains global feature extraction, a fusion layer, and feature reconstruction; the fusion layer is removed during training.
  • Network Overview: Fused global features are reshaped and reconstructed with a 1×1 convolution followed by Tanh activation to produce the fusion image.

B. Residual Swin Transformer Block

The residual Swin Transformer block stacks Swin Transformer layers with residual connections to extract global features while aggregating information across feature levels. Its layers combine local-window and shifted-window attention with MLP processing and residual modules.

  • B. Residual Swin Transformer Block: RSTB combines a series of Swin Transformer layers with a residual connection for feature extraction.The residual pathway supports low-level feature aggregation and information retention.
  • B. Residual Swin Transformer Block: Each Swin Transformer layer partitions features into non-overlapping N×N windows and computes local attention.The windows contain HW/N^2 local regions.
  • B. Residual Swin Transformer Block: The attention pipeline uses learnable Q, K, and V projections, followed by MLP processing with GELU nonlinearity.LayerNorm precedes attention and MLP modules, each of which uses a residual connection.
  • B. Residual Swin Transformer Block: Shifted windows bridge information across neighboring regions, enabling the architecture to model broader dependencies.The layer alternates regular-window and shifted-window multi-head self-attention.

C. Fusion Strategy

SwinFuse fuses infrared and visible global features using L1-norm activity measurements computed along both row and column dimensions. The resulting features are added, reconstructed into an image, and the fusion layer is used only during testing.

  • C. Fusion Strategy: The fusion strategy measures infrared and visible sequence-matrix activity levels with L1-norms along row and column dimensions.Softmax converts the L1-based row and column weights into activity levels.
  • C. Fusion Strategy: Row activity levels weight the corresponding global features to produce row-dimension fused features.The same weighting principle is separately applied to infrared and visible feature matrices.
  • C. Fusion Strategy: Column activity levels are computed analogously and used to obtain column-dimension fused global features.The column operation complements the row-dimension fusion.
  • C. Fusion Strategy: Element-wise addition combines the row- and column-dimension fused features into the final fused global representation.A convolutional layer then reconstructs the fusion image, while the fusion layer is removed during training and retained during testing.

D. Loss Function

SwinFuse trains with SSIM and L1 losses: SSIM supervises structural information, while L1 compensates for SSIM’s sensitivity to color and brightness variation. Their weighted sum forms the total loss.

  • D. Loss Function: SSIM and L1 are used together as training losses to supervise network optimization.They are defined as separate components in the loss formulation.
  • D. Loss Function: SSIM reflects scene details and structural texture but is prone to color deviation and brightness variation.The paper describes SSIM as independent of image brightness and contrast.
  • D. Loss Function: L1 loss compensates for SSIM’s shortcomings, and λ adjusts the difference in magnitude between the two loss terms.The total loss is defined as Ll1 + λLssim.

IV. EXPERIMENTS AND ANALYSES

The experiments section presents the experimental setup and then discusses and analyzes the relevant experiments.

  • IV. EXPERIMENTS AND ANALYSES: The section first introduces the experimental setup.
  • IV. EXPERIMENTS AND ANALYSES: It then focuses on discussion of the relevant experiments.
  • IV. EXPERIMENTS AND ANALYSES: The section also analyzes the relevant experimental results.

A. Experimental Setup

The experiments train SwinFuse on MS-COCO and evaluate it on TNO, Roadscene, and OTCBVS against nine representative methods using subjective, objective, and ablation comparisons.

  • Training and testing setup: SwinFuse is trained on more than 80000 grayscale MS-COCO images resized to 224×224 with three RSTBs and six STLs.Training uses Adam for 50 epochs with learning rate 1×10^-5 and batch size 4.
  • Evaluation protocol: The evaluation compares SwinFuse with nine traditional and deep-learning methods using eight objective evaluation indexes.The listed baselines include MDLatLRR, IFCNN, DenseFuse, RFN-Nest, FusionGAN, GANMcC, PMGI, SEDRFuse, and Res2Fusion.
  • Ablation studies: The loss-function ablation varies λ from 1e0 to 1e4 to examine its effect on fusion performance using TNO and eight evaluation indexes.Red and blue labels identify optimal and suboptimal average index values.
  • Ablation studies: λ=1e3 achieves optimal SF, SD, MI, MS SSIM, FMI W, and SCD values, while AG and VIFF are suboptimal, establishing the setting used subsequently.The study reports λ=1e3 as the best overall ablation setting.

2) The impact of RSTL framework:

The RSTL ablation examines architectural depth, residual connections, and row- or column-only fusion, then compares SwinFuse with nine methods on TNO.

  • RSTL ablation: The ablation varies the numbers of RSTBs m∈{2,3,4,5} and STLs n∈{5,6,7,8}, alongside residual-connection and fusion-strategy choices.The alternatives include without residual connection, Only row, and Only col.
  • RSTL ablation: Only-row and Only-col fusion produce relatively poor, low-contrast visual results, whereas m=3 and n=6 retain conspicuous infrared targets and unambiguous visible details.Visual differences among tested RSTB, STL, and no-residual settings are described as inconspicuous.
  • RSTL ablation: With m=3, SwinFuse obtains optimal AG, SD, MS SSIM, FMI w, SCD, and VIFF values, with suboptimal SF and MI values.These objective results identify three RSTBs as the better tested setting.
  • RSTL ablation: With n=6, SwinFuse obtains optimal AG, SD, MI, FMI w, and VIFF values, with suboptimal SF, MS SSIM, and SCD values.These results identify six STLs as the better tested setting.
  • TNO comparison: Across TNO examples, SwinFuse retains high-brightness infrared targets and unambiguous visible details, while several baselines favor one modality or blur details.The comparison covers pedestrians, roof structure, trees, and trench edges.
  • TNO comparison: On TNO, SwinFuse ranks first for AG, SF, SD, MS SSIM, SCD, and VIFF, and second for MI and FMI w among the compared methods.The objective comparison is reported as consistent with the subjective analysis.

D. Experiments on Roadscene Dataset

On Roadscene, SwinFuse is compared visually and objectively with nine methods, showing stronger retention of pedestrian brightness and scene details while balancing multiple fusion indexes.

  • Subjective comparisons: On FLIR 08835 and FLIR 08094, SwinFuse produces brighter pedestrian targets and complete, clear street-lamp and car details than the other nine methods.Figures 10 and 11 provide the subjective comparisons.
  • Objective comparisons: The Roadscene objective comparison evaluates eight indexes across the nine competing methods, representing SwinFuse with a red dotted line.Figure 12 summarizes the index comparisons.
  • Objective comparisons: The largest SCD and FMI w indicate stronger feature retention, while the largest SF and MS SSIM indicate abundant structural texture and edge details.These interpretations are stated for the fused results in the multi-index analysis.
  • Objective comparisons: The largest AG, SD, and VIFF indicate higher definition and contrast, attributed in the discussion to attentional maps and the L1-norm fusion strategy.The method is described as retaining infrared brightness and visible texture details.
  • Objective comparisons: Although MI is only competitive, the multi-index evaluation reports SwinFuse as achieving the best fusion performance.The discussion associates the MI tradeoff with row- and column-vector normalization.

E. Experiments on OTCBVS Dataset

On OTCBVS, SwinFuse shows strong subjective and objective fusion results across 31 OSU color-thermal image pairs, while maintaining competitive computational efficiency.

  • Subjective and objective comparisons: On video 1036, SwinFuse provides better pedestrian intensity distribution, clear target edges, and more realistic parking-lock scene detail than other methods.The subjective comparison is drawn from the OSU color-thermal database.
  • Subjective and objective comparisons: On OTCBVS, SwinFuse ranks first for AG, SF, SD, MS SSIM, SCD, and VIFF, and ranks second and third for MI and FMI w, respectively.The dataset uses 31 selected image pairs.
  • Subjective and objective comparisons: Across TNO, Roadscene, and OTCBVS, SwinFuse’s optimal indexes are described as almost consistent, with fusion performance superior to the nine compared methods.This is the paper’s reported cross-dataset experimental conclusion.
  • Computational efficiency: SwinFuse’s computational efficiency follows IFCNN, DenseFuse, and RFN-Nest, whose convolutional architectures and average-addition fusion strategies are simpler.Table III reports efficiency comparisons in seconds on the three datasets.
  • Computational efficiency: The authors conclude that SwinFuse combines better fusion performance, stronger generalization ability, and competitive computational efficiency.The conclusion presents it as a simple and strong fusion baseline.
Loading 2204.11436v1…