Source-linked AI summary
Image Fusion Transformer
Vibashan VS, Jeya Maria Jose Valanarasu, Poojan Oza, Vishal M. Patel
TL;DR
Image fusion must combine complementary sensor information, yet CNN-based methods largely model local features and miss long-range dependencies. IFT introduces multi-scale Spatio-Transformer fusion with CNN and transformer branches, and reports better results than existing methods across multiple benchmark datasets, with ablation evidence for the strategy.
Problem
CNN-based image-fusion methods improve generalization through local-feature learning but do not capture long-range dependencies and global context.
Method
IFT extracts multi-scale deep features and fuses them with a Spatio-Transformer network combining spatial CNN and transformer branches for local and long-range information.
Results
IFT achieves better results than existing fusion methods across multiple benchmark datasets, and ablation analysis shows the effectiveness of extracting local and long-range information.
Takeaways & Limitations
Integrating local and long-range features enables IFT to preserve global contextual information while reconstructing fine details in fused images.
Takeaways & Limitations
Traditional fusion methods remain constrained by poor generalization, time-consuming dictionary learning, and source-specific strategies.
Abstract
from arXiv · showhide
In image fusion, images obtained from different sensors are fused to generate a single image with enhanced information. In recent years, state-of-the-art methods have adopted Convolution Neural Networks (CNNs) to encode meaningful features for image fusion. Specifically, CNN-based methods perform image fusion by fusing local features. However, they do not consider long-range dependencies that are present in the image. Transformer-based models are designed to overcome this by modeling the long-range dependencies with the help of self-attention mechanism. This motivates us to propose a novel Image Fusion Transformer (IFT) where we develop a transformer-based multi-scale fusion strategy that attends to both local and long-range information (or global context). The proposed method follows a two-stage training approach. In the first stage, we train an auto-encoder to extract deep features at multiple scales. In the second stage, multi-scale features are fused using a Spatio-Transformer (ST) fusion strategy. The ST fusion blocks are comprised of a CNN and a transformer branch which capture local and long-range features, respectively. Extensive experiments on multiple benchmark datasets show that the proposed method performs better than many competitive fusion algorithms. Furthermore, we show the effectiveness of the proposed ST fusion strategy with an ablation analysis. The source code is available at: https://github.com/Vibashan/Image-Fusion-Transformer.
1. INTRODUCTION
Image fusion combines complementary information from different sensors, but existing CNN-based methods miss long-range dependencies. IFT addresses this gap with multi-scale fusion that integrates local and global information.
- Motivation: Visible and thermal sensors provide complementary information: visible images preserve texture and colour, while thermal images distinguish objects from backgrounds in poor lighting.The paper notes visible sensors operate at 300-530 µm and thermal sensors at 8-14 µm.
- Limitations of prior methods: Traditional fusion methods have poor generalization, time-consuming dictionary learning, and source-specific fusion strategies.These shortcomings are attributed to handcrafted feature extraction and differing source-image characteristics.
- Motivation: CNN-based fusion improves generalization by learning local features but fails to extract long-range dependencies and essential global context.The paper motivates combining local features with long-range information to improve fusion performance.
- Proposed approach: IFT uses a multi-scale Spatio-Transformer strategy to learn local features and long-range information for fusing complementary source-image information.The proposed strategy is presented as the central response to the limitations of existing image-fusion methods.
- Contributions: IFT employs a spatial CNN branch and a transformer branch to use local and global features, and achieves competitive results on multiple benchmark datasets.The contribution statement also identifies the method as a response to limited global contextual understanding in recent fusion work.
2. RELATED WORKS
Related image-fusion methods evolved from handcrafted and source-specific techniques toward deep learning, but existing approaches still primarily model spatially local features. Transformer architectures provide a basis for incorporating long-range information.
- Traditional methods: Traditional methods use transforms, sparse representation, principal component analysis, and related feature extraction techniques, but lack generalizability.Traditional strategies may also be source-specific because different image sources require different fusion procedures.
- Deep learning methods: Deep learning methods improved computer-vision performance relative to traditional feature-extraction approaches.The supplied related-work passage introduces deep learning as a promising direction after discussing traditional limitations.
- Remaining gap: Existing image-fusion methods focus on spatial local features between source images and do not consider long-range dependencies within them.The paper positions extracting long-range features alongside local features as a way to enhance fusion quality.
- Transformer background: Transformer models are motivated by their ability to capture long-range information more effectively than recurrent neural networks and CNNs.This motivation underlies the paper’s use of transformer-based modeling for image fusion.
3. PROPOSED METHOD
IFT extracts multi-scale features and fuses them through a Spatio-Transformer combining CNN-based local features with transformer-based long-range information. Its training objective combines feature similarity and structural similarity losses to preserve salient structures and fine details.
- 3.1. Image Fusion Transformer (IFT): IFT uses an encoder, multi-scale Spatio-Transformer fusion network, and nested decoder to generate a fused image from source images.The encoder extracts features at multiple scales, which are fused before decoding.
- 3.2. Self-attention and axial-attention: Axial attention applies self-attention separately along the height and width axes, reducing the computational complexity of full self-attention.Learnable positional embeddings make the affinities sensitive to positional information.
- 3.3. Spatio-Transformer fusion: The ST fusion block adds spatial and transformer features to produce a fused feature map containing enhanced local and global-context information.ST fusion is applied at multiple scales before the fused features are forwarded to the decoder.
- 3.2. Self-attention and axial-attention: The spatial branch uses convolutional layers and a bottleneck to capture local features, while the transformer branch uses axial attention for long-range dependencies.The two branches provide complementary local and global-contextual features.
- 3.4. Loss function: IFT training combines feature similarity loss with structural similarity loss through Lfuse = Lfeat + αLdet.The structural term uses SSIM, while the feature term constrains fused deep features to preserve salient structures and fine details.
4. EXPERIMENTS AND RESULTS
Experiments evaluate IFT on infrared/visible and MRI/PET fusion benchmarks, with qualitative comparisons and an ST-fusion ablation. IFT combines local and long-range information, producing stronger reported fusion quality than competing approaches.
- Experimental setup: IFT is trained on 80,000 KAIST visible/infrared pairs and tested on 21 TNO pairs.MRI/PET experiments use 9,981 cropped training patches and 20 test pairs from Harvard datasets.
- Infrared/visible fusion: IFT assigns consistent intensity across humans while reconstructing fine details in infrared/visible fusion.The qualitative comparison attributes these properties to capturing both long-range and local information.
- MRI/PET fusion: IFT outperforms existing methods on Entropy and CC for MRI/PET fusion while remaining competitive on SD and MG.The reported metrics are Entropy, Standard Deviation (SD), Correlation Coefficient (CC), and Mean Gradient (MG).
- MRI/PET fusion: In MRI/PET qualitative results, IFT produces PET-like color variation and brighter colors than Structure-aware and comparable or better variation than DDcGAN.The passage links IFT’s PET-like color variation to its high CC performance.
- Ablation study: The ST fusion network outperforms spatial-only and transformer-only fusion across all ablation metrics.The ablation compares local-feature spatial fusion, long-range-dependency transformer fusion, and their combined ST strategy.
5. CONCLUSION
The paper proposes IFT with an ST fusion strategy that combines local and long-range dependencies through CNN and transformer branches. Evaluations on multiple benchmark datasets report better results than existing fusion methods, while ablation results support extracting both types of information.
- Conclusion: IFT uses an ST fusion strategy that attends to local and long-range dependencies.Its ST strategy introduces CNN and transformer branches to fuse local and global features.
- Conclusion: IFT achieves better results than existing fusion methods across multiple fusion benchmark datasets.The conclusion also reports an ablation study evaluating local and long-range information extraction during fusion.