Source-linked AI summary
NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Models
Hui Li, Xiao-Jun Wu, Tariq Durrani
TL;DR
The paper addresses limitations in deep-learning infrared and visible image fusion, especially incomplete multi-scale feature use and insufficiently designed deep-feature fusion. It proposes a nest-connection encoder–fusion strategy–decoder framework with spatial and channel attention, and reports state-of-the-art fusion performance under visual and objective evaluation, including improved RGBT tracking performance.
Problem
Existing fusion frameworks do not fully exploit multi-scale deep features, and their fusion strategies do not consider attention for deep features.
Method
NestFuse uses a nest connection-based encoder and decoder with spatial and channel attention models to fuse multi-scale deep features.
Results
The proposed framework demonstrates state-of-the-art fusion performance, while an additional RGBT tracking experiment shows improved algorithm performance.
Takeaways & Limitations
NestFuse provides a multi-scale infrared and visible image fusion framework whose fusion strategy is also reported effective for an RGBT tracking task.
Abstract
from arXiv · showhide
In this paper we propose a novel method for infrared and visible image fusion where we develop nest connection-based network and spatial/channel attention models. The nest connection-based network can preserve significant amounts of information from input data in a multi-scale perspective. The approach comprises three key elements: encoder, fusion strategy and decoder respectively. In our proposed fusion strategy, spatial attention models and channel attention models are developed that describe the importance of each spatial position and of each channel with deep features. Firstly, the source images are fed into the encoder to extract multi-scale deep features. The novel fusion strategy is then developed to fuse these features for each scale. Finally, the fused image is reconstructed by the nest connection-based decoder. Experiments are performed on publicly available datasets. These exhibit that our proposed approach has better fusion performance than other state-of-the-art methods. This claim is justified through both subjective and objective evaluation. The code of our fusion method is available at https://github.com/hli1221/imagefusion-nestfuse
I. INTRODUCTION
Image fusion combines salient and complementary information from source images, but existing deep-learning frameworks leave multi-scale extraction, network topology, and deep-feature fusion insufficiently addressed. NestFuse responds with a nest-connection architecture and spatial/channel attention fusion strategy, reporting better visual and objective performance than existing methods.
- Motivation: Image fusion aims to generate one image containing salient features and complementary information from multiple sources.The technique is used in applications including self-driving vehicles, visual tracking, and video surveillance.
- Existing Methods: Deep-learning fusion methods generally seek better performance through trainable models, including CNN, dense-block autoencoder, and GAN approaches.Prior work includes CNN-based multi-focus fusion, multi-level feature fusion, DenseFuse, and FusionGAN.
- Limitations: Existing deep-learning frameworks lack multi-scale feature extraction, need improved network topology, and do not carefully design deep-feature fusion.One identified framework has no down-sampling operator, leaving deep features incompletely utilized.
- Proposed Framework: NestFuse applies a nest connection architecture with encoder, fusion strategy, and decoder components to preserve information from encoder-extracted features at different scales.The architecture is designed to make fuller use of deep features across scales.
- Proposed Framework: The proposed fusion strategy combines spatial attention and channel attention models to fuse multi-scale deep features.These models are introduced specifically for multi-scale deep-feature fusion.
- Results: The fusion framework reports better performance than existing state-of-the-art methods under both visual and objective assessment.The paper presents this as its comparative experimental outcome.
II. RELATED WORKS
Related fusion research spans traditional signal-processing and representation-learning methods as well as trained and untrained deep-learning approaches. NestFuse adapts nest connections from medical image segmentation to image fusion and adds a dedicated fusion strategy.
- Deep Learning-based Fusion Methods: Deep-learning fusion methods can be divided into approaches with a training phase and approaches that use pre-trained networks without backpropagation.The latter extract deep features and generate a decision map.
- Deep Learning-based Fusion Methods: A CNN decision-map method achieved better multi-focus fusion performance but was limited to multi-focus images by its training strategy.Its labels distinguish clear and blurred image patches.
- Deep Learning-based Fusion Methods: DenseFuse uses an encoder, fusion layer, and decoder to fuse infrared and visible images, while FusionGAN uses a generator trained with content and discriminator losses.DenseFuse trains the encoder and decoder as an autoencoder before applying fusion.
- The Nest Connection Architecture: Nest connection replaces long skip connections with up-sampling and several short skip connections to constrain semantic-gap effects and preserve information.The architecture was originally proposed for medical image segmentation.
- The Nest Connection Architecture: NestFuse introduces this architecture into image fusion and combines it with a novel fusion strategy.The paper presents this as a modified nest-connection-based fusion framework.
III. PROPOSED FUSION METHOD
The proposed fusion method combines an encoder, fusion strategy, and nest connection-based decoder to process and reconstruct multi-scale deep features. Its training phase uses an auto-encoder objective combining pixel and structural similarity losses.
- The framework contains an encoder, fusion strategy, and decoder, with nest connections processing the encoder’s multi-scale deep features.
- Two source images are encoded into multi-scale features, fused at each scale, and reconstructed by the nest connection-based decoder.
- During training, the fusion strategy is discarded so the encoder and decoder learn an auto-encoder reconstruction pathway.
- The encoder uses convolutional blocks with max pooling to extract features at different scales, while the decoder uses nest-connected convolutional blocks without pooling.
- The total training loss combines pixel loss and SSIM loss, with λ controlling their trade-off.
- The auto-encoder is trained on 80,000 grayscale, resized MS-COCO images, and its training input type is not limited to infrared and visible images.
C. Fusion Strategy
The fusion strategy addresses limitations of weighting-based methods by applying two attention-model stages to multi-scale deep features. Spatial and channel attention produce fused features that are passed to the decoder.
- Weight-average fusion strategies depend critically on the choice of a weighting map.
- Existing test-phase fusion strategies are not designed for deep features and do not yet incorporate attention mechanisms.
- The proposed strategy applies two attention-model stages to multi-scale deep features across M = 4 levels.
- The attention models generate final fused multi-scale deep features that serve as input to the decoder network.
- Spatial and channel attention models process extracted multi-scale deep features in two phases.
1) Spatial Attention Model:
The spatial attention model extends spatial-based fusion to multi-scale deep features by computing position-dependent weighting maps and combining enhanced features.
- Spatial attention extends existing spatial fusion to multi-scale deep features.
- Weighting maps are computed from feature l1-norms and a soft-max operation.
- Each spatial position carries a C-dimensional feature vector used in weighting-map computation.
- The enhanced deep features are weighted by the resulting maps and added to produce fused features.
2) Channel Attention Model:
The channel attention model accounts for channel-wise information in three-dimensional deep features by deriving channel weights through global pooling and soft-max normalization.
- Channel attention is introduced because deep features contain channel information in addition to spatial information.
- Global pooling produces initial weighting vectors for corresponding channels.
- Average, max, and nuclear-norm pooling respectively summarize channel averages, maxima, and singular-value sums.
- A soft-max operator produces final weighting vectors, which weight enhanced features before fused-feature calculation.
IV. EXPERIMENTAL RESULTS
The experiments evaluate NestFuse against existing fusion methods using subjective assessment and seven objective quality metrics on infrared-visible image pairs.
- The testing phase combines subjective evaluation with objective quality-metric comparisons.
- The test set contains 21 pairs of infrared and visible images collected from two cited sources.
- Twelve typical and state-of-the-art fusion methods are selected as comparison methods.
- Seven metrics measure information, feature, structural, and visual fidelity properties of the fused outputs.
- Higher numerical values indicate better fusion performance across the seven metrics.
B. Ablation Study
The ablation study examines loss-parameter effects, convergence behavior, and deep supervision using training curves, metric comparisons, and dedicated evaluation tables.
- Parameter(λ) in Loss Function: The network uses λ values of 1, 10, 100, and 1000, with epoch and batch size set to 2 and 4.
- Parameter(λ) in Loss Function: During the first 400 iterations, increasing λ produces faster convergence, especially for Lpixel, Lssim, and Ltotal at λ = 100 or λ = 1000.
- Parameter(λ) in Loss Function: After more than 600 iterations, optimal network weights are obtained regardless of the selected λ.
- Parameter(λ) in Loss Function: λ = 100 achieves better fusion performance than the other tested λ values, so the experiments use λ = 100.
- Deep Supervision: The study also compares outputs from deep supervision with outputs produced without deep supervision.
2) The Influence of Multi-scale Deep Features:
The study examines how deep-feature scale and training strategy affect NestFuse fusion performance. Global optimization makes multi-scale features effective, while shallow and deep scales show different strengths under deep supervision.
- Training strategy: Deep supervision generates outputs O1, O2 and O3 at different deep-feature scales for training the fusion network.The associated loss uses Q = 3 and a total loss function.
- Scale analysis: With deep supervision, metric values are close across O1, O2 and O3, limiting the apparent advantage of multi-scale features.The comparison uses seven fusion-quality metrics.
- Scale analysis: Shallow scale O2 performs better than deeper O3 on En, SD, MI and V IF, indicating stronger detail representation.Deeper features instead provide more structure information for some metrics.
- Scale analysis: Deeper features achieve the best FMIdct and SSIMa values and a comparable FMIw value because fused images contain more structure features.This contrasts with the detail-oriented strengths of shallow features.
- Optimization strategy: Global optimization boosts fusion performance and obtains all best values, demonstrating the effectiveness of the multi-scale mechanism.The authors note that deep supervision may still perform better for image segmentation than for image fusion.
C. Results Analysis
NestFuse is evaluated visually and objectively against existing fusion methods across infrared-visible image examples. It achieves strong subjective quality and five best plus five second-best metric values, while pooling choices affect channel-attention fusion.
- Visual comparison: CBF, DCHWT, JSR and JSRSD produce more noise and less-clear details, while GTF, WLS, ConvSR, VggML and FusionGAN blur some regions.These observations are reported for the red-boxed regions in the visual comparisons.
- Visual comparison: Across man, street and umbrella examples, NestFuse uses average, max and nuclear channel-attention pooling variants for comparison.The figures also compare eleven existing fusion methods with the proposed variants.
- Visual comparison: DeepFuse, DenseFuse, IFCNN and NestFuse perform better subjectively than the other compared methods, with NestFuse producing more reasonable luminance.The comparison concerns the fused images shown in the three image experiments.
- Objective comparison: Five best and five second-best values show that NestFuse preserves more detail and feature information, excluding FMIdct and SSIMa.The cited metrics are En, SD, MI, FMIw and V IF for the preserved information claim.
- Objective comparison: NestFuse achieves the best En value because its fused images combine reasonable luminance with less noise, unlike noise-generating alternatives.En measures the amount of information, but noise can also increase it.
- Objective comparison: Nuclear and average channel-attention operators achieve almost the best objective values and capture more structure information than max pooling.The three operators are the global pooling choices listed for the channel-attention fusion strategy.
D. An Application to Visual Objective Tracking
The paper applies its feature-level fusion strategy to SiamRPN++ for RGB-thermal tracking on VOT-RGBT2019. The fused multi-scale representation improves EAO relative to single-modality RGB or infrared tracking.
- Task and setup: VOT-RGBT2019 contains 60 video sequences with RGB and thermal infrared modalities for short-term tracking.Tracking examples are drawn from the benchmark sequences shown in Fig. 13.
- Task and setup: SiamRPN++ is selected as the base tracker, with NestFuse applied for feature-level fusion of multi-scale deep features.SiamRPN++ is described as a deep-learning tracker with state-of-the-art tracking performance in 2019.
- Evaluation: Tracking performance is evaluated using Expected Average Overlap, Accuracy and Failures.EAO estimates average overlap, Accuracy measures predicted-ground-truth overlap, and Failures evaluates robustness.
- Results: EAO improves over RGB-only and infrared-only SiamRPN++ when NestFuse fuses multi-scale deep features.EAO is the primary measure in the VOT challenge.
- Scope: The paper identifies applying the proposed fusion method to other computer vision tasks as future work.This marks the current application scope beyond the reported tracking experiment.
- Results: The authors report that the fusion strategy improves tracking performance in the RGBT task in addition to image fusion.This conclusion is stated as an additional computer-vision-task result.