PaperlayerView exact revision on arXiv

Source-linked AI summary

RFN-Nest: An end-to-end residual fusion network for infrared and visible images

Hui Li, Xiao-Jun Wu, Josef Kittler

arXiv:2103.04286v2cs.CV

TL;DR

Image fusion needs a task-appropriate, learnable strategy for combining infrared and visible information because existing approaches can rely on handcrafted choices. RFN-Nest introduces a residual end-to-end fusion framework trained in two stages with detail-preserving and feature-enhancing losses. The authors report the best subjective and objective fusion performance among seventeen existing methods, with better tracking performance when applied to RGBT tracking.

  • Problem

    Traditional fusion methods depend on handcrafted features and strategies, while complex images and limited training data constrain existing approaches.

  • Method

    RFN-Nest combines an encoder, four residual fusion networks, and a nest-connection decoder, using two-stage training and detail-preserving and feature-enhancing losses.

  • Results

    Compared to seventeen existing fusion methods, RFN-Nest achieves the best performance in both subjective and objective evaluation, and its tracker improves over single-modality tracking across all measures on VOT2019 and VOT2020.

  • Takeaways & Limitations

    The reported results support RFN-Nest's applicability to infrared-visible image fusion and extension of its RFN and losses to multimodal RGBT tracking.

  • Takeaways & Limitations

    The method's development was constrained by limited labeled training data, which required simplifying the RFN architecture and fixing AFAT modules in the tracking application.

Abstract

from arXiv · show

In the image fusion field, the design of deep learning-based fusion methods is far from routine. It is invariably fusion-task specific and requires a careful consideration. The most difficult part of the design is to choose an appropriate strategy to generate the fused image for a specific task in hand. Thus, devising learnable fusion strategy is a very challenging problem in the community of image fusion. To address this problem, a novel end-to-end fusion network architecture (RFN-Nest) is developed for infrared and visible image fusion. We propose a residual fusion network (RFN) which is based on a residual architecture to replace the traditional fusion approach. A novel detail-preserving loss function, and a feature enhancing loss function are proposed to train RFN. The fusion model learning is accomplished by a novel two-stage training strategy. In the first stage, we train an auto-encoder based on an innovative nest connection (Nest) concept. Next, the RFN is trained using the proposed loss functions. The experimental results on public domain data sets show that, compared with the existing methods, our end-to-end fusion network delivers a better performance than the state-of-the-art methods in both subjective and objective evaluation. The code of our fusion method is available at https://github.com/hli1221/imagefusion-rfn-nest

1. Introduction

Image fusion must combine complementary infrared and visible information, but traditional methods rely on handcrafted features and task-specific fusion strategies. RFN-Nest addresses this with a learnable end-to-end framework, staged training, specialized losses, and reported improvements over existing methods.

  • Image fusion reconstructs one image by combining complementary information from multiple source images for applications including tracking and surveillance.
  • Traditional methods depend on handcrafted features, task-specific fusion strategies, costly dictionary learning, and struggle with complex source images.
  • RFN-Nest combines an encoder, learnable residual fusion network, and nest-connection decoder in an end-to-end framework.The RFN replaces handcrafted feature fusion and extracts fused multi-scale deep features.
  • The network uses two-stage training: first an auto-encoder is trained, then fixed encoder-decoder components support RFN training with appropriate losses.
  • Detail-preserving and feature-enhancing losses are designed to preserve image details and salient features in the fused image.
  • RFN-Nest reportedly outperforms state-of-the-art fusion methods on public datasets in both subjective visual assessment and objective evaluation.

2. Related Works

Deep fusion methods have progressed from feature extraction and decision-map pipelines toward end-to-end architectures. Prior approaches include CNNs, auto-encoders, attention models, GANs, and general fusion networks, but handcrafted strategies and detail preservation remain recurring concerns.

  • Deep learning fusion methods are organized around feature extraction, fusion strategy, and end-to-end training, with non-end-to-end and end-to-end categories.
  • Early methods used pretrained deep networks or decision maps to extract features and determine how source images should be fused.
  • Auto-encoder approaches combine an encoder, fusion layer, and decoder, while NestFuse adds dense features and spatial/channel attention for infrared-visible fusion.
  • End-to-end methods seek to eliminate handcrafted fusion strategies, including GAN-based, general CNN, and related frameworks.
  • FusionGAN was reported to produce fused images resembling infrared inputs and failing to preserve image detail despite discriminator constraints.
  • FusionGANv2 deepened its networks and added detail and target edge-enhancement losses, producing more scene details and sharper target edges.

3. The Proposed Fusion Framework

RFN-Nest is an end-to-end infrared-visible fusion framework combining multi-scale feature extraction, learnable residual fusion, and nest-connected reconstruction. A two-stage training strategy first equips the encoder-decoder with reconstruction ability, then trains RFNs to fuse multi-scale features using detail-preserving and feature-enhancing objectives.

  • Architecture: RFN-Nest combines an encoder, four scale-specific residual fusion networks, and a nest connection-based decoder for end-to-end image fusion.The encoder extracts multi-scale features, RFNs fuse features at each scale, and the decoder reconstructs the fused image.
  • Residual Fusion Network: RFN replaces handcrafted fusion strategies with a learnable residual architecture that produces fused deep features for the decoder.Its residual design concatenates outputs from early convolutional layers before generating initial fused features.
  • Decoder Network: The nest-connected decoder uses short within-row connections and cross-layer links to combine fused multi-scale features during reconstruction.The decoder is simplified relative to UNet++ to remain light while reconstructing the fused image.
  • Two-Stage Training: The first training stage trains the encoder-decoder as an autoencoder to reconstruct input images from four-scale deep features.Max pooling extracts the multi-scale features, while short cross-layer connections help the decoder use them for reconstruction.
  • Auto-Encoder Training: The autoencoder loss combines pixel and SSIM losses, constraining reconstructed images to match inputs at pixel and structural levels.The parameter λ balances the pixel loss and structural similarity loss.
  • RFN Training: With the encoder and decoder fixed, the second stage trains RFNs using detail-preservation and feature-enhancement losses.The detail loss preserves visible-image detail and structure, while the feature loss constrains fused features to retain salient infrared structures.

4. Experimental Validation

The experiments evaluate RFN-Nest using ablations and objective metrics on public infrared–visible image datasets. Testing uses TNO and VOT2020-RGBT image pairs, while the two training stages use MS-COCO and KAIST.

  • Experimental Design: The study investigates network elements through ablation studies and compares fusion algorithms using subjective and objective evaluations.Six quality metrics are used for objective evaluation.
  • Training Data: The first training stage uses 80,000 grayscale 256 × 256 MS-COCO images for auto-encoder training.Training uses batch size 4, two epochs, and learning rate 1 × 10−4.
  • Training Data: The second training stage uses 80,000 grayscale 256×256 infrared–visible pairs from KAIST to train the RFN networks.The KAIST dataset contains almost 90,000 image pairs overall.
  • Test Data: Testing uses 21 infrared–visible pairs from TNO and 40 pairs collected from TNO and VOT2020-RGBT.The test images are publicly available through the cited source.
  • Evaluation: The evaluation metrics are entropy, standard deviation, mutual information, Nabf, SCD, and MS-SSIM.The paper states that fusion performance improves as the numerical index increases for the listed metrics.

4.3. Ablation Study for Ldetail and L feature

The ablation study examines detail-preserving and feature-enhancing losses, together with their parameters. Detail preservation is necessary for acceptable fusion, while α = 700 provides the selected overall setting.

  • Parameter Effects: Setting wvi = 0 prevents convergence because Ldetail and Lfeature impose conflicting constraints, so wvi is kept non-zero.The loss weights are chosen to balance visible detail preservation and infrared salient-feature retention.
  • Parameter Effects: Large wir values retain salient infrared features, whereas visible-image detail is not preserved when α = 0.For small wir, fused images resemble the visible image and suppress infrared salient features.
  • Loss Functions: When α = 700, Ldetail preserves detail information, while Lfeature promotes salient source-image features.The selected weights require wvi to be smaller than wir.
  • Ldetail: Without Ldetail (α = 0), the fusion network fails to obtain acceptable results and shows degradation in subjective and objective assessments.The comparison is made against the optimal parameter combinations.
  • Parameter Selection: With Ldetail enabled, RFN-Nest obtains comparable metric values at wir = 6.0 and wvi = 3.0, which are used in later experiments.The study evaluates weight combinations over the range [0.5, 6.0].
  • Parameter Selection: At α = 700, the proposed network achieves better subjective and objective performance and records the best value in four of seven metrics.The parameter α is therefore set to 700 in subsequent experiments.

4.5. Ablation Study for Training Strategy

The study compares simultaneous one-stage training with the proposed two-stage strategy. Two-stage training better preserves salient infrared objects while retaining visible-spectrum detail.

  • Training Strategies: Two-stage training is treated as a critical operation and is evaluated against simultaneous training of the encoder, RFN, and decoder.The comparison uses the same loss function and parameter settings for the one-stage strategy.
  • One-Stage Training: One-stage training enhances visible-spectrum detail but loses salient infrared objects, so the premise of image fusion is not realised.In this strategy, the encoder and decoder remain free to adapt their weights.
  • Two-Stage Training: Two-stage training preserves salient infrared objects and contains more visible-image detail than the one-stage result.The qualitative comparison is shown in Figure 11.
  • Interpretation: The authors attribute the one-stage shortcoming to insufficient feature-extraction and reconstruction ability in the encoder and decoder, while emphasizing careful RFN training.RFN is identified as the key component of the fusion network.
  • Two-Stage Training: The first stage trains the encoder and decoder for multi-scale feature extraction and image reconstruction, after which they are fixed for RFN training.The second stage trains RFNs to enhance visible detail and preserve salient infrared features.

4.6. Ablation Study for Nest Connection in Decoder

The decoder ablation removes its nest connections while keeping the training strategy and loss functions unchanged. The nest-connected RFN-Nest decoder performs better across the reported metric rankings.

  • Ablation Setup: The “No-nest” ablation removes short connections between decoder convolutional blocks and uses the same training strategy and loss functions.The comparison isolates the decoder’s nest connection architecture.
  • Results: RFN-Nest obtains one best, three second-best, and one third-best metric values compared with “No-nest”.Table 3 reports seven quality metrics for the ablation studies.
  • Interpretation: The results indicate that nest connections boost decoder reconstruction ability and help preserve information from multi-scale deep features.The reported preserved-information metrics include MI, FFMIdct, and FFMIw.

4.7. Ablation Study for Fusion Strategy

The study compares handcrafted fusion rules with the learnable RFN strategy. RFN-Nest uses adaptive RFN networks to perform fusion within its autoencoder.

  • Five handcrafted strategies—add, max, l1-norm, l∗-norm, and SCA—are selected for comparison with the adaptive RFN fusion mechanism.
  • The trained encoder extracts multi-scale deep features, while the decoder reconstructs the final image from fused features.
  • The add strategy directly sums source features, whereas max selects feature values using max(·).
  • Table 4 lists the formulas for the five handcrafted fusion strategies and RFN-Nest.
  • Figure 13 compares fusion results produced with different fusion strategies, identifying RFN-Nest as adaptive RFN-based fusion.

4.8. Fusion Results Analysis on 21 pairs Images

On 21 infrared-visible image pairs, RFN-Nest is compared with eleven representative methods using visual examples and seven averaged quality metrics. The proposed method preserves background detail, infrared contrast, and salient visible information while achieving strong objective scores.

  • Eleven representative methods, including DCHWT, GTF, ConvSR, DenseFuse, NestFuse, PMGI, DDcGAN, and U2Fusion, are used as baselines.
  • Visual results: In the man examples, RFN-Nest makes the tree, street lamp, and bushes clearer while maintaining infrared contrast.
  • Visual results: In the umbrella examples, RFN-Nest preserves pavilion features and reconstructs clearer tree detail than the other methods.
  • RFN-Nest produces a better balance between visible background information and infrared features than the compared fusion methods.
  • RFN-Nest obtains the best SCD value and third-best S_D, Nab_f, and MS-SSIM values across the averaged metrics.
  • The proposed method produces sharper content and higher visual information fidelity, while DDcGAN’s larger En, S_D, and MI values are attributed to noise and artefacts.

4.9. Further Analysis on 40 Pairs Images

A separate 40-pair dataset from TNO and VOT2020-RGBT is used to test generalization. RFN-Nest achieves strong visual quality and competitive objective performance on this expanded evaluation.

  • The new test set contains 40 infrared-visible image pairs collected from TNO and VOT2020-RGBT, extending evaluation beyond the original 21 pairs.
  • DenseFuse, NestFuse, PMGI, DDcGAN, and U2Fusion are selected as deep-learning fusion baselines.
  • RFN-Nest produces natural-looking fused images and preserves more detail from both infrared and visible images than existing methods.
  • RFN-Nest achieves the best SCD value, second-best Nab_f value, and third-best S_D and MS-SSIM values on the 40-pair evaluation.
  • The results are described as demonstrating better generalization from the two-stage training strategy and novel loss function, with performance comparable to DDcGAN.

5. Experiments on RGBT Object Tracking

The paper integrates RFN into the AFAT tracker for RGBT object tracking and evaluates it on VOT2019-RGBT and VOT2020-RGBT. Despite limited training data requiring a simplified RFN, the RFN-based tracker improves over single-modality inputs and remains competitive with strong trackers.

  • Experimental setup: The RGBT experiments use VOT2019 and VOT2020 protocols, where each sequence provides RGB and infrared modalities.
  • Method: RFN and its proposed loss function are incorporated into the AFAT base tracker for RGBT tracking.
  • Method: Limited labeled training data forces a reduced RFN architecture, while AFAT modules remain fixed during training.
  • Experimental setup: RFN is trained on GTOT, VT821, and VT1000, which together contain 17.6k frames.
  • Evaluation: The evaluated tracking measures are Accuracy, Robustness, and Expected Average Overlap, with VOT2020 using redefined Anew, Rnew, and EAOnew measures.
  • Results: The RFN-based AFAT tracker outperforms AFAT with only RGB or only infrared input on all measures across both VOT-RGBT2019 and VOT-RGBT2020.
  • Results: On VOT2019-RGBT, RFN-based AFAT is comparable to mfDiMP in EAO and has better accuracy; on VOT2020-RGBT, it is competitive with winning tracker DFAT.

6. Conclusions

RFN-Nest combines a nest-connected auto-encoder with residual fusion networks trained using detail-preserving and feature-enhancing losses. It achieves the best fusion performance among seventeen methods and generalizes to multimodal tracking.

  • 6. Conclusions: The framework targets image-detail preservation through a residual fusion network with nest connections and dedicated detail-preserving and feature-enhancing losses.
  • 6. Conclusions: RFN-Nest uses a two-stage strategy: an auto-encoder is trained first, followed by four residual fusion networks using detail and feature losses.The encoder, RFN networks, and decoder then reconstruct the fused image.
  • 6. Conclusions: RFN-Nest achieves the best subjective and objective fusion performance compared with seventeen existing fusion methods.
  • 6. Conclusions: RFN and its loss functions also improve tracking performance over single-modality tracking on every measure in VOT2019 and VOT2020.The RFN-based tracker also performs very well against state-of-the-art RGBT trackers.
Loading 2103.04286v2…