PaperlayerView exact revision on arXiv

Source-linked AI summary

LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images

Hui Li, Tianyang Xu, Xiao-Jun Wu, Jiwen Lu, Josef Kittler

arXiv:2304.05172v2cs.CV

TL;DR

Fusion networks achieve promising results, but their architectures are difficult to design systematically and are often developed through time-consuming trial and testing. LRRNet formulates fusion through a learnable low-rank representation, converts its optimization into a feed-forward convolutional network, and trains it with a detail-to-semantic loss. On public datasets, it outperforms state-of-the-art fusion methods while remaining substantially more lightweight.

  • Problem

    Fusion-network architecture design remains largely empirical and time-consuming, while representation-learning methods rely on iterative optimization and cumbersome fusion procedures.

  • Method

    LRRNet uses a learned low-rank representation to decompose source images, transform matrix multiplications into convolutions, and guide lightweight network construction with a multi-level loss.

  • Results

    LRRNet achieves better fusion performance than state-of-the-art methods on public datasets and is significantly more lightweight than competing models.

  • Takeaways & Limitations

    LRRNet preserves more visible-image detail, enhances infrared features, and demonstrates utility for RGBT tracking.

  • Takeaways & Limitations

    The loss function constrains the balance between infrared and visible features, requiring the visible-feature weight to be much smaller than the infrared-feature weight.

Abstract

from arXiv · show

Deep learning based fusion methods have been achieving promising performance in image fusion tasks. This is attributed to the network architecture that plays a very important role in the fusion process. However, in general, it is hard to specify a good fusion architecture, and consequently, the design of fusion networks is still a black art, rather than science. To address this problem, we formulate the fusion task mathematically, and establish a connection between its optimal solution and the network architecture that can implement it. This approach leads to a novel method proposed in the paper of constructing a lightweight fusion network. It avoids the time-consuming empirical network design by a trial-and-test strategy. In particular we adopt a learnable representation approach to the fusion task, in which the construction of the fusion network architecture is guided by the optimisation algorithm producing the learnable model. The low-rank representation (LRR) objective is the foundation of our learnable model. The matrix multiplications, which are at the heart of the solution are transformed into convolutional operations, and the iterative process of optimisation is replaced by a special feed-forward network. Based on this novel network architecture, an end-to-end lightweight fusion network is constructed to fuse infrared and visible light images. Its successful training is facilitated by a detail-to-semantic information loss function proposed to preserve the image details and to enhance the salient features of the source images. Our experiments show that the proposed fusion network exhibits better fusion performance than the state-of-the-art fusion methods on public datasets. Interestingly, our network requires a fewer training parameters than other existing methods. The codes are available at https://github.com/hli1221/imagefusion-LRRNet

1 INTRODUCTION

Image fusion combines complementary information from multiple modalities, but designing effective fusion architectures remains time-consuming and empirical. LRRNet addresses this with a representation-learning-guided architecture, a learnable decomposition model, and a multi-level loss function.

  • Image fusion synthesizes a single image containing complementary information from multiple input images.
  • Representation-learning-based fusion methods extract spatial-domain features but are slowed by iterative optimization and cumbersome traditional fusion procedures.They also struggle to handle complex, high-resolution images directly.
  • End-to-end fusion networks can achieve good performance, but their architectures are typically constructed through time-consuming empirical trial and testing.The design of an appropriate fusion network remains a critical research issue.
  • LRRNet uses a learnable image decomposition model to guide fusion-network architecture construction for infrared and visible images.The proposed loss is designed to preserve visible-image details and extract salient infrared features.
  • The detail-to-semantic information loss function contains pixel-, shallow-feature-, middle-feature-, and deep-feature-level loss terms for training the fusion network.
  • The proposed fusion network achieves better performance than state-of-the-art methods on public benchmarking datasets.

2 RELATED WORKS

Prior fusion methods use learned representations, iterative algorithms, or empirically designed end-to-end networks. Related work motivates replacing iterative or trial-and-test design with a learnable model-guided construction scheme.

  • 2.1 Learned Iterative Shrinkage-Thresholding Algorithm: LISTA learns representation coefficients within a limited number of iteration steps and realizes sparse-code inference as feed-forward processing.Its coefficients are updated by back propagation.
  • 2.1 Learned Iterative Shrinkage-Thresholding Algorithm: Learned convolutional sparse coding replaces LISTA matrix multiplications with convolutional layers and short connections.Stochastic gradient descent is used for back propagation.
  • 2.1 Learned Iterative Shrinkage-Thresholding Algorithm: Common LISTA-based objective functions are not ideal for image fusion, motivating a novel objective function for multi-modal image decomposition.
  • 2.2 End-to-end Learning Fusion Networks: End-to-end fusion networks include FusionGAN, FusionGANv2, U2Fusion, CUNet, SwinFusion, and YDTR.
  • 2.2 End-to-end Learning Fusion Networks: FusionGAN preserves infrared pixel information and visible-image gradients, while FusionGANv2 modifies the architecture and loss to preserve more detailed features.FusionGANv2 is reported not to generalize well to different tasks.
  • 2.2 End-to-end Learning Fusion Networks: U2Fusion uses elastic weight consolidation and sequential training to support multiple fusion tasks, but does not address fusion-network architecture design.
  • 2.2 End-to-end Learning Fusion Networks: CUNet guides architecture with optimization problems and multi-modal convolutional sparse coding, although its optimal formulas are only approximate.
  • 2.2 End-to-end Learning Fusion Networks: SwinFusion and YDTR explore new structures, but network design remains experimental and requires many experiments to identify a good architecture.The paper proposes a learnable model-guided construction scheme to address this problem.

3 LEARNED LOW-RANK REPRESENTATION MODEL (LLRR)

The LLRR model uses an LRR/SR-based decomposition whose optimization is converted into a learned convolutional architecture. LISTA and convolutional sparse coding produce LLRR blocks that extract low-rank and sparse image components.

  • Decomposition model: The decomposition objective uses the nuclear norm for the base part and the l1 norm for the salient part.The resulting model is designed specifically for multi-modal image decomposition.
  • Decomposition model: LRR decomposes each input image patch into low-rank and sparse coefficients projected by dictionaries D1 and D2.X contains vectorized image patches; D1L represents base information, while D2S preserves salient features such as texture, edges, and targets.
  • Optimization: Replacing the nuclear norm with a Frobenius-norm formulation enables faster optimization of the decomposition model.The original nuclear-norm optimization requires matrix decomposition and is described as time-consuming.
  • Optimization: LISTA solves the proposed model, and its iterative computation is implemented with convolutional layers through learned convolutional sparse coding.Matrix multiplications are replaced by convolutional operations, with C1 and C2 as trainable convolutional layers.
  • LLRR architecture: The LLRR block follows the transformed LISTA update and contains two convolutional layers with two short connections.A stack of LLRR blocks produces Z, which is split into low-rank and sparse coefficients before reconstructing the corresponding image parts.
  • LLRR architecture: The activation hθ applies soft thresholding, with θ learned as a parameter in the framework.The thresholding operator shrinks input values toward zero while retaining their signs.

4 THE FUSION NETWORK BASED ON THE LLRR BLOCKS

LRRNet uses LLRR blocks to guide an end-to-end infrared–visible fusion architecture, decomposing source images into low-rank and sparse components before convolutional fusion. A four-level detail-to-semantic loss balances visible detail preservation with infrared salient-feature enhancement.

  • LLRR blocks decompose infrared and visible source images into low-rank and sparse coefficients for subsequent fusion.
  • Convolutional layers fuse corresponding low-rank and sparse features, whose sum produces the final fused image.The architecture applies four convolutional layers to obtain parts, then concatenates and further fuses corresponding features.
  • The framework contains two LLRR branches with identical structures but different parameters, alongside six convolutional layers for component construction and fusion.
  • The total loss combines pixel, shallow-feature, middle-feature, and deep-feature terms extracted using a pre-trained network.VGG-16 provides features for the shallow, middle, and deep loss terms.
  • The loss is designed to preserve visible-image details while enhancing salient infrared features in the fused result.The deep term uses Gram-matrix information to represent abstract semantic features.

5 EXPERIMENTAL ANALYSIS

The experimental analysis evaluates LRRNet through controlled studies, subjective fusion examples, and downstream RGBT tracking experiments.

  • The experiments analyze the loss function and LLRR blocks, show fusion examples for subjective evaluation, and test LRRNet in RGBT tracking.

5.1 Experimental Settings

LRRNet is evaluated on public infrared–visible datasets using six objective quality metrics, with separate test data used to assess generalization.

  • The dataset examples show visible-spectrum images in the first row and infrared images in the second row.
  • Two public datasets, TNO and VOT2020-RGBT, are used to evaluate fusion performance.Twenty-one TNO pairs are used for testing, and 40 pairs from VOT2020-RGBT and TNO form a new generalization test dataset.
  • Evaluation uses Entropy, Standard Deviation, Mutual Information, modified SSIM, modified VIF, and modified fusion artifacts measure.Higher values indicate better performance for the first five metrics, whereas lower Nabf indicates less injected noise.

5.2 Ablation Study

Ablation studies examine loss-term weights, feature contributions, LLRR-block architectures, and training-loss behavior. The selected configuration balances visible detail preservation with infrared saliency and achieves strong objective and subjective results.

  • 5.2.1 The Impact of Lshallow and Lmiddle: Larger wir emphasizes infrared salient features, whereas larger γ2 emphasizes visible-spectrum detail information.
  • 5.2.1 The Impact of Lshallow and Lmiddle: γ2 = 1.5 and wir = 3.0 yield three best metrics and one second-best metric on TNO, so these values are selected.The best values are En, SD, and MI; Nabf is second-best.
  • 5.2.2 The Influence of Lpixel and Ldeep: γ4 = 2000 produces five best objective values, supporting preservation of infrared features alongside image details.
  • 5.2.2 The Influence of Lpixel and Ldeep: Removing Lpixel reduces detail information, while removing Ldeep weakens infrared-feature enhancement compared with the complete loss.
  • 5.2.2 The Influence of Lpixel and Ldeep: The complete loss achieves comparable six-metric performance and produces fused images with more detail and salient features, despite not being best on every metric.
  • 5.2.3 The Influence of LLRR Blocks: LLRR blocks outperform convolutional-layer and dense-block alternatives subjectively, while four LLRR blocks obtain five best metric values.
  • 5.2.4 The plot of the loss during the training phase: Lpixel and Lshallow stabilize within 2000 iterations, Lmiddle converges after 5000, and Ldeep stabilizes after 2500 iterations.The loss behavior reflects balancing visible detail information against infrared salient features and abstract deep features.

5.3 Fusion Results Analysis

LRRNet is evaluated against nine fusion methods on two public IR-VI datasets using visual, objective, and parameter-count comparisons. It preserves visible details and infrared salient features while producing lightweight, competitive fusion results.

  • Comparative evaluation: Nine classical and state-of-the-art fusion methods are compared with LRRNet on the TNO and VOTRGBT-TNO datasets.The evaluation includes visual comparisons, objective metrics, and training-parameter counts.
  • Visual results: LRRNet preserves visible-image texture and infrared salient features, producing clearer targets and enhanced tree textures in the TNO “man” example.Other methods either blur texture details or preserve texture while introducing noise.
  • Objective evaluation: LRRNet obtains three best objective values—En, SD, and MI—and one second-best value, VIFm, on TNO.The authors associate these results with more details, less noise, and more natural fused images.
  • Objective evaluation: On the expanded VOTRGBT-TNO dataset, LRRNet obtains two best values and one second-best value, indicating robust performance.Compared with SwinFusion, it reduces Nabf while preserving more information according to En and MI.
  • Model efficiency: LRRNet uses 0.04920M training parameters, fewer than IFCNN’s 0.08359M and Res2Fusion’s 0.09834M.It also achieves a better MI value with fewer parameters than RFN-Nest’s 7.52425M model.

5.4 Experiments on RGBT Object Tracking Task

The paper embeds LRRNet’s feature-fusion module into the DFAT RGBT tracker and evaluates it on VOT2020-RGBT. The resulting tracker achieves the best reported EAO performance among the compared trackers.

  • Tracker integration: The RGBT tracking experiment embeds LRRNet’s LLRR-based fusion module into the DFAT tracker.DFAT provides the baseline, while LRRNet replaces decision-level-only fusion with multi-modal feature fusion.
  • Training setup: Four limited RGBT datasets—GTOT, VT821, VT1000, and LasHeR—are used to train the embedded LRRNet module.The authors motivate embedding the fusion framework into a general tracker because RGBT tracking data are insufficient for training from scratch.
  • Training setup: The additional tracking loss has shallow, middle, and deep components corresponding to three DFAT backbone outputs.The components supervise feature fusion at multiple levels.
  • Tracking results: DFAT+LRRNet achieves the best tracking performance among the compared RGBT trackers according to EAO.The comparison includes single-modality AFAT, RFNT, M2C2Frgbt, and DFAT on VOT2020-RGBT.
  • Tracking results: LRRNet outperforms RFN in both image fusion and RGBT tracking when the fusion module is the changed component.The paper attributes RFN’s weaker result to an overly simple network structure for the decision-level tracker and a loss function needing reconsideration.
  • Cross-task applicability: The experiment indicates that LRRNet improves performance beyond image fusion in the demonstrated RGBT tracking application.The paper states that this cross-task evidence supports effectiveness in multi-modal computer vision tasks.

6 CONCLUSION

The conclusion presents LRRNet as a task-driven lightweight fusion network whose architecture is derived from a low-rank representation model. Experiments show sharper, detail-preserving outputs on two public datasets and demonstrated benefit in RGBT tracking.

  • Conclusion: LRRNet derives its lightweight architecture from a fusion-task-driven optimal model rather than empirical trial and error.The model decomposes source images into base and detail features and uses an optimization algorithm to construct the network architecture.
  • Conclusion: A multi-level detail-to-semantic loss function trains the LLRR-block-based end-to-end fusion network.The loss is designed to support the proposed representation-guided architecture.
  • Conclusion: Across two public IR-VI datasets, LRRNet produces sharper outputs, preserves visible details, enhances infrared features, and has a significantly lighter network size than other methods.These are reported as the principal image-fusion conclusions relative to nine classical and state-of-the-art methods.
  • Conclusion: The LRRNet feature-fusion module also improves performance in the demonstrated RGBT tracking application.The authors state that broader computer-vision applications remain a direction for future exploration.
Loading 2304.05172v2…