PaperlayerView exact revision on arXiv

Source-linked AI summary

An Interactively Reinforced Paradigm for Joint Infrared-Visible Image Fusion and Saliency Object Detection

Di Wang, Jinyuan Liu, Risheng Liu, Xin Fan

arXiv:2305.09999v1cs.CV

TL;DR

Infrared-visible fusion and salient object detection both use complementary cues, but prior methods rarely connect fusion outputs to downstream semantic vision tasks. IRFS jointly optimizes them through feature-screened fusion, fused-image-guided SOD, and interactive loop learning, and experiments report mutual gains across both tasks. The framework remains limited in dynamically adapting fusion for low-light or adverse-weather SOD scenarios.

  • Problem

    Prior infrared-visible fusion methods often focus on visual quality and struggle to transmit precise semantic information to downstream high-level tasks, while multimodal SOD does not perform pixel-level fusion.

  • Method

    IRFS uses FSFNet to preserve saliency-related fusion features, feeds the fused image as a third modality to FGC2Net, and interactively optimizes fusion and SOD.

  • Results

    IRFS mutually enhances infrared-visible fusion and SOD, with the predicted saliency maps ranking first on four commonly used SOD metrics and fusion gains of 45.8% in VIF over the second-best method.

  • Takeaways & Limitations

    A single framework can maintain a collaborative relationship between infrared-visible fusion and SOD for unmanned-system visual perception.

  • Takeaways & Limitations

    IRFS is not yet adapted to dynamically adjust fusion for SOD requirements in low-light or adverse-weather conditions.

Abstract

from arXiv · show

This research focuses on the discovery and localization of hidden objects in the wild and serves unmanned systems. Through empirical analysis, infrared and visible image fusion (IVIF) enables hard-to-find objects apparent, whereas multimodal salient object detection (SOD) accurately delineates the precise spatial location of objects within the picture. Their common characteristic of seeking complementary cues from different source images motivates us to explore the collaborative relationship between Fusion and Salient object detection tasks on infrared and visible images via an Interactively Reinforced multi-task paradigm for the first time, termed IRFS. To the seamless bridge of multimodal image fusion and SOD tasks, we specifically develop a Feature Screening-based Fusion subnetwork (FSFNet) to screen out interfering features from source images, thereby preserving saliency-related features. After generating the fused image through FSFNet, it is then fed into the subsequent Fusion-Guided Cross-Complementary SOD subnetwork (FC$^2$Net) as the third modality to drive the precise prediction of the saliency map by leveraging the complementary information derived from the fused image. In addition, we develop an interactive loop learning strategy to achieve the mutual reinforcement of IVIF and SOD tasks with a shorter training period and fewer network parameters. Comprehensive experiment results demonstrate that the seamless bridge of IVIF and SOD mutually enhances their performance, and highlights their superiority.

1. Inroduction

The paper identifies a gap between infrared-visible image fusion and downstream saliency object detection, then proposes IRFS to connect and mutually reinforce both tasks. IRFS combines feature-screened fusion, fused-image guidance, and interactive loop learning.

  • Problem: Existing fusion methods prioritize visual quality but seldom address linkage with downstream high-level vision tasks or precise semantic transfer.This limitation can cause downstream performance decline, while existing low-level methods provide only limited semantic conditioning.
  • Fusion-to-SOD bridge: The fused image becomes a third modality for FGC2Net, whose fusion-guided saliency-enhanced modules maintain cross-complementary guidance throughout SOD.The modules reweight source features using the fused image and suppress surrounding interference features.
  • Paradigm: IRFS is an interactively reinforced paradigm that explores the collaborative relationship between infrared-visible fusion and saliency object detection.The framework uses a bidirectional optimization stream between the two tasks.
  • Fusion-to-SOD bridge: FSFNet screens interfering features while preserving saliency-related and texture-informative features for fusion.The design targets fusion outputs that better support the downstream SOD task.
  • Learning strategy: Interactive loop learning tightly couples fusion and SOD, aiming for mutual reinforcement with fewer training costs and network parameters.The strategy alternates task optimization rather than relying only on simultaneous one-stage updates.
  • Results: Experiments report that IRFS bridges both tasks and that each benefits from the other, with superior capabilities.The introduction presents this as the main empirical outcome of the proposed paradigm.

2. Related Works

Prior work develops multimodal fusion and saliency detection largely as separate or one-directionally connected problems. The paper motivates an adaptive joint framework because pixel-level fused structures can complement feature-level multimodal SOD.

  • Multimodal Image Fusion: Traditional image fusion includes multi-scale transform, sparse representation, subspace, and saliency-based methods.These approaches use decompositions, learned dictionaries, dimensionality reduction, or salient-region and weight-based processing.
  • Multimodal Image Fusion: Deep learning fusion methods include autoencoder, deep CNN, GAN, and transformer-based approaches that preserve multimodal structures and details.Their strategies range from encoder-decoder reconstruction and intensity-gradient modeling to adversarial and long-range dependency modeling.
  • Fusion and High-Level Vision: Existing fusion methods often emphasize visual quality without establishing connections to high-level vision tasks.The paper argues that unmanned systems require joint infrared-visible fusion and high-level perception.
  • Multimodal Salient Object Detection: Thermal infrared-visible SOD methods use ranking, graph learning, or deep networks to extract complementary information and predict salient-object maps.VT821 and VT5000 are cited as multimodal SOD benchmarks associated with traditional and deep-learning approaches.
  • Multimodal Salient Object Detection: Existing multimodal SOD methods perform cross-modal interaction in feature space but do not perform pixel-level fusion, despite fused object structures aiding salient-object discrimination.This motivates combining image fusion and SOD within one framework.
  • Joint Frameworks of Multiple Vision Tasks: Joint low- and high-level vision frameworks include cascaded, semantic-conditioned, and parallel routes, but prior work does not establish an adaptive bridge between tasks.The paper links this absence to overfitting one task and deviating from the optimal balance between them.

3. The Proposed Method

IRFS jointly connects infrared-visible image fusion with saliency object detection through bidirectional interaction. FSFNet preserves saliency-related information for fusion, while FGC2Net uses the fused image as a third modality for precise SOD.

  • Overall framework: IRFS contains FSFNet and FGC2Net in a cascaded framework with bidirectional optimization between image fusion and SOD.Image fusion facilitates SOD bottom-up, while SOD facilitates fusion top-down.
  • 3.1. Feature Screening-based Image Fusion: FSFNet extracts and screens visible and infrared features to suppress interference while preserving saliency-related and texture-informative information.The visible input is converted to YCbCr and its Y channel is used in the visible branch; preserved features are reconstructed into the fused image.
  • 3.2. Fusion-Guided Cross-Complementary SOD: FGC2Net treats the fused image as a third modality and uses its sharp objects and object-background contrast to guide infrared-visible SOD.The fusion-guided design is described as the first departure from standard multimodal SOD configurations.
  • 3.2. Fusion-Guided Cross-Complementary SOD: The FGSE module combines saliency enhancement, cross-complementary feature transformation, and learnable feature selection within a siamese encoder.It is embedded behind each backbone feature scale to aggregate cross-modality features and enhance saliency-related representations.
  • 3.2. Fusion-Guided Cross-Complementary SOD: MSGD uses infrared, visible, and fusion-modality decoding branches, with fusion decoding producing a coarse map and the other branches producing coarse and precise maps.Precise modality-specific maps are aggregated into the final precise saliency map.
  • 3.3. Loss Functions: The training objective combines fusion and SOD losses, including intensity, gradient, weighted BCE, and weighted IoU terms with trade-off weights.The interactive formulation updates the trade-off weight η across learning loops while τ remains fixed at 1.

4. Experiments

The experiments evaluate implementation details, datasets, metrics, joint fusion and SOD performance, generalization, and the contribution of individual IRFS components.

  • 4. Experiments: The experimental section reports quantitative and qualitative evaluations of IRFS on joint multimodal image fusion and SOD tasks.It also evaluates generalization ability across the two subtasks.
  • 4. Experiments: Ablation studies assess the contribution of each key component of IRFS.The section presents these studies after evaluating the joint tasks and generalization.

4.1. Experimental Setup

The evaluation uses VT5000 for joint fusion and SOD, with additional fusion benchmarks for direct generalization. Fusion quality is measured by information transmission, visual fidelity, and correlation with source images.

  • Datasets: VT5000 contains 5,000 thermal-visible image pairs with binary SOD labels, split evenly into training and testing sets.The dataset uses 2,500 pairs for training and 2,500 for testing.
  • Datasets: Additional fusion evaluations use the TNO, RoadScene, and M3FD datasets without fine-tuning the pretrained FSFNet.The fused images are generated directly by the pretrained fusion subnetwork.
  • Implementation: The joint model is trained interactively with FSFNet and FGC2Net using PyTorch, Adam optimization, 352×352 inputs, and random horizontal flipping.Each batch contains eight randomly selected images.
  • Evaluation metrics: Fusion quality is evaluated with MI, VIF, and CC, covering information transmission, information fidelity, and similarity to source images.These metrics are selected to reflect source-to-fusion information preservation and human visual perception.
  • Evaluation metrics: MI quantifies information transmitted from infrared and visible source images to the fused image, with larger values indicating more transferred information.The measure is computed using a Kullback-Leibler formulation based on joint and marginal histograms.
  • Evaluation metrics: CC measures linear correlation between the fused image and source images, while higher CC indicates greater similarity.The formulation uses correlations with both infrared and visible inputs.

4.2. Joint Image Fusion and SOD Evaluation

On VT5000, IRFS is evaluated against existing fusion methods combined with CTDNet or FGC2Net. It performs favorably across fusion and SOD metrics, with qualitative results linking sharper contrast and reduced overexposure to more accurate saliency maps.

  • Experimental comparison: IRFS is compared with nine state-of-the-art fusion methods within temporary multi-task frameworks using CTDNet or FGC2Net.CTDNet receives each competing fusion result as input, while the original CTDNet settings remain unchanged.
  • Quantitative results: IRFS consistently outperforms existing methods on MI, VIF, CC, Sα, Fβ, Fw, and MAE evaluation metrics.The comparison covers both intermediate fusion quality and final saliency detection performance.
  • Quantitative results: 45.8% higher VIF, 5.2% higher MI, and 2.18% higher CC are reported for IRFS over the respective second-best methods.These gains concern the fusion metrics on VT5000.
  • Quantitative results: IRFS ranks first on four SOD metrics, with gains of 2.08% in mean Fβ and 5.56% in mean MAE over the second-best method.FGC2Net-based temporary frameworks also outperform those using CTDNet.
  • Qualitative results: IRFS produces more pronounced object-background contrast and suppresses overexposure, supporting more accurate saliency maps than competing temporary frameworks.The qualitative comparison is shown on VT5000 in Figure 5.
  • Conclusion: The quantitative and qualitative findings support a cooperative relationship between thermal-visible fusion and SOD within the interactively reinforced paradigm.The paper reports effectiveness for the joint image fusion and SOD design.

4.3. Generalization Analysis on Image Fusion

IRFS generalizes its fusion performance to TNO, RoadScene, and M3FD without fine-tuning. It achieves large advantages on VIF and CC, while TNO MI does not rank first.

  • Experimental comparison: IRFS is evaluated against existing fusion methods on TNO, RoadScene, and M3FD for intermediate fusion generalization.The evaluation compares fused outputs across three public datasets.
  • Quantitative results: IRFS outperforms existing infrared-visible fusion methods by large margins on VIF and CC metrics.The paper attributes this result to the reverse push from SOD, which better preserves source-image information.
  • Quantitative results: IRFS does not rank first on MI for TNO, but its results remain favorable under direct generalization evaluation.The model is evaluated without fine-tuning on the fusion datasets.
  • Qualitative results: Qualitatively, IRFS produces sharper salient objects and cleaner backgrounds than competing methods across the evaluated fusion datasets.The comparison contrasts IRFS with blurry, over-smoothed, noisy, or artifact-prone outputs from several baselines.

4.4. Generalization Analysis on SOD

IRFS is tested against ten thermal-visible SOD methods on three datasets and ranks first or second across them. On VT5000, it improves several metrics over OSRNet and handles challenging object-localization cases.

  • Experimental comparison: IRFS is compared with ten state-of-the-art thermal-visible SOD methods across VT5000, VT1000, and VT821.Performance is measured with four common SOD metrics.
  • Quantitative results: IRFS ranks either first or second on the three evaluated SOD datasets.This result is reported as a generalization analysis against existing thermal-visible SOD methods.
  • Quantitative results: On VT5000, IRFS gains 2.1% in mean Fβ, 1.6% in mean Eξ, and 13.16% in MAE over OSRNet.Its mean Sα is 0.11% below OSRNet.
  • Qualitative results: IRFS produces more accurate saliency maps with fewer false detections than OSRNet, including improved localization of small objects inside ring structures.The reported challenging cases include thermal crossover, small objects, and low contrast.

4.5. Analysis of Model Efficiency

IRFS is evaluated for efficiency against state-of-the-art fusion and SOD methods, with separate comparisons for its FSFNet and FGC2Net modules.

  • The efficiency analysis compares model size and inference time across state-of-the-art fusion and thermal infrared-visible SOD methods.The comparison evaluates FSFNet against fusion methods and the SOD component against SOD methods to account for the multi-task setting.
  • FSFNet and FGC2Net achieve the fastest inference speed despite not being the most compact models.The authors attribute this to FSFNet’s dual attention-guided feature screening and FGC2Net’s lightweight ResNet-34 Siamese encoder.
  • CSRN has the smallest model size among the compared methods, but this compactness comes at the cost of performance.

4.6. Discussion for Weight 휏of Fusion Loss

The fusion-loss weight τ affects both fusion and SOD outcomes, with τ = 1.0 selected because it provides superior fusion results while nearly matching the best SOD precision.

  • τ = 1.0 produces fusion results that are noticeably superior to those from the alternative tested weights.Fusion is evaluated with CC and VIF metrics.
  • Fβ precision peaks at τ = 0.5, but improves by only 0.004 over τ = 1.0.This small difference supports selecting τ = 1.0 for the overall framework.

4.7. Ablation Studies

Ablation studies examine interactive loop learning, the fused image as an SOD modality, and FGSE components. The results support mutual task improvement and the contribution of the proposed fusion guidance.

  • Qualitative comparisons: Qualitative comparisons assess IRFS against state-of-the-art fusion and SOD methods, including challenging thermal-crossover and low-contrast cases.
  • Interactive loop learning: Increasing interactive training loops progressively improves fusion and SOD performance, supporting a collaborative relationship between the tasks.The comparison examines one-stage training against interactive training at the 1st, 5th, and 9th intervals, with qualitative results showing stronger object highlighting and weaker background interference.
  • Pixel-level fusion for SOD: Using FSFNet’s fused image to guide FGC2Net yields more precise saliency maps with coherent edges and complete objects than using RGB or thermal infrared guidance.The fused image also achieves more desirable quantitative improvements across Sα, Fβ, Eξ, and MAE.
  • FGSE ablation: Removing C2FTL lowers the MAE score by 12.1%, while removing LFS lowers mean Fβ by 1.9% on VT5000.
  • FGSE ablation: Removing both C2FTL and LFS causes performance to drop by more than 2.5% in Fβ and 9.0% in MAE.These ablations indicate contributions from FGSE components to saliency-related feature enhancement and interference suppression.

5. Conclusion

The paper presents IRFS as an interactively reinforced framework jointly addressing infrared-visible image fusion and SOD for unmanned-system object discovery. Its experiments support collaboration between the tasks, while dynamic adaptation to low-light and adverse weather remains unresolved.

  • IRFS jointly performs infrared-visible image fusion and saliency object detection within an interactively reinforced paradigm for unmanned systems.Fusion highlights saliency-related features, while SOD propagates semantic loss back to fusion to prevent semantic information loss.
  • Comprehensive experiments reveal that fusion and SOD maintain a collaborative relationship within a single framework.
  • The paradigm has not yet been adapted to dynamically adjust fusion for low-light or adverse-weather SOD scenarios.The authors identify this adaptation as an important direction for improving real-world utility.
Loading 2305.09999v1…