PaperlayerView exact revision on arXiv

Source-linked AI summary

IT-TextFusion: Iterative Text-Image Interaction with Text-Guided Residual Refinement for Degradation-Aware Image Fusion

Siyang Liu, Peiyi Zhou, Tianle Jin, Rongrong Bian, Zheke Jin, Mengze Gao

arXiv:2609.01092v1cs.CV

TL;DR

Existing text-guided fusion methods provide limited semantic-visual interaction and cross-modal conditioning under complex degradations. IT-TextFusion maintains degradation-aware global text conditioning across hierarchical fusion and residual refinement. Experiments report improvements on several fusion and perceptual-quality metrics, alongside trade-offs on others.

  • Problem

    Existing text-guided fusion methods often use shallow semantic-visual interaction and limited cross-modal conditioning, restricting their use of textual guidance under complex degradations.

  • Method

    IT-TextFusion combines degradation-aware text engineering, stage-specific text-conditioned modulation, deepest-level Cross-Attention, multi-scale Cross-Gate Fusion, and text-guided residual refinement.

  • Results

    The method achieves competitive overall performance, improving several fusion and perceptual-quality metrics while showing trade-offs on others.

  • Takeaways & Limitations

    Global text conditioning is maintained across hierarchical decoding and residual refinement to support detail preservation and visual consistency under degradations.

  • Takeaways & Limitations

    On the EMS dataset, the method is visually comparable to Text-IF in some cases, while metric comparisons remain mixed across NIQE, BRISQUE, MUSIQ, and other criteria.

Abstract

from arXiv · show

Text-guided image fusion has recently emerged as an effective paradigm for integrating multi-modal information while enabling flexible and task-oriented fusion control. However, existing text-guided fusion methods often rely on shallow semantic-visual interaction and limited attention mechanisms, which restrict their ability to robustly handle complex degradations and fully exploit textual guidance. In this paper, we propose an iterative text-guided image fusion framework that incorporates text-conditioned feature interaction across multiple fusion and refinement stages. The proposed method integrates deepest-level Cross-Attention, multi-scale Cross-Gate Fusion, and stage-specific text-conditioned modulation, allowing the global text embedding to condition hierarchical feature fusion and residual refinement. By repeatedly injecting the pooled text embedding across hierarchical decoder and refinement stages, the proposed framework provides degradation-aware global semantic conditioning while preserving complementary information from the visible and infrared modalities. Experiments on several benchmark datasets show that the proposed method improves several information-preservation and perceptual-quality metrics, while exhibiting metric-dependent trade-offs on some datasets.

1 School of Automation, Southeast University, Nanjing, China

The paper presents IT-TextFusion as a degradation-aware framework for text-guided infrared-visible image fusion.

  • IT-TextFusion uses hierarchical text conditioning to guide fusion across multiple decoder stages.
  • Degradation-aware prompts provide scenario-specific global semantic guidance.
  • Cross-Gate Fusion adaptively aggregates visible and infrared features at multiple hierarchical scales.
  • Text-guided residual refinement improves high-resolution feature reconstruction under degradations.

1. Introduction

The introduction motivates infrared-visible fusion for robust perception under challenging conditions and identifies limitations in existing text-guided methods. IT-TextFusion addresses these limitations through degradation-aware prompts, hierarchical text conditioning, adaptive feature aggregation, and text-guided refinement.

  • Adverse conditions and complex traffic scenes reduce camera-based perception reliability, motivating multi-modal fusion.
  • Infrared-visible fusion combines complementary visual and thermal information for safety-critical perception tasks.
  • Existing text-guided methods often introduce text as a high-level signal with shallow semantic-visual interaction and limited cross-modal interaction.
  • Existing refinement approaches may not propagate semantic guidance consistently, leaving residual artifacts under severe or spatially heterogeneous degradations.
  • The proposed framework strengthens hierarchical text-conditioned fusion and residual refinement using degradation-aware text engineering.
  • ITIM applies textual guidance across multiple fusion stages, while TG-RRM adds text-conditioned modulation during refinement.

2. Related Work

Prior fusion systems use convolutional, residual, transformer, probabilistic, and text-guided strategies to combine modalities and semantics. IT-TextFusion distinguishes itself by maintaining stage-specific global text conditioning through hierarchical fusion and high-resolution residual refinement.

  • Encoder-decoder and residual architectures preserve complementary information from multiple sources, while transformers model long-range dependencies through self-attention.
  • Existing alternatives include feature-statistics alignment, probabilistic late fusion, and text-guided transformer or convolutional fusion.
  • Vision-language methods use shared image-text representations, pixel-text similarity, contextual prompting, or textual conditioning for visual control.
  • TextFusion, TeRF, TITFormer, MGFusion, and Text-IF explore affine modulation, region-aware control, cross-modal attention, CLIP-guided modulation, and degradation-adaptive fusion.
  • Existing text-guided fusion methods differ in the location, representation, and frequency of textual conditioning.
  • IT-TextFusion maintains global text conditioning across hierarchical decoding and residual refinement through stage-specific textual conditioning.

3. Method

The framework combines degradation-aware text engineering with hierarchical visible-infrared fusion and text-conditioned residual refinement. Cross-modal attention and gating operate alongside repeated global text conditioning across fusion and refinement stages.

  • Text Engineering: Degradation-aware prompts encode the fusion objective, degradation characteristics, and scenario-specific semantic priorities.Category-specific prompts provide global conditioning without changing the fusion-network architecture.
  • Image Fusion Pipeline: The image pipeline uses independently encoded visible and infrared features with hierarchical representations for subsequent fusion and decoding.Four encoder stages provide multi-scale features at progressively reduced spatial resolutions.
  • Image Fusion Pipeline: Bidirectional deepest-level Cross-Attention exchanges information between visible and infrared features through cross-modal affinities.Each modality queries the other modality's keys and values, with projected responses added through residual connections.
  • Image Fusion Pipeline: Cross-Gate Fusion adaptively regulates cross-attended features to preserve informative cues and suppress degraded or irrelevant responses.Learned gates modulate modality contributions, and the gated features are combined and projected through convolution.
  • Text Semantic Encoding and Feature Modulation: The framework repeatedly injects one pooled global text embedding into hierarchical decoder features through stage-specific conditioning.This design provides global semantic conditioning at each hierarchical stage rather than tokenwise language-vision attention.
  • Text-Guided Residual Refinement: TG-RRM uses text-guided residual refinement to enhance fine-grained details and structural coherence while improving visual consistency and suppressing degradation-related artifacts.The module is described as a multi-stage refinement component within the overall image fusion workflow.

4. Results

The results section evaluates IT-TextFusion through qualitative and quantitative comparisons, followed by ablations of its components.

  • The evaluation combines qualitative and quantitative comparisons of the proposed method for text-guided image fusion.
  • Ablation experiments analyze the contributions of different components in the proposed framework.
  • Table 1 compares IT-TextFusion with existing fusion methods and Text-IF on MSRS, LLVIP, and RoadScene using a fixed generic prompt.

4.1. Implementation Details

The implementation uses PyTorch training with AdamW, scheduled optimization, data augmentation, and validation-based checkpoint selection.

  • Models are implemented in PyTorch and trained on an NVIDIA RTX A5000 GPU using AdamW with a 1×10−4 learning rate.Weight decay is 5×10−2, batch size is 8, and training lasts 120 epochs.
  • The final checkpoint is selected by the lowest validation loss.
  • Training uses random 96×96 crops plus horizontal and vertical flips applied with probability 0.5.

4.2. Datasets

The study evaluates standard infrared-visible fusion datasets and constructs an extended multi-degradation dataset with paired prompts for diverse imaging artifacts.

  • Standard evaluation uses MSRS, LLVIP, MFNet, and RoadScene, which primarily represent single-degradation conditions.MSRS and LLVIP mainly contain low-light visible images, while MFNet focuses on low-contrast infrared imagery.
  • Because MSRS and LLVIP lack ground-truth fused images, their original infrared-visible pairs serve as supervision targets.
  • The extended multi-degradation dataset is built from MSRS, LLVIP, and MFNet.
  • It includes nine degradation types spanning low light, rain, blur, overexposure, haze, noise, low contrast, and infrared stripe noise.
  • Each degraded sample contains an RGB–IR pair and a textual description for text-conditioned fusion training.
  • Additional prompts follow existing templates for degradation types without predefined descriptions.

4.3. Evaluation Metrics and Comparison Methods

The study uses established fusion and no-reference perceptual metrics, compares against text-guided and conventional fusion methods, and defines SSIMsum from both source modalities.

  • Evaluation Metrics: Evaluation reports entropy, standard deviation, spatial frequency, mutual information, SCD, VIFF, QAB/F, and SSIMsum.
  • Evaluation Metrics: SSIMsum sums structural similarity between the fused image and each infrared or visible source image.It is not bounded by 1, and higher values indicate greater joint structural preservation.
  • Evaluation Metrics: For datasets without reference images, CLIP-IQA, NIQE, MUSIQ, and BRISQUE assess no-reference perceptual quality and naturalness.CLIP-IQA uses only the fused image here, not the degradation-aware prompt.
  • Comparison Methods: The proposed method is compared with Text-IF and conventional methods including U2Fusion, MetaFusion, DDFM, MURF, ReCoNet, TarDAL, and UMF-CMGR.
  • Comparison Methods: On the synthesized EMS dataset, comparison is limited to Text-IF.

4.4. Comparison with a Fixed Generic Prompt

With a fixed generic fusion prompt, the proposed architecture is evaluated against Text-IF to separate architectural effects from degradation-specific prompt content. Results show dataset- and metric-dependent gains, while repeated hierarchical text conditioning and residual refinement remain beneficial.

  • The fixed generic prompt describes only the infrared-visible fusion task, without degradation-aware or scene-specific semantics.
  • The architecture itself provides improved fusion capability before explicit degradation-aware text guidance is introduced.
  • The proposed method improves different metric subsets across datasets rather than uniformly outperforming Text-IF.On MSRS, it improves SCD, EN, VIFF, and QAB/F but has lower SD; the supplied comparison also reports metric trade-offs on RoadScene.
  • Repeated global text conditioning across four decoding stages and residual refinement regulates features at multiple resolutions.
  • The results suggest that hierarchical text-conditioned interaction and residual refinement contribute to performance beyond degradation-aware textual semantics alone.

4.5. Comparison with Degradation-Aware Text Guidance

Under degradation-aware text guidance, IT-TextFusion is compared with Text-IF and restoration-plus-fusion pipelines across standard and EMS datasets. The method delivers selected information-preservation, perceptual-quality, and qualitative gains, but results remain metric- and degradation-dependent rather than uniformly superior.

  • Degradation-aware prompts explicitly describe source-image conditions such as low illumination, noise, and low contrast.
  • Across four standard datasets, the proposed method exhibits different advantages and trade-offs across datasets and evaluation metrics.
  • On LLVIP, IT-TextFusion achieves the highest EN (7.502) and MUSIQ (54.067), together with the lowest NIQE (3.336).
  • On MFNet, IT-TextFusion obtains the highest SD (65.141), while Text-IF has higher EN (6.683) and IT-TextFusion has lower MUSIQ (33.942) than the best competing score.
  • On RoadScene, the proposed method achieves the lowest NIQE (3.330) and BRISQUE (27.571), but its SF (15.332) is below the best competing score.
  • On EMS, the method outperforms Text-IF on EN, SD, SF, SCD, QAB/F, and VIFF, while Text-IF is slightly higher on MI and SSIMsum.
  • No-reference perceptual metrics assess fused-image quality from statistical or learned perspectives and do not directly measure prompt alignment or multimodal information preservation.
  • Qualitatively, the method improves brightness, structural details, color consistency, and target-background separation across several degradations, while low-contrast infrared cases remain challenging.The EMS comparison reports more stable results under diverse degradations, but both methods are visually comparable for low-contrast infrared images.

4.6. Ablation Study

Ablation results show metric-dependent contributions from iterative interaction, residual refinement, Cross-Gate Fusion, and degradation-aware prompts. The complete model provides the strongest overall balance, although VIFF does not exceed the baseline.

  • The ablation progressively introduces degradation-aware text engineering, ITIM, Cross-Gate Fusion, and TG-RRM from a basic fusion backbone.
  • TG-RRM further improves SF and QAB/F and lowers NIQE, supporting its role in structural refinement and artifact suppression.
  • Cross-Gate Fusion slightly improves VIFF but decreases several other metrics, indicating a trade-off rather than uniform improvement.
  • Qualitative comparisons show clearer structures and better visual consistency, while the components jointly balance information preservation, structural fidelity, and perceptual naturalness.

5. Conclusion

The paper proposes a unified degradation-aware framework that maintains global text conditioning throughout hierarchical fusion and residual refinement. Experiments show competitive overall performance, with complementary component contributions and improvements on several metrics alongside trade-offs on others.

  • The framework introduces global textual guidance into hierarchical feature fusion and refinement for infrared-visible fusion under diverse degradations.
  • Task- and degradation-aware text engineering, stage-specific modulation, and Cross-Gate Fusion provide expressive and content-adaptive multimodal integration.
  • The Text-Guided Residual Refinement Module improves detail preservation and visual consistency under severe degradations.
  • Experiments demonstrate competitive overall performance, improving several fusion and perceptual-quality metrics while exhibiting trade-offs on others.
  • Ablations indicate complementary and synergistic contributions from hierarchical text conditioning, Cross-Gate Fusion, TG-RRM, and degradation-aware prompts.
Loading 2609.01092v1…