Source-linked AI summary
Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion
Xunpeng Yi, Han Xu, Hao Zhang, Linfeng Tang, Jiayi Ma
TL;DR
Existing image fusion methods struggle with degraded source images and cannot flexibly satisfy changing user and task requirements. Text-IF integrates an image fusion pipeline with semantic text encoding and interaction guidance for degradation-aware, interactive fusion. Experiments report advantages in fusion performance and degradation treatment, while the text-feature design assumes semantically similar texts produce nearby features.
Problem
Existing fusion methods struggle with degradations in low-quality infrared and visible images and provide relatively fixed results for changing user and application needs.
Method
Text-IF couples an image fusion pipeline with a text semantic encoder and semantic interaction guidance module to guide degradation-aware, interactive fusion.
Results
Text-IF demonstrates advantages over existing methods in fusion performance and degradation treatment and can generate fusion images from interactive user text input.
Takeaways & Limitations
The framework supports flexible, user-required fusion outcomes and extends image fusion to multi-modal information fusion.
Takeaways & Limitations
The text-feature design assumes semantically similar texts produce nearby features in the reduced Euclidean space.
Abstract
from arXiv · showhide
Image fusion aims to combine information from different source images to create a comprehensively representative image. Existing fusion methods are typically helpless in dealing with degradations in low-quality source images and non-interactive to multiple subjective and objective needs. To solve them, we introduce a novel approach that leverages semantic text guidance image fusion model for degradation-aware and interactive image fusion task, termed as Text-IF. It innovatively extends the classical image fusion to the text guided image fusion along with the ability to harmoniously address the degradation and interaction issues during fusion. Through the text semantic encoder and semantic interaction fusion decoder, Text-IF is accessible to the all-in-one infrared and visible image degradation-aware processing and the interactive flexible fusion outcomes. In this way, Text-IF achieves not only multi-modal image fusion, but also multi-modal information fusion. Extensive experiments prove that our proposed text guided image fusion strategy has obvious advantages over SOTA methods in the image fusion performance and degradation treatment. The code is available at https://github.com/XunpengYi/Text-IF.
1. Introduction
Image fusion combines complementary infrared and visible information, but degradations, model-switching burdens, and fixed outputs limit existing methods. Text-IF integrates degradation-aware processing with semantic text guidance to support interactive, flexible fusion.
- Motivation: Infrared images provide thermal information, while visible images provide reflectance-based visual information for complementary fusion.Their combination yields a more comprehensive scene representation than either modality alone.
- Challenges: Visible images can suffer low light and overexposure, while infrared images can suffer noise and diminished contrast.Existing fusion methods do not adaptively resolve these degradations, and manual preprocessing reduces flexibility and efficiency.
- Challenges: Handling individualized degradations with separate restoration models requires frequent switching and can disrupt harmony between enhancement and fusion.This separation approach brings operational trouble and unsatisfactory overall performance in complex, around-the-clock conditions.
- Challenges: Fixed, non-interactive fusion results can fall short when users and application tasks require different outcomes.Real-world fusion is complex, flexible, and task-oriented, so requirements may vary across subjective and objective needs.
- Proposed approach: Text-IF integrates an image fusion pipeline with a text semantic encoder and semantic interaction guidance module for degradation-aware, interactive fusion.Its Transformer-based image extraction, cross fusion, and coupled text-image semantic features target high-quality, user-guided results without predefined rules.
2. Related Work
Prior work advanced general image fusion and text-guided image processing, but existing fusion methods remained inadequate for degraded complex scenes and interactive user control. Text-IF brings semantic text guidance into image fusion to address these limitations.
- General Image Fusion Methods: Deep-learning image fusion progressed from autoencoder and CNN strategies to data-driven, multi-task, and diffusion-based methods.U2Fusion is described as an early all-in-one fusion method using learned weights and continuous learning.
- Text-Image Models: Text-image models use aligned representations and text guidance to customize image generation and processing.CLIP provides zero-shot recognition and robust text-image feature extraction that supports such applications.
- Text-IF: Text-IF uses text semantic features to guide image fusion through the Semantic Interaction Guidance Module.The workflow combines an image fusion pipeline with a text semantic feature encoder.
- Research gap: PromptIR removes various degradations with learnable visual prompts, but it does not provide text guidance designed for multi-modal degradation and fusion.This leaves a gap between prompt-based restoration and interactive text-guided image fusion.
3. The Proposed Method
Text-IF extends image fusion with text-semantic guidance to address degraded infrared and visible inputs and produce more customized fusion results. Its architecture combines multimodal image features, text interaction, and semantically regulated fusion losses.
- Problem formulation: Text-IF rewrites fixed image fusion as text-guided fusion, enabling customized results and restoration-aware fusion for degraded source images.The text semantic information guides the fusion function and allows flexible processing according to user-provided text.
- Image fusion pipeline: The image pipeline separately encodes visible and infrared inputs with Transformer/Restormer-based feature extractors.Visible and infrared images have different channel structures, and their encoder features are later integrated across modalities.
- Cross fusion layer: Cross-attention exchanges information between visible and infrared features before concatenating the resulting representations into fusion features.The cross fusion layer uses cross-attention and query exchange for interaction across modalities.
- Semantic interaction fusion decoder: The semantic interaction fusion decoder couples text features with image fusion features through Transformer decoder blocks and a cascaded Semantic Interaction Guidance Module.The decoder performs dense regulation and guidance across multiple stages, with upsampling between decoder levels to match encoder downsampling.
- Text semantic encoder: A frozen CLIP text encoder converts task and degradation descriptions into semantic features that guide the fusion network.Semantically similar texts are intended to produce nearby features in a reduced Euclidean space.
- Loss functions: The loss framework uses text-regulated weights over intensity, SSIM, maximum-gradient, and color-consistency losses.These losses respectively emphasize salient intensity, structural similarity, source-image edges, and visible-image color consistency, using high-quality visible and infrared images as constraints.
4. Experiments
Experiments evaluate Text-IF against existing fusion methods without semantic guidance, under source-image degradations, and on object detection. Results indicate strong fusion quality, degradation handling, and downstream detection performance.
- Implementation Details: Text-IF is trained on text-guided fusion data using 3,618 image pairs for training and 1,135 for testing.Training uses AdamW with a learning rate of 0.0001, batch size 16, and 96 × 96 crops.
- Comparison without Text Guidance: Qualitatively, Text-IF produces more prominent thermal targets, appropriate brightness, clearer details, and more natural colors than competing fusion methods.These comparisons cover MSRS, LLVIP, and RoadScene data.
- Comparison without Text Guidance: Without additional semantic information, Text-IF performs best on all five metrics for MSRS and LLVIP and on three metrics for RoadScene.The reported metrics are SCD, SD, EN, VIF, and QAB/F.
- Comparison with Text Guidance: With simple text guidance, Text-IF handles low-light, low-contrast, noisy, and over-exposed source images without switching among degradation-specific restoration models.The method uses the same model parameters across degradation scenarios, while comparison systems combine restoration and fusion methods.
- Comparison with Text Guidance: Text-IF achieves overall optimal performance across all metrics on MSRS, LLVIP, MFNet, DN-MSRS, and RoadScene degraded-image datasets.SD, EN, and SF reflect information transfer, while CLIP-IQA, NIQE, MUSIQ, and BRISQUE assess fusion quality under degradations.
- Performance on High-level Task: On LLVIP object detection, Text-IF detects all scene objects qualitatively and obtains the best quantitative detection performance.The experiments use YOLOv8 fine-tuned on LLVIP infrared-visible source images.
- Ablation Experiment: Ablation results show that each loss contributes to quantitative evaluation, while the complete method achieves the best qualitative and quantitative results.The analyzed losses include intensity, SSIM, maximum gradient, and color consistency losses.
5. Conclusion
Text-IF extends image fusion with text-semantic guidance to address degradations and user-directed interactivity. Experiments report advantages in fusion performance and degradation treatment.
- Text-IF combines an image fusion pipeline, text semantic feature extraction, and semantic interaction guidance for text-guided fusion.
- The framework targets complex-scene fusion with degraded inputs and user-required outputs through interactive text input.
- Extensive experiments demonstrate advantages in both fusion performance and degradation treatment.