Source-linked AI summary
One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion
Chunyang Cheng, Tianyang Xu, Zhenhua Feng, Xiaojun Wu, ZhangyongTang, Hui Li, Zeyang Zhang, Sara Atito, Muhammad Awais, Josef Kittler
TL;DR
The paper addresses the semantic gap and resource costs of high-level task supervision in image fusion. GIFNet instead uses low-level digital photography fusion tasks, shared reconstruction, and cross-task interaction, achieving generalised fusion with one model and extending to single-modality enhancement.
Problem
High-level supervision for image fusion can create semantic mismatch, increased computational cost, and limited generalisation across fusion contexts.
Method
GIFNet uses low-level digital photography fusion to supervise unsupervised multimodal fusion through cross-task interaction, shared reconstruction, and cross-fusion gating.
Results
GIFNet demonstrates robust fusion performance and generalisation across fusion tasks while also producing enhanced outputs for single-modality inputs.
Takeaways & Limitations
The framework supports task-independent image fusion and extends image-fusion techniques beyond multi-modal scenarios to single-modality enhancement.
Abstract
from arXiv · showhide
Advanced image fusion methods mostly prioritise high-level missions, where task interaction struggles with semantic gaps, requiring complex bridging mechanisms. In contrast, we propose to leverage low-level vision tasks from digital photography fusion, allowing for effective feature interaction through pixel-level supervision. This new paradigm provides strong guidance for unsupervised multimodal fusion without relying on abstract semantics, enhancing task-shared feature learning for broader applicability. Owning to the hybrid image features and enhanced universal representations, the proposed GIFNet supports diverse fusion tasks, achieving high performance across both seen and unseen scenarios with a single model. Uniquely, experimental results reveal that our framework also supports single-modality enhancement, offering superior flexibility for practical applications. Our code will be available at https://github.com/AWCXV/GIFNet.
1. Introduction
GIFNet replaces high-level semantic task interaction with low-level digital photography fusion tasks, using pixel-level supervision to support shared feature learning. Its three-branch design targets versatile, task-agnostic fusion while also extending to single-modality enhancement.
- Motivation: High-level supervision introduces a semantic gap with pixel-focused fusion, requiring resource-heavy bridging modules or pretrained models that generalise poorly.High-level tasks encode object categories, shapes, and scene layouts rather than fine-grained image details.
- Motivation: Low-level digital photography fusion provides pixel-level feature alignment and supervision more naturally aligned with multimodal fusion.The approach uses these tasks to enhance task-shared image features without relying on high-level semantics.
- Method: GIFNet uses main, auxiliary, and reconciling branches to interact across multimodal and digital photography fusion while learning a universal representation through reconstruction.The main and auxiliary branches alternate between task features, while the reconciling branch centres on shared reconstruction.
- Contributions: The method integrates multimodal and digital photography fusion tasks in one model, broadening applicability across fusion scenarios without task-specific adaptation.Its contributions include collaborative low-level task training, a reconstruction task, and an augmented RGB-focused joint dataset.
- Contributions: GIFNet also extends fusion techniques to single-modality enhancement, supporting applications beyond multimodal inputs.The paper presents this as an extension of image fusion models beyond the multi-modal domain.
2. Related work
Related work has pursued generalised fusion through high-level semantics, continual learning, and diverse architectures, but often retains task-specific data or models. GIFNet instead uses low-level cross-task interaction and a cross-fusion gating mechanism to support generalised fusion with one model.
- Semantic task interaction: High-level semantic methods can improve downstream multimodal tasks but depend on costly labels and large vision models that may reduce relevance to new fusion tasks.Their semantic gap with image-fusion features can also impair fused-image quality.
- GIFNet: GIFNet combines low-level vision tasks to establish cross-task interaction without scenario-specific cooperation mechanisms designed to reduce semantic gaps.The approach uses diverse fusion tasks to extract foundational and targeted features.
- Generalised fusion: Existing generalised methods often require extensive multi-task training data, multiple fusion models, or task-specific fusion rules for different scenarios.These requirements limit true generalisation without further task-specific training.
- GIFNet: A cross-fusion gating mechanism interacts across representative multimodal and digital photography fusion tasks, enabling one model for task-independent generalised fusion.The resulting hybrid image features and low-level representations support this unified use.
3. The Proposed GIFNet
GIFNet uses low-level interaction between multi-modal and digital photography fusion tasks to learn shared and task-specific features in a unified three-branch architecture. Its training alternates task branches with cross-fusion gating, while inference uses one image pair for fusion or identical inputs for single-modality enhancement.
- Single-modality enhancement: By setting both inputs to the same image, GIFNet performs fusion-like single-modality enhancement focused on refining image details.The enhanced output is denoted by X̂, and the model is intended to extend beyond strictly multi-modal fusion scenarios.
- Three-branch architecture: GIFNet combines main, auxiliary, and reconciling branches to separate task-specific extraction from shared foundational features.The reconstruction branch derives universal features from common RGB data, while the shared encoder is the only foundational feature extractor shared across tasks.
- Cross-task interaction: The cross-fusion gating mechanism alternates the multi-modal and digital photography branches, selectively routing hybrid features to the global decoder.The main branch is updated while the auxiliary branch is frozen, and their representations interact through cross-attention controlled by a learnable parameter λ.
- Training objectives: The reconstruction branch uses public loss to enforce consistency between its output and the shared RGB modality, while private losses are defined for each task branch.The framework uses structural similarity and mean squared error components for the public loss, with task-specific optimization applied during iterative interaction.
- Training and inference: During inference, one image pair supplies the task inputs, shared features are extracted, CFGM fuses task-specific representations, and the global decoder reconstructs the fused image.This differs from training, where the branches are optimized through iterative task interaction.
4. Experimental Results
GIFNet is evaluated across ablations, seen and unseen fusion tasks, and single-modality enhancement. Results support low-level cross-task interaction, adaptive CFGM, and shared representations as contributors to broad performance.
- Experimental Settings: GIFNet is evaluated directly on seen and unseen fusion tasks after training with IVIF data and augmented digital-photography data, without adaptation or fine-tuning.The evaluation includes IVIF, MFIF, medical image fusion, and other tasks.
- Ablation Experiments: Adding another task without CFGM or REC prevents convergence, whereas either component produces valid fusion results and both yield GIFNet’s best performance.The combined components optimise cross-task interaction and feature alignment.
- Ablation Experiments: Additional digital-photography supervision consistently improves multimodal fusion, with MFIF providing more compatible pixel-level supervision than MEIF.The authors attribute this difference to MFIF’s emphasis on higher-clarity content and information preservation.
- Ablation Experiments: Adaptive CFGM provides superior control over task interaction and more robust fused images than conventional fusion operations.The comparison is reported both quantitatively and qualitatively.
- Feature Visualisation: The shared encoder captures foundational contours and structural details, while the MM and DP branches respectively preserve salient source information and enhance fine image details.These branch-specific patterns are observed across seen and unseen fusion tasks.
- Fusion Results: GIFNet achieves strong results across MFIF, IVIF, MEIF, NIR-VIS, and medical fusion, including a 25% VIF increase in MFIF and higher performance on all reported MEIF metrics.For MEIF, the reported gains include VIF (+46.7%) and AG (+37.8%); GIFNet generally achieves the best IVIF performance and outperforms existing NIR-VIS methods.
- Single Modality: the Classification Task: GIFNet improves visual quality on CIFAR100 images and is the only compared method to surpass the original training setting in classification performance.The visual results include blur mitigation and edge enhancement, while the classification comparison is reported in Table 3.
5. Conclusion
GIFNet uses low-level task interaction, shared reconstruction, and an RGB-based joint dataset to reduce task and domain discrepancies in generalised image fusion. It achieves robust fusion generalisation and extends fusion to single-modality enhancement.
- GIFNet introduces low-level task interaction for generalised image fusion.
- Shared reconstruction and an RGB-based joint dataset reduce task and domain discrepancies through collaborative training.
- A cross-fusion gating mechanism supports superior generalisation and robust fusion performance.
- GIFNet extends fusion techniques to single-modality enhancement.