PaperlayerView exact revision on arXiv

Source-linked AI summary

HP-UniIF: Hierarchical Prompt Learning for Unified Image Fusion

Xingxin Xu, Siqi Zhao, Xin Li, Xinjie Yao, Yiming Sun, Pengfei Zhu

arXiv:2608.21786v1cs.CV

TL;DR

Existing fusion systems do not adequately unify heterogeneous tasks, degradation restoration, and downstream perception, despite deployments requiring all three. HP-UniIF uses diffusion priors with hierarchical prompt learning to separate these conditions across network stages. It consistently improves visual quality and downstream task performance across diverse fusion, degradation, and application settings.

  • Problem

    Existing methods typically address fusion quality, degradation robustness, and downstream adaptability separately, while practical systems require them jointly.

  • Method

    HP-UniIF uses a shared diffusion backbone with hierarchical prompt learning through task prompt modulation, degradation prompt routing, and an application prompt bank.

  • Results

    HP-UniIF consistently improves visual quality and downstream task performance across diverse fusion settings, degradation types, and application scenarios.

  • Takeaways & Limitations

    The framework jointly supports heterogeneous fusion tasks, diverse degradations, and multiple downstream applications while organizing conditions across complementary representation stages.

  • Takeaways & Limitations

    The related-work scope notes that existing application-oriented methods are often tied to predefined applications or fixed task networks, limiting flexible reuse.

Abstract

from arXiv · show

General image fusion seeks to integrate complementary information from multiple source images, yet real-world applications often require a single system to support heterogeneous fusion, degradation restoration, and task-oriented perception simultaneously. Existing unified frameworks struggle with these orthogonal objectives, resulting in entangled representations and degraded performance across subtasks. We propose HP-UniIF, a unified vision framework that leverages diffusion priors to bridge heterogeneous fusion, visual restoration, and downstream perception. To address the limited adaptability of diffusion models to domain-, degradation-, and task-level objectives within one pipeline, HP-UniIF introduces a depth-wise hierarchical conditional modulation strategy that decouples these objectives across network stages. Task prompt modulation at bottleneck layers adapts the backbone to different fusion paradigms, the degradation prompt router at shallow layers injects degradation-aware constraints for local restoration, and the application prompt bank at decoding stages aligns generation with downstream tasks. This hierarchical design enables HP-UniIF to produce visually faithful results while preserving task-relevant semantics. Extensive experiments across multiple fusion tasks, diverse degradations, and various downstream applications demonstrate the superior performance of HP-UniIF.

Introduction

Image fusion must increasingly handle heterogeneous tasks, degraded inputs, and downstream semantic requirements within one deployment setting. HP-UniIF addresses this M3 challenge by coordinating task, degradation, and application prompts across a shared diffusion prior.

  • Introduction: Modern deployment demands fusion models that remain visually effective, degradation-robust, and adaptable to downstream perception.Relevant applications include object detection, semantic segmentation, and medical diagnosis.
  • Introduction: Existing methods typically treat fusion quality, degradation robustness, and downstream adaptability as separate problems requiring specialized models.
  • Introduction: The M3 setting combines multiple fusion tasks, degradation conditions, and downstream applications, whose differing supervision can interfere when optimized together.The proposed solution requires a strong shared visual prior with structurally separated adaptation paths.
  • Introduction: HP-UniIF uses a pretrained diffusion framework with hierarchical prompt learning for task adaptation, degradation-aware restoration, and application-specific semantic guidance.TPM adapts fusion stages, DPR retrieves input-adaptive degradation prompts, and APB injects application guidance.
  • Introduction: HP-UniIF consistently improves visual quality and downstream task performance across diverse fusion settings, degradation types, and application scenarios.

Related Work

Prior work separately advances unified fusion, degradation-aware restoration, and task-oriented fusion, but downstream methods remain difficult to reuse across applications. HP-UniIF instead coordinates multiple conditions within one diffusion-based framework.

  • General Image Fusion Across Fusion Tasks: Unified-fusion methods share cross-task knowledge through task-specific rules, unsupervised learning, feature decomposition, cross-domain interaction, or reusable memory.
  • Image Fusion in Complex Environments: Degradation-aware fusion progresses from methods targeting specific illumination or noise conditions to approaches handling multiple corruptions.
  • Image Fusion for Downstream Applications: Task-oriented fusion incorporates supervision for object detection or segmentation and later strengthens semantic alignment through meta-learning and higher-order interactions.
  • Image Fusion for Downstream Applications: HP-UniIF coordinates task, degradation, and application prompts within a shared diffusion prior rather than treating these requirements as isolated settings.
  • Image Fusion for Downstream Applications: Existing application-oriented methods are predefined for one application or tightly coupled to a fixed task network, limiting reuse across multiple downstream applications.

Method

HP-UniIF uses a pretrained latent diffusion backbone with hierarchical prompts to coordinate task-specific fusion, degradation restoration, and application adaptation. Its modules operate across the encoder–decoder pipeline, with staged training separating fusion learning, restoration, and application-specific optimization.

  • Diffusion Backbone: HP-UniIF builds on a pretrained latent diffusion model whose VAE extracts latent and hierarchical skip features, while the denoising U-Net performs iterative refinement before VAE reconstruction.The two source latent codes initialize the diffusion process, and denoising proceeds from zT to z0 in a coarse-to-fine reconstruction pipeline.
  • Hierarchical Processing: At each decoder level, DPR restores skip features, TPM performs task-conditioned modulation, and APB further adapts representations when an application is specified.This depth-wise ordering separates degradation, fusion-task, and downstream-application conditioning within the decoder pipeline.
  • Task Prompt Modulation: Task Prompt Modulation progressively adapts shared latent and decoder representations to the specified fusion task using feature-guided, stage-specific prompts.A learnable task prompt is refined from coarse semantics to fine spatial details, then produces affine parameters and an injection gate for controlled modulation.
  • Degradation Prompt Router: The Degradation Prompt Router restores source skip features before cross-source fusion by combining reusable degradation prompts with input-adaptive conditions.Its descriptor uses the two source features, their discrepancy, and decoder context; a restoration block predicts source-specific corrections and gates their injection.
  • Application Prompt Bank: The Application Prompt Bank selects application-specific prompts to adapt decoder representations while preserving shared fusion and restoration paths.It uses gated prompt-injection and content-retention branches, is bypassed without an application identity, and is instantiated for object detection and semantic segmentation.
  • Training Strategy: HP-UniIF uses three training stages that successively optimize fusion, degradation restoration, and application adaptation while freezing earlier pathways.New applications require only an application-specific prompt and lightweight adapters, optimized with the corresponding training data and objective.

Experiments

Experiments evaluate HP-UniIF across clean and degraded VIF, MEF, and MFF fusion, plus downstream semantic segmentation and object detection. Across these settings, the method shows strong qualitative robustness, broad metric performance, and task generalization.

  • Experimental setup: HP-UniIF is compared with general, generalized or degradation-aware, and task-specific fusion baselines across multiple evaluation settings.The experiments cover VIF, MEF, and MFF, with downstream evaluation using common detection and segmentation models.
  • Multi-Task Fusion without Degradation: Under clean conditions, HP-UniIF preserves salient targets, appearance, focused structures, and local details across VIF, MEF, and MFF.Qualitative comparisons report preserved infrared targets and pavement structures, balanced exposure, and clear foreground and background contours.
  • Multi-Task Fusion under Degraded Conditions: Under degraded conditions, HP-UniIF suppresses noise, balances exposure, and sharpens structures while adapting to the distinct objectives of VIF, MEF, and MFF.The qualitative comparisons cover noise, color distortion, structural blur, dark-region underexposure, saturation artifacts, residual defocus, and brightness shifts.
  • Quantitative fusion results: HP-UniIF achieves the best performance on all 17 metrics across the three fusion tasks under diverse degraded conditions.The reported gains span fidelity, structural preservation, information retention, and perceptual quality.
  • Downstream applications: HP-UniIF achieves the best overall results in object detection and semantic segmentation, including the highest mIoU and mAcc and the best class-wise IoU on five of eight categories.Its more pronounced improvement in mAP@.5:.95 is associated with more accurate object localization, while APB provides application-specific semantic guidance.
  • Ablation studies: Ablations show that jointly using TPM and DPR gives the strongest overall fusion performance, while APB improves downstream tasks with minor intrinsic-quality variation.The results support complementary roles for task-aware fusion, degradation-aware restoration, and application-specific semantics.

Conclusion

HP-UniIF unifies heterogeneous fusion, degradation handling, and downstream application adaptation through hierarchical prompt learning over diffusion priors. Experiments across fusion tasks, degraded conditions, object detection, and semantic segmentation support its effectiveness.

  • HP-UniIF jointly supports heterogeneous fusion tasks, diverse degradations, and multiple downstream applications within one diffusion-prior framework.
  • Hierarchical prompt learning organizes task-aware fusion, degradation-aware restoration, and application-specific adaptation across complementary representation stages.This design is intended to mitigate interference among heterogeneous objectives.
  • Experiments on VIF, MEF, and MFF demonstrate effectiveness under clean and degraded conditions.
  • Downstream evaluations validate advantages for object detection and semantic segmentation.
  • Future work will address unseen degradations, broader applications, lightweight prompt learning, and efficient real-world deployment.
Loading 2608.21786v1…