PaperlayerView exact revision on arXiv

Source-linked AI summary

CRT-HMAR: Causal Requirement Tracing-Guided Hierarchical Multi-Agent Regulation for Open-Task-Aware Infrared-Visible Image Fusion

Zengyi Yang, Shuai Yuan, Zhong-Cheng Wu, Juan Cheng, Huafeng Li, Yu Liu

arXiv:2610.09330v1cs.CV

TL;DR

Existing IR-VIS fusion methods struggle to generalize from fixed seen-task sets to unseen downstream tasks. CRT-HMAR traces task requirements and uses collaborative agents to regulate fusion and multi-task optimization; it reports stronger unseen-task generalization while maintaining seen-task balance.

  • Problem

    Existing task-oriented IR-VIS fusion methods struggle to adapt to unseen downstream tasks beyond their fixed task sets.

    Source: §I · p. 2, B. Task-Aware IR-VIS Image Fusion · p. 4

  • Method

    CRT-HMAR traces task-network responses to map semantic preferences into image-level requirements, then uses agents to guide fusion, balance tasks, and mitigate gradient conflicts.

    Source: §I · p. 2

  • Results

    CRT-HMAR improves generalization to unseen tasks while maintaining multi-task balance among seen tasks.

    Source: §I · p. 3

  • Takeaways & Limitations

    The framework presents requirement-oriented modeling as an approach to extending IR-VIS fusion from predefined closed-task settings toward open-task scenarios.

    Source: §I · p. 2

    CRT-HMAR is designed mainly for IR-VIS fusion, so performance on medical, multi-exposure, and remote-sensing image fusion may be affected without specific optimization.

    Source: G. Limitations and Future Work · p. 15

Abstract

from arXiv · show

Infrared and visible (IR-VIS) image fusion integrates complementary multimodal information into a single fused image to support downstream vision tasks. However, existing methods are typically tailored to seen tasks within a fixed task set and struggle to generalize to unseen tasks, which restricts their applicability in real-world open-task scenarios. To address this issue, this paper proposes CRT-HMAR, a Causal Requirement Tracing-Guided Hierarchical Multi-Agent Regulation Framework for open-task-aware IR-VIS image fusion. CRT-HMAR introduces a Causal Requirement Tracing Task Localization mechanism, which actively intervenes in key image information and observes task-network response variations to map task-specific semantic preferences into image-level causal requirement maps. Based on these maps, a requirement analysis agent aggregates task-specific requirement knowledge to adaptively guide requirement-customized image fusion. Moreover, CRT-HMAR incorporates History-Analysis Multi-Objective Balancing and Task-Level-Correction Conflict Mitigation mechanisms, jointly constructing a hierarchical regulation chain of "requirement interpretation - task balancing - conflict mitigation". Through multiple collaborative agents, CRT-HMAR dynamically regulates key processes including open-task requirement modeling, multi-task balanced optimization, and gradient conflict mitigation. Extensive experiments on open-task scenarios involving five downstream tasks demonstrate that CRT-HMAR significantly improves generalization to unseen tasks while maintaining the performance and balance of seen tasks. Overall, CRT-HMAR shifts IR-VIS image fusion from task-oriented modeling toward requirement-oriented modeling, promoting its extension from closed-task settings to real-world open-task scenarios.

I. INTRODUCTION

IR-VIS fusion combines thermal information with visible-image texture, but existing task-oriented methods struggle to generalize beyond predefined downstream tasks. CRT-HMAR shifts toward requirement-oriented modeling for open-task fusion.

II. RELATED WORK

  • Single-task-aware methods use task supervision or semantic injection, while multi-task methods such as IDF-TDDT and CLDyN adapt to multiple tasks.Single-task examples include SeAFusion, TarDAL, TIMFusion, MRFS, SAGE, and UAAFusion.Source: §I · pp. 1–2

I. INTRODUCTION

  • Task-oriented methods struggle on unseen downstream tasks because directly task-driven fusion can become coupled to the seen-task set.CRT-HMAR instead characterizes task requirements to guide fusion.Source: §I · p. 2
  • CRT-HMAR maps downstream semantic preferences into image-level requirements and uses collaborative mechanisms for requirement interpretation, task balancing, and conflict mitigation.The proposed framework is evaluated across five downstream tasks, and its stated aim is stronger unseen-task generalization while maintaining seen-task balance.Source: §I · pp. 2–3

A. Task-Agnostic IR-VIS Image Fusion

  • Traditional IR-VIS fusion uses handcrafted transforms and sparse representations to extract and reconstruct multimodal features.Deep methods later adopted CNN, Transformer, Mamba, and hybrid architectures, including approaches addressing registration, resolution, and extreme-condition degradations.Source: A. Task-Agnostic IR-VIS Image Fusion · p. 3
  • Task-agnostic methods primarily optimize fused-image visual quality, leaving downstream task requirements insufficiently addressed.The paper links this limited attention to limited improvement in task-driven applications.Source: A. Task-Agnostic IR-VIS Image Fusion · p. 3

B. Task-Aware IR-VIS Image Fusion

Task-aware fusion methods optimize for downstream tasks, but methods trained for fixed task sets struggle to transfer to unseen tasks. CRT-HMAR combines requirement tracing with agents that regulate fusion and multi-task optimization.

B. Task-Aware IR-VIS Image Fusion

  • Fixed-task methods can adapt to seen downstream tasks but suffer significant performance degradation on unseen tasks beyond the training set.Single-task methods include supervision and semantic-injection approaches; IDF-TDDT and CLDyN target multiple tasks.Source: B. Task-Aware IR-VIS Image Fusion · pp. 3–4

A. Overview

  • CRT-HMAR targets adaptation to disjoint training and evaluation task sets without parameter tuning.The overview defines the training tasks as Ts and evaluation tasks as Td, with Ts ∩ Td = ∅.Source: A. Overview · p. 5
  • Causal Requirement Tracing intervenes in fused-image information and observes task-network responses to construct causal requirement maps.An interpretation agent aggregates requirement knowledge from the RKB to guide TAFN in generating requirement-customized fused images.Source: A. Overview · p. 5
  • History-Analysis Multi-Objective Balancing monitors task gains and imbalance to regulate the multi-task loss, while Task-Level-Correction Conflict Mitigation corrects joint gradients using task-specific gradients.Together, the mechanisms form a requirement-interpretation, task-balancing, and conflict-mitigation regulation chain.Source: A. Overview · p. 5

B. Task-Adaptive Fusion Network

TAFN produces either visually dominated fusion or task-customized fusion by conditioning semantic restoration on requirement knowledge vectors. Its encoder, task-driven semantic modules, and decoder extract, restore, and reconstruct image features.

B. Task-Adaptive Fusion Network

  • TAFN supports visually dominated fusion for the image-fusion task and requirement-conditioned semantic restoration for a specific downstream task.Task-specific requirement vectors guide semantic restoration of IR-VIS features before the fused image is reconstructed.Source: B. Task-Adaptive Fusion Network · p. 5, Figure 3 · p. 5
  • The encoder extracts features, IR/VIS-TSE modules generate dynamic convolution kernels from requirement vectors, and the decoder reconstructs the fused image.Scene features are combined with the requirement vector to generate the kernels used to extract task-specified semantic features.Source: B. Task-Adaptive Fusion Network · p. 5
  • For image-fusion mode, the encoder features and their gradient maps are concatenated and passed to the decoder to produce the fused image.The paper uses a fusion loss to encourage the fused image to retain multimodal information.Source: B. Task-Adaptive Fusion Network · p. 5
  • For task-oriented fusion, semantic features are reinjected into IR/VIS features, then concatenated with a gradient map for decoding into a task-specific fused image.The task-aware loss compares downstream task results after semantic restoration with task ground truth; task-loss weights are dynamically adjusted by the balancing mechanism.Source: B. Task-Adaptive Fusion Network · pp. 5–6

C. Causal Requirement Tracing Task Localization

CRT traces how downstream task networks respond to changes in image frequencies, turning task preferences into causal requirement maps that guide requirement-customized fusion. A requirement analysis agent then aggregates task-specific knowledge from those maps.

C. Causal Requirement Tracing Task Localization

  • A requirement analysis agent uses the map to aggregate task-specific knowledge vectors from the Requirement Knowledge Base for requirement-customized image generation.The aggregated vector guides semantic restoration in the fusion network.Source: C. Causal Requirement Tracing Task… · pp. 6–7
  • CRT decomposes fused images into high-, middle-, and low-frequency components, then tests enhancement or suppression of each frequency.The adjustment strength and the frequency and operation choices are explicitly defined.Source: C. Causal Requirement Tracing Task… · p. 6
  • Task-network responses identify frequency sensitivity, while information-quality measures help distinguish adjustments likely to improve downstream performance.The quality measures assess sparsity, uncertainty, and saliency; larger values across all three indicate stronger discriminability and a greater likelihood of positive performance improvement.Source: C. Causal Requirement Tracing Task… · pp. 6–7
  • The CRT module uses effective frequency adjustments and response strength to construct a causal requirement map, representing task needs across six frequency-adjustment channels.The channels correspond to enhancement or suppression of low-, middle-, and high-frequency information; ineffective adjustments and insensitive frequencies receive zero maps.Source: C. Causal Requirement Tracing Task… · p. 7
  • The maps translate task networks’ implicit semantic preferences into image-level requirements through active intervention, response observation, and requirement interpretation.The paper describes this as a shift from task-oriented to requirement-oriented modeling.Source: C. Causal Requirement Tracing Task… · p. 7

D. History-Analysis Multi-Objective Balancing

The task balancing agent monitors recent multi-task performance improvements and imbalance, then adaptively regulates the multi-task loss weights. Its actions are evaluated asynchronously using later performance statistics.

D. History-Analysis Multi-Objective Balancing

  • Agentl uses recent task-performance improvement magnitudes and their variance as historical statistics for regulating the multi-task loss.The recorded history spans the most recent X optimization steps.Source: D. History-Analysis Multi-Objective… · pp. 7–8
  • The agent takes performance-improvement information as its state and generates balancing parameters β for the tasks.The improvement measures compare task performance before and after semantic restoration.Source: D. History-Analysis Multi-Objective… · pp. 7–8
  • Agentl is evaluated asynchronously: its β at step y updates the fusion network and historical statistics, which are used to construct a pseudo ground truth at step y+1.The optimized agent then monitors imbalance at step y+2.Source: D. History-Analysis Multi-Objective… · p. 8
  • A pseudo ground truth estimates imbalance from average task improvements over the previous M steps, giving less-improved tasks more emphasis.The negative KL divergence between Agentl’s output and the pseudo ground truth provides its reward.Source: D. History-Analysis Multi-Objective… · p. 8

E. Task-Level-Correction Conflict Mitigation

The conflict mitigation agent uses relationships among task-specific and joint gradients, along with performance information, to correct the multi-task gradient and alleviate gradient conflicts.

E. Task-Level-Correction Conflict Mitigation

  • Agentg compares inter-task gradients and each task-specific gradient with the multi-task joint gradient to assess their relationships.Positive cosine similarity indicates synergy, negative similarity indicates conflict, and zero indicates approximate orthogonality without explicit conflict or synergy.Source: E. Task-Level-Correction Conflict… · p. 8
  • The agent receives gradient relationships, gradient norms, and task performance improvements as state inputs to produce gradient-correction weight χ.Source: E. Task-Level-Correction Conflict… · p. 8
  • Agentg aggregates task-specific gradients using χ to correct the joint gradient, then updates TAFN with the corrected gradient.Its reward increases with the mean and minimum cosine similarities between the corrected gradient and task-specific gradients, which indicate smaller gradient conflicts.Source: E. Task-Level-Correction Conflict… · pp. 8–9

A. Datasets

The experiments train on two downstream-task datasets and evaluate generalization using three unseen-task datasets across five task types.

A. Datasets

  • Training uses two datasets selected from M3FD, FMB, and VT5000; the remaining dataset and LLVIP and VTD are used for testing.Source: A. Datasets · p. 9
  • M3FD, FMB, VT5000, LLVIP, and VTD represent object detection, semantic segmentation, salient object detection, pedestrian detection, and depth estimation, respectively.Source: A. Datasets · p. 9
  • Official splits are used for FMB, VT5000, LLVIP, and VTD; M3FD is randomly split into 3,150 training pairs and 525 test pairs.Source: A. Datasets · p. 9

B. Implementation Details

Training uses AdamW for 160 epochs with specified batch size, learning rate, and hyperparameters. Four downstream task networks are used, and the implementation runs in PyTorch on one RTX 4090 GPU.

B. Implementation Details

C. Evaluation Metrics

The section defines metrics for downstream-task and fusion performance, then evaluates open-task generalization across three seen/unseen task configurations. Against task-network retraining methods, the proposed method is comparable or better on seen tasks and generalizes better to unseen tasks.

C. Evaluation Metrics

  • mAP50→95 evaluates OD and PD, mIoU evaluates Seg, mFβ and Em evaluate SOD, and ∆< 1.252 and ∆< 1.253 evaluate DE.Source: C. Evaluation Metrics · p. 9
  • Fusion quality is assessed with QCB, SSIM, CC, and NAB/F; NAB/F is negative, so smaller values are better, while larger values are better for the other metrics.Source: C. Evaluation Metrics · p. 9

D. Open-Task Adaptability Comparison

1) Comparison with Task-Network Retraining Methods:

  • Across the open-task scenarios, the proposed method is comparable to or better than task-network retraining methods on seen tasks and generalizes better to unseen tasks.The method is reported to perform favorably whether tasks are seen or unseen, while retraining methods usually degrade significantly on unseen tasks.Source: 1) Comparison with Task-Network… · p. 10, Table I · p. 10, Table II · p. 10
  • Qualitative comparisons report higher-confidence OD and PD detections, better-preserved Seg and SOD structures, and less foreground-background depth confusion.The cited figures compare downstream-task results across the OD-Seg, OD-SOD, and Seg-SOD scenarios against ground truth.Source: 1) Comparison with Task-Network… · p. 10

2) Comparison with Multi-Task-Aware Methods:

The proposed method shows stable, competitive performance across seen and unseen tasks, with favorable fusion quality across datasets. Ablations associate its generalization with causal requirement maps, requirement interpretation and knowledge support, balanced task optimization, and gradient correction.

2) Comparison with Multi-Task-Aware Methods:

E. Fusion Performance Comparison

F. Ablation Study

  • Removing CRT or Agentv with its correlated RKB degrades unseen-task performance, linking causal requirement maps and their interpretation and knowledge support to generalization.Without CRT, SOD boundaries blur, PD confidence falls, and DE predictions deviate; removing Agentv causes greater unseen-task degradation than removing RKB.Source: F. Ablation Study · pp. 12–13
  • Removing Agentl and replacing adaptive loss regulation with fixed weights leads to greater Seg degradation than OD degradation, indicating optimization imbalance among seen tasks.Qualitative results show incomplete semantic partitioning or incorrect Seg categories without Agentl.Source: F. Ablation Study · p. 13, Figure 11 · p. 14, Table V · p. 15
  • Without Agentg's task-specific gradient correction, performance favors detection: OD changes little, while Seg and other unseen tasks degrade more substantially.The unseen PD task is less affected than the other unseen tasks.Source: Figure 11 · p. 14, F. Ablation Study · p. 15, Table V · p. 15

G. Limitations and Future Work

CRT-HMAR maintains seen-task performance and balance while generalizing more strongly to unseen tasks across open-task scenarios involving five downstream tasks. Its current design is mainly for IR-VIS fusion; future work proposes a unified framework for multiple image types and their downstream requirements.

G. Limitations and Future Work

  • The method is mainly designed for IR-VIS fusion, so without specific optimization its performance on medical, multi-exposure, and remote-sensing image fusion may be affected.The paper proposes exploring a unified framework for multiple image types and their downstream task requirements.Source: G. Limitations and Future Work · p. 15
  • Future work will explore an open-task-aware unified fusion framework for multiple image types and their downstream task requirements.The stated aim is to extend image-fusion application potential across domains.Source: G. Limitations and Future Work · p. 15

V. CONCLUSION

  • Across open-task scenarios involving five downstream tasks, CRT-HMAR maintains seen-task performance and balance while achieving stronger generalization to unseen tasks.The conclusion characterizes this as a shift from task-oriented to requirement-oriented modeling.Source: §V · p. 15
Loading 2610.09330v1…