Task-Customized Mixture of Adapters for General Image Fusion

1. Introduction

• MEF and VIF involve relatively balanced fusion, whereas MFF tends to make polarized source choices.MEF integrates information across exposure levels, VIF combines infrared intensity with visible texture and gradient, and MFF selects one source per patch.Source: §1 · p. 1
• Task-specific fusion methods often generalize weakly, while unified methods can favor dominant-task bias or commonality at the expense of task individuality.These limitations motivate a more dynamically compatible approach across fusion scenarios.Source: §1 · pp. 1–2
• TC-MoA uses a fixed pre-trained foundation model with task-specific routing networks and shared tuning adapters to generate fusion prompts, constrained by mutual-information regularization.The regularization is intended to preserve complementarity across sources while supporting compatibility across tasks.Source: §1 · p. 2
• The proposed unified model adds 2.8% learnable parameters and is reported to outperform competing methods while showing controllability and generalizability.The paper presents it as an MoE-based flexible adapter approach for general image fusion.Source: §1 · p. 2

2. Related Work

• General image fusion methods seek to handle multiple tasks in one model, but prior approaches may use separate training, adjust fusion conditions, assess source information, or neglect task-specific individuality.The paper positions dynamic adapter customization as an alternative to trading individuality for cross-task commonality.Source: §2 · p. 2
• Parameter-efficient fine-tuning includes adapters and prefix tuning, with prior methods adapting pre-trained models to downstream tasks or modifying feature scaling.The paper distinguishes its focus as adapting a foundation model to general image fusion.Source: §2 · p. 2
• Prior mixture-of-experts work applies gated experts to increase model capacity or handle multi-task and vision-language learning, while TC-MoA treats experts as adapters for fusion.Its overview shows task-specific routers and a shared adapter bank producing prompts within a frozen ViT backbone.Source: §2 · p. 2, Figure 2 · p. 3

3.1. Overview

• TC-MoA inserts task-customized mixtures of adapters into a ViT encoder-decoder to adaptively prompt multiple fusion tasks in one model.The foundation model acts as a fixed encoder, and the TC-MoA modules are inserted into every four transformer blocks.Source: §3.1 · p. 3

3.2. Task-Customized Mixture of Adapters

• For each source pair, TC-MoA concatenates and reduces source features, then uses a task-specific router to select adapters and weight their outputs into a prompt.The routers use top-2 routing, and their adapter mixtures vary by task.Source: §3.2 · p. 3
• The prompt weights each source’s features, which are shifted by learnable source embeddings and fused to produce features for the next transformer block.The fusion layer adds refined source features and applies convolutional layers to introduce local receptive fields and reduce checkerboard artifacts.Source: §3.2 · p. 4
• Mutual-information regularization constrains the prompt to retain complementary source information and helps identify each source’s dominant intensity.The prompt represents the proportion of important information in each source.Source: §3.2 · p. 4

3.3. Task-Customized Loss Function

• Task-specific unsupervised losses target VIF detail retention, MEF average luminance with gradients, and MFF patch-wise selection of the source with maximum gradient.The VIF losses constrain structural, intensity, and gradient information; MEF uses a task-specific SSIM loss, while MFF masks pixel and gradient losses by patch.Source: §3.3 · p. 4

4.1. Experimental Setting

• The experiments use LLVIP for VIF, SCIE for MEF training and MEFB for testing, and RealMFF plus MFI-WHU for MFF training; evaluation uses task-specific metric sets.The model is jointly trained for 20 epochs using the pre-trained MAE-large backbone.Source: §4.1 · pp. 4–5

4.2. Evaluation on Multi-Modal Fusion

• TC-MoA outperforms all general image fusion methods on VIF and improves on most task-specific methods, with gains in perceptual, information, structural, and gradient metrics.The qualitative comparison also reports clearer dark-area contours and background texture.Source: Table 1 · p. 5, §4.2 · p. 5

4.3. Evaluation on Multi-Exposure Fusion

• TC-MoA achieves state-of-the-art performance among general image fusion methods on MEF and competitive results against task-specific methods.The paper reports improved texture details and color retention in both high- and low-illumination regions.Source: Table 2 · p. 5, §4.3 · p. 6

4.4. Evaluation on Multi-Focus Fusion

• On MFF, TC-MoA surpasses general image fusion methods across all metrics and significantly outperforms IFCNN on NMI, MI, Qcb, and Qcv.The qualitative comparison reports better texture and color consistency and fewer flower-region distortions.Source: Table 3 · p. 6, §4.4 · p. 6

4.5. Ablation Study

• Multiple adapters improve performance across all metrics, while direct multi-task training shows inter-task competition in MEF; TC-MoA also outperforms FrozenBackbone and AdaptFormer.The ablation reports that the multi-task versus single-task advantage reverses for MEF.Source: §4.5 · pp. 6–7, Table 4 · p. 7

4.6. Analysis and Discussion

• After shifted-window ViT optimization, the base and large models accelerate by 178% and 167%, respectively, for multi-task inputs of arbitrary size.The paper describes the resulting FPS as acceptable relative to other methods.Source: §4.6 · p. 7, Table 6 · p. 7
• Prompt scale and shift control source dominance, and prompt manipulation yields reasonable zero-shot fusion results on medical-image and pan-sharpening samples.The paper reports that increasing the shift factor changes global source resemblance, while scale changes regional source preference.Source: §4.6 · p. 7, Figure 6 · p. 8
• Performance peaks at four experts and at an interval of four between TC-MoA modules; adding experts beyond the peak does not necessarily help.These settings were selected using Qabf and SSIM across multi-task experiments.Source: §4.6 · p. 8

• Different fusion tasks customize their respective adapter mixtures to obtain prompt guidance for multi-source image fusion in one model.Source: §5 · p. 8
• Mutual information regularization constrains the adapters to support compatibility across tasks while maintaining complementarity between source images.Source: §5 · p. 8
• TC-MoA achieves advanced performance against competing methods and demonstrates strong prompt and router controllability for flexible fused-image manipulation.Source: §5 · p. 8

• The supplement provides an overview of task-customized loss functions and associated comparisons.Source: Abstract · p. 11
• It adds network structure and data-flow details, shifted-window implementation, network-design ablations, and parameter analysis.Source: Abstract · p. 11
• It discusses properties of different fusion tasks and examines task-specific routing.Source: Abstract · p. 11
• It supplies additional quantitative results on TNO and qualitative fused-image results across multiple tasks.Source: Abstract · p. 11

• Across tasks, the network uses tailored unsupervised losses to constrain fused-image structure, intensity, and gradients.Source: §7 · p. 11
• For VIF, maximum-pixel and maximum-gradient losses target distinct high- and low-frequency source information while preserving gradient direction.Source: §7 · p. 11

Qualitative and Quantitative Comparisons

• The task-customized losses yield fused images rich in high-frequency and structural details across tasks, while qualitative comparisons show better contrast, color saturation, and texture than the unified-loss model.Source: Table 7 · p. 11, §7 · p. 12

• For MEF, average-pixel and maximum-gradient losses preserve average luminance and gradient information, with MEF-SSIM replacing conventional SSIM.Source: §7 · p. 11
• For MFF, masked pixel and gradient losses select one source patch per fused patch according to which source has the larger maximum gradient.Source: §7 · pp. 11–12

Detail Architecture of TC-MoA

• TC-MoA has prompt-generation and prompt-driven fusion stages; fusion features are added to both branches, whose outputs are considered approximately identical so either can produce the fused image.Source: §8 · pp. 12–13, Figure 10 · p. 13

Shifted Windows

• Shifted windows partition features into 14 × 14-patch windows, use local position embeddings and cyclic shifts, and support variable-size inputs while capturing global context and avoiding checkerboard artifacts.Source: §8 · p. 13, Figure 11 · p. 13

Ablation Studies about Network Design

• Ablations report that shifted windows preserve image-wide coherence, convolutional local receptive fields alleviate checkerboard effects, and signed-gradient losses avoid incorrect gradient directions.Source: §8 · p. 13, Figure 12 · p. 14

• Figure 13 compares tasks using average dominant-intensity bias and the absolute difference between the X- and Y-source biases.Source: Figure 13 · p. 14

More Analysis of Parameters

• The ViT-large model uses 2.77% trainable parameters; shifted windows require 0.06%, and adapters plus routing networks use 0.63%.Source: §8 · p. 14, Table 8 · p. 14

Analysis on Top K

• Table 10's Top K ablation indicates better overall performance with more routed adapters, but only minor sensitivity to K; the model uses Top K = 2 to balance performance and inference cost.The passage reports the trend but supplies no specific ablation values.Source: Table 10 · p. 15, §9 · p. 15

The Properties of Various Fusion Tasks

• MFF has higher mean dominant intensity bias than MEF and VIF, while VIF has a larger source-level dominant-bias difference than the other tasks.MFF fusion is typically more unbalanced per token pair; MEF and VIF are more balanced, and the authors suggest LLVIP's mostly nocturnal scenes may contribute to VIF's source-level imbalance.Source: §9 · p. 15
• 0.7467 and 0.1607 are the VIF dominant and auxiliary intensity biases, respectively, for source X.The average dominant intensity bias reflects token-level fusion patterns, while the difference between dominant biases reflects source-level patterns.Source: §9 · p. 15

Details of Task-Specific Routing

• VIF, MEF, and MFF routers select different adapters according to task and dominant source, with MFF routing also tending to separate high- and low-frequency areas.For MFF, red tends to handle high-frequency areas and green low-frequency areas.Source: §9 · p. 15

10. More Quantitative Comparisons

• On the TNO dataset, the method achieves superior overall performance compared with the most recent methods.The passage does not provide metric values for this comparison.Source: §10 · p. 15

Adapter selection visualization

• Figure 14 colors identify the highest-routing-weight adapter at the encoder's last TC-MoA, but the map may contain unclear noise and reflects only a general trend.The figure does not show adapter selections across every encoder TC-MoA.Source: Figure 14 · p. 16

VIF

• VIF results retain visible-image details such as license plate numbers while highlighting infrared information, with bright and high-contrast overall images.Figure 15 provides additional VIF qualitative comparisons.Source: §11 · pp. 15–16, Figure 15 · p. 17

MEF

• MEF results preserve structural and color details from both sources while avoiding blurry halos in overexposed areas.Figure 16 provides additional MEF qualitative comparisons.Source: §11 · p. 16, Figure 16 · p. 18

MFF

• MFF results avoid structural and color distortions, especially around text, and are visually comparable to supervised IFCNN.Figure 17 provides additional MFF qualitative comparisons; the paper also reports substantial fusion controllability for manipulating results to particular requirements.Source: §11 · p. 16, Figure 17 · p. 19