Source-linked AI summary
Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging
Pengyang Zhou, Xiaobin Tu, Zhengxi Liu, Rongkun Xue, Haochen Li, Miancan Liu, Ziyuan Chen, Yinggui Wang, Jinkui Ren, Xiantao Zhang
TL;DR
Layer-wise merging does not fully characterize how expert updates propagate through depth and alter visual–textual computation, nor how those updates should be jointly weighted. TAC-Merge traces this influence and jointly optimizes regional updates; across two backbones, its merged models improve seen- and unseen-task averages over compared merging methods.
Problem
Layer-wise multimodal merging leaves unclear how individual updates influence downstream representations and image–instruction interactions, and how to jointly weight updates whose effects depend on one another.
Source: §1 · p. 2Method
TAC-Merge combines multimodal influence mapping, which measures update effects using Ricci curvature and expert predictions, with coupled merge control, which models coefficient interactions to select regional weights.
Source: §1 · p. 2Results
Across two evaluated backbones, TAC-Merge improves both seen- and unseen-task averages over the compared merging methods.
Source: §5 · p. 9Takeaways & Limitations
Gains on unseen tasks indicate that consolidated capabilities extend beyond the datasets used for expert training.
Source: §1 · p. 2The method assumes access to a small calibration set.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Multimodal model merging aims to consolidate task experts into a single model that retains their complementary capabilities. Most unimodal model merging methods combine expert updates within individual layers, and multimodal approaches largely follow this design. However, an expert update changes the representations passed to subsequent layers, allowing its influence to propagate across depth and affect how visual and textual information interact. When visual and language updates are combined, later updates act on inputs already modified by earlier ones, coupling their effects. This poses two challenges: (1) how to characterize the multimodal influence of individual expert updates across depth, and (2) how to jointly combine expert updates based on their multimodal influence. To address these challenges, we propose TAC-Merge for tracing and coordinating cross-layer influence in multimodal model merging. It contains two modules, i.e., multimodal influence mapping (MIM) and coupled merge control (CMC). MIM constructs graphs of update effects and uses Ricci curvature together with expert predictions to define a shared fusion objective. CMC models interactions among coefficient adjustments and jointly optimizes regional weights to synthesize one shared model. Experiments across diverse multimodal tasks demonstrate the effectiveness of TAC-Merge in consolidating complementary expert capabilities and supporting generalization to unseen tasks.
1 INTRODUCTION
Multimodal merging seeks one model that retains complementary expert capabilities, but layer-wise updates can obscure how influence propagates across depth and affects visual–textual computation. TAC-Merge traces this influence and jointly coordinates regional updates; it achieves the best average performance among compared methods on seen and unseen tasks.
1 INTRODUCTION
- A shared merged model offers an alternative to storing and deploying separate experts for tasks such as question answering, visual reasoning, chart understanding, and object localization.The goal is to preserve complementary capabilities and visual–textual interactions in one model.Source: §1 · p. 1
- Layer-wise multimodal merging can miss how local expert updates propagate to downstream representations and alter image–instruction interactions.Visual evidence and textual instructions jointly shape hidden representations, so token labels or final predictions alone do not distinguish the sources of an observed change.Source: §1 · p. 2
- Joint fusion must choose visual and language update weights together because each preferred weight depends on the weights assigned to other updates.The shared objective evaluates both multimodal influence structure and agreement with expert predictions.Source: §1 · p. 2
- TAC-Merge combines MIM, which measures update effects across depths and image–instruction responses, with CMC, which jointly selects regional weights for one shared model.MIM uses Ricci curvature and expert predictions; CMC models interactions among coefficient adjustments in a compact subspace.Source: §1 · p. 2
- TAC-Merge achieves the best average performance among compared methods on both seen and unseen tasks, with unseen-task gains extending beyond expert-training datasets.The evaluation covers two multimodal backbones, eight seen tasks, and four unseen tasks.Source: §1 · p. 2
2 RELATED WORK
Prior merging work combines updates through parameter aggregation, interference handling, or data-dependent estimation, and multimodal methods adapt these approaches to model architecture and task needs. Other work models dependencies across depth, while multimodal studies trace interactions between visual and textual information.
Multimodal Model Merging
- Single-modality merging methods aggregate parameters, transform updates to manage interference, or estimate merged parameters using data-dependent statistics and objectives.Multimodal methods adapt these strategies to modality-specific encoders and shared language components, heterogeneous architectures, uncertainty-guided composition, or denoised task vectors.Source: §2 · p. 2
Layer and Modality Dependencies
- Cross-depth merging methods refresh activation statistics sequentially, model correlations within and between layers, or optimize output-space coefficients across layers.These approaches reflect that updates change intermediate representations used by subsequent layers.Source: §2 · pp. 2–3
- Multimodal analyses use attention interventions and generation tracing to study where and how visual and textual information interact across architectures and tasks.The cited work considers image captioning and chain-of-thought reasoning as well as information transfer among visual tokens, textual tokens, and answer-prediction positions.Source: §2 · p. 3
3 METHOD
TAC-Merge selects regional expert-update weights by mapping multimodal effects across depth and coordinating coupled adjustments in one shared model.
3.1 PROBLEM STATEMENT
- Using calibration data, the method selects regional expert-update weights for one shared model that preserves experts’ complementary multimodal capabilities.Each expert is adapted from the same base model, and the selected regional blocks are combined in the original parameter order.Source: §3.1 · p. 3
3.2 FRAMEWORK OVERVIEW
- MIM maps regional update effects across depth, while CMC uses the resulting reference objective to coordinate coefficients and synthesize a shared model.The selected inference coefficients are fixed rather than input-dependent.Source: §3.2 · p. 3
3.3 MULTIMODAL INFLUENCE MAPPING
- MIM builds graphs linking expert–region updates to reachable readouts, measuring sensitivity across original, image-change, and instruction-change channels.Readouts pool hidden representations by region and token-position group; architectural connections exclude observations independent of the changed modality.Source: §3.3 · pp. 3–4
- It defines a shared fusion objective by matching task-expert graph curvature and next-token predictions on original calibration inputs.Curvature matching constrains relative influence organization across regions and modalities, while output agreement penalizes deviations from expert predictions.Source: §3.3 · p. 5
3.4 COUPLED MERGE CONTROL
- CMC derives a compact control subspace from MIM responses and fits a local quadratic objective to individual and paired coefficient variations.Paired variations estimate interactions between control directions, allowing adjustments to be evaluated in the context of other adjustments.Source: §3.4 · pp. 5–6
- CMC proposes coefficients within a prescribed neighborhood and accepts them only when the measured fusion objective improves over the uniform reference.The proposal is checked on the same calibration data and expert targets without iterative refitting of the response model.Source: §3.2 · p. 3, §3.4 · p. 6
4 EXPERIMENTS
Experiments evaluate TAC-Merge on two backbones using eight seen and four unseen datasets, comparing it with merging baselines and testing its components and hyperparameter sensitivity. TAC-Merge achieves the highest seen and unseen averages across both backbones, while ablations show contributions from both modules.
4.1 EXPERIMENTAL SETUPS
- The evaluation uses eight seen datasets spanning question answering, referring expressions, classification, and captioning, plus four unseen datasets to test transfer beyond expert adaptation.The same merged coefficients are evaluated on unseen datasets outside the expert-adaptation set.Source: §4.1 · p. 6
- Comparisons include reference settings, single-modality, cross-layer, and multimodal merging baselines, alongside TAC-Merge’s two-backbone evaluation.The baselines include Task Arithmetic, TIES, DC-Merge, Chain of Merges, RegMean++, RobustMerge, and OptMerge.Source: §4.1 · pp. 6–7, §4.2 · p. 7
4.2 PERFORMANCE EVALUATION
- 3.43 and 2.34 points are TAC-Merge’s gains over the strongest merging baselines on seen and unseen tasks, respectively, for Qwen3.5-4B; gains are 2.42 and 1.94 on Qwen3-VL-8B.It leads the merging methods on seven of eight seen tasks and three of four unseen tasks for each backbone.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 8
4.1 EXPERIMENTAL SETUPS
- Each backbone uses one jointly optimized LoRA expert per seen task, with adapters in vision-encoder and language-backbone attention and MLP modules.Pretrained weights remain frozen; the LoRA rank and scaling factor are both 16.Source: §4.1 · p. 7
4.2 PERFORMANCE EVALUATION
- Removing MIM lowers seen averages by 2.38 and 1.47 points on 4B and 8B, while removing only modality contrasts lowers them by 1.04 and 0.41 points.The ablations compare modality-specific image- and instruction-change responses with original representations as guidance for influence matching and coefficient-direction selection.Source: §4.2 · p. 8, Table 3 · p. 8
- Removing CMC lowers unseen averages by 0.52 and 0.24 points on 4B and 8B; removing directional coupling lowers them by 0.39 and 0.12 points.Directional coupling represents interactions between control directions; the full configuration has the highest mean in all four seen/unseen settings.Source: §4.2 · pp. 8–9, Table 3 · p. 8
- The sensitivity scans vary λ and R separately around (λ, R) = (1, 0.25); λ has a broad intermediate plateau, while larger R can yield accepted proposals with greater unseen-task decline.The figure reports average-score changes from the shared default, with dotted lines marking default values.Source: §4.2 · p. 9, Figure 2 · p. 9
5 CONCLUSION
TAC-Merge traces multimodal expert influence across depth and coordinates regional updates into one shared model. Across two evaluated backbones, the merged model improves seen and unseen averages over compared merging methods; future work targets more efficient influence estimation and scalable coordination.
5 CONCLUSION
- TAC-Merge combines multimodal influence mapping, which defines an objective using Ricci curvature and expert predictions, with coupled merge control, which selects regional weights while modeling coefficient interactions.The resulting static model serves all tasks using the base inference architecture.Source: §5 · p. 9
- Across two evaluated backbones, the merged model improves both seen and unseen averages over the compared merging methods.Ablations support contributions from modality-specific responses and interactions between control directions.Source: §5 · p. 9
- Future work will explore more efficient influence estimation and scalable coordination when merging larger collections of task experts.Source: §5 · p. 9