PaperlayerView exact revision on arXiv

Source-linked AI summary

Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs

Hoigi Seo, Byung Hyun Lee, Minjun Kim, Dohyun Mah, Jongho Lee, Se Young Chun

arXiv:2610.04580v1cs.AIcs.CL

TL;DR

Improving an MLLM’s modality capability typically requires costly modality-specific training, while same-modality variants may be scarce. DCAT transfers alignment from a strong donor using a small calibration set; it reports average gains on audio and video tasks over original recipient models.

  • Problem

    Improving modality-specific MLLM capability typically requires costly training data and compute, while same-modality variants are scarcer for some domain-specific modalities.

    Source: §1 · pp. 1–2

  • Method

    DCAT transfers alignment from a high-alignment donor to a recipient using a few hundred calibration samples and a closed-form weight update.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    DCAT reports 29.55% average improvement for audio tasks and 24.92% for video tasks over original recipient models, and consistently outperforms existing model-merging methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analyses support the Alignment score as a broadly predictive indicator for cross-modal transfer, though not for every evaluation setting.

    Source: §1 · p. 2, G Limitation · p. 43

    Alignment–performance correlation is weak or negligible on a small number of benchmarks, where other task or evaluation factors may dominate.

    Source: G Limitation · p. 43

Abstract

from arXiv · show

Multi-modal large language models (MLLMs) achieve strong modality understanding by pairing a large language model (LLM) with an encoder for a target modality such as vision, video, or audio. However, improving an MLLM's capability for a given modality typically requires additional training on large modality-specific datasets, incurring substantial data collection and compute costs. Model merging offers an alternative, but it is often infeasible for data-scarce, large per-sample size, or domain-specific modalities (\textit{e.g.}, audio and video), where same-modality model variants are rarely available. In this work, we characterize an intriguing asymmetric phenomenon: merging a well-aligned, data-rich source-modality MLLM into a data-scarce target-modality MLLM substantially improves the target on its own benchmarks. Our theoretical and empirical analyses show that this gain stems from enhanced alignment between modality-specific and textual tokens, induced by the stronger donor modality. Specifically, we derive a mutual-information lower bound that is monotonic in alignment-related quantities and strongly correlated with downstream MLLM performance. Building on this principle, we propose Directional Cross-modal Alignment Transfer (DCAT), a novel framework that transfers textual alignment from a strong, well-aligned source (donor) modality to a weak target (recipient) modality, boosting target-modality performance without further fine-tuning. We further show that the alignment-enhancing objective admits a closed-form weight-space solution computed from only a small calibration set. DCAT outperforms existing model-merging methods, offering an efficient path toward cross-modal alignment transfer. Project page with code is available at \url{https://seohoiki3215.github.io/DCAT_project_page}

1 Introduction

Modality-specific training is costly, and same-modality variants are scarce for some modalities, motivating cross-modal transfer. DCAT transfers alignment from a strong donor to a weaker recipient, improving target-modality performance.

1 Introduction

  • Cross-modal merging is a targeted alternative because data and same-modality model variants are scarcer for some domain-specific modalities.The paper asks whether a donor can improve a recipient on its own modality-specific benchmarks without additional training.Source: §1 · pp. 1–2
  • The authors study merging gains through mutual information and define an Alignment score from Subspace Energy Overlap and Spectral Diversity.Their theoretical and empirical analyses find the score strongly and significantly correlated with MLLM performance.Source: §1 · p. 2
  • 29.55% average improvement for audio tasks and 24.92% for video tasks over original recipient models are reported for DCAT.The authors also report that DCAT consistently outperforms existing model-merging methods.Source: §1 · p. 2
  • DCAT transfers alignment from a high-alignment donor to a recipient using only a few hundred calibration samples.The proposed framework is motivated by analyses of cross-modal merging and alignment.Source: §1 · p. 2

2 Related Work

Related work describes MLLM architectures and training costs, then reviews model-merging approaches and the comparatively unexplored problem of directional cross-modal transfer.

Multi-modal large language models

  • MLLMs pair an LLM with modality-specific encoders and projectors, while training can be compute-intensive and modality-specific data costly to collect.The passage identifies training-free and data-efficient alternatives as motivation for improving capabilities.Source: §2 · pp. 2–3

Model merging

  • Model-merging methods address interference or update structure through approaches including TA, TIES, ISO, TSV, WUDI, and OptMerge.The passage frames merging as a training-free way to consolidate independently fine-tuned models.Source: §2 · p. 3
  • Directional transfer from a source-modality MLLM to a designated target-modality MLLM remains insufficiently examined in prior work.Table 1 compares audio-vision and video-vision merges on MMAU and Video-MME, respectively.Source: §2 · p. 3, Table 1 · p. 3

3 Analysis

Cross-modal merging improves modality-specific benchmark performance, and the analysis links those gains to an Alignment score motivated by a mutual-information lower bound.

3.1 Model merging increases modality performance

  • Vision-model merging improves audio and video benchmark performance by up to 18.45% over the original modality-specific model.The reported checkpoints are Vicuna-based and use OptMerge.Source: §3.1 · p. 3

3.2 On the lower bound of Mutual Information

  • The Alignment score combines Subspace Energy Overlap and Spectral Diversity to represent modality-token energy entering and spreading across the text principal subspace.The score is motivated by a lower bound on mutual information between text and the other modality.Source: §3.2 · pp. 3–4

3.3 Empirical analysis on Alignment score

  • Alignment increases significantly under merging (p ≪10^-10), and 114 Vicuna-based checkpoints show strong positive alignment–performance correlations across modalities.The reported correlations use the 21st layer; the authors also observe high correlations beyond the 19th layer.Source: §3.3 · p. 5
  • The 21st layer is used for analysis because it exhibits the lowest standard deviation among the examined layers.High alignment–performance correlation is observed beyond the 19th layer across modality merge pairs.Source: §3.3 · p. 5

4 Method

DCAT transfers donor-induced alignment while preserving recipient behavior, and pairs cross-modal alignment with self-alignment of recipient modality activations. Its layer-wise solution is closed-form, with an adaptive schedule balancing alignment objectives.

4 Method

  • Alignment score and performance show consistently strong positive correlation from the middle layers onward across the measured MLLM combinations and modalities.Source: Figure 3 · p. 5
  • Donor-guided cross-modal alignment uses text inputs to transfer donor-compatible directions, while recipient anchoring preserves the recipient’s original behavior and modality-specific capabilities.The donor term is not simply a pull toward donor weights; it transfers alignment directions through text space.Source: §4 · pp. 5–6
  • DCAT constructs a recipient activation target in the principal text subspace, balancing preservation of recipient geometry against broader spectral diversity while preserving activation norm.The parameter α controls this trade-off; the target’s norm-preserving property holds for all α.Source: §4 · p. 6
  • Matching recipient activations to the constructed target encourages both Subspace Energy Overlap and spectral diversity.The loss reduces activation energy outside the principal text subspace and shapes the projected representation within it.Source: F | {z } Encouraging SEO · p. 6, §4 · p. 6

F | {z } Encouraging SEO

  • The combined quadratic objective admits a closed-form layer-wise update, applied progressively with recalibrated activations to reduce accumulated activation drift.The update combines donor-guided cross-modal alignment with self-alignment matching.Source: F | {z } Encouraging SEO · p. 6
  • The adaptive α schedule is more conservative when recipient SEO is weak and more isotropic when its projected representation is spectrally concentrated.Larger α preserves more of the recipient’s projected spectrum, while smaller α encourages spectral diversity.Source: F | {z } Encouraging SEO · p. 7

5 Experiment

Experiments compare DCAT with merging baselines across audio-plus-vision and video-plus-vision settings, then examine alignment changes and ablations. DCAT improves target-modality benchmark performance, while measured alignment quantities rise and ablations support the method’s design choices.

5.1 Experimental settings

  • Experiments compare DCAT with six representative merging baselines—Task Arithmetic, TIES-merging, ISO-C, Task Singular Vector, WUDI, and OptMerge—across Audio+Vision and Video+Vision settings on Vicuna and LLaMA 3.1 backbones.Calibration uses AVQA audio tracks for audio MLLMs and video tracks for video MLLMs, with 256 samples each.Source: §5.1 · p. 7

5.2 Quantitative comparison

  • DCAT improves audio and video benchmark performance over the original models, with gains of 7.62% on MMAU, up to 100.92% on MuCho-Music, 16.37% on Video-MME, and 23.83% on MVBench.The comparison covers audio+vision and video+vision settings; the table caption also reports gains up to 53.88% on MLVU.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 Does DCAT increase SEO and SD? Yes.

  • DCAT significantly increases SEO and SD while keeping the modality-token norm ratio close to 1.0, consistent with its intended alignment changes.The reported SEO and SD increases are statistically significant at p < 0.05.Source: §5.3 · pp. 7–8, Figure 4 · p. 9

5.4 Ablation study

  • Ablations indicate that each tested DCAT objective component and the adaptive α schedule contributes meaningfully to performance.The ablation study uses Vicuna models, the full MMAU benchmark for audio, and 900 sampled Video-MME examples for video.Source: §5.4 · p. 8, Table 3 · p. 9

6 Discussion

The authors conjecture that vision models are often stronger donors because visual data is more abundant, making DCAT promising for relatively rare, data-scarce modalities.

6 Discussion

  • The authors conjecture that vision models are often better aligned because visual data is more abundant, positioning rare, data-scarce modalities as the likely recipients of transfer.They report that vision models lose performance under merging across multiple models and architectures, motivating their use as donors.Source: §6 · p. 9

7 Conclusion

The paper presents cross-modal merging as a performance-improving phenomenon and introduces DCAT to transfer alignment in closed form. Across audio and video MLLMs, DCAT improves average relative performance over the original models.

7 Conclusion

  • DCAT achieves average relative improvements of 29.55% on audio and 24.92% on video over the original models across multiple MLLMs and benchmarks.The conclusion presents these results as evidence of DCAT’s effectiveness.Source: §7 · p. 9

A Code

  • The authors provide a code implementation of DCAT for reproducibility.Source: A Code · p. 15

B.1 Proof of Theorem 1

The proof connects subspace energy overlap (SEO) and spectral diversity (SD) to a weighted mutual-information lower bound under stated assumptions. It also derives DCAT’s alignment objective and closed-form update, then checks the spectral approximation empirically.

B.1 Proof of Theorem 1

  • The proof relates SEO to projected covariance and SD to the determinant and trace of projected modality covariance.For Lemma 1, the text subspace is formed from the top-k right singular vectors of centered text activations.Source: §B.1 · pp. 15–16

Assume that the projected spectrum satisfies the condition

  • Theorem 1 gives a weighted lower bound on mutual information in terms of alignment-related quantities, including SEO and SD.The proof assumes positive-definite covariance matrices and a spectral non-degeneracy condition.Source: Assume that the projected spectrum… · pp. 17–18

B.2 Derivation of Eq. 8

  • The derivation of Eq. 8 constructs a target for modality-token activations using the text subspace and a uniform singular value that preserves total output energy.The text defines the projector onto the top-k text directions and its orthogonal complement before deriving properties of the target.Source: §B.2 · p. 19

B.3 Derivation of Eq. 9

F | {z } Encouraging SD

  • The SD term encourages spectral diversity while the update remains constrained to the text subspace, with the proof showing the cross term vanishes.This derivation uses the target’s relation to the top-k projector and its orthogonal complement.Source: F | {z } Encouraging SD · pp. 20–21

B.4 Derivation of closed form for DCAT

  • The layer-wise DCAT objective is quadratic in the weight update and has a unique minimizer obtained by setting its gradient to zero.The derivation assumes the text-activation matrix has positive-definite covariance.Source: §B.4 · p. 21

B.5 Empirical validation of δ

  • δeff is close to one in both settings: 0.9704 for Audio+Vision and 1.1046 for Video+Vision, indicating only mild relaxation in the MI lower bound.The text describes δeff as a spectral conversion factor, not a per-eigenvalue probability lower bound.Source: §B.5 · p. 22

C.1 Models and computation

  • The Vicuna comparison uses models trained from the same base and recipe except for modality in OptMerge, while the solver takes about 25–30 minutes per setting on one L40S GPU.The experiments also include independently trained models sharing LLaMA-3.1-8B-Instruct backbones.Source: §C.1 · p. 22

C.2 Calibration and alignment-target data

DCAT calibrates its alignment target with a small paired AVQA sample set and applies the closed-form update progressively across layers and dependent projection groups. The objective is solved as a regularized linear system without gradient-based optimization.

C.2 Calibration and alignment-target data

  • DCAT uses 256 AVQA samples per modality for calibration, with evaluation benchmarks disjoint from AVQA.The paired audio and video streams share question-and-answer prompts, allowing the same text source to be used across recipient modalities.Source: §C.2 · p. 22

C.3 Progressive, dependency-aware merging

  • DCAT merges layers sequentially and updates four dependency-aware projection groups in order, recollecting activations after each group.The groups are query/key/value, output, up/gate, and down projections; later solves use activations reflecting earlier updates.Source: §C.3 · p. 23

C.4 Hyperparameter values

  • The text-subspace dimension k is 128 for Vicuna and 256 for LLaMA-3.1-8B-Instruct.The paper attributes the larger value to the wider LLaMA hidden states and their need for a larger effective rank to span principal text directions.Source: §C.4 · p. 23

C.5 Closed-form solve and engineering notes

  • The quadratic objective becomes a layer-wise SPD system solved with Cholesky and a small diagonal ridge, with no gradient-based optimization.The solver uses float64, adds ϵsolve = 10^-6 tr(A)/d for numerical stability, and applies the update in the recipient checkpoint’s native bfloat16 precision.Source: §C.5 · p. 23

D.1 Analysis details

The analysis compares alignment with benchmark accuracy across merging methods, model families, and modality benchmarks, using full merges and SVD-derived variants. The evaluation includes Vicuna and LLaMA 3.1 models and benchmarks spanning audio, video, and vision.

D.1 Analysis details

  • The analysis measures Pearson and Spearman alignment–accuracy correlations across Audio+Vision and Video+Vision settings on Vicuna and LLaMA 3.1 backbones.It includes six merging baselines and evaluates full-merge checkpoints alongside low-rank variants reconstructed at ranks 1 through 128.Source: §D.1 · pp. 23–24

D.2.1 Baselines

  • The six comparison baselines are Task Arithmetic, TIES-Merging, ISO-C, Task Singular Vectors, WUDI-Merging, and OptMerge.Their described strategies include task-vector operations, interference reduction, subspace balancing, and low-rank optimization.Source: §D.2.1 · pp. 24–25

D.2.2 Base models

  • Vicuna models share a base and retain modality-specific encoders and connectors, whereas the LLaMA 3.1 models are independently trained but share a compatible language-model architecture.Both settings apply merging to the shared language-model parameters while preserving each modality’s encoder and connector.Source: §D.2.2 · pp. 25–26

D.2.3 Benchmarks

  • Evaluation spans audio, video, and vision benchmarks, including MMAU, Video-MME, and TextVQA.The broader benchmark lists include five audio and six video tasks, plus GQA, MMBench, MMStar, POPE, and TextVQA for vision.Source: §D.2.3 · pp. 26–27

E.1 Additional correlation analysis

The appendix extends the alignment–performance analysis across benchmarks and backbones, examines SEO and SD changes across transformer components, and explains how DCAT schedules α in response to text effective rank.

E.1 Additional correlation analysis

  • Across additional benchmarks and both backbones, alignment is most strongly correlated with performance in middle layers, with positive Pearson and Spearman correlations at layer 21.The analysis covers additional vision, audio, and video benchmarks in both Audio+Vision and Video+Vision settings.Source: §E.1 · p. 27, Figure A1 · p. 28, Figure A2 · p. 29

E.2 Additional layer-wise SEO and SD analysis

  • DCAT consistently increases SEO and SD across examined MLP and attention projections and layers in both Vicuna merged models.Figures A5 and A6 show the component-level analysis for Audio+Vision and Video+Vision settings.Source: §E.2 · p. 32, Figure A5 · p. 33, Figure A6 · p. 34

E.3 Rationale of scheduling α

  • When text effective rank is small, DCAT uses larger α to emphasize SEO; when rank is larger, α decreases to permit stronger encouragement of SD.The schedule is motivated by the risk that increasing SD in a low-rank text subspace can spread modality information along weak or noisy directions.Source: §E.3 · p. 35, Figure A7 · p. 35

F.1 Statistical significance of presented results

DCAT’s improvements are statistically significant across reported settings, extend across subcategories and additional benchmarks, and generally outperform other merging methods. The appendix also examines vision-side effects, cross-audio/video transfers, calibration robustness, and language ability.

F.1 Statistical significance of presented results

  • DCAT yields statistically significant improvements in all reported Audio+Vision and Video+Vision settings, while baseline gains are not consistently significant.The analysis uses McNemar’s test on paired benchmark samples.Source: §F.1 · pp. 35–36, Table A3 · p. 36, Table A4 · p. 36

F.2 Additional quantitative results

  • DCAT consistently improves performance across diverse subcategories and additional benchmarks, maintaining a clear advantage over other merging methods.The additional results include detailed MVBench subcategories and MLVU, TempCompass, EgoSchema, and LongVideoBench results.Source: §F.2 · p. 36, Table A5 · p. 37, Table A6 · p. 38

F.3 Results on vision benchmarks

  • With an audio MLLM as recipient, vision benchmark performance is largely preserved or slightly improved; with a video recipient, some degradation occurs.The paper frames this as alignment transfer rather than a goal of one model performing well on both modalities.Source: §F.3 · p. 37, Table A7 · p. 39

F.4 Results on audio-video DCAT

  • Video-to-Audio DCAT improves audio performance, whereas improvement in Audio-to-Video is limited; the paper hypothesizes that donor alignment quality explains the difference.The accompanying table reports these transfers on MMAU and Video-MME, respectively.Source: §F.4 · pp. 37–38, Table A8 · p. 39

F.5 Calibration-matched baselines

  • With 256 AVQA calibration samples, DCAT beats the strongest calibration-aware baseline by 1.9 points on MMAU and 1.3 points on Video-MME overall.The comparison includes LoRA fine-tuning, RegMean, and RegMean++ and reports advantages across all categories in both benchmarks.Source: §F.5 · p. 38

F.6 Calibration sample robustness

  • Across 10 calibration subsets, overall-score standard deviations are 0.33 on MMAU and 0.14 on VideoMME, indicating robustness within AVQA.The subsets were formed by random sampling with 10 seeds.Source: §F.6 · p. 38

F.7 Language ability on alignment-transferred models

  • After DCAT, Audio+Vision shows no significant language-ability change (p = 0.506), while Video+Vision shows significant improvement (p ≪ 0.01).Language ability was evaluated on 13 text benchmarks.Source: §F.7 · p. 39

F.8 Additional ablation study

The ablations find that balancing coefficients matter, while performance is robust to subspace dimension and calibration-set size. DCAT’s merging cost remains practical in the reported settings.

F.8 Additional ablation study

  • Appropriate choices of β and η are important for strong target-modality performance, whereas DCAT is robust to subspace dimension k and calibration sample count.The sweeps were conducted with Vicuna-based models over ranges where marked degradation was observed on MMAU and Video-MME.Source: §F.8 · p. 39

F.9 Efficiency study

  • On one L40S GPU, DCAT takes 28.45/34.57 minutes for LLaMA-Audio/Video and about 24–25 minutes for Vicuna, with peak memory of 16.72GB and 14.56GB, respectively.For LLaMA settings, merging time is comparable to TSV and OptMerge; Vicuna costs more than lightweight baselines because it merges full weights after LoRA adapters.Source: §F.9 · p. 43

G Limitation

This section covers DCAT’s limitations and broader-impact disclosures, including the scope of its alignment score and model compatibility, potential societal benefits and risks, and statements about human subjects, crowdsourcing, and LLM use in benchmark evaluation.

G Limitation

  • Alignment does not explain performance uniformly: some benchmarks show weak or negligible correlation, where perception, reasoning format, dataset bias, or evaluation protocol may matter more.The authors frame the score as a useful, broadly predictive indicator rather than a complete explanation, and identify exceptional cases as future work.Source: G Limitation · p. 43
  • DCAT requires compatible parameter shapes, limiting its use with in-the-wild models whose language backbones differ in vocabulary, embeddings, LM head, or other components.The authors avoid ad-hoc resizing, padding, partial copying, or retraining, and identify extending DCAT to heterogeneous backbones as future work.Source: G Limitation · p. 44

H Broader Impacts

  • DCAT could reduce data and compute costs and improve accessibility, but stronger audio and video understanding could enable surveillance, profiling, unfair decisions, or amplified bias and privacy risks.The authors recommend demographic and domain evaluation, privacy-preserving calibration data, safety testing before and after merging, restrictions on sensitive uses, and human oversight in high-risk settings.Source: H Broader Impacts · pp. 44–45

10. Broader impacts

  • The paper reports no crowdsourcing or human-subject research, marking participant-instruction, compensation, and IRB disclosures as not applicable.Source: §10 · pp. 52–53
  • The paper declares LLM use because some experiments use an LLM as a judge and for parsing.Source: §10 · p. 53
Loading 2610.04580v1…