PaperlayerView exact revision on arXiv

Source-linked AI summary

VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation

Yixia Li, Yaqing Shi, Zhiwen Ruan, Dongdong Zhang, Lingjie Jiang, Shaohan Huang, Yun Chen, Guanhua Chen, Furu Wei

arXiv:2608.26155v1cs.CLcs.AI

TL;DR

Most MLLMs remain English-centric because multilingual multimodal supervision is scarce and costly, while direct text tuning can disrupt vision-language alignment. VFA fine-tunes a shared base LLM on multilingual text and merges the resulting task vector with a vision-aligned MLLM. Across five MLLMs and six benchmarks, it consistently improves multilingual performance while largely retaining general multimodal and text-only capabilities.

  • Problem

    Most open-source MLLMs remain English-centric, while multilingual multimodal adaptation is constrained by scarce, expensive non-English image–text supervision and direct text tuning can disrupt cross-modal alignment.

  • Method

    VFA fine-tunes the original LLM on multilingual text to derive a multilingual task vector, then fuses it with the MLLM’s vision-aligned task vector.

  • Results

    VFA consistently enhances multilingual performance across five MLLMs and six benchmarks while largely retaining general multimodal and text-only capabilities; using 100K text samples, it narrows the gap to models trained with millions of image–text pairs.

  • Takeaways & Limitations

    VFA provides a practical, resource-efficient pathway for broadening MLLMs’ linguistic capabilities without additional text-image paired datasets.

  • Takeaways & Limitations

    VFA’s text-only approach primarily enhances language-intensive reasoning, while perception-heavy tasks such as OCR may still benefit from supplementary image–text supervision.

Abstract

from arXiv · show

Multimodal large language models have advanced rapidly, yet most remain English-centric, as scaling multilingual multimodal instruction tuning is limited by the scarcity and high cost of high-quality non-English image-text supervision. Although multilingual text data is abundant, naive textual fine-tuning can disrupt vision-language alignment and induce catastrophic forgetting. We propose Vision-Free Adaptation (VFA), a framework that decouples multilingual language enhancement from visual alignment by composing complementary task vectors over a shared LLM backbone. Specifically, we fine-tune a base LLM on multilingual text data to derive a multilingual task vector, which is then merged with the vision-aligned task vector of an MLLM. Experiments on five MLLMs across six multilingual multimodal benchmarks show consistent improvements while preserving both general multimodal and text-only capabilities. Moreover, using less than 2% of the text data, VFA narrows the gap to the fully multimodal-trained model, demonstrating its data efficiency.

1 Introduction

Most open-source MLLMs remain English-centric, while multilingual multimodal tuning is limited by scarce, costly supervision and direct text tuning can disrupt visual alignment. VFA addresses these constraints by merging multilingual and vision-aligned task vectors, improving multilingual performance while retaining capabilities.

  • Most open-source MLLMs degrade substantially when prompts, outputs, or benchmarks shift beyond English.
  • High-quality non-English image–text supervision is scarce, expensive to curate, unevenly distributed, and computationally costly to use at scale.Visual tokens increase attention and memory costs during multimodal training.
  • Directly fine-tuning an MLLM on multilingual text can interfere with cross-modal alignment and cause catastrophic forgetting of multimodal capabilities.
  • VFA derives a multilingual task vector by fine-tuning the original LLM on multilingual text, then fuses it with the MLLM’s vision-aligned task vector.The composition injects multilingual competence while preserving existing vision-language alignment.
  • +2.99 and +0.57 improvements were obtained on LLaVA-OneVision-1.5 at 8B and 4B scales, respectively, across six multilingual multimodal benchmarks.VFA also largely retained general multimodal and text-only capabilities, and narrowed the gap to models trained with millions of image–text pairs using only 100K text samples.

2 Related Work

Related work establishes multilingual MLLMs, model merging, and the resource constraints of multimodal tuning. VFA applies model merging to multilingual adaptation, combining text-only fine-tuning with efficient task-vector composition.

  • Multimodal LLMs: Open-source MLLMs commonly align vision encoders with LLMs through a projection layer, with later systems improving training strategies.
  • Multilingual MLLMs: Multilingual MLLM research expands data scale, but high-quality image–text data shortages and high training costs remain significant challenges.Pangea and Aya Vision illustrate efforts to broaden language coverage and use synthetic data.
  • Model Merging for MLLM: Model merging offers a cost-efficient way to expand model capabilities without increasing inference overhead.
  • VFA Architecture: VFA uses multilingual text-only data to obtain a task vector and merges it into a vision-aligned MLLM while preserving visual grounding.When MLLMs share an underlying LLM, the multilingual task vector can be trained once and reused through lightweight merging.
  • Model Merging for MLLM: VFA addresses the limited use of model merging for multilingual adaptation by combining fine-tuning with merging instead of relying on large-scale image–text datasets.

3 Methodology

VFA combines a multilingual task vector learned from text with a vision-aligned MLLM update, aiming to improve multilingual capability without disrupting visual grounding. It uses task-vector merging and frozen visual modules to provide a resource-efficient alternative to joint multimodal fine-tuning.

  • Vision-Free Adaptation: VFA treats multilingual and multimodal adaptation as task vectors over a shared LLM backbone.A multilingual task vector is composed with the vision-aligned MLLM update.
  • Vision-Free Adaptation: VFA injects the multilingual parameter update into the MLLM backbone through a scaling factor α.The update is the parameter difference between the multilingual fine-tuned LLM and the base LLM.
  • Merged MLLM: VFA keeps the vision encoder and projection modules frozen while integrating the merged language backbone with the original visual modules.The visual stream remains fixed to retain vision–language grounding, and the resulting model adds no inference latency or memory overhead compared with the original MLLM.
  • Merged MLLM: When MLLMs share an underlying LLM, one multilingual task vector can be trained once and reused through lightweight merging across models.This contrasts with separate end-to-end multimodal adaptation for each MLLM.
  • Merging Methods: Three merging operators—Weight Averaging, Task Arithmetic, and TIES-Merging—compose multilingual updates with the vision-aligned MLLM.They provide different trade-offs between multilingual capability injection and preservation of vision–language alignment.
  • Merging Methods: The mixing coefficient α balances general multimodal capabilities against task-specific expertise and controls the strength of injected capabilities.For TIES-Merging, trimming, electing, and sign-merging are used to mitigate parameter interference.

4 Experiments

VFA is evaluated across five MLLMs, multiple model families and scales, and multilingual multimodal, general multimodal, and text-only benchmarks. It improves multilingual multimodal performance while largely preserving existing capabilities, though gains vary by benchmark and model.

  • Experimental Setup: VFA is evaluated on five MLLMs spanning multiple model families and 4B–8B scales, using multilingual multimodal, general multimodal, and text-only evaluation axes.The experiments use a 100K-example subset of the Multilingual-SFT dataset and compare merging configurations with CVQA validation.
  • Multilingual Multimodal Results: +12.33 on Qwen2.5-VL-7B and +36.17 on Idefics3-8B are reported on MaRVL, with average gains of +2.39, +10.64, and +6.35 for three evaluated model configurations.The largest improvements occur on culturally grounded visual reasoning, while Qwen3-based LLaVA-OV models gain +2.99 at 8B and +0.57 at 4B.
  • Multilingual Multimodal Results: Direct Text SFT reduces Qwen2.5-VL-7B’s overall average by 11.92 and reduces MaRVL from 53.50 to 0.00, whereas VFA improves multilingual multimodal performance without degrading existing capabilities.For LLaVA-Next-8B, Direct Text SFT also produces a −39.83 MaRVL change and a 3.40-point overall-average drop.
  • General Multimodal Results: VFA maintains or improves general-purpose vision–language performance, including +1.36 on Idefics3-8B and +1.00 on Qwen2.5-VL-7B, while LLaVA-OV-1.5-8B changes by −0.19.The results show no systematic degradation, but improvements are concentrated mainly on reasoning-centric benchmarks such as MMMU and MathVista.
  • Text-only Multilingual Results: VFA preserves or improves most text-only multilingual benchmarks, with LLaVA-Next-8B gaining +6.27 on average but Idefics3-8B declining on FLORES by −13.94 and XNLI by −3.08.The FLORES regression is attributed to over-generation caused by failure to emit the EOS token, which lowers BLEU through precision dilution.

5 Analysis

The analysis studies merging choices and compares VFA with large-scale multimodal training and a visual-coding case study. VFA’s selected configurations are architecture-dependent, while its text-only training uses substantially less data than the multimodal baseline and also transfers to visual coding.

  • Merging Strategies: WA, TA, and TIES show architecture-dependent performance across mixing coefficients α, so no single merging operator dominates all backbones.WA and TA outperform TIES for Qwen2.5-VL-7B and LLaVA-OV-1.5-4B, whereas TIES is stronger for Idefics3-8B.
  • Efficiency Comparison: 47.84 from VFA using 100K text-only samples trails Pangea-7B’s 50.34 by 2.50 points despite Pangea using 6M multimodal samples.This corresponds to about 2% of Pangea’s training size and shifts the performance–data-cost trade-off toward lower data requirements.
  • Broader Applicability: VFA improves visual coding performance across all three evaluated backbones, with gains of +4.59 on InternVL3-8B and +6.62 on Idefics3-8B.The case study presents vision-free injection as applicable beyond multilingual adaptation.

6 Conclusion

VFA broadens MLLMs’ multilingual capabilities without additional text-image pairs while preserving generic multimodal robustness. Across five architectures and six benchmarks, it can surpass models trained on massive multimodal datasets in some tasks.

  • VFA enhances multilingual MLLM capabilities without relying on additional text-image paired datasets.
  • The framework decouples linguistic knowledge injection from visual representation learning to avoid catastrophic forgetting of visual alignment.
  • Evaluations across five diverse architectures and six multilingual multimodal benchmarks show preserved generic multimodal robustness.
  • VFA surpasses models trained on massive multimodal datasets in some tasks, providing an efficient pathway to broaden MLLMs’ linguistic capabilities.

Limitations

The evaluation uses modest, unfiltered multilingual text data and focuses mainly on 4B–8B models. VFA may therefore remain less suitable for perception-heavy tasks and larger-scale settings.

  • The multilingual adaptation data lacks explicit quality filtering, and its scale is intentionally modest to highlight VFA’s efficiency.
  • As a text-only approach, VFA primarily enhances language-intensive reasoning, while perception-heavy tasks such as OCR may benefit from supplementary image-text supervision.
  • The empirical evaluation mainly focuses on models in the 4B–8B parameter range.
  • Future work includes larger model and data scales, stricter data curation, and advanced merging algorithms to more consistently mitigate interference.

A.1 Datasets

The dataset appendix describes multilingual instruction resources and the construction and training setup for VFA’s 100K multilingual text subset. It also contrasts VFA’s text-only efficiency with standard multimodal supervised fine-tuning.

  • The 100K multilingual text subset is deterministically filtered from agentlans/multilingual-sft using field validation and removal of multimodal-tagged instances.The fine-tuning setup uses total batch size 128, learning rate 1e-5, and one epoch; VFA avoids visual-token processing and reduces training costs relative to multimodal SFT.
  • xP3mt is a large-scale multilingual instruction-tuning dataset covering 46 languages and 13 training tasks.It uses machine translation to produce prompts in 20 diverse languages.
  • Bactrian-X contains approximately 3.4 million instruction-response pairs across 52 languages.Translated English instructions are paired with responses generated by gpt-3.5-turbo.
  • Aya Dataset comprises approximately 204K human-curated instruction-response pairs covering 65 languages.Its curation targets biases and artifacts in machine-translated corpora, especially for low-resource languages.
  • Tagengo-GPT4 provides approximately 76,000 multilingual prompt-response pairs covering 74 languages.
  • Multilingual-SIFT is derived from English instruction corpora translated into multiple target languages using GPT-3.5 Turbo.

B Evaluation

The evaluation covers multilingual language, reasoning, visual, and multimodal capabilities using several benchmarks and qualitative comparisons. It includes model-backbone and merge-coefficient tables, dataset summaries, and training-efficiency comparisons.

  • The evaluation includes TyDiQA and multilingual image-captioning cases, with one captioning example comparing the base model’s output against a ground-truth caption.The reported base output describes a single camera, while the ground truth mentions two cameras.
  • Table 9 compares standard multimodal SFT with VFA on training time, GPU memory, and data requirements.VFA uses text-only data and avoids visual-token processing.
  • P-MMEval evaluates core linguistic and reasoning capabilities across 10 typologically distinct languages using parallelized samples and task-specific metrics.
  • PangeaBench assesses multilingual and multicultural MLLM capabilities across 47 languages and 14 datasets, including culturally specific tasks.Its categories range from multimodal chat to multi-subject reasoning.
  • A FLORES Chinese case study examines EOS failure in Idefics3-8B, where hallucinated continuation causes BLEU regression, alongside a case with proper EOS emission.
  • The appendix reports model–LLM backbone pairings and selects merge operators and coefficients per model using CVQA validation.

C PangeaBench Results by Language

The section presents performance comparisons between VFA-enhanced MLLMs and base MLLMs across six multilingual benchmarks.

  • VFA-enhanced MLLMs are compared with base MLLMs across MaXM, xGQA, xMMMU, MaRVL, M3Exam, and XM100.
  • The reported comparisons cover multilingual visual and examination benchmarks.

D Qualitative Analysis

Qualitative examples show that VFA improves the precision and overall quality of non-English visual question-answering generations while avoiding the typical vision-language alignment tax. Additional analyses examine EOS failures, model-merging guidelines, and targeted monolingual adaptation.

  • Qualitative comparisons: VFA improves the precision and overall quality of non-English generations across multilingual visual question-answering examples.The qualitative comparisons are illustrated in Figure 4.
  • Qualitative comparisons: The qualitative gains are reported without the typical vision-language alignment tax.
  • EOS-failure analysis: In Idefics3-8B, VFA preserves accurate first-sentence translation but frequently fails to emit EOS, causing hallucinated overgeneration until the length limit.Figure 5 presents raw examples of this EOS-failure pattern.
  • Model-merging guidelines: The authors establish practical model-merging guidelines from their empirical findings.These guidelines are reported in Table 13.
  • Monolingual adaptation: VFA remains effective for targeted language-specific adaptation on Hindi and Romanian, which are outside the main multilingual suite.The monolingual results are reported in Table 20.
Loading 2608.26155v1…