PaperlayerView exact revision on arXiv

Source-linked AI summary

Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework

Grzegorz Statkiewicz, Alicja Dobrzeniecka, Karolina Seweryn, Aleksandra Krasnodębska, Karolina Piosek, Katarzyna Bogusz, Sebastian Cygert, Wojciech Kusa

arXiv:2602.14073v2cs.CLcs.AI

TL;DR

English-centric training data limits VLM performance in other languages and cultural contexts. This paper adapts LLaVA-NeXT to Polish using translated and filtered multimodal data, supplemented with synthetic data. The models improve over LLaVA-family baselines on Polish MMBench, while caption evaluations show mixed comparisons with stronger models.

  • Problem

    English-centric training data limits VLM performance in other languages and cultural contexts.

    Source: §1 · p. 1

  • Method

    The authors adapt LLaVA-NeXT to Polish with automated translation and filtering of multimodal datasets, supplemented by synthetic data for tasks such as OCR.

    Source: §1 · pp. 1–2

  • Results

    +9.55 percentage point gain over the best LLaVA-1.6 baseline on the Polish split, while caption evaluations show mixed comparisons with stronger baselines.

    Source: §6 · p. 7

  • Takeaways & Limitations

    Large-scale automated translation combined with basic filtering can effectively bootstrap multimodal models for languages with limited native resources.

    Source: §7 · p. 8

    Training data relies heavily on automatic translation, and the downstream impact of translation artefacts was not directly measured.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Most vision-language models (VLMs) are trained on English-centric data, limiting their performance in other languages and cultural contexts. This restricts their usability for non-English-speaking users and hinders the development of multimodal systems that reflect diverse linguistic and cultural realities. In this work, we reproduce and adapt the LLaVA-Next methodology to create a set of Polish VLMs. We rely on a fully automated pipeline for translating and filtering existing multimodal datasets, and complement this with synthetic Polish data for OCR and culturally specific tasks. Despite relying almost entirely on automatic translation and minimal manual intervention to the training data, our approach yields strong results: we observe a +9.5% improvement over LLaVA-1.6-Vicuna-13B on a Polish-adapted MMBench, along with higher-quality captions in generative evaluations, as measured by human annotators in terms of linguistic correctness. These findings highlight that large-scale automated translation, combined with lightweight filtering, can effectively bootstrap high-quality multimodal models for low-resource languages. Some challenges remain, particularly in cultural coverage and evaluation. To facilitate further research, we make our models and evaluation dataset publicly available.

1 Introduction

The paper adapts LLaVA-NeXT to Polish to address the limits of English-centric VLM training, testing automated translation with minimal manual intervention in training data. It introduces Polish models and a human-revised Polish MMBench, and reports improvements over LLaVA-family baselines and strong Polish caption quality.

1 Introduction

  • The authors report improvements over LLaVA-family baselines on Polish and English MMBench versions and linguistically correct Polish captions that match or surpass several open-access models.Caption comparisons include PaliGemma2-10B, Pixtral-12B, and Qwen2.5-VL-7B.Source: §1 · pp. 1–2
  • A fully automated dataset pipeline translates and filters multimodal data, with synthetic data for tasks such as OCR that are difficult to translate.The training mixture covers general VQA, OCR, and counting tasks.Source: §1 · pp. 1–2
  • The paper presents a Polish VLM family built with LLaVA-NeXT and Polish-native 11B–12B language backbones.The models are named LLavA-PLLuM and LLaVA-Bielik.Source: §1 · pp. 1–2
  • The Polish MMBench adaptation includes linguistic reannotation that identifies issues in nearly 4% of samples.The paper also releases its models and Polish MMBench version for future research.Source: §1 · p. 2

2 Related Work

The related work situates Polish VLM adaptation among multilingual VLMs, instruction datasets, and Polish language models. The paper distinguishes its approach through automated translation, Polish-native backbones, and human intervention limited to evaluation benchmark curation.

2.1 Vision-language models

  • Earlier VLMs developed aligned image-text representations and later advanced pretraining and architecture for cross-modal reasoning and generation.The discussion cites CLIP and ALBEF as contrastive-learning examples, followed by BLIP, BLIP-2, and SigLIP2.Source: §2.1 · pp. 2–3
  • Prior LLaVA adaptations include Korean-English, Italian, and multilingual models, while this work focuses on one morphologically rich language using automated translation and Polish-native backbones.The approach omits vocabulary modification and cross-lingual pretraining, with human intervention limited to evaluation benchmark curation.Source: §2.1 · p. 3

2.2 Visual instruction following datasets

  • Instruction-style datasets support conversational and reasoning-oriented VLMs, while MMBench evaluates multimodal perception, reasoning, and knowledge and XM3600 covers multilingual image-text understanding.The discussion also traces dataset development from MS-COCO to instruction-oriented resources such as LLaVA-Instruct, The Cauldron, and WIT.Source: §2.2 · p. 3

2.3 Polish large language models

  • Polish’s inflection, seven grammatical cases, agreement patterns, and relatively free word order challenge language understanding and generation, yet Polish VLM datasets and models were absent to the authors’ knowledge.The paper addresses this gap with Polish-adapted VLMs and benchmarks, restricting human annotation to evaluation.Source: §2.3 · pp. 3–4

3 Model

The model follows LLaVA-NeXT, combining a pretrained vision encoder and language model through a lightweight projector. It uses Polish-native instruction-tuned language backbones, SigLIP2, and parameter-efficient LoRA adaptation.

3 Model

  • LLaVA-NeXT aligns a pretrained visual encoder with an LLM through a two-layer MLP projector and supports higher resolutions and dynamic tiling.These features have been observed to improve fine-grained perception and OCR performance.Source: §3 · p. 4
  • The model uses Polish-native, instruction-tuned PLLuM and Bielik language backbones in the 11–12B parameter range and replaces the usual CLIP-like vision encoder with SigLIP2.SigLIP2 was selected for multilingual image-text alignment and text localization.Source: §3 · p. 4
  • Training first updates only the projector for visual-language alignment, then jointly fine-tunes the projector and vision encoder while adapting the language model with LoRA.The authors choose parameter-efficient adaptation to reduce computational resource use while preserving the language backbone’s pretrained linguistic competencies.Source: §3 · p. 4

4 Data

The data strategy combines translated and filtered multimodal training data, synthetic examples for OCR, and a linguistically and content-reviewed Polish evaluation benchmark. The instruction mixture covers general visual tasks, OCR, cultural knowledge, and counting.

4 Data

  • The data pipeline combines large-scale automated translation, metric-based filtering, and synthetic data for Polish multimodal training and benchmark refinement.It prepares a captioning corpus for pretraining and a visual instruction mixture for broader capabilities.Source: §4 · p. 4, §4.1 · p. 4

4.1.1 Polish language adaptation

  • English multimodal dialogues are translated with Tower+ 72B, assessed with reference-less COMET, and filtered by removing QA pairs below dataset-specific thresholds.Thresholds vary from 0.4 to 0.8 based on preliminary manual inspection.Source: §4.1.1 · p. 4

4.1.2 Pre-training data

  • The pretraining corpus adapts LLaVA-LCS-558K into a Polish-majority mixture to help the visual tokenization and projector interface with Polish prompts early in training.The corpus contains 595K multi-turn samples with broad image-text coverage and simple conversational templates.Source: §4.1.2 · p. 5

4.1.3 Instruction data

  • The instruction mixture preserves an 85:15 balance of Polish-adapted and original English samples across general VLM supervision, synthetic OCR, Polish-captioned WIT knowledge examples, and translated counting data.OCR examples render Polish and English text on natural-image backgrounds; counting examples come from TallyQA.Source: §4.1.3 · p. 5

4.2 MMBench adaptation

  • The Polish MMBench dev set is machine-translated and reviewed by native professional linguists for linguistic and content corrections.The adapted benchmark contains 1,292 questions; 3.56% had inaccuracies and 3.02% were rooted in foreign contexts, with problematic questions corrected when possible.Source: §4.2 · p. 5

5 Experiment setup

The experiments evaluate model training and compare Polish VLMs with open-weight baselines using MMBench and image-captioning assessments.

5.1 Model training

  • Stage 1 pretraining used A100 GPUs for approximately 336 GPU-hours per model, while Stage 2 visual instruction tuning used GH200 nodes for approximately 1,344 GPU-hours per model.The training description covers three models.Source: §5.1 · p. 5

5.2 Baselines

  • Five comparable-size open-weight VLMs provide architectural and leading-model baselines, including LLaVA-1.6-Mistral-7B, LLaVA-1.6-Vicuna-13B, Qwen2.5-VL-7B, PaliGemma2-10B, and Pixtral-12B.The comparison is intended to assess competitiveness against leading multilingual systems.Source: §5.2 · pp. 5–6

5.3 Evaluation procedure

  • The evaluation covers a multimodal benchmark and image-captioning quality.The following protocols assess MMBench and XM3600, respectively.Source: §5.3 · p. 6

5.3.1 MMBench

  • MMBench V1.1 uses its dev split in English and Polish-adapted variants, with accuracy measured after direct answer generation and regular-expression extraction of choices A–D.The two language variants assess cross-lingual transfer and native alignment.Source: §5.3.1 · p. 6

5.3.2 XM3600

  • XM3600 caption quality is assessed with open-source LLM/VLM judges on 3,600 images, Claude Sonnet 4.5 on 500 images, and native Polish linguists on 400 image-caption pairs.Judges and annotators compare linguistic correctness and image-content quality; each manually evaluated item is scored by one annotator.Source: §5.3.2 · p. 6

6 Results and Discussion

On the Polish-adapted MMBench, LLaVA-PLLuM-12B-NC outperforms the best LLaVA-1.6 baseline while maintaining similar English performance. Caption evaluations show that human judgments favor the models more strongly on linguistic correctness than automatic judgments do, while content-description results vary across evaluators.

6 Results and Discussion

  • +9.55 percentage points on the Polish split separates LLaVA-PLLuM-12B-NC from the best LLaVA-1.6 baseline, with similar English performance.The results table covers both Polish and English MMBench variants; scores generally drop when switching from English to Polish.Source: §6 · pp. 6–7, Table 2 · p. 7
  • 48% win rate on XM3600 linguistic correctness made LLaVA-PLLuM-12b-nc comparable to Pixtral-12B under Llama-3.3-70B-Instruct evaluation.The models consistently outperform LLaVA-1.6-Mistral-7B, LLaVA-1.6-Vicuna-13B, and PaliGemma2-10B in this evaluation.Source: §6 · p. 7, Table 3 · p. 7
  • 49.29% and 48.75% win rates put LLaVA-PLLuM-12b-nc on par with Pixtral-12B and Qwen2.5-VL-7B, respectively, for content-description quality.The reported comparison also places it ahead of LLaVA-1.6-Mistral-7B, LLaVA-1.6-Vicuna-13B, and PaliGemma2-10B.Source: §6 · p. 7, Table 4 · p. 7
  • Claude-based evaluation found stronger linguistic win-tie rates against Pixtral but lower performance than Qwen, while competitors generally led on content description.LLaVA-Bielik-11b-v2.6 reached a 48.1% linguistic-quality win-tie rate, comparable to Qwen2.5-VL-7B.Source: §6 · p. 7, Figure 3 · p. 7, Figure 4 · p. 8
  • Human reviewers favored the models on linguistic correctness, with at least 64% wins and up to 84% for LLaVA-PLLuM-12b-nc against Pixtral-12B.For content description, that model won 39% versus Pixtral’s 34% but still underperformed Qwen; the linguistic advantage was less apparent automatically.Source: §6 · pp. 7–8, Figure 6 · p. 8, Figure 5 · p. 8

6.1 Qualitative analysis

  • In qualitative examples, LLaVA-PLLuM-12b-nc correctly identified a cat in a dishwasher and listed Poland’s seven neighboring countries, where competitors made errors.The examples illustrate gains in content recognition and language accuracy.Source: §6.1 · p. 8

7 Conclusion and Future Work

The paper adapts the LLaVA-NeXT recipe to Polish using mostly translated multimodal data and limited synthetic data, with improvements over English-centric baselines. It concludes that automated translation with basic filtering can bootstrap VLMs for languages with limited native resources, while cultural coverage remains a target for future work.

7 Conclusion and Future Work

  • The Polish adaptation uses large-scale automated translation, complemented by limited synthetic data.The approach targets a non-English, morphologically rich language.Source: §7 · p. 8
  • The models show consistent improvements over English-centric baselines on a Polish-adapted MMBench and in human evaluations of caption quality.Source: §7 · p. 8
  • Automated translation with basic filtering may effectively bootstrap multimodal models for languages with limited native resources, while future work may improve cultural coverage with language- or region-specific data.Source: §7 · pp. 8–9

Limitations

The limitations concern translation quality, validation of Polish-specific capabilities, cultural coverage, evaluation comparability, and untested design choices. The appendix also describes the training configuration and computational resources used.

Limitations

  • Automatic translation may introduce translationese, semantic drift, and unnatural phrasing, while threshold ablations and downstream effects of these artefacts were not measured.Source: Limitations · p. 9
  • Polish OCR and Poland-specific knowledge objectives lack dedicated evaluations, limiting direct validation of those capabilities.Source: Limitations · p. 9
  • English-centric source datasets and a narrow set of benchmarks may leave performance in authentic Polish contexts, deeper reasoning, factual grounding, cultural robustness, and real-world usability underexplored.Source: Limitations · p. 9
  • Preference rates across evaluation modes are not directly comparable because the XM3600 automatic-judge setup disallows ties while human and Claude evaluations allow them.Source: Limitations · p. 9
  • Key choices—including SigLIP2 versus CLIP, language-mix ratio, two-stage training, and LoRA versus full fine-tuning—were not comprehensively ablated.The authors call for systematic evaluation of these factors and their interactions.Source: Limitations · p. 9

A Traning Details

  • Table 5 organizes training hyperparameters and computational resources into pre-training and instruction-tuning columns.It covers batch sizes, context lengths, learning rates, LoRA configuration, and hardware.Source: A Traning Details · p. 13

B Manual Evaluation

Manual evaluation used an anonymous subset of observations from the automatic Claude Sonnet evaluation. Table 6 reports the number of samples assessed by human annotators.

B Manual Evaluation

  • Human evaluation used a subset of the 500 observations included in the automatic Claude Sonnet evaluation.Source: B Manual Evaluation · p. 13
  • Annotators were not told which models generated the responses, a design intended to prevent bias and support fair comparison.Source: B Manual Evaluation · p. 13
  • Table 6 reports the number of sample pairs assessed manually.Source: Table 6 · p. 13

C Prediction Examples

The examples use a small curated image set to illustrate Polish cultural-context responses. They show cases where LLaVA-PLLuM-12B-nc answers correctly and cases where competing models make factual or linguistic errors.

C Prediction Examples

  • The examples come from a small curated and annotated image dataset intended to assess Polish cultural-context understanding.Source: C Prediction Examples · p. 13
  • LLaVA-PLLuM-12B-nc identifies the location correctly, while Qwen2.5-VL-7B and Pixtral-12B give an incorrect location and LLaVA-1.6-Vicuna-13B invents one.Source: Figure 7 · p. 13
  • LLaVA-PLLuM-12B-nc responds correctly, while Qwen2.5-VL-7B miscounts neighbors and Pixtral-12B and LLaVA-1.6-Vicuna-13B show linguistic errors and factual inaccuracies.Source: Figure 8 · p. 13
  • LLaVA-PLLuM-12B-nc and Qwen2.5-VL-7B answer correctly; Pixtral-12B names the tree rather than the fruit, and LLaVA-1.6-Vicuna-13B invents a fruit.Source: Figure 9 · p. 13
  • Qwen2.5-VL-7B identifies the object and quantity but uses the wrong grammatical inflection; Pixtral-12B gets the quantity wrong, and LLaVA-1.6-Vicuna-13B invents an animal.Source: Figure 10 · p. 14
  • LLaVA-PLLuM-12B-nc and Pixtral-12B identify the museum correctly, while Qwen2.5-VL-7B and LLaVA-1.6-Vicuna-13B give incorrect museum names.Source: Figure 11 · p. 14

D Discussion of MMbench issues

The MMBench review identifies flawed questions and items rooted in foreign cultural or linguistic contexts, with the latter posing localization challenges rather than substantive flaws. It groups flaws by their prevailing characteristic and provides examples of issues and adaptations.

D Discussion of MMbench issues

  • Categories 1d–1i cover ambiguous images, unknowable future consequences, potentially stereotypical phrasing, minor factual errors, missing context, and flawed image-question or answer relations.Each question was assigned to one category based on its prevailing characteristic, despite possible overlap.Source: D Discussion of MMbench issues · p. 14
  • 3.56% of unique MMBench questions were inaccurate or otherwise flawed, while 3.02% of the dev split required foreign cultural or linguistic knowledge.The foreign-context questions were not considered substantively flawed, but posed localization challenges; Chinese or Japanese text fragments in Category 2 were translated into Polish.Source: D Discussion of MMbench issues · p. 14
  • Figures 12–30 present MMBench V1.1 example problems, including cases with more than one correct answer, no correct answer, and tasks identifying people, buildings, or dishes foreign to European cultural context.Table 7 categorizes issues among 1,292 unique questions, and Table 8 covers questions rooted in foreign contexts. Questions requiring foreign cultural and linguistic context constitute 3.02% of the MMBench dev split.Source: D Discussion of MMbench issues · p. 14, Table 7 · p. 14, Table 8 · p. 14
  • Potentially stereotypical wording was revised to ask about a man’s clothing color, and another question’s answer choices were changed so colors referred to boys’ T-shirts.These examples show adaptations to phrasing and answer content in the Polish benchmark.Source: D Discussion of MMbench issues · p. 16

E Evaluation prompts

The evaluation prompts compare Polish captions for linguistic quality and image-content correspondence, with some prompts separating these judgments and others combining them. Evaluation procedures also address response-order effects and use different judgment formats for Claude Sonnet 4.5 and other evaluations.

E Evaluation prompts

  • For evaluations other than Claude Sonnet 4.5, both model-response orders are prompted and their scores are averaged; Claude’s response order is assigned randomly.These procedures address response ordering in the pairwise evaluations.Source: E Evaluation prompts · p. 18
  • The prompt evaluates two captions independently for Polish-language quality and image-content correspondence, allowing different category-specific decisions, including a tie when the captions are comparable.Source: E Evaluation prompts · p. 19
  • The language-only prompt judges Polish grammar, spelling, punctuation, syntax, and idiomatic naturalness while explicitly excluding image accuracy and factual content.It requires a choice between A and B even when both appear equally good or bad.Source: E Evaluation prompts · p. 20
  • The image-grounded caption prompt assesses image correspondence, detail, linguistic correctness, and clarity, instructing the judge not to guess details that are not visible.The prompt evaluates each caption independently before selecting the better one.Source: E Evaluation prompts · p. 21

Related papers

Loading 2602.14073v2…