PaperlayerView exact revision on arXiv

Source-linked AI summary

jina-vlm: Small Multilingual Vision Language Model

Andreas Koukounas, Georgios Mastrapas, Florian Hönicke, Sedigheh Eslami, Guillaume Roncari, Han Xiao

arXiv:2512.04032v4cs.CLcs.AIcs.CV

TL;DR

Multilingual degradation and high training and deployment costs remain challenges for VLMs, while small-model training data mixtures are not well understood. jina-vlm combines SigLIP2, Qwen3, image tiling, and attention pooling, and the authors use leave-one-out data-category ablations to examine category contributions and transfer. It achieves state-of-the-art multilingual VQA and leading English VQA results among open 2B-scale VLMs.

  • Problem

    Small VLMs face multilingual degradation and high training and deployment costs, while fine-grained training-mixture contributions remain poorly understood under limited budgets.

    Source: §1 · p. 1, §6 · p. 8

  • Method

    jina-vlm pairs SigLIP2 with Qwen3 using image tiling and attention pooling, while leave-one-out data-category ablations probe category necessity and cross-task transfer.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    jina-vlm achieves state-of-the-art multilingual VQA and leading English VQA among open 2B-scale VLMs, with a 72.3 average across English VQA benchmarks.

    Source: §1 · p. 1, §5.1 · p. 6, §7 · p. 9

  • Takeaways & Limitations

    The ablations indicate that task-specific, text-only, and multilingual data support their respective capabilities, while some categories trade off against multimodal performance.

    Source: §6.2 · p. 9

    The data-mixture ablations ran for 10% of full training, so their findings are preliminary evidence rather than definitive conclusions.

    Source: §7 · p. 10

Abstract

from arXiv · show

We present jina-vlm, a token-efficient 2.4B parameter vision-language model that achieves state-of-the-art multilingual VQA performance among open 2B-scale VLMs. The model couples a SigLIP2 vision encoder with a Qwen3 language decoder and makes use of image tiling and attention-pooling for token-efficient processing of arbitrary-resolution images. To understand the contribution of different training data categories, we conduct a leave-one-out data mixture ablation study-systematically removing task, domain, modality, and language categories-to diagnose which data types are necessary versus redundant and whether task benefits transfer across domains. Model weights and code are publicly released at https://huggingface.co/jinaai/jina-vlm.

1 INTRODUCTION

Vision-language models show strong multimodal results, but multilingual performance can degrade during vision adaptation and training and deployment costs remain high. jina-vlm addresses these challenges and examines which training-data categories matter and whether their benefits transfer.

1 INTRODUCTION

  • VLM deployment faces two challenges: multilingual capabilities can degrade during vision adaptation, and high-quality models remain expensive to train and deploy.The costs limit accessibility for researchers and practitioners with constrained resources.Source: §1 · p. 1
  • jina-vlm achieves state-of-the-art multilingual performance on MMMB and Multilingual MMBench, and the highest average score, 72.3, across eight English VQA benchmarks among 2B-scale VLMs.The English benchmarks span diagrams, charts, documents, and OCR; the model uses high-quality English and multilingual training data and a token-efficient image pipeline.Source: §1 · p. 1
  • The paper studies fine-grained training-data categories and whether their benefits transfer across tasks, aiming to distinguish essential from redundant data types.The authors identify dataset-mixture optimization as important for small-scale VLM training.Source: §1 · p. 1

2 RELATED WORK

Related work covers VLM architectures and training, resolution and token efficiency, connector designs, small models, multilinguality, and data-mixture analysis. jina-vlm’s approach combines overlapping tiling and learned attention pooling, while its leave-one-out ablations examine category-level contributions and transfer.

VLM architecture and training

  • Decoder-only VLMs pair pretrained vision encoders and connectors with language decoders; training recipes also vary in their use of text-only, multimodal, and curated VQA data.PaLI introduced an encoder-decoder pattern, while LLaVA popularized the decoder-only form (Chen et al., 2023; Liu et al., 2023).Source: §2 · p. 2

Efficient resolution-agnostic image processing

  • Resolution-agnostic approaches include overlapping tiling, dynamic resolution, and native-resolution encoders; token compression targets redundancy, while jina-vlm learns compact representations during training.The cited training-free HERO approach instead prunes tokens at inference (Li et al., 2025a).Source: §2 · p. 2, Figure 1 · p. 3

Vision-language connectors

  • Connectors reduce visual-token counts through learned queries or cross-attention; jina-vlm instead applies local 2×2 attention pooling to preserve spatial structure while compressing tokens 4×.Its connector also concatenates intermediate and near-final vision features to combine fine-grained and high-level information.Source: §2 · p. 2

Small VLMs

  • Small-VLM work targets efficiency through quantization and resolution reduction, edge deployment, or systematic exploration of compact model designs.The cited work includes models as small as 256M parameters (Marafioti et al., 2025).Source: §2 · p. 2

Multilinguality and text-only performance

  • Lightweight VLMs may lose performance outside English or on text-only tasks; targeted multilingual training data and text-focused training strategies are among proposed responses.Cited multilingual instruction-tuning data spans 39 languages (Yue et al., 2025).Source: §2 · p. 2

Data mixture analysis

  • Prior data-mixture studies examine modality ratios, source diversity, caption quality, and curation; jina-vlm complements them by removing whole categories to diagnose marginal contributions and cross-domain transfer.This leave-one-out perspective is framed as relevant to fixed-compute settings.Source: §2 · pp. 2–3

3 MODEL ARCHITECTURE

jina-vlm processes arbitrary-resolution images with overlapping tiles and attention-based pooling that reduces visual sequence length while preserving spatial information.

3 MODEL ARCHITECTURE

  • The model follows an architecture of a pretrained vision encoder, a connector that projects visual features into language-model embedding space, and a decoder that generates text conditioned on visual tokens.jina-vlm’s image pipeline combines overlapping tiling with attention-based token pooling.Source: §3 · p. 3

3.1 VISION ENCODER

  • Overlapping tiles and a global thumbnail let the SigLIP2 encoder process arbitrary-resolution images, with a default limit of 12 tiles that can be increased at inference or continued training.Memory scales linearly with tile count.Source: §3.1 · pp. 3–4

3.2 VISION-LANGUAGE CONNECTOR

  • The connector concatenates two intermediate vision layers, then attention-pools local 2×2 patch neighborhoods and projects pooled features to the language-model embedding dimension.The selected layers are the 24th and 18th; the 4× token reduction preserves local structure.Source: §3.2 · p. 4

3.4 EFFICIENCY ANALYSIS

  • 2.3× lower overall FLOPs with attention pooling, including shared ViT cost; LLM prefill FLOPs fall 3.9× and KV-cache memory 4×.For the default 12-tile setup plus thumbnail, pooling reduces visual tokens from 9,477 to 2,366; savings apply to the LLM, not ViT processing.Source: §3.4 · p. 4, Table 1 · p. 5

4 TRAINING

Training updates the encoder, connector, and decoder in two stages, using multilingual data throughout. Alignment emphasizes cross-language semantic grounding, followed by instruction fine-tuning across VQA and reasoning tasks.

4 TRAINING

  • Both training stages update the vision encoder, connector, and language decoder without freezing any component.The combined training data includes English, Chinese, Arabic, German, Spanish, French, Italian, Japanese, Korean, Portuguese, Russian, Turkish, Vietnamese, Thai, Indonesian, Hindi, and Bengali.Source: §4 · p. 5

4.1 STAGE 1: ALIGNMENT TRAINING

  • Stage 1 uses multilingual caption data for cross-language semantic grounding and includes 15% text-only data to mitigate text-only task degradation.Caption data spans natural scenes, documents, infographics, and diagrams; the connector receives a higher learning rate and shorter warmup than the encoder and decoder.Source: §4.1 · p. 5

4.2 STAGE 2: INSTRUCTION FINE-TUNING

  • Stage 2 combines public VQA and instruction datasets with text-only instruction data across academic VQA, document understanding, OCR, mathematics, and reasoning.Sources include LLaVA OneVision, Cauldron, Cambrian, PangeaIns, FineVision, and Singh et al. (2024).Source: §4.2 · p. 5
  • Instruction fine-tuning runs 30K steps with single-source batches, then 30K steps with mixed-source batches.The authors report that single-source batches were more effective initially, likely because of the heterogeneous data mixture.Source: §4.2 · p. 5

5 EVALUATION

Evaluation finds jina-vlm strong on multilingual and general VQA benchmarks, while also showing uneven performance across reasoning, multi-image, and text-only tasks. Its multilingual results lead among 2B-scale VLMs, and its general VQA average is the highest reported in the comparison.

5 EVALUATION

  • The evaluation compares lightweight VLMs across six capability areas using VLMEvalKit and English prompts matching the training format.The areas include general VQA, multimodal comprehension, multi-image reasoning and hallucination control, mathematical reasoning, text-only performance, and multilingual understanding.Source: §5 · p. 6

5.1 GENERAL VQA TASKS

  • 72.3 is the highest average across eight VQA benchmarks, with particularly strong diagram interpretation and text extraction.The benchmarks cover diagrams, charts, scene text, documents, OCR, and diverse scenes.Source: §5.1 · p. 6

5.2 MULTIMODAL AND REAL-WORLD UNDERSTANDING

  • 68.2 on RealWorldQA is the best result, alongside scores of 67.4 on multimodal tasks and 61.9 on real-world tasks.Source: §5.2 · p. 6

5.4 MATHEMATICAL REASONING

  • jina-vlm performs comparably to InternVL3-2B and outperforms Qwen2-VL-2B on structured reasoning benchmarks.These benchmarks cover multidisciplinary comprehension, visual mathematics, and logical reasoning.Source: Table 6 · p. 7, §5.4 · p. 7

5.3 MULTI-IMAGE REASONING AND HALLUCINATION

  • 47.3 on multi-image tasks contrasts with the best POPE score of 90.3, indicating low hallucination rates.The authors describe the multi-image result as expected given limited multi-image training data.Source: §5.3 · p. 7, Table 5 · p. 7

5.5 TEXT-ONLY PERFORMANCE

  • 46.4 →30.3 on MMLU-Pro marks substantial degradation, while jina-vlm matches or exceeds Qwen3-1.7B on ARC-C and HellaSwag and retains most MMLU and GSM-8K performance.The authors suggest MMLU-Pro’s emphasis on extended multi-step reasoning conflicts with instruction-tuning toward concise visual responses.Source: §5.5 · p. 7, Table 7 · p. 7

5.6 MULTILINGUAL UNDERSTANDING

  • 78.8 on MMMB and 74.3 on Multilingual MMBench are the highest averages among 2B-scale VLMs.The setup uses English system prompts with target-language questions, answer options, and image text, testing cross-lingual visual understanding but not target-language instruction-following.Source: Table 8 · p. 8, §5.6 · p. 8

6 DATA MIXTURE ANALYSIS

The section tests how training-mixture categories affect downstream capabilities through leave-one-category-out ablations. It reports task-specific benefits and trade-offs, while treating the findings as preliminary diagnostics.

6 DATA MIXTURE ANALYSIS

  • The analysis addresses uncertainty about fine-grained mixture composition and cross-task transfer, which the authors consider especially relevant when small-scale training budgets are limited.They hypothesize that mixture composition matters alongside data quality.Source: §6 · p. 8

6.1 EXPERIMENTAL DESIGN

  • The study removes nine task, domain, modality, or language categories from the original mixture in separate variants and measures downstream effects across capabilities.The removed categories include VQA, OCR, real-world photos, math/reasoning, charts/diagrams, documents, code, text-only data, and non-English data.Source: §6 · p. 8, §6.1 · p. 9
  • Training data are grouped by modality, language, task, and domain, with language and modality classified automatically and task and domain assigned through manual inspection.The task and domain labels follow a high-level taxonomy.Source: §6.1 · pp. 8–9
  • Because all runs use 10% of full training, category removals can increase exposure to remaining data, and the results are correlational diagnostics rather than definitive conclusions.Training order and hyperparameters are held fixed; separating order from composition effects would require further controlled experiments.Source: §6.1 · p. 9, §6.2 · p. 9

6.2 RESULTS

  • −39.4% ChartQA for Xcharts and −4.6% MathVista for Xmath; removing VQA data chiefly harms VQA, while removing OCR data harms OCRBench but improves other tasks.The VQA effect is described as highly task-specific, with minimal impact elsewhere.Source: §6.2 · p. 9
  • Removing Xrealworld or Xdocs improves HellaSwag (+9.1%) and MMMU (+7.3%), while Xcode removal lowers HellaSwag (−10.6%).The authors suggest noise or gradient conflict may explain the improvements at this scale.Source: §6.2 · p. 9
  • Removing text-only data improves multimodal benchmarks but hurts HellaSwag; removing multilingual data improves most tasks but lowers MTVQA by −16.1%.The authors regard text-only and multilingual data as necessary for their respective capabilities under fixed compute budgets.Source: §6.2 · p. 9

7 CONCLUSION

The conclusion presents jina-vlm as a 2.4B model with strong multilingual and English VQA results among open 2B-scale VLMs. It also emphasizes limits in resolution handling and the preliminary status of the data-mixture findings.

7 CONCLUSION

  • jina-vlm is presented as state-of-the-art for multilingual VQA and leading for English VQA among open 2B-scale VLMs.Source: §7 · p. 9
  • Multi-tile processing adds overhead with image resolution and can fragment global spatial context, potentially impairing counting and precise spatial reasoning across tile boundaries.A global thumbnail partly mitigates this; the authors say native-resolution approaches may suit these tasks better.Source: §7 · p. 9
  • Table 9 summarizes ablation results; scores are accuracy percentages except OCRBench, and parentheses show relative differences from baseline.Bold indicates the best score per column, and some evaluation tasks are omitted due to insignificant variance.Source: §7 · p. 10, Table 9 · p. 10
  • The data-mixture analysis is preliminary evidence for future investigation because its ablations used 10% of full training.Source: §7 · p. 10

A.1 PSEUDOCODE FOR CREATING OVERLAPPING TILES

The pseudocode converts an image into a thumbnail plus a grid of overlapping tiles, resizing the image to fit the selected tiling.

A.1 PSEUDOCODE FOR CREATING OVERLAPPING TILES

  • The algorithm takes an image, base input size, patch size, maximum tile count, and overlap margins, and returns a thumbnail-plus-grid tile list.The default maximum is 12 tiles, with overlap margins of four patches on each side.Source: §A.1 · p. 16
  • It computes overlap sizes and stride, selects a tiling, resizes the image to fit that tiling with margins, then extracts overlapping tiles.Source: §A.1 · p. 16
  • The final list places a resized global thumbnail before the extracted grid tiles and returns the list with the row-and-column tiling.Source: §A.1 · p. 16

A.2 TRAINING SET EXAMPLES

The examples illustrate image and text inputs for VQA, OCR, spatial reasoning, and multilingual text-only tasks.

A.2 TRAINING SET EXAMPLES

A.3 INSTRUCTION FINE-TUNING STAGE DATASET COMPOSITION

The section presents the instruction fine-tuning dataset composition alongside reference tables for task types and domains.

A.3 INSTRUCTION FINE-TUNING STAGE DATASET COMPOSITION

Loading 2512.04032v4…