PaperlayerView exact revision on arXiv

Source-linked AI summary

jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval

Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, Han Xiao

arXiv:2506.18902v3cs.AIcs.CLcs.IR

TL;DR

jina-embeddings-v4 addresses the challenge of building one embedding model for multilingual text, images, code, and visually rich documents. It unifies modalities and output formats with task-specific LoRA adapters, and reports strong performance across retrieval settings, especially visually rich document retrieval, while introducing Jina-VDR for evaluation.

  • Problem

    The paper addresses the challenge of supporting multilingual text, images, coding languages, and visually rich data within a multimodal embedding model.

  • Method

    The model uses unified multimodal representations, single- and multi-vector outputs, and task-specific LoRA adapters for retrieval, similarity, and code tasks.

  • Results

    The model achieves strong performance across text, image, cross-modal, and code retrieval, with particularly greater effectiveness from multi-vector representations on visually rich document retrieval.

  • Takeaways & Limitations

    The unified model supports semantic text, text-to-image, visually rich document, and code search, while Jina-VDR expands evaluation to multilingual visual retrieval.

  • Takeaways & Limitations

    Future work will further enhance multilingual capabilities and explore smaller, more efficient variants.

Abstract

from arXiv · show

We introduce jina-embeddings-v4, a 3.8 billion parameter multimodal embedding model that unifies text and image representations through a novel architecture supporting both single-vector and multi-vector embeddings in the late interaction style. The model incorporates task-specific Low-Rank Adaptation (LoRA) adapters to optimize performance across diverse retrieval scenarios, including query-document retrieval, semantic text similarity, and code search. Comprehensive evaluations demonstrate that jina-embeddings-v4 achieves state-of-the-art performance on both single-modal and cross-modal retrieval tasks, with particular strength in processing visually rich content such as tables, charts, diagrams, and mixed-media formats. To facilitate evaluation of this capability, we also introduce Jina-VDR, a novel benchmark specifically designed for visually rich image retrieval.

1 Introduction

jina-embeddings-v4 is introduced as a unified multimodal embedding model for text, images, and visually rich documents. The work combines multi-task embedding outputs with task-specific adapters and presents Jina-VDR to broaden visual retrieval evaluation.

  • Model and scope: jina-embeddings-v4 processes text and image data into semantic embeddings for broad applications, including visually rich mixed-media documents.The supported visual materials include tables, charts, diagrams, and other mixed media.
  • Task specialization: Task-specific LoRA extensions support information retrieval, semantic text similarity, programming-language embeddings, technical question answering, and code retrieval.The adapters are intended to enhance support for specific domains and task types.
  • Model and scope: The model jointly represents texts and images as both single-vector and multi-vector embeddings through a unified multi-task learning paradigm.
  • Evaluation: jina-embeddings-v4 shows particularly strong progress on visually rich images and outperforms other multimodal models by a significant margin on this material.The paper emphasizes uses beyond the question-answering focus of the existing ViDoRe benchmark.
  • Evaluation: Jina-VDR is a multilingual, multi-domain screenshot-retrieval benchmark covering multilingual retrieval, more query types, and diverse materials such as maps, diagrams, and advertisements.

2 Background

Embedding models map digitally encoded objects into a semantic vector space, but existing systems often specialize by modality, input type, task, or output format. jina-embeddings-v4 instead uses one base model across modalities and tasks, with practical savings from avoiding multiple deployments.

  • Embedding-model foundations: Embedding models transform digitally encoded objects into high-dimensional vectors where shared semantic features correspond to greater proximity.Similarity is typically measured with cosine similarity.
  • Embedding-model specialization: Existing embedding systems commonly specialize in particular modalities or inputs, including text, images, code, legal documents, and visually rich data.
  • Task and output specialization: Embedding models also vary by task and representation, including information retrieval, clustering, classification, single-vector outputs, and multi-vector late interaction.Late interaction is generally more precise for retrieval but requires substantially more storage and computation.
  • Unified approach: jina-embeddings-v4 provides competitive text, image, and cross-modal performance with both single-vector and multi-vector outputs from one base model.LoRA extensions target information retrieval, multimodal semantic similarity, and computer code retrieval.
  • Unified approach: The single-model approach entails significant practical savings compared with deploying multiple AI models for different tasks and modalities.

3 Related Work

Related work establishes transformer embedding methods, truncatable representations, multimodal encoders, and late-interaction retrieval. The paper positions jina-embeddings-v4 as a multilingual model spanning modalities, retrieval formats, and task types beyond narrower prior systems.

  • Embedding methods: Transformer-based semantic embedding architectures and multi-stage contrastive training provide established foundations for this work.Techniques for supporting longer texts are also identified as relevant.
  • Embedding methods: Matryoshka Representation Learning motivates training embedding models whose vectors can be truncated while retaining performance.
  • Multimodal models: Vision-language models can outperform dual-encoder architectures on relevant multimodal tasks and may suffer less from modality gaps.
  • Multimodal models: jina-embeddings-v4 differs from cited alternatives by using multilingual data, supporting single- and multi-vector retrieval, and avoiding task-specific instructions.
  • Late interaction: Late-interaction similarity sums each query token’s maximum cosine similarity to any document token rather than using one simple cosine score.This asymmetric comparison operates over two sequences of token embeddings.
  • Visual retrieval benchmarks: ViDoRe evaluates English and French question answering over charts, tables, and PDF pages, while the paper broadens visual-document benchmarking beyond that scope.

4 Model Architecture

jina-embeddings-v4 uses a unified vision-language pathway that jointly processes text and image tokens, then offers dense or late-interaction outputs. Task-specific LoRA adapters provide specialized retrieval behavior while leaving the backbone frozen.

  • Unified multimodal pathway: Text and images share a processing pathway in which image tokens and text are jointly processed by the language-model decoder with contextual attention.The architecture is built on Qwen2.5-VL-3B-Instruct and is described as eliminating the modality gap of dual encoders.
  • Dual output modes: The model supports single-vector embeddings and ColBERT-style multi-vector embeddings for late-interaction retrieval.
  • Dual output modes: Single-vector outputs have 2048 dimensions and can be truncated to 128 with minimal loss of precision through Matryoshka Representation Learning.
  • Dual output modes: Multi-vector outputs contain one 128-dimensional vector per input token, including image tokens, preserving token-level contextual representations.
  • Output construction: Mean pooling produces single-vector outputs, while an additional projection layer produces multi-vector outputs from the base model.
  • Task-specific adapters: Three LoRA adapters target asymmetric query-document retrieval, semantic similarity and symmetric retrieval, and code retrieval.Each adapter has 60M parameters, can be selected at inference time, and supports image and text encoding.

5 Training Method

Training combines a frozen Qwen2.5-VL-3B-Instruct backbone with jointly trained single- and multi-vector objectives, followed by independently trained task-specific LoRA adapters. The method uses text and multimodal pairs, contrastive losses, hard negatives, and truncatable single-vector representations.

  • Training setup: Training initializes a Qwen2.5-VL-3B-Instruct backbone and updates only randomly initialized projection and LoRA adapter parameters.The backbone weights remain unchanged throughout training.
  • Joint pair training: A first phase jointly trains one adapter on text and text-image pairs for single-vector and multi-vector similarity using contrastive InfoNCE.The model samples separate text-pair and multimodal batches and constructs similarity matrices for both embedding types.
  • Task-specific adaptation: The resulting adapter is duplicated into three independently trained task-specific adapters for retrieval, semantic similarity, and code retrieval.Each adapter receives task-specific text and text-image triplets, while Matryoshka loss supports truncatable single-vector embeddings.
  • Asymmetric retrieval: Hard negatives extend contrastive training with closely related but incorrect documents, using an LNCE+ objective for retrieval.Multimodal hard negatives come from existing and mined datasets.
  • Semantic similarity: Semantic similarity training uses ground-truth similarity pairs with CoSENT3, supplemented by InfoNCE training for pairs without known scores.The loss compares pairs whose ground-truth similarities are ordered.
  • Code retrieval: Code adapters target natural-language-to-code retrieval, code-to-code similarity, and technical question answering without affecting the model’s vision portion.Code training uses specialized sources and a triplet-based method.

Rich Document Retrieval Benchmark

Jina-VDR is a multilingual, multi-domain visual-document retrieval benchmark extending ViDoRe with 30 additional tests. It covers diverse document types, query forms, languages, and both human-annotated and synthetic data.

  • Benchmark scope: Jina-VDR evaluates retrieval over rendered documents combining textual and visual elements such as charts, tables, and running text.The benchmark targets integration of textual and visual document understanding.
  • Dataset diversity: The collection spans legal texts, historic documents, marketing materials, charts, tables, manuals, printed text, and maps.These datasets cover multiple domains and material types.
  • Query diversity: Queries include questions, facts, descriptions, and non-question formats such as GitHub descriptions paired with rendered Markdown images.Map images are also paired with accompanying textual descriptions.
  • Benchmark construction: 30 additional tests extend ViDoRe with real-world and synthetic data adapted for retrieval.The suite includes repurposed datasets, manually annotated datasets, and generated synthetic data.
  • Annotation and filtering: The benchmark combines human-annotated resources with LLM-generated queries and LLM-based filtering for relevance and real-world querying.Synthetic generation helps broaden coverage where human annotation is unavailable.
  • Multilingual coverage: Jina-VDR provides multilingual coverage, with some datasets spanning up to 20 languages.The collection also includes bilingual examples for assessing cross-lingual retrieval.

7 Evaluation

The evaluation spans multilingual text retrieval, long-document retrieval, semantic similarity, text-to-image search, visually rich document retrieval, and code retrieval. jina-embeddings-v4 generally improves on jina-embeddings-v3, remains broadly comparable with state-of-the-art models, and is especially strong on visually rich documents.

  • Evaluation scope: The evaluation covers multilingual text, long-document, semantic similarity, text-to-image, visually rich document, and code retrieval benchmarks.MTEB and MMTEB assess text retrieval; LongEmbed assesses long documents; STS assesses semantic similarity; CLIP Benchmark assesses text-to-image search; ViDoRe and Jina-VDR assess visually rich documents; MTEB-CoIR assesses code retrieval.
  • Text retrieval: jina-embeddings-v4 generally outperforms jina-embeddings-v3 and is broadly comparable with state-of-the-art performance on MTEB and MMTEB.The MTEB benchmarks are English-only, while MMTEB is multilingual.
  • Long-document retrieval: Long-document performance significantly outpaces competing models except the voyage-3 series and improves dramatically over jina-embeddings-v3.These results are reported for the LongEmbed benchmark.
  • Semantic similarity: The model is competitive with state-of-the-art semantic similarity systems and best-in-class for English similarity tasks.The evaluation uses MTEB STS and MMTEB STS benchmarks.
  • Text-to-image retrieval: jina-embeddings-v4 has a higher average CLIP Benchmark score than jina-clip-v2 and nllb-siglip-large, while nllb-siglip-large performs higher on Crossmodal3600.The Crossmodal3600 difference is attributed to low-resource languages unsupported by the model’s Qwen2.5-VL-3B-Instruct backbone.
  • Visually rich document retrieval: jina-embeddings-v4 achieves current state-of-the-art performance on Jina-VDR in both single-vector and multi-vector modes.The analysis states that multi-vector late-interaction matching remains more precise than single-vector matching for Jina-VDR.
  • Code retrieval: On MTEB-CoIR, jina-embeddings-v4 is competitive with general-purpose state-of-the-art embedding models, but voyage-code performs somewhat better.MTEB-CoIR contains 10 tasks spanning text-to-code, code-to-text, code-to-code, and hybrid code retrieval.

8 Analysis of the Embedding Space

The analysis examines modality gaps and embedding-space geometry using similarity distributions and cross-modal alignment scores. jina-embeddings-v4 substantially reduces the modality gap and separates positive from negative image-text matches more clearly, although alignment is weaker on CIFAR-100.

  • Cross-Modal Alignment: Figure 2 compares cosine-similarity distributions for paired image-text and text-text embeddings across OpenAI CLIP, jina-clip-v2, and jina-embeddings-v4.The distributions are drawn from Flickr8K14.
  • Cross-Modal Alignment: jina-embeddings-v4 dramatically reduces the modality gap through its crossmodel encoder, which biases modalities toward a shared embedding-space region.The cited analysis contrasts this with separate encoders in CLIP-style architectures.
  • Cross-Modal Alignment: Table 4 compares cross-modal alignment scores for jina-embeddings-v4 and OpenAI CLIP on random samples from Flickr30K, MSCOCO, and CIFAR-100.The score is defined as the average cosine similarity of matching image-text pairs.
  • Cross-Modal Alignment: The results confirm that jina-embeddings-v4 produces a better-aligned cross-modal embedding space than CLIP-style models.This conclusion follows the cross-modal alignment comparison.
  • Cross-Modal Alignment: Alignment is much poorer on CIFAR-100 than on MSCOCO and Flickr30K because CIFAR-100 labels are less descriptive.CIFAR-100 is a classification dataset, whereas the other datasets provide more informative descriptive texts.
  • Cone Effect: Figure 3 shows small correct-versus-incorrect similarity differences for OpenAI CLIP, larger differences for jina-clip-v2, and distinctly separated peaks for jina-embeddings-v4.The jina-embeddings-v4 distributions also show greater spread and overlap between image and text embeddings.

9 Conclusion

The paper concludes that jina-embeddings-v4 is a state-of-the-art multimodal and multilingual embedding model spanning text, image, visually rich document, and code-search tasks. It reports especially strong multi-vector performance for visually rich document retrieval while reducing structural gaps between modalities.

  • 9 Conclusion: jina-embeddings-v4 is designed for semantic text retrieval, text-to-image retrieval, text-to-visually-rich document retrieval, and code search.The model supports both multimodal and multilingual embedding tasks.
  • 9 Conclusion: The model performs strongly with single-vector representations and is more effective with multi-vector representations, particularly for visually rich document retrieval.Its representations are aligned across modalities in a shared semantic space.
  • 9 Conclusion: The authors plan to enhance multilingual capabilities and develop smaller, more efficient variants.These are identified as future-work directions.

A Appendix

The appendix documents the dataset collection and reports retrieval, cross-modal, semantic-similarity, and code-search benchmark evaluations. It also defines the model abbreviations and task-level averaging conventions used throughout these results.

  • Datasets: The dataset collection includes Finance Charts and mixed maps, with query/document counts and supported languages listed for each dataset.Finance Charts contains 6278 queries and 10000 documents across nine languages, while the mixed-maps dataset contains 443 queries and 451 documents in English.
  • Retrieval benchmarks: The appendix reports visually rich retrieval results through JinaVDR, ViDoRe, Wikimedia Commons, GitHub README, and additional benchmark tables.The listed evaluations cover visually rich retrieval and specialized retrieval settings, including Wikimedia Commons and GitHub README retrieval.
  • Models: The evaluation compares BM25 with OCR, jina-embeddings-v3 with OCR, jina-clip-v2, ColPALI-v1.2, DSE-Qwen2-2b-MRL-V1, and jina-embeddings-v4 variants.The jina-embeddings-v4 variants are distinguished as single-vector and multi-vector models.
  • Metrics: Task averages are defined using mean nDCG@10%, mean Recall@5%, or mean Spearman correlation % over the specified task sets.The appendix provides separate task lists for retrieval, cross-modal retrieval, semantic similarity, and code-search evaluations.
  • Cross-modal retrieval: Cross-modal evaluation includes text-to-image Recall@5% results on CLIP, xtd10, and crossmodal3600 across supported languages.Separate tables cover CLIP, multilingual xtd10, and multilingual crossmodal3600 benchmarks.
  • Text and code evaluation: The appendix also evaluates MTEB and MMTEB retrieval, LongEmbed, MTEB v2 and MMTEB v2 semantic similarity, and MTEB-CoIR code information retrieval.The reported metrics are nDCG@10% for retrieval and code information retrieval, and Spearman correlation % for semantic textual similarity.
Loading 2506.18902v3…