Source-linked AI summary
jina-embeddings-v3: Multilingual Embeddings With Task LoRA
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, Han Xiao
TL;DR
General-purpose embedding models can require task-specific fine-tuning, while large LLM-based embeddings are difficult to deploy efficiently. jina-embeddings-v3 combines a 570-million-parameter multilingual, long-context encoder with task-specific LoRA adapters and synthetic failure-case data. It reports strong English and multilingual benchmark performance, with adjustable output dimensions and a noted limitation in evaluating synthetic examples.
Problem
General-purpose embedding models often require task-specific fine-tuning, while LLM-based embeddings are typically much larger and offer marginal metric improvements over encoder-only models.
Method
jina-embeddings-v3 adapts XLM-RoBERTa with task-specific LoRA adapters, long-context techniques, Matryoshka Representation Learning, and synthetic data targeting retrieval failures.
Results
jina-embeddings-v3 outperforms OpenAI and Cohere embeddings on English tasks and multilingual-e5-large-instruct across multilingual tasks, while using 570 million parameters.
Takeaways & Limitations
The model is presented as a cost-efficient option for production and on-edge computing, with output dimensions configurable below 1024 through Matryoshka Representation Learning.
Takeaways & Limitations
Synthetic evaluation examples may be too closely aligned with training data, making the corresponding failure cases easier for the model to resolve.
Abstract
from arXiv · showhide
We introduce jina-embeddings-v3, a novel text embedding model with 570 million parameters, achieves state-of-the-art performance on multilingual data and long-context retrieval tasks, supporting context lengths of up to 8192 tokens. The model includes a set of task-specific Low-Rank Adaptation (LoRA) adapters to generate high-quality embeddings for query-document retrieval, clustering, classification, and text matching. Evaluation on the MTEB benchmark shows that jina-embeddings-v3 outperforms the latest proprietary embeddings from OpenAI and Cohere on English tasks, while achieving superior performance compared to multilingual-e5-large-instruct across all multilingual tasks. With a default output dimension of 1024, users can flexibly reduce the embedding dimensions to as low as 32 without compromising performance, enabled by Matryoshka Representation Learning.
1 Introduction
Text embeddings support retrieval, classification, clustering, and RAG, but general-purpose models often need task-specific fine-tuning. jina-embeddings-v3 addresses this with a compact multilingual model, task-specific LoRA, synthetic failure-case data, and long-context and dimensionality techniques.
- Text embeddings convert semantic relationships between documents into spatial relationships between vectors for retrieval, classification, clustering, and RAG.They serve as the primary retrieval technique in RAG systems.
- General-purpose embedding models often require task-specific fine-tuning and struggle with common failure cases.
- 570 million parameters enable jina-embeddings-v3 to target multilingual data, long-context retrieval, and multiple tasks.The model is described as outperforming several prior and proprietary embedding baselines in the supplied evaluation summary.
- Task-specific LoRA adapters generate embeddings optimized for different use cases and outperform prior instruction-based approaches.
- Synthetic training data addresses four identified retrieval failure types and improves robustness on edge cases.
- Matryoshka Representation Learning, instruction tuning, and long-context retrieval are integrated as core model advances.
2 Related Work
Related work improves embedding versatility through task-oriented fine-tuning, multilingual encoders, long-context mechanisms, compact representations, and LLM-derived knowledge. The paper positions task-specific LoRA as an alternative to complex instructions and large LLM backbones.
- Masked language modeling is not optimal for high-quality embeddings, motivating embedding-specific fine-tuning and model extensions.
- Multi-stage and multi-task weakly supervised contrastive training improves embedding versatility across diverse applications.
- AliBi and RoPE enable models such as jina-embeddings-v2 to handle sequences up to 8192 tokens.
- Matryoshka Representation Learning enables embedding truncation without compromising downstream performance by modifying the training loss.
- Multilingual encoder development progressed from mBERT through XLM and XLM-R, with later work adding multilingual fine-tuning, distillation, and longer-sequence support.
- LLM-based multilingual embeddings offer broad language support but are computationally inefficient, motivating knowledge distillation into compact encoders.
- Asymmetric retrieval benefits from distinct query and document prefixes, while later approaches use more complex relevance instructions.
- LoRA freezes original weights and typically uses less than 1% of their memory, but had not previously been explored as an alternative to instruction-based embedding methods.
3 Model Architecture
jina-embeddings-v3 adapts XLM-RoBERTa for long-text encoding, task-specific representations, and efficient operation. RoPE, FlashAttention 2, and dynamically selected LoRA adapters provide the main architectural changes.
- The architecture adapts XLM-RoBERTa to encode long sequences, support task-specific embeddings, and increase efficiency while retaining its tokenizer.
- LoRA adapters add less than 3% of total parameters, while FlashAttention 2, activation checkpointing, and DeepSpeed reduce computational and memory costs.
- RoPE replaces absolute positional embeddings to encode relative positional dependencies within self-attention for long-text processing.
- A rotary base frequency of 10,000 during training and 20,000 during inference improves long-text performance without degrading short-text performance.
- Task-specific LoRA adapters are dynamically selected from the input task type to provide different embedding behavior.
4 Training Method
The training procedure initializes an XLM-RoBERTa backbone, performs multilingual pre-training and embedding-task fine-tuning, and then trains task-specific adapters. The architecture supplies task descriptors, multiple adapters, and adjustable output dimensions for downstream use.
- Pre-Training: XLM-RoBERTa initialization accelerates pre-training compared with random initialization, despite the original MLM objective being imperfectly aligned with the modified training objectives.
- Training Stages: Training uses a three-stage paradigm comprising multilingual MLM pre-training, embedding-task fine-tuning, and task-specific adapter training.
- Fine-Tuning for Embedding Tasks: Pooling aggregates token representations into a single embedding vector, followed by fine-tuning on pairs of semantically related texts.
- Training Task-Specific Adapters: The model input combines text with a task type, which selects among five LoRA adapters supporting four task categories.
- Output Representation: 1024 is the maximum output dimension, while Matryoshka Representation Learning allows lower dimensions such as 16 or 32 for a space-efficiency and performance trade-off.
4.1 Pre-Training
Pre-training adapts the transformer for multilingual and long-context inputs before embedding-specific fine-tuning. Training progresses from masked language modeling on multilingual data to sequences up to 8192 tokens, with additional long-text training improving retrieval performance.
- The transformer is initially trained with whole-word masking, while the LoRA adapters and pooling layer remain excluded.
- 89-language CulturaX data supports multilingual pre-training, with each batch containing one language and languages rotated between batches.
- 100,000 steps use sequences truncated to 512 tokens, followed by 60,000 steps with sequences truncated to 8192 tokens and a reduced batch size.
- Long-document encoding initially lagged behind jina-embeddings-v2, but extended long-text training improved retrieval performance on tasks such as NarrativeQA.
4.2 Fine-Tuning for the Embedding Task
Embedding-task fine-tuning converts encoded sequences into single vectors and trains them on diverse text pairs with bidirectional contrastive learning. Filtering and staged sequence lengths are used to make training harder and support longer texts.
- A mean pooling layer aggregates output token vectors into one text representation, and fine-tuning uses a bidirectional InfoNCE loss on text pairs.
- The training corpus contains over one billion text pairs from more than 300 subdatasets spanning domains and languages.
- Filtering removes pairs where at least 80% of the shorter text’s words, with a minimum of four, are substrings of the longer text.
- Training moves from short text pairs to longer texts with larger sequence lengths and reduced batch sizes, using datasets containing sufficiently long texts.
4.3 Training Task-Specific Adapters
The paper trains independently selected LoRA adapters for distinct embedding tasks, replacing prompt-dependent behavior with task- and role-specific encoding. Retrieval uses jointly trained query and passage adapters, while other adapters target classification, matching, and separation.
- Adapter design: Task-specific LoRA adapters are trained independently with frozen base-model weights; retrieval query and passage adapters are trained jointly.
- Adapter design: Users select an adapter according to downstream task and input role, avoiding the need to learn task-specific prompts whose behavior can be harder to predict.
- Classification adapter: The classification adapter learns high similarity for same-class text pairs and low similarity against seven different-class negatives using an extended InfoNCE objective.
- Text matching adapter: The matching adapter treats query and corpus texts symmetrically for similarity tasks such as STS and duplicate detection, using CoSent training data and losses.
- Retrieval adapter: Asymmetric retrieval uses distinct prefixes and two jointly trained specialized adapters for queries and documents, with hard negatives emphasizing subtle relevance distinctions.
- Failure analysis: Training data analysis identifies retrieval failures involving misleading syntactic similarity, named-entity misinterpretation, and ineffective handling of polar questions.
F4. Preference for Low-Quality Documents:
The low-quality-document failure case motivates preference-based retrieval training that ranks higher-quality answers above sufficiently worse alternatives. The separation adapter instead supports clustering and reranking through label-based pair construction, with additional pair-training data improving performance.
- Failure case: Low-quality documents can be retrieved because embedding models focus on similarity and relevance rather than document quality.
- Training data: Synthetic examples target retrieval failures F1–F3, while preference datasets supply quality-ranked question-and-answer examples for F4.
- Training data: The highest-quality answer becomes positive, and answers at least 0.3 points lower in quality become negatives; additional negatives are sampled when necessary.
- Separation adapter: The separation adapter distinguishes same-group from different-group texts for clustering and separates relevant from irrelevant documents for reranking.
- Separation adapter: Because this training format has limited data, adding pair-training data to the mix improves performance.
5 Evaluation
The evaluation spans multilingual MTEB tasks, long-document retrieval, retrieval-failure cases, and ablations of adapters and embedding dimensions. Across these tests, jina-embeddings-v3 shows strong multilingual and English performance, competitive long-context retrieval, improved handling of several failure cases, and practical efficiency trade-offs.
- 5.1 Performance of Jina-XLM-RoBERTa: 76.05% on monolingual English tasks and 67.12% on multi-/cross-lingual tasks were achieved by the backbone after short embedding training, outperforming XLM-R and mBERT across all tasks.Training used 1000 steps on a single GPU node and approximately 2 million pairs, without adapter tuning.
- 5.2 Performance on MTEB: 65.52 average MTEB score was achieved across all tasks, while English Classification Accuracy reached 82.58 and Sentence Similarity reached 85.80.The overall average exceeded text-embedding-3-large, multilingual-e5-large-instruct, and Cohere-embed-multilingual-v3.0; the English task scores were the highest reported in their categories.
- 5.2 Performance on MTEB: 66.63 average across 56 English MTEB tasks was achieved by e5-mistral-7b-instruct, approximately 1.03% higher than jina-embeddings-v3, but with 4096-dimensional embeddings and 7.1 billion parameters.Jina-embeddings-v3 uses 570 million parameters and 1024-dimensional embeddings, with lower dimensions available through MRL with a manageable performance trade-off.
- 5.2 Performance on MTEB: Jina-embeddings-v3 outperformed multilingual-e5-large on all multilingual tasks except reranking and approached multilingual-e5-large-instruct.The multilingual results are weighted averages over a broad selection of multilingual and cross-lingual MTEB tasks.
- 5.3 Performance on LongEmbed MTEB: Jina-embeddings-v3 with the text-matching adapter achieved the highest average performance on six MTEB LongEmbed retrieval tasks.The results are attributed to RoPE-based positional embeddings outperforming the fixed positional embeddings of bge-m3 and the ALiBi-based approach of jina-embeddings-v2.
6 Conclusion
jina-embeddings-v3 combines task-specific adapter tuning with failure-aware synthetic data augmentation to achieve competitive performance across English and multilingual tasks while maintaining a reasonable parameter size. The authors identify low-resource language performance and failures caused by limited data as areas for future improvement.
- 6 Conclusion: Task-specific adapter tuning and failure-aware synthetic data augmentation support competitive performance across a wide range of tasks.The approach combines a robust backbone with task-specific adapters and synthetic data designed to address retrieval failures.
- 6 Conclusion: Extensive evaluations on English and multilingual datasets highlight strong performance while maintaining a reasonable parameter size.
- 6 Conclusion: The authors plan to improve performance on low-resource languages and analyze systematic failures caused by low data availability.
A Appendix
The appendix documents the model’s scaling comparison, training hyperparameters, ablations, and task-specific MTEB evaluation tables. It also defines model and adapter labels used across the reported multilingual, English, retrieval, similarity, classification, clustering, pair-classification, and reranking results.
- A Appendix: Figure 2 compares 100 embedding models after excluding models without size information and filtering identified outliers or trolling submissions.The figure reports jina-embeddings-v3’s performance relative to similarly sized models and its predecessor, jina-embeddings-v2.
- A Appendix: The appendix includes hyperparameter tables, an MLR ablation study, and performance tables spanning English and multilingual MTEB tasks.The task tables cover retrieval, STS, pair-classification, classification, clustering, and reranking.
- A Appendix: Separate appendix tables report multilingual retrieval, STS, pair-classification, classification, clustering, and reranking performance.
- A Appendix: The evaluation tables distinguish jina-embeddings-v3, the jina-embeddings-v2 bilingual suite, multilingual-e5-large, multilingual-e5-large-instruct, and other comparison models.A footnote states that ArguAna, CQADupstackRetrieval, and QuoraRetrieval use the text matching adapter.