PaperlayerView exact revision on arXiv

Source-linked AI summary

jina-embeddings-v5-text: Task-Targeted Embedding Distillation

Mohammad Kalim Akram, Saba Sturua, Nastia Havriushenko, Quentin Herreros, Michael Günther, Maximilian Werk, Han Xiao

arXiv:2602.15547v2cs.CL

TL;DR

Small embedding models need to retain task performance while reducing model and embedding costs. This paper combines model distillation with task-specific contrastive training, and reports that the resulting compact multilingual models outperform simpler alternatives while remaining robust to truncation and binary quantization.

  • Problem

    Compact embedding models require training approaches that preserve high performance across semantic-similarity and retrieval tasks.

  • Method

    The method combines model distillation with task-specific contrastive and specialized training objectives.

  • Results

    The resulting models perform on-par with or better than recent comparable-size models and remain robust under truncation and binary quantization.

  • Takeaways & Limitations

    The approach yields compact multilingual embedding models with state-of-the-art performance among comparable models and publicly released weights.

  • Takeaways & Limitations

    The reported benchmark scores for one retrieval benchmark cover only public tasks because the authors lacked access to its private tasks.

Abstract

from arXiv · show

Text embedding models are widely used for semantic similarity tasks, including information retrieval, clustering, and classification. General-purpose models are typically trained with single- or multi-stage processes using contrastive loss functions. We introduce a novel training regimen that combines model distillation techniques with task-specific contrastive loss to produce compact, high-performance embedding models. Our findings suggest that this approach is more effective for training small models than purely contrastive or distillation-based training paradigms alone. Benchmark scores for the resulting models, jina-embeddings-v5-text-small and jina-embeddings-v5-text-nano, exceed or match the state-of-the-art for models of similar size. jina-embeddings-v5-text models additionally support long texts (up to 32k tokens) in many languages, and generate embeddings that remain robust under truncation and binary quantization. Model weights are publicly available, hopefully inspiring further advances in embedding model development.

1 Introduction

The paper introduces compact embedding models trained by combining model distillation with task-specific contrastive objectives. This approach is reported to outperform simpler alternatives and match or exceed comparable models on MTEB.

  • Text embeddings support retrieval, question-answering, recommendation, clustering, classification, and semantic-similarity applications.
  • Transformer encoders with pooling layers are the prevailing embedding-model architecture.
  • Knowledge distillation trains small models to mimic one or more teacher models, while recent embedding models commonly use contrastive learning.
  • The proposed training method combines model distillation with task-specific contrastive loss training.
  • The combined approach improves over naive contrastive and pure distillation-based training and performs on-par with or better than similarly sized MTEB models.
  • The resulting model weights are publicly released to foster further embedding-model research.

2 Related Work

Related work covers language-model distillation, embedding-specific distillation, and methods for jointly optimizing embedding models across tasks. Prior approaches include teacher alignment, projection-based distillation, multi-teacher training, and task-specific adaptation.

  • Related work spans general language-model distillation, embedding-model distillation, and contrastive multi-task learning.
  • Distillation aligns a compact student model with a teacher using specialized losses or by mimicking hidden-layer or self-attention behavior.
  • Embedding-distillation methods have aligned teacher similarity scores, projected teacher and student spaces, and combined embedding and score-based multi-teacher distillation.
  • Multi-task embedding training addresses task conflicts through combined losses, changing objectives, model soups, instructions, or LoRA adapters.

3 Model Architecture

The model uses a transformer with last-token pooling and task-specific LoRA adapters. Query and document prefixes support asymmetric retrieval, while Matryoshka training enables embedding truncation.

  • The transformer converts each text input into one embedding using the end-of-sequence token through last-token pooling.
  • LoRA adapters support retrieval, semantic similarity, clustering, and classification, with the appropriate adapter selected at inference time.
  • Asymmetric retrieval distinguishes queries and documents with “Query:” and “Document:” prefixes, while other tasks use “Document:”.
  • Matryoshka Representation Learning during training allows embeddings to be truncated for downstream efficiency.

4 Training

The models are trained in two main stages: distillation from a larger embedding model, followed by task-specific LoRA adapters with specialized objectives. Training also addresses multilingual data, long contexts, asymmetric retrieval, and embedding-space geometry.

  • 4 Training: The method uses EuroBERT-210m for jina-embeddings-v5-text-nano and Qwen3-0.6B-Base for jina-embeddings-v5-text-small.
  • 4.1 First-Stage: Embedding Distillation: Embedding distillation transfers knowledge from the much larger Qwen3-Embedding-4B teacher while using minimal instructions to support knowledge transfer.
  • 4.2 Task-Specific Adapter Training: The second stage freezes the base model and trains separate LoRA adapters for retrieval, semantic similarity, clustering, and classification.Separate adapters avoid conflicting optimization objectives across task categories.
  • 4.1.2 Loss Function: Student embeddings are projected into the teacher’s higher-dimensional space, where cosine similarity supplies the distillation signal.The projection is ψ(z)=Wz+b from R^n to R^m.
  • 4.1.3 Long Context Training: Long-context training uses curated synthetic and natural long texts, while modified positional-embedding settings support extended context windows.The training includes book chapters, long-form articles, and LLM-generated queries.
  • 4.2.1 Asymmetric Retrieval Adapter: Retrieval training combines InfoNCE with hard negatives, distillation, and a global orthogonal regularizer in a weighted objective.The regularizer encourages uniformly distributed embeddings, improving quantization robustness and approximate-nearest-neighbor retrieval efficiency.

5 Evaluation

The evaluation compares the proposed models with similarly sized multilingual embeddings across MTEB, retrieval benchmarks, training objectives, projection configurations, and quantization robustness. The models achieve strong size-adjusted performance, while results also identify task-specific trade-offs and design choices affecting retrieval and compressed embeddings.

  • Evaluation setup: The evaluation uses English and multilingual MTEB, additional retrieval benchmarks, ablations, and truncation-robustness tests, with comparable multilingual models as primary baselines.Metrics include nDCG@10 for most retrieval tasks, Spearman correlation for STS and summarization, V-measure for clustering, and accuracy or precision for classification and reranking.
  • MTEB performance: Both jina-embeddings-v5-text models achieve the highest average scores in their size category, while the larger Qwen3-4B teacher substantially outperforms them.Qwen3-4B has more than five times as many parameters as jina-embeddings-v5-text-small and sixteen times as many as jina-embeddings-v5-text-nano.
  • MTEB performance: Jina-embeddings-v5-text-nano exceeds all other sub-0.5B models in retrieval, reranking, and STS, although KaLM-mini-v2.5 has slightly higher overall average scores and Gemma-300M leads summarization.Voyage-4-nano narrowly exceeds jina-embeddings-v5-text-nano in retrieval, while KaLM-mini-v2.5 performs slightly better on clustering.
  • Retrieval benchmarks: Jina-embeddings-v5-text-small achieves the highest task-level average across the tested retrieval benchmarks and outperforms comparably sized Qwen3-0.6B on three of five benchmarks.Qwen3-0.6B scores higher on English MTEB and LongEmbed, suggesting an advantage on English and long-document retrieval.
  • Training objectives: Embedding distillation converges more slowly initially but reaches the highest final retrieval performance in both S2ORC and full-mixture training regimes.Score-based distillation and InfoNCE improve faster at first but plateau earlier, whereas embedding alignment provides a stronger sustained signal.
  • Projection configurations: Teacher projection without freezing fails, while the other projection configurations perform comparably; freezing the student projection accelerates convergence and leaving it unfrozen gives the best final results.The comparison uses average nDCG@10 on MTEB English v2 for S2ORC.
  • Training objectives: Adding task-specific losses is necessary beyond embedding distillation alone, which scores 63.16 on MTEB and 64.37 on RTEB in the tested combinations.The paper describes distillation as strong initialization for stage 1 and task-specific losses as critical in stage 2.
  • Quantization robustness: GOR loss modestly improves full-precision scores but provides greater quantization robustness, reducing degradation from -3.08 to -1.90 on MTEB and from -3.92 to -2.51 on RTEB.With GOR, full-precision scores improve from 64.21 to 64.50 on MTEB and from 66.16 to 66.45 on RTEB.

6 Conclusion

The paper introduces two compact multilingual embedding models and a training method combining distillation with task-specific objectives. The models achieve strong comparable-model performance while remaining robust to truncation and binary quantization.

  • jina-embeddings-v5-text-small and jina-embeddings-v5-text-nano are compact multilingual embedding models.
  • The proposed training method combines distillation-based and task-specific training.
  • The models achieve state-of-the-art performance among comparable multilingual embedding models.
  • The models remain robust under truncation and binary quantization, with only minimal performance degradation despite increased storage and computational efficiency.
  • The authors released the models publicly with integrations for Sentence Transformers and vLLM, plus quantized variants for llama.cpp.

A.1 Hyperparameters

The appendix documents hyperparameters for the different models and training stages, including the LoRA adapter settings used throughout training.

  • All LoRA adapters use rank 32 and alpha 32.
  • Table A1 reports hyperparameters for the different models and training stages.

A.2 English MTEB Benchmarks

The English MTEB appendix reports benchmark evaluations across retrieval, reranking, semantic similarity, pair classification, classification, and clustering tasks.

  • English MTEB evaluations use default metrics, combining leaderboard results with self-evaluated scores from the mteb package.
  • Retrieval and reranking: Table A2 reports English MTEB retrieval results using nDCG@10, while Table A3 reports reranking results using MAP@1000.
  • The listed English tasks include retrieval, similarity, pair classification, classification, and clustering benchmarks, with Avg denoting the average over all tasks.
  • Similarity and classification: Table A4 evaluates semantic textual similarity with Spearman correlation [%].
  • Similarity and classification: Table A5 uses Max Average Precision [%] for pair classification, Table A6 uses Accuracy [%] for classification, and Table A7 uses V-measure [%] for clustering.

A.3 Multilingual MTEB (MMTEB) Benchmarks

The multilingual MTEB appendix evaluates MMTEB performance across retrieval, reranking, semantic similarity, pair classification, bitext mining, classification, multi-label classification, clustering, and instruction reranking.

  • Multilingual evaluations use default metrics, combining MTEB leaderboard results with self-evaluated missing scores.
  • Retrieval and reranking: Table A8 reports multilingual retrieval results using nDCG@10%, and Table A9 reports reranking results using MAP@1000 [%].
  • Similarity and classification: Table A10 evaluates semantic textual similarity with Spearman correlation [%], while Table A11 evaluates pair classification with Max Average Precision [%].
  • Cross-lingual and classification tasks: Table A12 uses F1 Score [%] for bitext mining, Table A13 uses Accuracy [%] for classification, and Table A14 uses Accuracy [%] for multi-label classification.
  • Clustering and instruction reranking: Table A15 reports clustering with V-measure [%], and Table A16 reports instruction reranking with p-MRR [%].
  • The listed multilingual tasks span retrieval, reranking, similarity, classification, clustering, bitext mining, and instruction reranking, with Avg denoting the average over all tasks.

A.4 Other Retrieval Benchmark

This section reports retrieval performance across BeIR, LongEmbed, and RTEB (Public), with task-level scores summarized by averages and named datasets.

  • Table A17 reports retrieval performance on BeIR using nDCG@10 percentages.
  • The BeIR task list includes an average across all tasks and datasets such as ArguAna, CQADupstackRetrieval, and Natural Questions.
  • Table A18 reports retrieval performance on LongEmbed using nDCG@10 percentages.
  • The LongEmbed task list includes an average across tasks such as NarrativeQA Retrieval, Needle Retrieval, and Passkey Retrieval, with scores in nDCG@1.
  • Table A19 reports retrieval performance on RTEB (Public) using nDCG@10 percentages.
  • The RTEB (Public) task list includes an average across tasks spanning legal, finance, multilingual, coding, and biomedical retrieval.

A.5 Learning Rate Ablation

The learning-rate ablation compares InfoNCE, feature-based distillation, and score-based distillation under two rates. The objectives favor different settings: feature distillation performs better at 1×10−4, while InfoNCE and score distillation favor 1×10−5.

  • The ablation uses S2ORC data, a fixed trainable student projection, two GPUs, batch size 512, and maximum sequence length 512.
  • The experiments compare InfoNCE, feature-based distillation, and score-based distillation at learning rates 1×10−4 and 1×10−5.
  • Feature-based distillation performs significantly better at 1×10−4, while 1×10−5 converges more slowly and does not reach the same performance ceiling within the tested steps.
  • InfoNCE starts faster at 1×10−4 but eventually degrades or plateaus below the more consistent 1×10−5 run.
  • Score-based distillation is unstable at 1×10−4, dropping sharply after an initial peak, whereas 1×10−5 produces steady, sustained improvement.
  • The main results use 1×10−4 for feature-based distillation and 1×10−5 for InfoNCE and score-based distillation.
  • Figure 7 visualizes language-specific MMTEB performance using heatmaps based on each language’s mean and standard deviation.
Loading 2602.15547v2…