PaperlayerView exact revision on arXiv

Source-linked AI summary

Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks

Yauhen Babakhin, Radek Osmulski, Ronay Ak, Gabriel Moreira, Mengyao Xu, Benedikt Schifferer, Bo Liu, Even Oldridge

arXiv:2511.07025v1cs.CLcs.IR

TL;DR

The paper addresses the lack of a fully open universal text embedding model for diverse multilingual and cross-lingual tasks. It develops llama-embed-nemotron-8b with open weights, diverse training data, and instruction-aware embeddings, achieving first place on MMTEB as of October 21, 2025. The authors report that synthetic data improves general classification performance but does not fully replace small amounts of high-quality in-domain data.

  • Problem

    Existing strong text embedding models often do not fully disclose their training data or methodologies, motivating a universal model for diverse multilingual and cross-lingual tasks.

  • Method

    The paper develops an open-weights instruction-aware embedding model using a bidirectional Llama-3.1-8B encoder, a 16M-pair training mix, synthetic data from diverse open-weight LLMs, and model merging.

  • Results

    llama-embed-nemotron-8b achieves Rank 1 on MMTEB, with 39,573 Borda votes across 131 tasks and 250+ languages.

  • Takeaways & Limitations

    The released model provides an instruction-aware text embedding tool for retrieval, classification, and STS, with planned release of its curated data mix.

  • Takeaways & Limitations

    Synthetic data improves general classification performance but is not a complete substitute for even small amounts of high-quality in-domain data.

Abstract

from arXiv · show

We introduce llama-embed-nemotron-8b, an open-weights text embedding model that achieves state-of-the-art performance on the Multilingual Massive Text Embedding Benchmark (MMTEB) leaderboard as of October 21, 2025. While recent models show strong performance, their training data or methodologies are often not fully disclosed. We aim to address this by developing a fully open-source model, publicly releasing its weights and detailed ablation studies, and planning to share the curated training datasets. Our model demonstrates superior performance across all major embedding tasks -- including retrieval, classification and semantic textual similarity (STS) -- and excels in challenging multilingual scenarios, such as low-resource languages and cross-lingual setups. This state-of-the-art performance is driven by a novel data mix of 16.1 million query-document pairs, split between 7.7 million samples from public datasets and 8.4 million synthetically generated examples from various open-weight LLMs. One of our key contributions is a detailed ablation study analyzing core design choices, including a comparison of contrastive loss implementations, an evaluation of synthetic data generation (SDG) strategies, and the impact of model merging. The llama-embed-nemotron-8b is an instruction-aware model, supporting user-defined instructions to enhance performance for specific use-cases. This combination of top-tier performance, broad applicability, and user-driven flexibility enables it to serve as a universal text embedding solution.

1 Introduction

The paper targets a universal text embedding model that remains robust across diverse tasks, domains, and languages. It introduces llama-embed-nemotron-8b and reports first place on MMTEB as of October 21, 2025.

  • Dense text embeddings support information retrieval and applications including web search, question answering, semantic textual similarity, and recommendation.
  • Text-only embedding models remain efficient for text-native data and OCR-converted documents despite growing interest in multimodal and omni-modal models.
  • The central challenge is building a universal text model that performs robustly across diverse tasks, domains, and multiple languages.
  • llama-embed-nemotron-8b ranks first on the comprehensive MMTEB leaderboard as of October 21, 2025.

2 Model

llama-embed-nemotron-8b is an instruction-tuned embedding model that adapts outputs to different tasks. It converts Llama-3.1-8B into a bidirectional encoder and uses task-specific architectures for retrieval, STS, and classification.

  • Instruction-aware model: The model uses task-specific instructional prefixes to generate specialized embeddings for retrieval, classification, and STS.Its shared encoder adapts embedding outputs according to the target task or use-case.
  • Encoder architecture: Replacing causal attention with bidirectional attention lets every token attend to every other input token.The model is initialized from Llama-3.1-8B and modifies attention masking throughout the transformer.
  • Embedding construction: Global average pooling over final-layer hidden states produces a single fixed-size embedding vector.The hidden-state dimension is 4096.
  • Task architectures: Retrieval uses a bi-encoder with cosine similarity, while STS and classification use a uni-encoder and task-specific embeddings.Queries and documents are independently encoded for retrieval; classification uses embeddings as classifier features.
  • Task architectures: The instruction-driven design allows one model to handle the varied MMTEB task types.

3 Training

Training uses InfoNCE contrastive learning with task-dependent triplet construction, followed by two-stage adaptation and fine-tuning. Six diverse checkpoints are merged into the final model, which performs best among the individual checkpoints without increasing inference time.

  • Contrastive objective: InfoNCE contrastive learning maximizes similarity for related items and minimizes it for unrelated items.The objective uses cosine similarity and a temperature hyperparameter.
  • Task-specific training: Training triplets vary by task: retrieval uses query-document pairs, classification uses correct and incorrect label names, and STS uses symmetric text pairs.STS negatives are mined from the dataset corpus, while retrieval uses mined hard negatives without extra negatives.
  • Training stages: The model is trained in two stages: retrieval pretraining on Web data, followed by fine-tuning across retrieval, classification, STS, and bitext mining.The stages comprise approximately 70% and 30% of the overall data mix, respectively.
  • Model merging: Model merging averages parameters from multiple runs with varied data mixes and hyperparameters.The approach is used to combine diverse checkpoints.
  • Model merging: The final model averages six diverse checkpoints and achieves the best evaluation results versus individual checkpoints without increasing inference time.

4 Datasets

The training data mix contains roughly 16 million query-document pairs from pretraining, public datasets, synthetic generation, translation, and hard-negative mining. Synthetic data covers multiple multilingual and cross-lingual task types and is generated through complementary strategies.

  • Data composition: The complete training mix contains approximately 16 million query-document pairs spanning pretraining and fine-tuning sources.The data mix is organized by pretraining, non-synthetic fine-tuning, and synthetic fine-tuning data.
  • Pretraining data: Pretraining contributes approximately 11.8M pairs from Nemotron-CC-v2 using existing questions and newly generated synthetic questions.The existing-question strategy yields 5.0M pairs, while generated questions contribute 6.8M pairs.
  • Fine-tuning data: Fine-tuning combines public datasets such as MIRACL, HotpotQA, MS MARCO, Natural Questions, and SQuAD with synthetic data.Most data overlapping evaluation sources was removed to preserve zero-shot evaluation.
  • Synthetic data generation: Synthetic data generation covers retrieval, classification, STS, and bitext mining, with emphasis on multilingual and cross-lingual diversity.
  • Synthetic data generation: Synthetic examples are created either from complete query-positive-negative triplets or by generating queries from seed documents and mining hard negatives.
  • Synthetic data generation: Existing datasets are translated into target languages using a diverse set of open-weight LLMs, whose synthetic-data quality is evaluated.The models include gpt-oss, Mixtral, and Llama instruction-tuned variants.
  • Hard-negative mining: Hard-negative mining selects top relevant negatives whose similarity is below 95% of the query-positive similarity.This threshold is intended to retain challenging negatives while filtering overly similar candidates.

5 Results

The model is evaluated on MMTEB’s broad multilingual and cross-lingual benchmark, using task-specific instructions where available. It ranks first among leading models under the official leaderboard’s Borda-count evaluation.

  • MMTEB covers 131 tasks across 9 task types and more than 250 high- and low-resource languages.
  • The evaluation uses task-specific instructions from MMTEB datasets and adapted instructions for tasks without defaults.
  • The comparison includes the other top-10 MMTEB leaderboard models, including Gemini, Qwen3-Embedding, and multilingual-e5-large-instruct.
  • 39,573 Borda votes place llama-embed-nemotron-8b at Rank 1, more than 200 votes ahead of Gemini Embedding and Qwen3-Embedding-8B.
  • The leaderboard’s Borda count aggregates each task’s relative model preferences, while Qwen3-Embedding-8B has a higher Mean (Task) score of 70.58 versus 69.46.

6 Ablation Study

The ablations evaluate contrastive-loss formulations, synthetic-data generation, the role of in-domain data, and model merging. Results favor a hard-negative-only loss, diverse synthetic-data mixtures, retained value from in-domain data, and merging complementary checkpoints.

  • 6.1 Contrastive Loss Formulations: A hard-negative-only InfoNCE formulation achieved 38,225 Borda votes and won the most individual task types, while alternative formulations performed similarly overall.The comparison fixed hyperparameters and tuned only each loss’s learning rate; adding in-batch or same-tower negatives provided minimal-to-no significant benefit.
  • 6.2 Choice of LLM for Synthetic Data Generation: Synthetic-data quality varied by domain and language, but mixing examples equally across six LLMs achieved the best results.The findings indicate that synthetic-data diversity mattered more than using the largest single generator.
  • 6.2 Choice of LLM for Synthetic Data Generation: 100k synthetic classification examples improved performance by +464 Borda votes and +0.94 Mean points over a baseline without synthetic classification data.The comparison was between 37,812 and 37,348 Borda votes, and 62.89 and 61.95 Mean points, respectively.
  • 6.3 Impact of Synthetic vs. In-Domain Data: In-domain data outperformed the synthetic mix across all five evaluated classification tasks, with 1.5k TweetTopicSingleClassification samples surpassing about 1M synthetic samples.The in-domain dataset was constructed only for this ablation and was not used in the final leaderboard model.
  • 6.3 Impact of Synthetic vs. In-Domain Data: Synthetic data improved general classification performance but was not a complete substitute for small amounts of high-quality in-domain data.The ablation identifies a practical boundary for relying on synthetic classification examples alone.
  • 6.4 Model Merging: The final model averaged six diverse checkpoints, improving over the best individual model by +119 Borda votes and +0.84 Mean points.The checkpoints varied in data mixes and hyperparameters, and the merged model achieved the strongest overall results by combining complementary task specializations.

7 Conclusion

The paper presents llama-embed-nemotron-8b as an open-weights universal text embedding model with state-of-the-art multilingual performance and broad task applicability. Releasing its weights, and planning to release its curated data mix, supports further research and development.

  • #1 on the MMTEB leaderboard, the model outperforms other top models across 131 tasks and 250+ languages by Borda count.The ranking is reported as of October 21, 2025.
  • The reported performance combines a Llama-3.1-8B bidirectional encoder, a 16M-pair training mix, and a robust training methodology.
  • A diverse mix of open-weight LLMs for synthetic data generation yields more robust results than any single LLM.The ablation emphasizes data diversity as an important design choice.
  • Released model weights provide an instruction-aware tool for retrieval, classification, and STS, while the curated data mix is planned for future release.

A Implementation Details

The implementation details summarize the training resources and hyperparameter documentation for llama-embed-nemotron-8b.

  • Pretraining took 25.0 hours and fine-tuning took 21.5 hours on 64 NVIDIA A100 80GB GPUs.The hyperparameters for both stages are summarized in Table 8.
  • Table 8 lists the main hyperparameters used during llama-embed-nemotron-8b training.

B Fine-Tuning Data Mix for llama-embed-nemotron-8b

The fine-tuning data mix contains multilingual and cross-lingual data from public and synthetic sources, totaling 4.3 million samples.

  • 4.3 million samples make up the curated fine-tuning dataset, sourced from diverse multilingual and cross-lingual corpora.
  • 2.7 million non-synthetic and 1.6 million synthetic samples compose the fine-tuning data mix.The component datasets and sample counts are detailed in Table 9.
  • Table 9 lists the component datasets and sample counts in the llama-embed-nemotron-8b fine-tuning mix.
Loading 2511.07025v1…