Source-linked AI summary
F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World
Ziyin Zhang, Zihan Liao, Hang Yu, Peng Di, Rui Wang
TL;DR
F2LLM-v2 addresses English-centric evaluation and limited transparency in embedding research by introducing multilingual models trained on public data and released with their training resources. Its eight model sizes and efficiency techniques support strong performance across benchmarks while offering deployment trade-offs across hardware constraints.
Problem
Embedding research remains English-centric, while many leading models have limited training transparency through closed-source or incomplete releases.
Method
F2LLM-v2 uses a 60-million-sample multilingual corpus, unified contrastive formats, and a two-stage training pipeline across eight model sizes from 80M to 14B parameters.
Results
The 14B model achieves state-of-the-art performance on 11 of 17 evaluated benchmarks, while smaller models provide efficient performance with limited degradation.
Takeaways & Limitations
Releasing the models, data, code, and intermediate checkpoints supports transparent embedding research and enables performance-cost-storage trade-offs for varied deployment settings.
Abstract
from arXiv · showhide
We present F2LLM-v2, a new family of general-purpose, multilingual embedding models in 8 distinct sizes ranging from 80M to 14B. Trained on a newly curated composite of 60 million publicly available high-quality data samples, F2LLM-v2 supports more than 200 languages, with a particular emphasis on previously underserved mid- and low-resource languages. By integrating a two-stage LLM-based embedding training pipeline with matryoshka learning, model pruning, and knowledge distillation techniques, we present models that are far more efficient than previous LLM-based embedding models while retaining competitive performances. Extensive evaluations confirm that F2LLM-v2-14B ranks first on 11 MTEB benchmarks, while the smaller models in the family also set a new state of the art for resource-constrained applications. To facilitate open-source embedding model research, we release all models, data, code, and intermediate checkpoints.
1 Introduction
F2LLM-v2 addresses English-centric evaluation and limited training transparency in frontier embedding research with a multilingual, open family of models.
- Motivation: Frontier embedding research remains limited by English-centric training and evaluation, which can produce models that perform well in English but offer less global utility.The passage identifies high-resource languages such as English and Chinese as receiving disproportionate attention.
- Motivation: Many leading embedding models do not disclose their training data or methodologies, hindering reproducibility and understanding of inclusive embedding systems.
- Contribution: F2LLM-v2 introduces a general-purpose multilingual embedding family designed to address language imbalance and training opacity.
- Contribution: The family combines 60 million publicly available samples across 282 natural languages and over 40 programming languages with 8 model sizes from 80M to 14B parameters.
- Contribution: The 14B model achieves state-of-the-art results on 11 MTEB benchmarks, while smaller models outperform similarly sized previous frontier models.
2 Related Work
Recent embedding research has shifted toward decoder-based models, but multilingual evaluation remains English-centric and training transparency is limited.
- Model Paradigms: Decoder-based embedding models have become dominant after earlier multilingual encoder-based models, benefiting from capabilities acquired during large-scale pre-training.
- Evaluation Bias: MTEB covers over 500 tasks and more than 250 languages, yet 35 of 131 MTEB-Multilingual tasks focus exclusively on English.
- Evaluation Bias: Language-specific benchmarks receive less attention than English or Multilingual benchmarks; the Polish MTEB benchmark previously had only one model with complete results.
- Transparency: Several top-performing multilingual embedding models are closed-source APIs or open-weight releases without training transparency.
- Transparency: KaLM-Embedding discloses training data but focuses on the Multilingual leaderboard and is not evaluated on language-specific benchmarks.
3 F2LLM-v2
F2LLM-v2 builds a multilingual, task-diverse embedding family from public data using unified contrastive formats, staged training, and efficiency-oriented compression methods.
- Training Data: The training corpus contains 60 million samples from 157 public sources, spanning 282 natural languages and over 40 programming languages.Curation prioritizes real-world data availability rather than benchmark-specific optimization and includes low-resource languages and code.
- Training Data: The dataset covers retrieval, clustering, sentiment analysis, and intent or domain classification tasks.
- Training Data: All data are consolidated into retrieval, clustering, and two-way classification formats for a unified contrastive learning objective.
- Model Family: F2LLM-v2 provides 8 dense decoder models from 80M to 14B parameters, with the three smallest models pruned from the 0.6B model.
- Two-stage Training: Stage 1 trains five 0.6B–14B models on 27 million samples from 7 retrieval datasets to establish a robust semantic foundation.
- Two-stage Training: Stage 2 uses 18 million samples and task-specific instructions to sharpen performance on classification, reranking, and paraphrase detection.
- Efficiency Methods: Pruning reduces the 0.6B model along hidden size, MLP intermediate size, and layer dimensions, while knowledge distillation mitigates performance losses in compact students.
- Efficiency Methods: Matryoshka Representation Learning is applied in both training stages with a minimum representation dimension of 8.
4 Experiments
F2LLM-v2 is evaluated across 17 MTEB benchmarks and 430 tasks, with results showing strong scalability from the 80M model to 14B. Ablations support the effectiveness of knowledge distillation and matryoshka representation learning for efficient deployment.
- Evaluation setup: F2LLM-v2 is evaluated on 17 MTEB benchmarks totaling 430 tasks across ten task types.The evaluation covers multilingual, domain-specific, and language-specific benchmarks.
- Main results: The 14B model achieves state-of-the-art performance on 11 of the 17 evaluated benchmarks.The results are presented with leaderboard rankings in Table 2.
- Main results: The 330M and 0.6B models consistently outperform Qwen3-Embedding and EmbeddingGemma on most language-specific benchmarks and the code benchmark.These smaller models provide a performance-efficiency tradeoff for edge deployment.
- Ablation studies: The ablation studies use a subset of 350 tasks selected solely by evaluation time to accelerate model iteration.
- Ablation studies: Knowledge distillation prevents the consistent performance drop observed when training pruned models without distillation across five model scales.The ablation averages results over 350 tasks.
- Ablation studies: Matryoshka representation learning concentrates the most critical semantic information in leading dimensions, with the steepest gains from 8 to 128 dimensions before performance plateaus.Embeddings are evaluated after truncation to dimensions ranging from 8 to the full size.
- Ablation studies: A 330M model with a full 896-dimensional embedding performs comparably to 8B and 14B models truncated to 32 dimensions.This supports dynamically balancing performance, inference cost, and storage cost.
5 Conclusion
F2LLM-v2 addresses language imbalance and training opacity in embedding research through a multilingual model family and open release of its training resources.
- F2LLM-v2 addresses language imbalance and training opacity in embedding model research.The paper frames this as a step toward democratizing high-performance embedding models.
- The release includes eight models, the complete training recipe, and intermediate checkpoints.These resources are intended to facilitate transparency in frontier embedding research.
A Training Data Details
The paper provides detailed tabulations of the natural-language, programming-language, and sample-count distributions in the F2LLM-v2 training data.
- Tables 5 and 6 report the natural-language distribution of the F2LLM-v2 training data.
- Table 7 reports the programming-language distribution in the F2LLM-v2 training data.
- Tables 8 and 9 report the number of samples in the collected training dataset.
B Details on MTEB Evaluation
MTEB organizes a broad evaluation suite into task categories and benchmarks grouped by language, domain, or both. F2LLM-v2 is evaluated across 17 benchmarks totaling 430 tasks.
- Task categories: MTEB evaluates embedding models across retrieval, reranking, classification, clustering, pair classification, STS, bitext mining, summarization, instruction reranking, and multilabel classification.These categories cover document retrieval, relevance ordering, semantic similarity, translation-pair identification, and classification variants.
- Task categories: Summarization measures semantic similarity between a generated summary and a reference summary.
- Task categories: Instruction reranking requires following a detailed natural-language instruction to determine relevance.
- Task categories: Multilabel classification allows each document to receive multiple labels.
- Benchmark organization: Benchmarks are grouped by language, domain, or combinations of both, including language-specific, domain-specific, and aggregated multilingual sets.This hierarchy enables fine-grained analysis of model capabilities.
- Evaluation scope: F2LLM-v2 is evaluated on 17 benchmarks totaling 430 unique tasks, including multilingual, code, medical, European, Scandinavian, and multiple language-specific benchmarks.