PaperlayerView exact revision on arXiv

Source-linked AI summary

Gemini Embedding: Generalizable Embeddings from Gemini

Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, Feng Han, Andreas Doumanoglou, Nithi Gupta, Fedor Moiseev, Cathy Yip, Aashi Jain, Simon Baumgartner, Shahrokh Shahi, Frank Palma Gomez, Sandeep Mariserla, Min Choi, Parashar Shah, Sonam Goenka, Ke Chen, Ye Xia, Koert Chen, Sai Meher Karthik Duddu, Yichang Chen, Trevor Walker, Wenlei Zhou, Rakesh Ghiya, Zach Gleicher, Karan Gill, Zhe Dong, Mojtaba Seyedhosseini, Yunhsuan Sung, Raphael Hoffmann, Tom Duerig

arXiv:2503.07891v1cs.CLcs.AI

TL;DR

General-purpose embedding models must support diverse languages, domains, and tasks. Gemini Embedding uses Gemini-derived capabilities and training data, achieving state-of-the-art results across multilingual, English, and code evaluations.

  • Problem

    Existing research seeks general-purpose embeddings that perform well across diverse languages, domains, modalities, and downstream tasks.

  • Method

    Gemini Embedding adapts Gemini with curated multilingual and code-task data, synthetic generation, filtering, hard-negative mining, contrastive learning, and task prompts.

  • Results

    Gemini Embedding achieves state-of-the-art performance across multilingual, English, and code evaluations, including MTEB(Multilingual) Task Mean 68.32 and Borda rank #1.

  • Takeaways & Limitations

    The unified model provides versatile representations for classification, similarity search, clustering, ranking, and retrieval across many languages and task types.

  • Takeaways & Limitations

    The report leaves extending Gemini Embedding to image, video, and audio modalities as future work requiring suitable multimodal training data.

Abstract

from arXiv · show

In this report, we introduce Gemini Embedding, a state-of-the-art embedding model leveraging the power of Gemini, Google's most capable large language model. Capitalizing on Gemini's inherent multilingual and code understanding capabilities, Gemini Embedding produces highly generalizable embeddings for text spanning numerous languages and textual modalities. The representations generated by Gemini Embedding can be precomputed and applied to a variety of downstream tasks including classification, similarity, clustering, ranking, and retrieval. Evaluated on the Massive Multilingual Text Embedding Benchmark (MMTEB), which includes over one hundred tasks across 250+ languages, Gemini Embedding substantially outperforms prior state-of-the-art models, demonstrating considerable improvements in embedding quality. Achieving state-of-the-art performance across MMTEB's multilingual, English, and code benchmarks, our unified model demonstrates strong capabilities across a broad selection of tasks and surpasses specialized domain-specific models.

1. Introduction

The paper introduces Gemini Embedding, a Gemini-initialized model trained on heterogeneous embedding tasks with Gemini-assisted data curation. It evaluates the model across over 100 tasks and 250+ languages, reporting state-of-the-art results on multilingual and other benchmarks.

  • 1. Introduction: Embedding models represent text as dense vectors that place semantically similar texts near one another in embedding space.They are used to capture semantic information across domains and modalities.
  • 1. Introduction: Large language models improve embedding development by refining training data through hard-negative mining and synthetic-data generation, and by providing strong initialization.These approaches can distill LLM knowledge into smaller embedding models and yield superior performance despite higher computational demands.
  • 1. Introduction: Gemini Embedding is initialized from Gemini and trained on a comprehensive suite of embedding tasks using a heterogeneous, high-quality dataset.Gemini assists with filtering low-quality examples and identifying relevant positive and negative retrieval passages.
  • 1. Introduction: 68.32 mean score on MTEB(Multilingual) leads multilingual-e5-large-instruct by +5.09, while its 59.64 task-type mean leads gte-Qwen2-7B-instruct by +3.64.Gemini Embedding also achieves the highest public-leaderboard Borda rank and establishes new state-of-the-art results on multiple other benchmarks, including XOR-Retrieve (Asai et al., 2021).

2. Related Work

Prior work developed general-purpose text embeddings, LLM-based embedding encoders, and synthetic-query methods for downstream NLP tasks. Recent LLM-initialized models generally outperform earlier BERT- or T5-based approaches, while raising generalization concerns.

3. Gemini Embedding

Gemini Embedding is initialized from Gemini and refined into holistic representations for retrieval, clustering, classification, and ranking. Its design combines bidirectional transformer encoding, contrastive training, multi-dimensional embeddings, and a two-stage training pipeline with model averaging.

  • Architecture: The model uses a bidirectional transformer initialized from Gemini, followed by a pooler that converts token embeddings into a single input representation.This initialization leverages knowledge already encoded in Gemini’s parameters.
  • Training objective: Gemini Embedding is trained with noise-contrastive estimation using in-batch negatives, task strings, and masking tailored to tasks such as classification.The method omits same-tower negatives because potential false negatives were found to reduce performance for most tasks.
  • Embedding dimensions: The model supports 3,072-dimensional embeddings and uses multi-resolution loss to train overlapping 768- and 1,536-dimensional subspaces.This allows one model to support multiple embedding dimensions.
  • Training recipe: Training proceeds through noisy-pair pre-finetuning, task-specific fine-tuning with hard negatives, and model-soup averaging of checkpoints to improve generalization.Pre-finetuning uses large batches without hard negatives, while fine-tuning uses smaller dataset-homogeneous batches and searches training configurations.

4. Datasets

The training mixture combines diverse multilingual embedding and code-retrieval tasks, with Gemini improving data through synthetic generation, filtering, and hard-negative mining. Pre-finetuning uses billion-scale web title–passage pairs, while fine-tuning targets task, language, and coding diversity.

  • Training Data: Gemini improves a diverse multilingual and code-retrieval training mixture through synthetic data generation, data filtering, and hard-negative mining.
  • Pre-finetuning: Pre-finetuning exposes the model to billion-scale web title–passage pairs, a simple approach found effective even with LLM initialization.
  • Fine-tuning: Fine-tuning uses mixtures designed for task diversity, language diversity, and coding capability, while excluding many in-domain MTEB datasets to reduce leakage-driven gains.The task-diversity mixture includes academic datasets used by Gecko and synthetic datasets introduced in Section 4.2.
  • Synthetic Data Generation: Synthetic datasets expand training for retrieval and classification, using Gemini-enhanced adaptations of FRet and SWIM-IR with few-shot prompting.
  • Data Filtering: Gemini filters human-annotated retrieval data by assessing examples with few-shot prompts and removing low-quality cases with incorrect targets.
  • Hard Negative Mining: Hard negatives are mined by retrieving nearest neighbors with an initial embedding model, scoring them with Gemini using two prompting strategies, and selecting the lowest-scoring neighbors.The scores from graded classification and query likelihood prompts are combined with Reciprocal Rank Fusion.

5. Evaluation

Gemini Embedding achieves state-of-the-art performance across multilingual, English, and code embedding benchmarks, while also advancing cross-lingual retrieval. Evaluations span 164 tasks across 250+ languages and multiple task types, including MMTEB (Enevoldsen et al., 2025), XTREME-UP (Ruder et al., 2023), and XOR-Retrieve (Asai et al., 2021).

  • MTEB(Multilingual) leaderboard: 68.32 Task Mean, 59.64 Task Type Mean, and Borda rank #1 establish Gemini Embedding as the top MTEB(Multilingual) model.These metrics summarize aggregate performance across tasks and task types on the March 10, 2025 leaderboard.
  • Overall benchmark results: Gemini Embedding ranks #1 on MTEB(Multilingual), MTEB(Eng, v2), and MTEB(Code), while advancing state-of-the-art general-purpose cross-lingual embeddings.The unified model uses a shared embedding space across multilingual, English, code, and cross-lingual evaluations.
  • XTREME-UP: Gemini Embedding demonstrates strong cross-lingual retrieval on XTREME-UP, mapping queries in 20 underrepresented languages to English passages.Figure 2 shows correct retrieval for Assamese and Hindi queries encoded without translation, including a Hindi query containing a typo.
  • Ablations: +17.6 is reported for self-training on Gemini-generated data in MTEB classification, while multilingual fine-tuning helps most on XTREME-UP long-tail languages.The training-mixture ablations also report good English-only performance on MTEB(Multilingual), MTEB(Eng, v2), and XOR-Retrieve.

6. Ablation Study

The ablation study attributes Gemini Embedding’s broad performance to staged pre-finetuning and fine-tuning, synthetic data generation, dataset filtering, and hard-negative mining. These components improve performance across languages, tasks, and retrieval datasets, although excessive hard negatives can cause overfitting.

  • Training Recipe: Pre-finetuning substantially improves performance across multiple benchmarks, while English-only fine-tuning remains strong on multilingual evaluation.The study compares no training, pre-finetuning only, and subsequent fine-tuning to analyze generalization across languages and tasks.
  • Data Filtering: LLM-filtered retrieval datasets consistently improve results across languages, with only minor drops for some MIRACL languages.The filtering experiment covers MIRACL training datasets spanning 18 languages.
  • Hard Negative Mining: Hard negatives generally enhance retrieval performance across four datasets, but excessive negatives cause overfitting and performance degradation.The analysis motivates future work on regularization and improved hard-negative sampling strategies.
  • Synthetic Data Generation: Training on zero-shot synthetic classification datasets greatly improves performance across all evaluated datasets and can match in-domain training.The synthetic datasets were generated without using examples from the original datasets, using a multi-stage prompting strategy.

7. Future Work

Future work will extend Gemini Embedding beyond text to image, video, and audio, aiming to represent multimodal combinations in a single embedding space. The authors will also explore training recipes that balance unimodal and multimodal capabilities.

  • 7. Future Work: The model will be extended to image, video, and audio, using Gemini’s multimodal capabilities to represent modality combinations in one embedding space.This aims to make Gemini Embedding more comprehensive across modalities.
  • 7. Future Work: Training recipes will be explored to balance performance across different unimodal and multimodal capabilities.

8. Conclusion

Gemini Embedding is presented as a unified, general-purpose embedding model that leverages Gemini’s multilingual and code-understanding capabilities across many languages, domains, and task types. MMTEB evaluations show substantial gains over previous top-performing models on multilingual, English, and code evaluations, with particular strengths in classification, clustering, and retrieval.

  • 8. Conclusion: Gemini Embedding is a unified, general-purpose embedding model built on Gemini’s multilingual and code-understanding capabilities.It generates versatile encodings for inputs spanning many languages, domains, and task types.
  • 8. Conclusion: MMTEB evaluations show substantial gains over previous top-performing models across multilingual, English, and code evaluations.These results advance the state of the art in embedding performance.
  • 8. Conclusion: Gemini Embedding particularly excels at classification, clustering, and retrieval tasks.Its unified capabilities are complemented by the ability to precompute representations.

9. Full Results

The section reports full Gemini Embedding results across multilingual, English, code, retrieval, and multilingual understanding benchmarks.

  • Full results are presented for Gemini Embedding on MTEB(Multilingual).
  • Full results are presented on MTEB(Eng, v2) and MTEB(Code).
  • Full results are also presented on XOR-Retrieve and XTREME-UP.

10. Contributions and Acknowledgments

The report identifies Jinhyuk Lee as a core contributor with equal contribution and lists additional contributors including Iftekhar Naim, Gustavo Hernández Ábrego, and others.

  • Core Contributors: Jinhyuk Lee is identified as a core contributor with equal contribution.
  • Core Contributors: Additional contributors include Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter, Vera Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, and Vikram Rao.
Loading 2503.07891v1…