PaperlayerView exact revision on arXiv

Source-linked AI summary

Gemini Embedding: Generalizable Embeddings from Gemini

Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, Feng Han, Andreas Doumanoglou, Nithi Gupta, Fedor Moiseev, Cathy Yip, Aashi Jain, Simon Baumgartner, Shahrokh Shahi, Frank Palma Gomez, Sandeep Mariserla, Min Choi, Parashar Shah, Sonam Goenka, Ke Chen, Ye Xia, Koert Chen, Sai Meher Karthik Duddu, Yichang Chen, Trevor Walker, Wenlei Zhou, Rakesh Ghiya, Zach Gleicher, Karan Gill, Zhe Dong, Mojtaba Seyedhosseini, Yunhsuan Sung, Raphael Hoffmann, Tom Duerig

arXiv:2503.07891v1cs.CLcs.AI

TL;DR

A key challenge is building general-purpose text embeddings that capture semantic information and perform across diverse downstream tasks. Gemini Embedding adapts a model initialized from Gemini and trains it on a curated suite of embedding tasks using contrastive learning. MMTEB evaluations report state-of-the-art results across multilingual, English, and code benchmarks.

  • Problem

    A key challenge is building general-purpose text embeddings that capture semantic information and perform across diverse downstream tasks.

    Source: §1 · p. 1

  • Method

    Gemini Embedding adapts a model initialized from Gemini and trains it on a curated suite of embedding tasks using contrastive learning.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across MMTEB’s multilingual, English, and code evaluations, Gemini Embedding advances state-of-the-art; MTEB(Multilingual) Task Mean is 68.32, +5.09 over multilingual-e5-large-instruct.

    Source: §1 · p. 3, §5.2 · p. 7, §8 · p. 12

  • Takeaways & Limitations

    Gemini Embedding’s representations span classification, similarity search, clustering, ranking, and retrieval, with precomputation supporting broader use by researchers and developers.

    Source: §8 · p. 12

    The reported work focuses on text embeddings; extending them to image, video, and audio remains future work.

    Source: §7 · p. 11

Abstract

from arXiv · show

In this report, we introduce Gemini Embedding, a state-of-the-art embedding model leveraging the power of Gemini, Google's most capable large language model. Capitalizing on Gemini's inherent multilingual and code understanding capabilities, Gemini Embedding produces highly generalizable embeddings for text spanning numerous languages and textual modalities. The representations generated by Gemini Embedding can be precomputed and applied to a variety of downstream tasks including classification, similarity, clustering, ranking, and retrieval. Evaluated on the Massive Multilingual Text Embedding Benchmark (MMTEB), which includes over one hundred tasks across 250+ languages, Gemini Embedding substantially outperforms prior state-of-the-art models, demonstrating considerable improvements in embedding quality. Achieving state-of-the-art performance across MMTEB's multilingual, English, and code benchmarks, our unified model demonstrates strong capabilities across a broad selection of tasks and surpasses specialized domain-specific models.

1. Introduction

Gemini Embedding uses Gemini’s capabilities to train a general-purpose embedding model for diverse tasks and languages, with state-of-the-art results reported across multilingual, English, and code benchmarks.

1. Introduction

  • Text embeddings represent semantic similarity through vector proximity, and general-purpose models aim to support tasks such as retrieval, clustering, and classification.Large language models’ pretraining knowledge has motivated their use in developing general-purpose embeddings (Anil et al., 2023a,b; Brown et al., 2020).Source: §1 · p. 1
  • Prior work improves embedding models by using LLMs to generate better training examples or initialize model parameters; direct initialization can increase computational demands while improving performance (Lee et al., 2025; Neelakantan et al., 2022; Wang et al., 2023).Examples include hard-negative mining (Lee et al., 2024) and synthetic data generation (Dai et al., 2022; Wang et al., 2023).Source: §1 · pp. 1–2
  • Gemini Embedding is initialized from Gemini (Anil et al., 2023a; Team, 2024) and trained on a curated heterogeneous dataset using contrastive learning, task prompts, pre-finetuning, and Model Soup checkpoint averaging.Gemini filters low-quality examples, selects relevant positive and negative retrieval passages, and generates synthetic datasets.Source: §1 · p. 2
  • 68.32 on MMTEB multilingual task-mean scores is +5.09 above multilingual-e5-large-instruct, while the task-type mean of 59.64 is +3.64 above gte-Qwen2-7B-instruct; Gemini Embedding ranks first by Borda rank.The report also describes state-of-the-art performance on MTEB(Multilingual).Source: §1 · pp. 2–3, Table 1 · p. 2
  • Gemini Embedding establishes state-of-the-art performance on XOR-Retrieve and performs strongly in both high-resource English and low-resource languages such as Macedonian.XOR-Retrieve evaluates cross-lingual retrieval; Table 1 also covers MTEB(English, v2), MTEB(Code), and XTREME-UP.Source: Table 1 · p. 2, §1 · p. 3

2. Related Work

Related work highlights embedding models’ generalization limits and reviews LLMs both as sources of training data and as embedding-model backbones.

Text Embedding Models

  • Prior general-purpose embedding models have struggled to generalize across diverse tasks and domains, motivating benchmarks that emphasize novel task and domain generalization.MTEB (Enevoldsen et al., 2025; Muennighoff et al., 2023) was developed in this context.Source: §2 · p. 3

LLMs for Embedding Data Generation

  • LLM-generated synthetic queries and task-conditioned triplets provide diverse training data, while LLM-based methods improve passage selection and scale generation across many languages and tasks.Prior work explores synthetic query generation (Dai et al., 2022; Jeronymo et al., 2023; Nogueira et al., 2019), better positive and negative passage selection (Lee et al., 2024), and task-conditioned (query, positive, hard negative) triplets across nearly one hundred languages and hundreds of thousands of tasks (Wang et al., 2023).Source: §2 · p. 3

LLMs as Embedding Models

3. Gemini Embedding

Gemini Embedding starts from Gemini parameters and adds a pooling and projection head to produce reusable representations. Its training combines contrastive objectives with staged adaptation and checkpoint averaging.

3.1. Model Architecture

  • Gemini Embedding uses a bidirectional transformer initialized from Gemini, mean-pools token embeddings, and applies a learned linear projection to produce one output embedding.The Gemini initialization supplies pretrained knowledge; the projection scales the pooled representation to the target dimension.Source: §3.1 · p. 4

3.2. Training Objective

  • Training uses noise-contrastive estimation with in-batch negatives on query–positive-target examples, accompanied by a prescribed task string.The task string describes the example’s task, such as question answering or fact checking.Source: §3.2 · p. 4
  • The objective omits same-tower negatives because they can create false negatives, which the authors report decreases performance on most tasks.The loss also includes a masking term particularly relevant to classification, where the number of target labels is small.Source: §3.2 · p. 5
  • Multi-resolution loss supports 768- and 1,536-dimensional subembeddings from the model’s 3,072-dimensional output.It applies separate losses across overlapping embedding subdimensions.Source: §3.2 · p. 5

3.3. Recipe

  • Pre-finetuning adapts the model from autoregressive generation to encoding on noisy query–target pairs, using large batches and substantially more steps than fine-tuning.The larger batches provide more stable gradients and mitigate noise; this phase omits the hard-negative loss term.Source: §3.3 · p. 5
  • Fine-tuning uses task-specific query, target, and hard-negative triples, with smaller batches limited to one dataset to strengthen within-task discrimination.The authors search training hyperparameters and mixture components to obtain candidate checkpoints.Source: §3.3 · p. 5
  • The final recipe averages parameters from selected fine-tuning checkpoints to seek additional generalization performance.The experiments include averages within and across runs, weighted averages, intentional data variation, and manual checkpoint selection.Source: §3.3 · p. 5

4. Datasets

Gemini Embedding’s training data combines multilingual embedding tasks and code retrieval, with Gemini used to generate synthetic data, filter low-quality examples, and mine hard negatives. The data pipeline spans billion-scale web pre-finetuning and fine-tuning mixtures designed for task, language, and coding diversity.

4. Datasets

  • The training mixture combines diverse multilingual embedding tasks and code retrieval, while Gemini supports synthetic data generation, data filtering, and hard negative mining.Source: §4 · p. 6

4.1. Training Data Mixture

  • Pre-finetuning exposes the model to a billion-scale web corpus using titles and passages as inputs and positive target pairs.The authors report that this simple approach is consistently effective even when the embedding model is initialized from an LLM.Source: §4.1 · p. 6
  • Fine-tuning mixtures target task, language, and coding diversity; the authors excluded many in-domain MTEB datasets because gains were largely limited to their test splits through leakage or dataset bias.Mixture rates were selected by fine-grained grid search, initialized from the optimal convergence steps for each training dataset.Source: §4.1 · p. 6

4.2. Improving Data Quality with Gemini

  • Synthetic retrieval data uses Gemini-generated web-passage queries filtered by an auto-rater, while synthetic English classification data covers counterfactual, sentiment, and review tasks.Multi-stage prompting conditions on synthetic user, product, or movie generations and samples from longer generation lists to increase diversity.Source: §4.2 · p. 6
  • Gemini assesses human-annotated retrieval examples and removes low-quality cases with incorrect positive or negative targets.Source: §4.2 · p. 6
  • For retrieval hard negatives, Gemini scores top-k neighbors from an initially trained embedding model using graded-classification and query-likelihood prompts combined with Reciprocal Rank Fusion; the lowest-scoring neighbor is selected.Source: §4.2 · p. 6

5. Evaluation

Gemini Embedding achieves top overall performance across multilingual, English, and code benchmarks, with strong cross-lingual retrieval results. Evaluations span diverse MMTEB tasks and cross-lingual benchmarks, and training-mixture results highlight gains from multilingual fine-tuning and synthetic-data self-training.

5.1. Benchmarks and Tasks

  • Evaluation covers 164 MMTEB tasks across 250+ languages and 10 task types, plus XOR-Retrieve queries in 7 languages and XTREME-UP queries in 20 underrepresented Indo-European languages.Source: §5.1 · p. 7

5.2. Overall Performance

  • 68.32 Task Mean, 59.64 Task Type Mean, and Borda rank #1 establish MTEB(Multilingual) state of the art; Gemini also ranks #1 on MTEB(Eng, v2) and MTEB(Code).The report also describes its cross-lingual retrieval performance on XOR-Retrieve and XTREME-UP as advancing state of the art for general-purpose embeddings.Source: §5.2 · p. 7
  • On MTEB(Multilingual), gains over the second-best model are +9.6 Classification, +3.7 Clustering, and +9.0 Retrieval; on MTEB(Eng, v2), they are +7.1, +5.3, and +4.3, respectively.Source: §5.2 · p. 8
  • Gemini leads MTEB(Code) in mean performance and Borda rank across eight tasks, and has the best Mean -COIR across seven; XTREME-UP results show improved retrieval across 20 underrepresented languages.Qualitative XTREME-UP examples show it retrieving relevant English passages for Assamese and Hindi queries without translation, including a Hindi query with a typo.Source: §5.2 · pp. 8–9, Table 5 · p. 9, Figure 2 · p. 9

5.3. Qualitative Examples

  • English-only training performs well on MTEB(Multilingual), MTEB(Eng, v2), and XOR-Retrieve; multilingual fine-tuning helps most on XTREME-UP long-tail languages, while synthetic-data self-training improves classification by +17.6.Source: Table 6 · p. 10, Table 7 · p. 10

6. Ablation Study

The ablation study systematically analyzes Gemini Embedding’s training recipe, finding that it generalizes across languages and tasks and benefits from synthetic data generation, retrieval-data filtering, and hard-negative mining.

6.1. Does Gemini Embedding Generalize to Multilingual Tasks?

  • English-only fine-tuning performs strongly on mostly zero-shot MTEB(Multilingual) and outperforms top embedding models on XTREME-UP, showing cross-language generalization.Pre-finetuning improves performance across multiple benchmarks, while the multilingual-only retrieval mixture’s lower score suggests task diversity matters more than language diversity for fine-tuning.Source: §6.1 · p. 10

6.2. How Does Gemini Improve Data Quality?

  • Gemini-filtered MIRACL training data consistently improves results across languages, with only minor drops for some languages.The English mixture also helps multilingual-task quality, making Gemini Embedding the best in Table 8.Source: Table 8 · p. 11, §6.2 · p. 11
  • Gemini-selected hard negatives generally improve retrieval performance across four datasets, but excessive hard negatives can cause overfitting and degrade retrieval performance.The study identifies regularization and better hard-negative sampling as future directions.Source: Figure 3 · p. 11, §6.2 · p. 11
  • Training on zero-shot synthetic classification datasets greatly improves performance across datasets and can match in-domain data, as with Gecko on AmazonPolarity.The multi-stage prompting strategy enables controllable generation and may reduce bias compared with real data.Source: §6.2 · p. 11

7. Future Work

Future work will extend Gemini Embedding beyond text to image, video, and audio, aiming to represent modality combinations in one embedding space. This requires suitable multimodal training data and recipes that balance unimodal and multimodal capabilities.

7. Future Work

  • Future work will extend Gemini Embedding to image, video, and audio, representing modality combinations in one embedding space.The authors plan to curate multimodal data tasks for learning generalizable representations and develop training recipes balancing unimodal and multimodal capabilities.Source: §7 · p. 11

8. Conclusion

Gemini Embedding adapts Gemini’s multilingual and code capabilities into reusable representations for a wide range of languages, domains, and tasks. MMTEB evaluations show substantial gains over previous top-performing models across multilingual, English, and code evaluations.

8. Conclusion

  • Gemini Embedding is a unified, general-purpose model that encodes inputs into versatile representations for classification, similarity search, clustering, ranking, and retrieval across many languages and domains.It builds on Gemini’s multilingual and code understanding capabilities.Source: §8 · p. 12
  • The model adapts Gemini’s capabilities to representation learning and uses Gemini itself to generate many of its training sets.Its representations can be precomputed, cached, and reused, enabling use in compute- and latency-sensitive settings.Source: §8 · p. 12
  • MMTEB evaluations show substantial gains over previous top-performing models across multilingual, English, and code evaluations, with particular strengths in classification, clustering, and retrieval.Its precomputable representations enable broader use by researchers and developers.Source: §8 · p. 12

9. Full Results

The section presents full results across multilingual, English, and code embedding benchmarks, as well as XOR-Retrieve and XTREME-UP.

Full Results

  • Table 9 reports Gemini Embedding’s full results on MTEB(Multilingual).Source: Table 9 · p. 16
  • Table 10 reports full results on MTEB(Eng, v2) and MTEB(Code).Source: Table 10 · p. 17
  • Table 11 reports full results on XOR-Retrieve and XTREME-UP.Source: Table 11 · p. 17

10. Contributions and Acknowledgments

The section names the paper’s core contributors and indicates equal contributions with an asterisk.

10. Contributions and Acknowledgments

  • Core contributors are Jinhyuk Lee*, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter, Vera Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, and Vikram Rao.The asterisk denotes equal contributions.Source: §10 · p. 18

Related papers

Loading 2503.07891v1…