EmbeddingGemma: Powerful and Lightweight Text Representations

1. Introduction

• The work targets the trade-off between large models’ capabilities and their computational cost, which limits use in low-latency, high-throughput and on-device applications.Such deployments may involve sensitive data or require offline access.Source: §1 · p. 2
• EmbeddingGemma initializes from a Gemma 3 encoder adapted through an encoder-decoder model, then distills Gemini Embedding with embedding matching and a spread-out regularizer.Training also uses task prompts, pre-finetuning, and model souping across checkpoints trained on different finetuning mixtures to strengthen generalizable representations.Source: §1 · p. 2
• First across all aggregate metrics—Borda count, mean task score, and mean task-type score—EmbeddingGemma leads sub-500M models on MTEB(Multilingual, v2), MTEB(Code), and MTEB(English, v2).On MTEB(Multilingual, v2), it ranks 8th overall and 17 places above the second-best sub-500M model; its performance is comparable to models nearly twice its size. The multilingual benchmark spans 250+ languages and over 100 tasks.Source: §1 · pp. 2–3, Figure 1 · p. 2
• Its lead persists with embeddings truncated to 128 dimensions or weights quantized to 4-bit precision.These results support the model’s suitability for low-latency, high-throughput settings such as on-device applications.Source: §1 · p. 3
• Ablations find encoder-decoder initialization stronger than decoder-only initialization, simple pooling stronger than attention pooling, and mixture-diverse checkpoint souping effective.The analysis also confirms that per-channel and per-block quantization-aware training yield models robust to quantization.Source: §1 · p. 3

2.1. Architecture

• EmbeddingGemma is initialized from the encoder of a T5Gemma-style encoder-decoder adaptation of a pretrained 300M Gemma 3 model.Its encoder uses bidirectional attention, mean-pools token embeddings, and projects the pooled representation to the target dimension.Source: §2.1 · p. 3

2.2. Training

• The contrastive NCE loss uses in-batch negatives and difficulty-weighted hard negatives, with the hardness weight controlled by α = 5.0.The stop-gradient operator prevents differentiation through the weight factor.Source: §2.2 · p. 4
• A spread-out regularizer encourages embedding diversity to support quantization robustness and efficient approximate-nearest-neighbor retrieval.It uses only the second-moment term, which the authors find sufficient to push the mean term toward its target.Source: §2.2 · p. 4
• Embedding matching aligns EmbeddingGemma’s embedding space with Gemini Embedding’s, including hard negatives, and the three losses are weighted uniformly.Matching hard negatives helps the model learn how the teacher distinguishes them from queries.Source: §2.2 · p. 4
• MRL applies separate contrastive and spread-out losses to overlapping embedding sub-dimensions, supporting 768-, 512-, 256-, and 128-dimensional outputs.Individual sub-dimension losses are added equally during training.Source: §2.2 · p. 5

2.3. Recipe

• The recipe first adapts Gemma 3 into an encoder-decoder model using UL2 pretraining, then uses its encoder as EmbeddingGemma’s backbone.This initialization transfers Gemma 3’s world knowledge, including knowledge from more than 100 languages.Source: §2.1 · p. 3, §2.3 · p. 5
• Pre-finetuning uses diverse, evenly weighted task and language data, while finetuning uses smaller, higher-quality task-specific mixtures with hard negatives.Pre-finetuning covers question answering, sentence similarity, code retrieval, and web search; each batch in both stages contains examples from one dataset.Source: §2.3 · p. 5
• Bayesian-optimized task mixtures specialize in different domains and are averaged into the final model, while quantization-aware finetuning produces int4, int8, and mixed-precision checkpoints.The paper reports that quantized-checkpoint quality is comparable to the bf16 checkpoint.Source: §2.3 · pp. 5–6

3.1. Initialization Strategy

• Encoder-decoder initialization outperforms decoder-only initialization across task types, while both strategies dramatically improve performance over random weights.The authors suggest encoder-decoder initialization produces more expressive representations.Source: §3.1 · p. 6

3.2. Pooling Types

• Mean pooling performs best despite attention pooling’s additional learnable parameters, extending simple pooling’s advantages to embedding tasks (Suganthan et al., 2025).Pooling aggregates token representations into one vector; the attention pooler uses multi-head attention with four heads.Source: Table 3 · p. 7, §3.2 · p. 7

3.3. Model Souping

• The souped checkpoint improves overall performance and outperforms its ingredients in each task type.Runs with different finetuning mixtures can be combined; each mixture yields an expert in different task types.Source: Table 4 · p. 7, §3.3 · p. 7

4.1. Tasks

• Across the evaluated subsets, coverage spans 250+ natural languages and 14+ programming languages, across 20 domains and 10 task types; evaluations also include cross-lingual retrieval.XOR-Retrieve pairs English passages with queries in 7 languages, while XTREME-UP pairs English passages with queries in 20 underrepresented Indo-European languages.Source: §4.1 · p. 8

4.2. Methods

• The study compares popular open models under 1B parameters and commercial APIs, excludes models trained on over 25% of MTEB data, and evaluates with bfloat16 weights.Most tasks use a 512-token context length, increased to 1024 or 2048 for tasks requiring longer inputs.Source: §4.2 · pp. 8–9

4.3. Results

• EmbeddingGemma ranks #1 on the MTEB multilingual, English, and code leaderboards for models under 500M parameters, and retains the highest Task Mean and Task Type Mean with 128-dimensional embeddings.The paper also reports excellent XOR-Retrieve and XTREME-UP results.Source: Table 5 · p. 8, §4.3 · p. 9
• EmbeddingGemma ranks #3, #2, and #2 on the MTEB multilingual, English, and code leaderboards among models under 1B parameters, and outperforms commercial APIs except Gemini Embedding.Its performance is described as comparable to state-of-the-art models nearly twice its size and with larger embedding dimensions.Source: Table 5 · p. 8, §4.3 · p. 9
• EmbeddingGemma leads aggregate scores and nearly all task types on MTEB(Multilingual, v2), outperforming Qwen3 Embedding 0.6B in instruction retrieval, multilabel classification, pair classification, and reranking.The comparison covers top-ranked multilingual models, including those below 500M parameters.Source: §4.3 · p. 9, Table 6 · p. 9
• EmbeddingGemma achieves the highest overall scores on MTEB(English, v2), improving classification (+8.5), clustering (+7.8), and summarization (+4.4) over all other models.It also tops the sub-1B-parameter leaderboard in these task types, excluding models potentially exposed to data leakage through training on over 25% of MTEB data.Source: §4.3 · p. 10, Table 7 · p. 10
• EmbeddingGemma achieves the best MTEB(Code) Mean -COIR and all other aggregate metrics, with gains over the second-best model of +37.6 on AppsRetrieval and +10.0 on CosQA.Mean -COIR averages the eleven remaining tasks because most top models lack COIR results.Source: §4.3 · p. 10, Table 8 · p. 10
• On XTREME-UP retrieval from 20 underrepresented languages to English passages, EmbeddingGemma outperforms top billion-parameter and commercial models, and leads open models in every language.Only a commercial model surpasses it in two languages; some strong MTEB multilingual models struggle on this benchmark.Source: §4.3 · p. 11, Table 9 · p. 11

Future Work

• EmbeddingGemma’s planned expansion beyond text includes image, audio, and video modalities.The paper intends to extend the model’s capabilities to these modalities.Source: §5 · p. 11
• The authors will explore recipes that perform across unimodal, cross-modal, and multimodal embedding use cases.Examples include text ⇔text, text ⇔image, and text+image ⇔image.Source: §5 · p. 11
• The goal is a lightweight, natively multimodal embedding model that can run on-device, addressing the billions of parameters used by prior multimodal embedding models (Jiang et al., 2025).Gemma 3 has demonstrated strong multimodal understanding, and prior work has shown promise in adapting multimodal large language models into multimodal embedding models.Source: §5 · p. 11

6. Conclusion

• The training recipe combines initialization from a rich T5Gemma encoder, distillation from Gemini Embedding, spread-out regularization, and model merging with multiple optimized mixtures.These choices aim to make representations expressive and generalizable.Source: §6 · p. 11
• EmbeddingGemma pushes state-of-the-art across multilingual, English, and code MTEB benchmarks for models under 500M parameters, with performance comparable to models twice its size.Its lead persists when weights are quantized or embeddings are truncated.Source: §6 · p. 11
• EmbeddingGemma’s compact size targets faster, private, offline-capable applications on user devices, and its checkpoints are released to accelerate research and development in efficient models.The model is built on the Gemma 3 language model family.Source: §6 · p. 11

• Table 10 reports EmbeddingGemma’s full results on MTEB (Multilingual, v2).Source: Table 10 · p. 15
• Table 11 reports full results on MTEB (English, v2) and MTEB (Code), comparing the top 20 sub-500M models under a stated training-data exclusion.Models trained on more than 25% of MTEB data are excluded from the English comparison to mitigate potential over-fitting.Source: Table 11 · p. 16
• Table 12 reports full results on XOR-Retrieve and XTREME-UP.Source: Table 12 · p. 16

• Core contributors are Henrique Schechter and Vera, with an asterisk marking equal contributions.Source: B. Contributions and Acknowledgments · p. 17
• Additional listed contributors are Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, and Daniel Cer.Source: B. Contributions and Acknowledgments · p. 17