Source-linked AI summary
EmbeddingGemma: Powerful and Lightweight Text Representations
Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, Daniel Cer, Alice Lisak, Min Choi, Lucas Gonzalez, Omar Sanseviero, Glenn Cameron, Ian Ballantyne, Kat Black, Kaifeng Chen, Weiyi Wang, Zhe Li, Gus Martins, Jinhyuk Lee, Mark Sherwood, Juyeong Ji, Renjie Wu, Jingxiao Zheng, Jyotinder Singh, Abheesht Sharma, Divyashree Sreepathihalli, Aashi Jain, Adham Elarabawy, AJ Co, Andreas Doumanoglou, Babak Samari, Ben Hora, Brian Potetz, Dahun Kim, Enrique Alfonseca, Fedor Moiseev, Feng Han, Frank Palma Gomez, Gustavo Hernández Ábrego, Hesen Zhang, Hui Hui, Jay Han, Karan Gill, Ke Chen, Koert Chen, Madhuri Shanbhogue, Michael Boratko, Paul Suganthan, Sai Meher Karthik Duddu, Sandeep Mariserla, Setareh Ariafar, Shanfeng Zhang, Shijie Zhang, Simon Baumgartner, Sonam Goenka, Steve Qiu, Tanmaya Dabral, Trevor Walker, Vikram Rao, Waleed Khawaja, Wenlei Zhou, Xiaoqi Ren, Ye Xia, Yichang Chen, Yi-Ting Chen, Zhe Dong, Zhongli Ding, Francesco Visin, Gaël Liu, Jiageng Zhang, Kathleen Kenealy, Michelle Casbon, Ravin Kumar, Thomas Mesnard, Zach Gleicher, Cormac Brick, Olivier Lacombe, Adam Roberts, Qin Yin, Yunhsuan Sung, Raphael Hoffmann, Tris Warkentin, Armand Joulin, Tom Duerig, Mojtaba Seyedhosseini
TL;DR
EmbeddingGemma addresses the need for strong text embeddings that are less costly to deploy than large models. It combines encoder-decoder initialization, teacher embedding distillation, regularization, and checkpoint merging, achieving state-of-the-art MTEB results for models under 500M parameters and comparable performance to substantially larger models. Its lead persists under low-bit quantization and truncated embeddings, supporting resource-efficient deployment.
Problem
Large state-of-the-art embedding models can be too computationally expensive for low-latency, high-throughput, and on-device applications.
Method
EmbeddingGemma uses an encoder initialized from an encoder-decoder adaptation of Gemma 3, embedding matching to Gemini Embedding, spread-out regularization, and checkpoint merging across finetuning mixtures.
Results
EmbeddingGemma achieves state-of-the-art performance across multilingual, English, and code MTEB leaderboards for models under 500M parameters, with performance comparable to models nearly twice its size.
Takeaways & Limitations
EmbeddingGemma provides a compact general-purpose embedding model whose lead persists with quantized weights or embeddings truncated to 128 dimensions.
Abstract
from arXiv · showhide
We introduce EmbeddingGemma, a new lightweight, open text embedding model based on the Gemma 3 language model family. Our innovative training recipe strategically captures knowledge from larger models via encoder-decoder initialization and geometric embedding distillation. We improve model robustness and expressiveness with a spread-out regularizer, and ensure generalizability by merging checkpoints from varied, optimized mixtures. Evaluated on the Massive Text Embedding Benchmark (MTEB) across multilingual, English, and code domains, EmbeddingGemma (300M) achieves state-of-the-art results. Notably, it outperforms prior top models, both proprietary and open, with fewer than 500M parameters, and provides performance comparable to models double its size, offering an exceptional performance-to-cost ratio. Remarkably, this lead persists when quantizing model weights or truncating embedding outputs. This makes EmbeddingGemma particularly well-suited for low-latency and high-throughput use cases such as on-device applications. We provide ablation studies exploring our key design choices. We release EmbeddingGemma to the community to promote further research.
1. Introduction
EmbeddingGemma targets the trade-off between embedding quality and computational cost with a lightweight model and a training recipe built from encoder-decoder initialization, distillation, regularization, and checkpoint merging. It achieves leading results across MTEB benchmarks for models under 500M parameters, with robustness to compression and truncation.
- Motivation: Lightweight embedding models are motivated by the need for low-latency, high-throughput inference, especially in on-device applications.Larger state-of-the-art models can be computationally expensive for applications involving sensitive data or offline access.
- Approach: EmbeddingGemma is a 308M-parameter model built on Gemma 3 and initialized from an encoder adapted through an encoder-decoder model.The recipe adapts Gemma 3 with the UL2 objective before using its encoder to initialize EmbeddingGemma.
- Approach: The training recipe combines embedding matching to Gemini Embedding, spread-out regularization, task prompts, pre-finetuning, and model souping across different finetuning mixtures.Model souping combines multiple finetuned checkpoints to produce stronger, more generalizable representations.
- Results: EmbeddingGemma ranks first across all aggregate metrics on multilingual, code, and English MTEB leaderboards among models under 500M parameters.The aggregate metrics are Borda count, mean over all task scores, and mean over all task type scores.
- Results: On MTEB(Multilingual, v2), EmbeddingGemma ranks 8th overall, 17 places above the second-best sub-500M-parameter model.Its performance is also comparable to models nearly double its size.
- Ablations: Ablations find encoder-decoder models provide a stronger starting point than decoder-only initialization because of richer contextual representations.The analysis attributes this advantage likely to bidirectional attention and encoder specialization for input understanding.
2. EmbeddingGemma
EmbeddingGemma is an encoder-only transformer adapted from Gemma 3 and trained to produce expressive, robust, generalizable text embeddings. Its recipe combines multi-stage training, several objectives, and checkpoint averaging across optimized task mixtures.
- Architecture: EmbeddingGemma is an encoder-only transformer initialized from a Gemma 3 encoder-decoder adaptation, inheriting Gemma 3 knowledge and contextual representations.The encoder-decoder model is trained with the UL2 objective before its encoder initializes EmbeddingGemma.
- Embedding construction: Mean pooling produces the final sequence representation, while MRL enables 768-, 512-, 256-, and 128-dimensional embeddings.The encoder outputs token embeddings, which are mean-pooled and projected before producing the embedding.
- Training objectives: EmbeddingGemma combines NCE contrastive learning, a spread-out regularizer, and embedding matching to train discriminative and expressive representations.The embedding-matching loss aligns the student embedding space with Gemini Embedding and also applies to hard negatives.
- Training objectives: The spread-out regularizer targets embedding distributions resembling independent points on the unit sphere, using only its second-moment term.The design is intended to support quantization robustness and efficient approximate-nearest-neighbor retrieval.
- Training recipe: Training spans pre-finetuning on diverse multilingual and programming-task data, followed by finetuning on smaller, higher-quality task-specific mixtures with hard negatives.Pre-finetuning uses large-scale unsupervised data and larger batches; finetuning uses hard negatives and dataset-specific batches.
- Model combination: Model souping averages checkpoints from finetuning runs based on Bayesian-optimized mixtures to improve final-model quality and robustness.The final model uses an unweighted average of checkpoints from runs with different optimized mixtures.
3. Ablation Studies
The ablations examine initialization, pooling, and model souping as contributors to EmbeddingGemma’s quality across tasks and languages. Encoder-decoder initialization, mean pooling, and combining checkpoints from different finetuning mixtures each improve performance.
- Initialization Strategy: Encoder-decoder initialization outperforms decoder-only initialization across various task types.Both initialization strategies dramatically improve performance over random weights.
- Pooling: Mean pooling yields the best performance despite attention pooling using substantially more learnable parameters.The result extends prior findings for encoder-only classification and regression to embedding tasks such as clustering.
- Model Souping: Model souping improves overall performance and outperforms its individual ingredients in each task type.The result indicates that souping can combine checkpoints trained with different finetuning mixtures, not only different hyperparameter configurations.
- Model Souping: Different finetuning mixtures produce experts in different task types, giving model souping complementary ingredients to combine.The final model averages parameters from checkpoints associated with multiple mixtures.
4. Evaluation
EmbeddingGemma is evaluated across multilingual, English, code, cross-lingual retrieval, and other general-purpose embedding benchmarks. It leads sub-500M-parameter models across MTEB aggregates, remains competitive with larger models, and performs strongly on low-resource languages.
- Overall Performance: EmbeddingGemma ranks first across all aggregate metrics on MTEB multilingual, English, and code leaderboards for models under 500M parameters.The aggregate metrics are Borda count, mean over all task scores, and mean over all task type scores.
- Overall Performance: 128-dimensional embeddings retain the highest MTEB Task Mean and Task Type Mean scores.The performance gap persists even when embedding outputs use fewer dimensions.
- Overall Performance: Under 1B parameters, EmbeddingGemma ranks #3 on MTEB multilingual, #2 on English, and #2 on code.It is comparable to state-of-the-art models nearly double its size and outperforms commercial API models except Gemini Embedding.
- MTEB(Multilingual, v2): On multilingual MTEB, EmbeddingGemma outperforms Qwen3 Embedding 0.6B in instruction retrieval, multilabel classification, pair classification, and reranking.Qwen3 Embedding 0.6B is identified as the top-ranked model with fewer than 1B parameters.
- MTEB(Eng, v2): On English MTEB, EmbeddingGemma improves classification by +8.5, clustering by +7.8, and summarization by +4.4 versus other models.It also tops the sub-1B-parameter leaderboard in these task types after excluding models trained on over 25% of MTEB data.
- XTREME-UP: EmbeddingGemma vastly outperforms billion-parameter and commercial models on XTREME-UP cross-lingual retrieval, demonstrating capability in low-resource languages.It achieves the strongest performance among selected open models for each evaluated language and is surpassed by a commercial model in only two languages.
5. Future Work
Future work extends EmbeddingGemma beyond text to image, audio, and video modalities. The goal is lightweight, natively multimodal embedding models that support unimodal, cross-modal, and multimodal use cases on-device.
- Multimodal Extensions: The authors plan to extend EmbeddingGemma beyond text into image, audio, and video modalities.They will explore unimodal, cross-modal, and multimodal use cases.
- On-Device Models: Future work targets lightweight, natively multimodal embedding models suitable for on-device execution.Existing multimodal embedding models use several billion parameters, while the proposed direction aims to reduce this scale.
6. Conclusion
EmbeddingGemma combines larger-model knowledge transfer, spread-out regularization, and checkpoint merging to produce lightweight, generalizable text embeddings. Evaluations show state-of-the-art performance across multilingual, English, and code benchmarks for models under 500M parameters, with implications for efficient on-device applications.
- EmbeddingGemma uses T5Gemma encoder initialization and Gemini Embedding distillation to transfer knowledge from larger models.
- Spread-out regularization and model merging across multiple optimized mixtures target expressive and generalizable representations.
- EmbeddingGemma pushes the state of the art across multilingual, English, and code MTEB benchmarks for models under 500M parameters.
- Its performance is comparable to models twice its size, and the lead persists after quantization or embedding truncation.
- Compact state-of-the-art performance supports faster, private, and offline-capable applications directly on user devices.
A. Full Results
The full-results materials cover multilingual, English, code, XOR-Retrieve, and XTREME-UP evaluations. The multilingual results span multiple languages, while the English and code table compares top sub-500M-parameter models under an over-fitting mitigation rule.
- MTEB(Multilingual, v2): Table 10 reports full EmbeddingGemma results on MTEB(Multilingual, v2).
- MTEB(English, v2) and MTEB(Code): MTEB(English, v2) and MTEB(Code) results are presented together, including comparisons among top 20 models under 500M parameters.The comparison excludes models trained on more than 25% of MTEB data to mitigate potential over-fitting.
- Languages: The multilingual materials list Arabic, Bengali, Finnish, Japanese, Korean, Russian, and Telugu.
- Additional evaluations: Table 12 reports full results on XOR-Retrieve and XTREME-UP.
B. Contributions and Acknowledgments
The acknowledgments identify the paper’s core contributors and note equal contributions for marked authors. The supplied contributor list includes Henrique Schechter and Vera, followed by additional named contributors.
- Core Contributors: Henrique Schechter and Vera are listed as core contributors, with Vera marked as an equal contributor.
- Contributors: Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, and Sara Smoot appear in the contributor list.
- Contributors: Iftekhar Naim, Joe Zou, Feiyang Chen, and Daniel Cer also appear in the contributor list.