PaperlayerView exact revision on arXiv

Source-linked AI summary

EmbeddingGemma 2: Unified and Efficient Multimodal Representations

Henrique Schechter Vera, Sahil Dua, Jovan Andonov, Sara Smoot, Sebastian Russo, Ivan Llanos, Max Matveev, Jae Hyeon Yoo, Kat Black, Shih-Cheng Huang, Josiah Aklilu, Maarten Grootendorst, Omar Sanseviero, Chandramouli Amarnath, Ian Ballantyne, Zhe Li, Daniel Salz, Ibrahim Alabdulmohsin, Ilya Philippov, Kristen Quan, Glenn Cameron, Edouard Yvinec, Luciano Martins, Doug Reid, Sindhu Raghuram Panyam, Weiyi Wang, Hengxuan Ying, Madhuri Shanbhogue, Tanmaya Dabral, Shanfeng Zhang, Juyeong Ji, Renjie Wu, Joe Zou, Xu Chen, Yunan Zhang, Yu-Hui Chen, Matthew Soulanille, Vinod Mamillapalli, Yi-Chun Kuo, Wai Hon Law, Matt Kreileder, Alice Zheng, Suril Shah, Mogan Shieh, Pooja Consul, Alberto Montes, Andreas Hess, Andres Sevilla, Babak Samari, Brendan McCloskey, Brian Potetz, Daniel Cer, Georgios Kathareios, Gustavo Hernández Ábrego, Hesen Zhang, Ji Ma, Jiaxing Wu, Keegan Mosley, Paul Suganthan, Rui Meng, Shen Gao, Shengcao Cao, Yi-Ting Chen, Yichang Chen, Zhongli Ding, Kendra Byrne, Cormac Brick, Olivier Lacombe, Qin Yin, Tris Warkentin, Armand Joulin, Tom Duerig, Mojtaba Seyedhosseini

arXiv:2610.37364v1cs.LGcs.CLcs.CVcs.SD

TL;DR

Text-based embedding systems rely on extra tools for audio and visual data, while large multimodal embedding models can be expensive to deploy. EmbeddingGemma 2 combines modality encoders with a shared embedding space and staged training, establishing a new Pareto frontier across modalities and often surpassing much larger models.

  • Problem

    Text embedding systems need ASR and image captioners to handle audio and visual data, while MLLM-derived embedding models that support mixed-modal inputs typically contain several billion parameters.

    Source: §1 · p. 1

  • Method

    EmbeddingGemma 2 uses modular modality encoders and a shared embedding space, trained through stages that progress across modalities and objectives.

    Source: §1 · p. 2

  • Results

    EmbeddingGemma 2 establishes a new Pareto frontier across modalities and often matches or exceeds specialized and multi-billion-parameter models on multimodal evaluations.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    Its modular encoder loading, truncated embeddings, and quantization support offer flexibility for resource-constrained and on-device settings.

    Source: §6 · p. 15

Abstract

from arXiv · show

We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.

1. Introduction

EmbeddingGemma 2 addresses the limits of text-based and large multimodal embedding systems with a modular model and staged training. Across benchmarks spanning modalities, it establishes a new Pareto frontier and often matches or exceeds much larger specialists.

1. Introduction

  • Text embedding systems rely on ASR and image captioners for audio and visual inputs, adding computation while missing fine-grained visual details and non-verbal or environmental sounds.Multimodal embedding models add vision towers but cannot natively process mixed-modal input; MLLM-derived models support interleaved inputs but typically contain several billion parameters.Source: §1 · p. 1
  • The 744M-parameter model maps text, images, video, audio, and interleaved combinations into one embedding space, with modular encoders enabling text-only, vision-language, or audio-language configurations.The respective configurations use 271M, 439M, and 577M parameters.Source: §1 · p. 2
  • A three-stage training regime spans tasks, modalities, and domains, separating text-only from multimodal training and distillation from contrastive and regularization objectives.The model also supports truncated embeddings and quantization-aware checkpoints for resource-constrained settings.Source: §1 · p. 2
  • Simplifying per-layer embeddings by removing the dedicated lookup table reduces cost and improves performance; ablations also examine vision-token budgets and their effects on training and inference.The reported effects significantly concern tasks such as document retrieval that require fine-grained visual understanding.Source: §1 · pp. 2–3
  • EmbeddingGemma 2 establishes a new Pareto frontier across seven benchmark suites and often matches or exceeds specialized and multi-billion-parameter models on multimodal evaluations.The suites span text, code, image, video, document, speech, and sound inputs.Source: Figure 2 · p. 3, §1 · p. 3
  • Ablations examine vision-encoder co-training, contrastive finetuning versus distillation, and native audio understanding versus ASR.The authors report that co-training the vision encoder improves multimodal embedding performance and that native audio understanding has value over ASR.Source: §1 · p. 3

2. EmbeddingGemma 2

EmbeddingGemma 2 combines a bidirectional transformer with projected vision and audio tokens, then trains shared representations through multimodal inputs and staged objectives.

2.1. Architecture

  • A bidirectional transformer pools token outputs into a normalized embedding, while per-layer inputs reuse the main embedding table instead of a dedicated lookup table.The final projection supports 768-dimensional embeddings.Source: §2.1 · pp. 3–4

2.2. Input

  • Text, images, audio, and video are encoded as ordered token sequences; video can interleave timestamped frames and audio segments.Training and evaluation use up to 32 sampled video frames; image and video token budgets are configurable at inference.Source: §2.2 · pp. 4–5

2.3. Objective

  • Training combines contrastive learning with in-batch, hard, and same-tower negatives and a spread-out regularizer that encourages embeddings to use the space.The regularizer uses the second-moment term of the global orthogonal regularizer.Source: §2.3 · pp. 5–6
  • Embedding matching distills the teacher’s embeddings, while Matryoshka Representation Learning supports 768-, 512-, 256-, and 128-dimensional outputs.The teacher embeddings are truncated rather than projected to match the student’s dimensionality.Source: §2.3 · pp. 6–7

2.4. Recipe

  • The recipe initializes from Gemma 4 and progresses from text-only distillation to multimodal distillation, then multimodal finetuning with contrastive and spread-out objectives.Training uses approximately 2.9T tokens across the three stages.Source: §2.4 · pp. 7–8
  • Checkpoint averaging mitigates degradation on tasks that overfit, while quantization-aware training narrows the quality gap for some evaluations.With Q4_C quantization, peak Pixel 11 Pro inference memory is 191 MB for text-only, 447 MB for text and vision, and 567 MB for the full multimodal model.Source: §2.4 · p. 8

3. Ablation Studies

The ablations examine architectural and training choices using an intermediate recipe, so their scores are not representative of the final model. They cover per-layer embeddings, finetuning, vision encoder training, and vision token budgets.

3.1. Per-Layer Embeddings Table

  • Removing the dedicated per-layer embedding lookup table reduces parameter overhead and improves performance across all three categories, including +0.69 on Video and +0.61 on VisDoc.The approach retains per-layer inputs by projecting the shared embedding space; similar improvements also appeared in text-only evaluations.Source: §3.1 · p. 9

3.2. Finetuning Stage

  • Switching from continued distillation to contrastive and spread-out finetuning boosts image and video performance while producing similar visual-document results.The compared runs used approximately compute-equivalent durations; distillation plateaued while finetuning improved the representation space beyond teacher imitation.Source: §3.2 · p. 9

3.3. Vision Encoder Training

  • Training the vision encoder greatly improves multimodal embedding performance compared with freezing it.Both runs start from a text pre-finetuning checkpoint; further audio-encoder training may yield analogous gains.Source: §3.3 · p. 10, Table 4 · p. 10

3.4. Vision Token Budgets

  • At inference, visual-document tasks continue improving beyond the 280-token training budget, while aggregate image quality begins to saturate at 280 tokens.Video performance degrades between 280 and 560 tokens, possibly because longer sequences dilute attention and mean-pooled representations.Source: Table 5 · p. 10, §3.4 · p. 11
  • Training with 70 rather than 140 tokens per video frame helps video tasks at 70-token inference, but hurts image and visual-document quality; 140 tokens is preferable across both evaluation regimes.The comparison uses different intermediate recipes; the authors note that higher training budgets may bring further gains at significant computational cost and with diminishing returns.Source: Table 5 · p. 10, §3.4 · p. 11

4. Evaluation

EmbeddingGemma 2 is evaluated across text, code, image, video, visual-document, speech, music, and environmental-sound benchmarks against modality specialists and general-purpose models. It shows strong results across these areas, with distinct strengths and trade-offs by modality.

4.1. Methods

  • The evaluation reports results across task types, domains, and languages for text/code, image/video/visual documents, and speech/music/environmental sounds; it excludes models trained on >25% of evaluation data.Baselines are under 1B parameters for MTEB, MAEB, and MIEB(lite), and under 2B for MMEB-v2. Visual inputs use up to 32 video frames at 140 tokens per frame and 280 tokens per image; Table 6 compares against a slightly larger multimodal peer and modality specialists.Source: §4 · p. 11, §4.1 · p. 11, Table 6 · p. 12

4.2. Text

  • EmbeddingGemma 2 improves over its predecessor on multilingual text despite fewer text parameters, while its code results establish a new Pareto frontier and come within 2 points of the 28× larger frontrunner.Its English text performance sees a modest regression; truncated representations degrade minimally, especially down to 256 dimensions.Source: §4.2 · p. 12, Table 7 · p. 13

4.3. Vision

  • On MIEB(lite), EmbeddingGemma 2 leads sub-1B models by +7.9 Task and +12.2 Type, and even its 128-dimensional embeddings outperform every other sub-1B model at full dimensionality.On MMEB-v2, it is the smallest leaderboard model and its overall score is exceeded only by models around 2B parameters.Source: §4.3 · p. 13, Table 8 · p. 13, Table 9 · p. 14

4.4. Audio

  • On MAEB, EmbeddingGemma 2 sets a new efficiency frontier, with state-of-the-art sub-1B scores on any-to-any retrieval and audio classification.Its mean task score is about 0.2 points behind the best model under 1B parameters and about 3.5 points behind the overall highest-scoring model; 128-dimensional embeddings beat every other sub-1B model.Source: §4.4 · p. 14, Table 10 · p. 14
  • Across audio-text specialists, EmbeddingGemma 2 performs differently by task: it can surpass them on non-speech tasks but remains behind on audio captioning.It struggles on zero-shot speaker-gender clustering (VoxPopuliGenderClustering: 0.40), but a linear probe reaches 66.40 (IEMOCAPGender).Source: §4.4 · p. 14
  • Native audio inputs improve MSEB retrieval beyond ASR-based embeddings, and even 256-dimensional native-audio embeddings outperform full-dimensional ASR embeddings.Native inputs preserve signals that transcription can lose, including accents, homophones, and artifacts.Source: §4.4 · p. 15, Table 11 · p. 15

5. Future Work

The paper identifies longer context, alternative architectures, and training changes as directions for future research, alongside evaluation frameworks for emerging multimodal search use cases.

5. Future Work

  • Extending context length, potentially with content-aware methods, could support indexing full-length media, improve long-horizon tasks, and enable agentic memory systems.The model already supports long sequences and variable vision token budgets.Source: §5 · p. 15
  • Encoder-free architectures are proposed as a direction that could improve quality while further reducing memory footprint and latency.Source: §5 · p. 15
  • End-to-end RAG training could further optimize embeddings for downstream use, while new evaluation frameworks will matter as interleaved all-to-all multimodal search grows.Source: §5 · p. 15

6. Conclusion

EmbeddingGemma 2 combines native multimodal capabilities with a smaller, stronger backbone and training and architecture changes. The authors report a new Pareto frontier across modalities and support flexible deployment and research use.

6. Conclusion

  • EmbeddingGemma 2 combines a smaller, stronger backbone with native multimodal capabilities.Source: §6 · p. 15
  • Simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression accompany its new Pareto frontier across text, vision, and audio benchmarks.The model often surpasses specialist and general-purpose models several times larger.Source: §6 · p. 15
  • Quantization, embedding truncation, and modular encoder loading support resource-constrained and on-device use; released checkpoints are intended for off-the-shelf use and further research.Source: §6 · p. 15

A. Task Instructions

The task instructions distinguish asymmetric tasks, which format queries and targets differently, from symmetric tasks, which apply the same prefix to all inputs.

A. Task Instructions

  • Asymmetric tasks use different formats for queries and targets.When no document title is available, the query format uses “title: none.”Source: Table 12 · p. 23
  • Symmetric tasks apply the same prefix to all inputs.Source: Table 12 · p. 23
  • The instructions catalog task-specific prefixes for EmbeddingGemma 2.Source: Table 12 · p. 23

B. Multilingual and English Leaderboards

The section presents leaderboards for top-ranking models under 1B parameters on multilingual and English MTEB v2.

B. Multilingual and English Leaderboards

  • The multilingual leaderboard covers top-ranking models under 1B parameters on MTEB(Multilingual, v2).Source: Table 13 · p. 23
  • Together, the tables report leaderboard performance for multilingual and English MTEB v2.Source: Table 13 · p. 23, Table 14 · p. 23
  • The English leaderboard covers top-ranking models under 1B parameters on MTEB(English, v2).Source: Table 14 · p. 23

C. Full Results

The full results section reports EmbeddingGemma 2 results across multilingual, English, code, and multimodal embedding benchmarks.

C. Full Results

Contributions and Acknowledgements

The listed core contributors include Henrique Schechter Vera and Sahil Dua, with additional contributors named in the following list.

Contributions and Acknowledgements

Related papers

Loading 2610.37364v1…