PaperlayerView exact revision on arXiv

Source-linked AI summary

Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini

Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang, Gustavo Hernández Ábrego, Shih-Cheng Huang, Aashi Jain, Daniel Salz, Sonam Goenka, Chaitra Hegde, Ji Ma, Feiyang Chen, Jiaxing Wu, Tanmaya Dabral, Babak Samari, Kevin Poulet, Daniel Cer, Kaifeng Chen, Paul Suganathan, Hui Hui, Jovan Andonov, Philippe Schlattner, Jay Han, Iftekhar Naim, Wing Lowe, Vladimir Pchelin, Albert Yang, Yi-Ting Chen, Zhongli Ding, Grace Zhang, Georg Heigold, Yichang Chen, Antoine Reveillon, Brendan Mccloskey, Wenlei Zhou, Dahun Kim, Rui Meng, Emma Wang, Jack Zheng, Halley Fede, Zhen Yang, Keegan Mosley, Brian Potetz, Sahil Dua, Henrique Schechter Vera, Shen Gao, Hesen Zhang, Andreas Hess, Hengxuan Ying, Alberto Montes, Karan Gill, Min Choi, Sebastian Russo, Anja Hauth, Jinhyuk Lee, Michael Boratko, Megan Barnes, Vikram Rao, Claudiu Musat, Cyril Allauzen, Ehsan Variani, Shankar Kumar, Tom Bagby, Junyi Jiao, Yang Gu, Tengxin Li, Ayush Agrawal, Roberto Santana, Dev Nath, Stephen Karukas, Shuoxuan Han, Lucia Loher, Alice Twu, Nidhi Vyas, Siddharth Bhai, Frank Palma Gomez, Wangyuan Zhang, Chaoren Liu, Jizheng Yang, Steve Qiu, Shijie Zhang, Sujay Kulkarni, Sascha Rothe, Sean Nakamoto, Raphael Hoffmann, Zach Gleicher, Yunhsuan Sung, Qin Yin, Tom Duerig, Mojtaba Seyedhosseini

arXiv:2605.27295v1cs.CV

TL;DR

Existing embedding models are limited in representing mixed-modality inputs and interactions coherently. Gemini Embedding 2 addresses this with unified embeddings for arbitrary combinations of video, audio, image, and text, achieving state-of-the-art performance across diverse embedding benchmarks.

  • Problem

    Existing embedding models have limited ability to represent mixed-modality inputs and interactions coherently across modalities.

  • Method

    Gemini Embedding 2 uses Gemini-based multimodal representations and multi-task training to embed arbitrary combinations of video, audio, image, and text in one space.

  • Results

    Gemini Embedding 2 achieves state-of-the-art performance across multilingual, code, cross-modal, and multimodal retrieval benchmarks.

  • Takeaways & Limitations

    The unified representations support downstream uses including document retrieval, video recommendation, audio search, and RAG.

  • Takeaways & Limitations

    The paper identifies the need for novel evaluation frameworks to benchmark expanding interleaved multimodal applications.

Abstract

from arXiv · show

We introduce Gemini Embedding 2, a native multimodal embedding model that allows embedding video, audio, image, and text modalities in a unified representation space. We leverage the multimodal capabilities of Gemini to produce embeddings for arbitrary combinations of interleaved inputs across all these modalities that generalize well across a wide variety of tasks. Applying large-scale contrastive learning in a multi-task multi-stage training setup, we achieve state-of-the-art performance on key embedding benchmarks including unimodal, cross-modal, and multimodal retrieval spanning a diverse set of tasks. We show that our embedding model demonstrates strong performance (with a score of 62.9 R@1 on MSCOCO, 68.8 NDCG@10 on Vatex, 69.9 on MTEB multilingual and 84.0 on MTEB Code) across a variety of tasks surpassing the performance of specialized models. These unified capabilities make Gemini Embedding 2 a promising candidate for downstream use cases such as RAG, recommendation and search. Furthermore, its robust zero-shot performance across distinct fields - from astronomy and bioscience to fine arts and the culinary arts - establishes it as a highly reliable, out-of-the-box representation even for specialized domains.

1. Introduction

Gemini Embedding 2 is introduced as a general-purpose multimodal embedding model that maps heterogeneous inputs and arbitrary modality combinations into one representation space. It uses Gemini capabilities and multi-task training to capture richer cross-modal interactions, achieving state-of-the-art performance across diverse benchmarks.

  • Motivation: Native multimodal embeddings are motivated by the need for coherent semantic representations within and across modalities for downstream applications.The introduction connects this capability to broad application performance enabled by dense semantic vector representations.
  • Contributions: Gemini Embedding 2 embeds video, audio, image, text, and arbitrary combinations thereof into a single representation space.The model is designed to capture interactions across modalities rather than treating them only through separate modality-specific encoders.
  • Motivation: Existing late-fusion models such as CLIP [1], ALIGN [2], SigLIP 2, and CoCa handle mixed-modality inputs poorly because they do not exploit interactions between modalities.These models use paired cross-modal data and modality-specific encoders, yielding good unimodal and cross-modal capabilities but less rich mixed-modality representations.
  • Evaluation: The model achieves state-of-the-art performance across a wide variety of academic-focused and enterprise-focused benchmarks.Evaluation includes multilingual text embedding tasks spanning retrieval, clustering, classification, and other downstream use cases.

2. Related Work

Related work has progressed from encoder-only text embedders and narrow image–text dual towers toward instruction-tuned, multimodal models spanning diverse modalities and enterprise documents. Gemini Embedding 2 is positioned as unifying these previously separate capabilities in one out-of-the-box model.

  • Large Language Models as Text Embedders: Text embedding models evolved from encoder-only architectures such as BERT [11] and RoBERTa [12] to decoder-only and massive LLM backbones, with BGE [13] and E5 [14] using task-specific prefixes.These instruction-tuned representations unify semantic search, clustering, and classification within a single model.
  • Evolution of Multimodal Embedders: Multimodal embedders expanded from CLIP [1] and ALIGN [2] dual towers trained on image–text pairs toward unified semantic spaces covering text, code, images, documents, audio, and video.The earlier paradigms were limited by narrow contrastive objectives over simple image–text pairs.
  • Architectural Adaptations for Bidirectional Attention: Architectural adaptations address causal LLMs’ unidirectional attention, which limits dense context-aware embeddings, through methods such as MoCa [21] and modality-aware continual pre-training.MoCa [21] uses a joint reconstruction objective to denoise interleaved text and image inputs.
  • Adaptation to Enterprise Use Cases: Enterprise-focused embedders process massive document contexts using specialized visual-document methods for complex PDFs, charts, and tables, while RAG quality also depends on pipeline choices such as chunking.These requirements arise from growing enterprise and agentic workloads centered on documents.
  • Novelty: Gemini Embedding 2 unifies multi-stage distillation, LLM backbone adaptation, and enterprise use cases that preceding architectures largely addressed in isolation.The paper presents this unification as supporting broad use cases with an out-of-the-box model.

3. Multimodal Gemini Embedding

Gemini Embedding 2 builds unified representations for individual and combined modalities using a Gemini-initialized transformer, modality-aware tokenization, and pooled embeddings. Its multi-task, multi-stage contrastive training combines modality-specific tasks, cross-modal tasks, dimensionality support, and checkpoint averaging.

  • Model Architecture: Gemini Embedding 2 creates holistic representations for multimodal inputs, supporting retrieval, clustering, classification, and ranking across different modalities.The model leverages Gemini’s multimodal and cross-modal capabilities and is initialized from Gemini to use knowledge already present in its parameters.
  • Model Architecture: Raw images, video, and audio are converted into token sequences using Gemini-supported format conversions, then processed by a bidirectional Gemini-initialized transformer and mean pooling.For an input sequence of L tokens, the transformer produces token embeddings before the pooler generates a single d_M-dimensional representation.
  • Training Objective: The model uses multi-task, multi-stage training spanning single-modality, multimodal, and cross-modal tasks with noise-contrastive estimation and in-batch negatives.Text-only tasks may include task strings such as question answering or fact checking, which are randomly dropped to improve robustness when task strings are unavailable.
  • Training Objective: Gemini Embedding 2 produces 3,072-dimensional embeddings, while MRL trains overlapping losses optimized for 768- and 1,536-dimensional subspaces.This adaptation supports using different embedding dimensions with a single model.
  • Training Recipe: Training proceeds through pre-fine-tuning on noisy image, text, and code pairs, fine-tuning across six modalities with task-specific batches, and model soup checkpoint averaging.Pre-fine-tuning adapts autoregressive parameters for encoding, while model soup averages checkpoints from individual fine-tuning runs to improve generalization across modalities.

4. Evaluation

Gemini Embedding 2 achieves strong or state-of-the-art performance across unimodal, cross-modal, and multimodal retrieval, while maintaining text-only quality and enabling robust zero-shot evaluation without manual prompt engineering. Native multimodal processing is especially beneficial for audio retrieval, outperforming transcription-based encoding across same-language and cross-lingual settings.

  • Multimodal retrieval: Gemini Embedding 2 achieves the highest global mean score and leads on unimodal image, text-to-image, image-to-text, and text-to-video retrieval tasks.It generalizes to Vatex, MSR-VTT, and YouCook2 without dataset-specific in-domain training splits, with particularly strong results on DOCCI and TextCaps.
  • Multimodal retrieval: 64.9 on ViDoRe V2 places Gemini Embedding 2 ahead of Amazon Nova MME (60.6) and near Voyage-3.5-multimodal (65.5) for document retrieval.The task requires page-level visual structure, layout, and embedded-text understanding; Gemini Embedding 2 also supports the full Video/Audio/Image/Text modality set.
  • Text and code benchmarks: 69.9 on MMTEB exceeds the previous text-only Gemini Embedding model’s 68.32 mean, showing that expanded multimodal capabilities do not compromise textual performance.Gemini Embedding 2 also sets new state-of-the-art performance levels on MTEB Code v1 and CoIR, outperforming the previous text-only model.
  • Audio retrieval: +2.0 points in PassageInLang (75.58 vs. 73.58) and +5.01 points in PassageCrossLang (72.56 vs. 67.55) show consistent native-audio gains across linguistic conditions.The larger cross-lingual improvement supports robust semantic alignment beyond the phonetic constraints of an intermediate ASR transcript.

5. Ablation Study

The ablations show that Gemini Embedding 2’s performance benefits from Gemini-generated training data, additional video training, and targeted fine-tuning, while model souping preserves robustness across tasks. Zero-shot evaluations further show state-of-the-art, domain-robust multimodal retrieval across specialized fields.

  • Zero-shot domain evaluation: Gemini Embedding 2 achieves state-of-the-art zero-shot image-to-text retrieval across all evaluated specialized domains, with R@5 scores of 64.4 in astronomy and 79.3 in microscopy.On Recipe1M, it reaches 90.2 for ingredient retrieval and 92.1 for instruction retrieval, surpassing SigLIP2-Giant’s 81.2 and 80.4.
  • Synthetic-data ablation: Gemini Embedding 2 improves over the text-only Gemini Embedding baseline even before synthetic data is added, demonstrating the benefit of the multimodal model.Gemini was also used to synthesize high-quality training data for the MTEB Code ablation tasks.
  • Model souping: Model souping balances task-specific gains with the original model’s robustness, in several cases outperforming the baseline across tasks.Targeted data can improve in-domain performance while slightly degrading out-of-domain tasks, including a 0.6% dip on YouCook2.
  • Fine-tuning ablation: Fine-tuning improves performance over pre-fine-tuning across almost all image and video benchmarks, with the largest gains concentrated in video tasks.Image improvements are consistent but relatively modest, attributed to additional video training data in the fine-tuned checkpoint.
  • Targeted-data ablation: Adding only a few thousand steps and modest O(k) quantities of targeted in-domain data significantly improves the corresponding tasks, including MSR-VTT reaching 76.1%.The passage also reports that adding MSR-VTT and Vatex training splits pushes those task metrics upward, but the Vatex value is truncated in the supplied text.

6. Future Work

Future work will explore Gemini Embedding 2 for enterprise applications enabled by native multimodal representations, while incorporating ranking signals from search systems to improve retrieval.

  • 6. Future Work: Native multimodal capabilities could support enterprise use cases such as agentic RAG, video recommendation, and interleaved multimodal retrieval.These applications would avoid converting inputs into intermediate modalities.
  • 6. Future Work: Incorporating ranking and other signals from search systems is proposed as a way to improve embedding retrieval capabilities.

7. Conclusion

Gemini Embedding 2 is presented as a state-of-the-art multimodal successor to Gemini’s text-only embedding model. It produces embeddings for arbitrary combinations of interleaved text, image, audio, and video inputs, generalizing across diverse tasks.

  • Gemini Embedding 2 is presented as a state-of-the-art multimodal successor to the text-only Gemini Embedding model.
  • It produces embeddings for arbitrary combinations of interleaved text, image, audio, and video inputs.
  • The model generalizes well across a wide variety of tasks by leveraging Gemini’s multimodal, multilingual, and code-centric foundations.

8. Full Results

This section presents Gemini Embedding 2’s full results on the MTEB Multilingual and MTEB Code benchmarks.

  • The full MTEB Multilingual results for Gemini Embedding 2 are reported in Table 7.
  • The full MTEB Code results for Gemini Embedding 2 are reported in Table 8.

9. Contributions and Acknowledgments

The paper identifies its core contributors, leadership team, and additional contributors and acknowledgements by name.

  • Core Contributors: Core contributors include Madhuri Shanbhogue and Gustavo Hernández Ábrego, marked as making equal contributions, alongside the broader contributor group.The remaining core contributors are listed across the contributor roster.
  • Core Contributors: Additional contributors listed are Dahun Kim, Rui Meng, Emma Wang, Jack Zheng, Halley Fede, Zhen Yang, Keegan Mosley, Brian Potetz, Sahil Dua, Henrique Schechter, Vera Shen, Gao, and Hesen Zhang.
  • Leadership: The leadership team comprises Raphael Hoffmann, Zach Gleicher, Yunhsuan Sung, Qin Yin, Tom Duerig, and Mojtaba Seyedhosseini.
  • Acknowledgement: The acknowledgements thank James Gan, Jon Matthews, Luciano Martins, Patrick Löber, Anna Kelly, Kristen Quan, Roxanne Daniel, Ryan Trostle, Tania Bedrax-Weiss, Srinivasan (Cheenu) Venkatachary, Howard Zhou, and Tomas Izo.
Loading 2605.27295v1…