Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini

1. Introduction

• Late-fusion models provide unimodal and cross-modal capabilities but struggle with mixed-modality inputs because they do not exploit interactions between modalities.CLIP [1], ALIGN [2], SigLIP 2, and CoCa train modality-specific encoders using paired cross-modal data. Coherent semantics within and across modalities motivate richer representations for applications such as video recommendation and document search.Source: §1 · pp. 1–2
• Gemini Embedding 2 leverages Gemini [5] and diverse multi-task training to embed text, images, video, audio, and arbitrary combinations into one representation space.Training captures interactions between modalities; multimodal large language models enable richer embeddings through deep fusion of cross-modal interactions.Source: §1 · p. 2
• Native audio and multimodal understanding outperform text-based alternatives such as ASR or captioning.The introduction reports this comparison without providing numerical results.Source: §1 · p. 2
• Interleaved image, text, and video inputs enable retrieval of specific temporal events in videos using combined visual and textual prompts.The curated training tasks also target document retrieval, video recommendation, audio-based search, and RAG applications [6].Source: §1 · p. 2
• Gemini Embedding 2 achieves state-of-the-art multilingual and code embedding performance, alongside strong cross-modal retrieval results and generalization to specialized domains.Evaluation spans academic and enterprise benchmarks, including the Massive Multilingual Text Embedding Benchmark (MMTEB) [7] and cross-modal retrieval benchmarks MSCOCO [8], Flickr30k [9], and MSR-VTT.Source: §1 · p. 2, Figure 2 · p. 3

Large Language Models as Text Embedders

• Text embedding research has shifted from encoder-only architectures toward instruction-tuned representations and LLM-based training, illustrated by BGE [13], E5 [14], Gecko [15], and NV-Embed [16].BGE [13] and E5 [14] unify downstream tasks through task-specific prefixes, while Gecko [15] uses two-step distillation from massive LLM teachers. NV-Embed [16] combines instruction-tuned contrastive learning with synthetic non-retrieval data; Gemini Embedding [18] achieves state-of-the-art MMTEB performance using synthetic data and strong multilingual generalization.Source: §2 · p. 3

Evolution of Multimodal Embedders

• Multimodal embeddings are evolving beyond image–text dual-tower models such as CLIP [1] and ALIGN [2] toward unified semantic spaces through multi-stage contrastive training.SAIL-Embedding [19] uses content-aware progressive training for industrial recommendation. Amazon Nova MME and SigLIP 2 demonstrate strong cross-modal retrieval performance.Source: §2 · p. 4

Architectural Adaptations for Bidirectional Attention

• MoCa [21] addresses causal attention limitations through modality-aware continual pre-training and joint reconstruction that denoises interleaved text and images to encourage bidirectional context-aware reasoning.MM-Embed instead addresses modality bias through modality-aware hard negative mining, preventing text-to-text similarity from dominating cross-modal relevance.Source: §2 · p. 4

Adaptation to Enterprise Use Cases

• Enterprise document embedding requires efficient processing of large contexts and complex visual documents, making RAG quality dependent on processing-pipeline choices such as chunking.Specialized visual-document processing, including tiled mixtures of vision encoders, supports PDFs, charts, and tables.Source: §2 · p. 4

2. Related Work

• Gemini Embedding 2 unifies capabilities from multi-stage distillation, LLM backbone adaptation, and enterprise applications into one model usable out-of-the-box across diverse use cases.The paper contrasts this integration with preceding architectures that predominantly address these capabilities in isolation.Source: §2 · p. 4

3.1. Model Architecture

• Gemini initialization supplies pretrained knowledge, while task-specific, modality-specific, and cross-modality fine-tuning builds holistic representations using Gemini’s native conversions for raw image, video, and audio inputs.The representations support retrieval, clustering, classification, and ranking.Source: §3.1 · pp. 4–5
• A Gemini-initialized bidirectional transformer produces token embeddings, which mean pooling combines before a randomly initialized linear projection maps them to the target embedding dimension.Mean pooling averages token embeddings along the sequence axis.Source: §3.1 · p. 5

3.2. Training Objective

• Noise-contrastive estimation with in-batch negatives [24] trains single-modality, multimodal, and cross-modal tasks, while random omission of text task strings targets robustness when those strings are absent.The exact loss varies by task; text task strings describe purposes such as question answering or fact checking.Source: §3.2 · p. 5
• 3,072-dimensional embeddings use MRL [25] losses over overlapping embedding prefixes, with support optimized for 768 and 1,536 dimensions.Separate losses operate on prefixes such as the first 768 and first 1,536 dimensions, enabling multiple output dimensions within one model.Source: §3.2 · p. 5

3.3. Recipe

• Pre-fine-tuning adapts generation to encoding using noisy image, text, and code pairs; fine-tuning adds document, audio, and video tasks with task-specific batch sizes.Pre-fine-tuning uses large batches to stabilize gradients; both stages construct single-task batches using specified sampling rates. Fine-tuning often uses query–target–hard-negative triplets, aligns modalities through single- and cross-modality batches, and is sensitive to sampling rates and batch sizes.Source: §3.3 · p. 6
• Model soup averages fine-tuning checkpoint parameters to improve generalization across modalities, exploring same-run checkpoint averaging [26], cross-run averaging [27], and weighted combinations.The supplied passages describe these combinations but do not quantify their gains or identify a winning combination.Source: §3.3 · p. 6

4.1. Multimodal Retrieval

• Gemini Embedding 2 achieves the highest global mean score and leads unimodal image, text-to-image, image-to-text, and text-to-video retrieval comparisons.Results are particularly strong on long-caption benchmarks DOCCI and TextCaps. Performance generalizes to Vatex, MSR-VTT, and YouCook2 without using those datasets’ specific in-domain training splits.Source: Table 1 · p. 7, §4.1 · p. 7
• 64.9 on ViDoRe Benchmark V2 [35] places Gemini Embedding 2 above Amazon Nova MME’s 60.6 and near Voyage-3.5-multimodal’s 65.5.This document retrieval benchmark requires understanding page-level visual structure, layout, and embedded text. Gemini Embedding 2 and Amazon Nova MME are the comparison’s only models supporting video, audio, image, and text.Source: Table 1 · p. 7, §4.1 · pp. 7–8

4.2. MMTEB

• 69.9 Mean (by task) for Gemini Embedding 2 surpasses the previous text-only Gemini Embedding model’s 68.32 on MMTEB.Gemini Embedding 2 also outperforms other multimodal models on this text-only benchmark. MMTEB [7] covers 250+ languages and 10 task types.Source: Table 2 · p. 8, §4.2 · p. 8
• Gemini Embedding 2 sets new state-of-the-art results on MTEB Code v1 [7] and CoIR [40], outperforming its text-only predecessor and voyage-code-3.MTEB Code v1 [7] contains 12 code retrieval tasks across 15 coding languages; CoIR [40] contains 10 tasks across 9 coding languages.Source: Table 2 · p. 8, §4.2 · p. 8

4.3. MSEB

• The MSEB [41] evaluation focuses on retrieving relevant text documents from a large corpus using spoken queries.The evaluation uses the benchmark’s retrieval split to assess Gemini Embedding 2’s auditory capabilities.Source: §4.3 · p. 9

4.3.1. Experimental Setup

• The audio experiment contrasts ASR transcription followed by text encoding against direct raw-audio encoding to isolate the unified architecture’s impact.The principal metric is mrr@10, evaluated separately on PassageInLang for same-language retrieval and PassageCrossLang for cross-language retrieval.Source: §4.3.1 · p. 9

4.3.2. Results

• 73.99 average retrieval mrr@10 with native audio exceeds the ASR-based approach’s 70.40 on MSEB.PassageInLang improves by +2.0 points (75.58 vs. 73.58), while PassageCrossLang improves by +5.01 points (72.56 vs. 67.55). Direct encoding preserves acoustic ambiguity, prosody, intonation, and emphasis rather than propagating incorrect transcription decisions.Source: Table 3 · p. 9, §4.3.2 · pp. 9–10

5.2. Impact of synthetic data

• +15.81 points on average over Gemini Embedding [18] are reported across three code-retrieval tasks after adding Gemini-generated synthetic training data.These ablations exclude souping and cover CodeFeedbackMT [47], SyntheticText2SQL, and CodeFeedbackST [40], with especially pronounced gains on CodeFeedbackMT [47]. The multimodal model already improves over its text-only predecessor before synthetic data is added.Source: Table 5 · p. 11, §5.2 · p. 11

5.1. Generalization to specialized domains

• 64.4 R@5 in astronomy and 79.3 in microscopy accompany state-of-the-art zero-shot image-to-text retrieval across all evaluated specialized domains.The evaluation also covers fine art and culinary retrieval. Culinary ingredient and instruction retrieval reach 90.2 and 92.1 R@5, respectively, versus SigLIP2-Giant’s 81.2 and 80.4.Source: §5.1 · pp. 10–11
• Gemini Embedding 2 maintains consistent cross-domain alignment, unlike baseline families whose performance fluctuates sharply between specialized domains.TIPS-G14 from the TIPS [46] family scores 65.2 R@5 on ArtCap but 20.0 on MicroVQA. The SigLIP2 lineage reaches 81.2 on Recipe1M but drops to 8.4 on ArtCap.Source: §5.1 · p. 11

5.3. Impact of Fine-Tuning and Pre-Fine-Tuning

• Final fine-tuning improves over pre-fine-tuning on almost all evaluated image and video benchmarks, with the largest gains attributed to additional video training data.Image-task improvements are consistent but relatively modest.Source: Figure 3 · p. 12, §5.3 · p. 12

5.4. Impact of In-Domain Video Data

• 76.1% MSR-VTT and 79.5% Vatex NDCG@10 follow targeted fine-tuning, while model souping balances task-specific gains with the original model’s robustness.One epoch over O(k) added data requires only a few thousand training steps, but YouCook2 falls to 55.3%. Interpolating the fine-tuned and original weights yields improvements over baseline across tasks in several cases.Source: Table 6 · p. 12, §5.4 · pp. 12–13

• Native multimodal capabilities could support agentic RAG, video recommendation, and interleaved multimodal retrieval without conversion to intermediate modalities.Source: §6 · p. 13
• Incorporating search-system signals such as ranking could improve embedding retrieval capabilities by leveraging highly capable LLM backbones.Source: §6 · p. 13
• End-to-end RAG training could fine-tune embeddings for enterprise use cases, motivated by agentic RAG applications.Source: §6 · p. 13
• The authors invite the academic community to develop evaluation frameworks that benchmark emerging interleaved multimodal capabilities.Source: §6 · p. 13

7. Conclusion

• Gemini Embedding 2 achieves landmark performance on MSCOCO, Vatex, and MMTEB, with a particularly significant improvement in code retrieval.The conclusion positions it as a state-of-the-art multimodal successor to the text-only Gemini Embedding model.Source: §7 · p. 13
• Gemini Embedding 2 embeds arbitrary combinations of interleaved text, image, audio, and video inputs, generalizing across a wide variety of tasks.It leverages Gemini’s multimodal, multilingual, and code-centric foundations.Source: §7 · p. 13
• Native audio input outperforms traditional ASR in retrieval tasks, while eliminating costly task-specific instructions further supports the model’s efficiency.Source: §7 · p. 13
• The model excels across specialized domains, including microscopy, astronomy, and the culinary arts, alongside its general-task capabilities.Source: §7 · p. 13
• The unified embedding approach enables cross-data retrieval, providing infrastructure for next-generation agentic systems operating in tandem with Gemini.Source: §7 · p. 13

8. Full Results

• Full-results tables cover Gemini Embedding 2 on MTEB (Multilingual) and MTEB(Code).Only table captions are supplied, so benchmark scores and comparative performance cannot be summarized.Source: Table 7 · p. 18, Table 8 · p. 19

Core Contributors

• Madhuri Shanbhogue, Zhe Li, Gustavo Hernández Ábrego, Shih-Cheng Huang, and Aashi Jain are credited with equal contributions.An asterisk denotes equal contributions.Source: §9 · p. 20
• Additional contributors include Daniel Salz, Sonam Goenka, Chaitra Hegde, Ji Ma, Feiyang Chen, and Jiaxing Wu.The extended roster also names contributors including Daniel Cer, Georg Heigold, and Wenlei Zhou.Source: §9 · p. 20

Leadership

• Leadership credits name Raphael Hoffmann, Zach Gleicher, Yunhsuan Sung, Qin Yin, Tom Duerig, and Mojtaba Seyedhosseini.Source: §9 · p. 20