Gemini: A Family of Highly Capable Multimodal Models

1. Introduction

• Gemini models are jointly trained across image, audio, video, and text to combine broad multimodal abilities with domain-specific understanding and reasoning.Source: §1 · p. 1
• Gemini 1.0 provides Ultra for highly complex tasks, Pro for scalable deployment, and Nano for on-device applications.The sizes address different computational limitations and application requirements.Source: §1 · p. 1
• Post-training produces chat-focused Gemini Apps models and developer-focused Gemini API models while improving quality, target capabilities, alignment, and safety.Source: §1 · p. 1
• Gemini Ultra achieves new state-of-the-art results on 30 of 32 reported benchmarks, including text, image, video, and speech tasks.The reported breakdown is 10 of 12 text and reasoning, 9 of 9 image, 6 of 6 video, and 5 of 5 speech benchmarks.Source: §1 · p. 2
• Gemini Ultra demonstrates crossmodal reasoning by interpreting handwritten physics work, locating an error, and producing a corrected worked solution.The example requires understanding images, problem formulation, mathematical typesetting, and reasoning instructions.Source: §1 · p. 2, Figure 1 · p. 3
• Gemini’s reasoning capabilities also support more general agents: AlphaCode 2 ranks within the top 15% of Codeforces entrants, versus the predecessor’s top 50%.Source: §1 · p. 2

2. Model Architecture

• Gemini models build on optimized Transformer decoders supporting 32k context length and efficient attention for stable training and inference at scale.Source: §2 · p. 3
• Gemini natively interleaves text with images, audio, and video as input and can produce text and image outputs.The models are multimodal from the beginning and use discrete image tokens for image generation.Source: §2 · p. 3, Figure 2 · p. 4
• Video is encoded as frames in the context window, while variable input resolution allocates more computation to fine-grained visual tasks.Source: §2 · p. 3
• Gemini directly ingests 16kHz audio features, preserving nuances that can be lost when audio is first mapped to text.Source: §2 · p. 4
• Nano models use distillation and training advances to provide small models for tasks such as summarization and reading comprehension on devices.Source: §2 · p. 4

3. Training Infrastructure

• At Gemini Ultra scale, hardware failures become common, requiring multi-datacenter TPU deployment and strategies for hot standbys, rolling maintenance, and recovery.Source: §3 · p. 5
• SuperPods are connected across datacenters to support synchronous training with model parallelism within superpods and data parallelism across them.Source: §3 · p. 5
• Jax, Pathways, GSPMD, and MegaScale XLA coordinate distributed computation and overlap communication with computation while maintaining consistent step times.Source: §3 · p. 5
• Redundant in-memory model-state copies increase the largest-scale training job’s goodput from 85% to 97%.Recovery proceeds directly from an intact replica after unplanned hardware failures, avoiding conventional persistent-storage checkpointing.Source: §3 · p. 5
• Deterministic replay, proactive silent-data-corruption scanners, and hot standbys rapidly detect faulty hardware and support stable training.The infrastructure was designed for silent data corruption events expected every week or two at this scale.Source: §3 · p. 5

4. Pre-Training Dataset

• Gemini’s pre-training dataset combines multilingual and multimodal data from web documents, books, code, images, audio, and video.Source: §4 · p. 5
• Training the tokenizer on a large sample of the full corpus improves vocabulary inference and supports efficient tokenization of non-Latin scripts.The authors report benefits for model quality, training speed, and inference speed.Source: §4 · pp. 5–6
• Dataset mixtures are quality- and safety-filtered, checked for evaluation contamination, and staged to increase domain-relevant data toward the end of training.Source: §4 · p. 6
• Mixture weights are selected through ablations on smaller models, while smaller models receive significantly more training tokens to improve performance under inference-budget constraints.Source: §4 · p. 6
• The authors identify data quality as important for high-performing models while leaving the optimal pre-training dataset distribution as an open question.Source: §4 · p. 6

5. Evaluation

• Gemini Ultra sets state-of-the-art results across academic, image, and video benchmarks, demonstrating strong capabilities across text and multimodal tasks.The evaluation covers reasoning, STEM, coding, image understanding, temporal video reasoning, and multimodal tasks.Source: §5 · p. 6, §5.1.1 · p. 6, §5.2.1 · p. 14

5.1.1. Academic Benchmarks

• 90.04% accuracy on MMLU lets Gemini Ultra exceed human-expert performance of 89.8% and the prior state of the art of 86.4%.Its strongest MMLU performance uses uncertainty-aware chain-of-thought consensus over 8 or 32 samples.Source: §5.1.1 · p. 6
• 74.9% on Natural2Code and 74.4% on HumanEval show Gemini Ultra’s leading Python-generation performance, while it reaches 94.4% on GSM8K and 53.2% on MATH.On the AMC-derived task, Gemini Ultra solves 32% of questions versus GPT-4’s 30%.Source: §5.1.1 · pp. 6–7

5.1.2. Trends in Capabilities

• Gemini Ultra is best across all six capabilities, while quality gains with increased model size are especially evident in reasoning, math/science, summarization, and long-context.Gemini Pro is also competitive while being more efficient to serve.Source: §5.1.2 · p. 8

5.1.4.1 Machine Translation

• 74.8 BLEURT on WMT23 out-of-English translation exceeds GPT-4’s 73.6 and PaLM 2’s 72.2, while Gemini Ultra reaches 79.0% on multilingual MGSM.It also achieves 27.0 chrF on very low-resource languages versus 25.3 for PaLM 2-L, and 17.6 ROUGE-L on XLSum versus 15.4 for PaLM 2.Source: §5.1.4.1 · p. 10, §5.1.4.2 · p. 10

5.1.5. Long Context

• 98% accuracy across the full 32K-token context shows Gemini Ultra can retrieve information throughout long inputs, while factuality post-training halves inaccuracy, raises attribution accuracy by 50%, and enables 70% accurate hedging.The factuality evaluation covers closed-book factuality, source attribution, and hedging on unanswerable prompts.Source: §5.1.5 · pp. 10–11, §5.1.6 · pp. 11–12

5.1.7. Complex Reasoning Systems

• 43% of Codeforces problems solved by AlphaCode 2 is a 1.7x improvement over AlphaCode’s 25%, corresponding to an estimated 85th-percentile competition ranking.AlphaCode 2 combines a specialized Gemini Pro with massive program search, filtering, clustering, and reranking.Source: §5.1.7 · p. 12

5.2.1. Image Understanding

• Gemini Ultra outperforms existing image-understanding approaches, advances MathVista by +3.1% and InfographicVQA by +5.2%, and exceeds the MMMU state of the art by more than 5 percentage points.It also achieves state-of-the-art video captioning and zero-shot video question answering, demonstrating temporal reasoning across frames.Source: Table 7 · p. 13, §5.2.1 · p. 14, Table 8 · p. 14

5.2.3. Image Generation

• Gemini models natively generate multiple images interleaved with text, enabling few-shot image suggestions without an intermediate natural-language description.In a one-shot example, Gemini Ultra transforms blue-and-yellow yarn examples into suggestions for a green avocado with a pink seed and a green bunny with pink ears.Source: §5.2.3 · pp. 16–17, Figure 6 · p. 17

5.2.4. Audio Understanding

• Gemini Pro significantly outperforms USM and Whisper across all evaluated ASR and AST tasks, while Gemini Nano-1 surpasses both except on FLEURS.ASR uses WER, where lower is better, and AST uses BLEU, where higher is better; Gemini Pro without FLEURS training still achieves a WER of 15.8 and outperforms Whisper.Source: §5.2.4 · p. 18, Table 11 · p. 18
• Gemini Pro produces more understandable ASR responses than USM, particularly for rare words and proper nouns.The comparison is illustrated through qualitative error analysis examples.Source: §5.2.4 · p. 18, Table 12 · p. 18

5.2.5. Modality Combination

• Gemini processes interleaved audio and images natively and reasonably accurately reasons about fine-grained visual details in an omelet-cooking interaction.The model evaluates whether the omelet is fully cooked while responding to turn-by-turn verbal questions and pictures.Source: Table 13 · p. 19, §5.2.5 · p. 19

6. Post-Training Models

• Post-training creates Gemini Apps and Gemini API variants tailored respectively to conversational services and developer products.The variants are specialized for Gemini and Gemini Advanced, or exposed through Google AI Studio and Cloud Vertex AI.Source: §6.1 · p. 20, §6.2 · p. 20

6.3. Post-Training Methods & Data

• Curated prompts support supervised fine-tuning, reward-model training from human preferences, and reinforcement learning from human feedback.The process uses diverse real-world prompts, demonstrations, candidate-response feedback, and iterative RM–RL improvement.Source: §6.3 · pp. 20–21

6.5.1. Instruction Following

• Gemini Advanced achieves close to 90% per-instruction accuracy but about 54% full-response accuracy on complex prompts.The gap indicates remaining difficulty in satisfying every sub-instruction simultaneously.Source: §6.5.1 · p. 22, Table 14 · p. 22

6.5.2. Tool Use

• Tools improve user preference outcomes, with tool-equipped models preferred 78% of the time on travel-planning and video-discovery tasks.Gemini represents tool calls as code and iteratively combines model sampling with tool execution.Source: §6.5.2 · pp. 22–23, Figure 8 · p. 23

6.5.3. Multilinguality

• Gemini Pro improves multilingual quality by more than 0.1 SxS score across all five evaluated languages.Coding and reasoning gains are preserved across languages, supported by localized and translated post-training data.Source: §6.5.3 · pp. 23–24, Table 16 · p. 24

6.5.4. Multimodal Vision

• RLHF improves multimodal image understanding by +0.223±0.06 SxS over SFT alone while preserving text-only quality at +0.01±0.01.SFT also improves several Gemini API vision benchmarks, including InfographicVQA, AI2D, and VQAv2.Source: §6.5.4 · p. 25, Table 17 · p. 25

6.5.5. Coding

• Post-training data significantly boosts coding quality and correctness, with Gemini Advanced outperforming Gemini Pro on internal coding evaluations.The coding data combines human and synthetic collection approaches.Source: §6.5.5 · p. 26, Table 18 · p. 26

7. Responsible Deployment

• The responsible-deployment process addresses foreseeable downstream societal impacts throughout the lifecycle of Gemini development.The report presents this approach as ongoing and commits to updated transparency about testing and practices.Source: §7 · p. 26

7.1. Impact Assessment

• Google assesses Gemini risks and impacts at both the model level and within broader products or services.Model assessments cover modalities including text-to-text, image-to-text, and video-to-text.Source: §7.1 · pp. 26–27, §7.1.1 · p. 27

7.1.1. Model Assessment

• Assessments draw on literature, external expertise, and in-house ethics and safety research, with review by the Google DeepMind Responsibility and Safety Council.This evidence base supports documenting societal benefits and harms across Gemini modalities.Source: §7.1.1 · p. 27
• Model assessments consider benefits such as information processing and accessibility alongside content, child-safety, representation, surveillance, environmental, economic, and dangerous-capability risks.The report describes evaluations across these domains as part of ongoing risk research.Source: §7.1.1 · p. 27

7.1.2. Product Assessments

• Product assessments and red teaming guide mitigations and deployment decisions before launch.For Gemini Advanced, testing covered safety, accountability, and inclusion; API Ultra reviews added enterprise-specific evaluations and safeguards.Source: §7.1.2 · pp. 27–28
• Product safeguards include accuracy disclosures, professional-advice warnings, feedback channels, safety filters, and developer guidance.These measures are implemented across Gemini products and Cloud-based API offerings.Source: §7.1.2 · pp. 27–28

7.2. Safety Policies

• Gemini model policies define standardized safety criteria and prioritization for responsible development and launch readiness.Products using Gemini apply an additional product-policy framework, including extra coverage for users under 18.Source: §7.2 · p. 28
• Policy areas include child sexual abuse content, hate speech, harassment, dangerous or malicious content, and bias reduction.The policies apply across consumer and enterprise contexts.Source: §7.2 · p. 28
• Gemini policies prioritize neutral answers grounded in authoritative consensus facts or multiple perspectives where consensus is absent.Source: §7.2 · p. 28

7.3.1. Data Curation Practices

• Training data is filtered for high-risk content, while human data creators and evaluators are considered across gender, age, and racial and ethnic diversity.The data-enrichment process also includes contractual requirements that workers receive at least a local living wage.Source: §7.3.1 · pp. 28–29

7.3.2. Model Mitigation

• Safety mitigation primarily uses supervised fine-tuning and RLHF focused on adversarial or harm-inducing queries.This safety-focused post-training is distinct from generic quality-oriented post-training for all user queries.Source: §7.3.2 · p. 29

7.3.2.1 Harm-inducing queries

• Approximately 20 harm types are covered using queries crafted by experts, generated by language models, and discovered through automated red teaming.The categories include hate speech, ungrounded medical advice, and dangerous behavior.Source: §7.3.2.1 · p. 29

7.3.2.2 Supervised fine-tuning

• Safety SFT combines human collections with policy-conditioned response generation and selection, using different constitutions for different harm types.For sensitive controversial conversations such as elections, the recipe encourages neutral responses.Source: §7.3.2.2 · p. 29
• Safety tuning must balance harmlessness with helpfulness while adapting quickly to changing harmful-query patterns and generalizing beyond observed examples.The authors use more advanced chain-of-thought recipes to operate at the level of safety-policy concepts rather than individual harm examples.Source: §7.3.2.2 · pp. 29–30

7.3.2.4 Beyond the general recipe

• Additional mitigations include locale-specific expertise, multimodal safety SFT, and internal harm-inducing query evaluations that track safety and helpfulness over time.Text-only safety SFT was ineffective for some harmful queries containing both text and images.Source: §7.3.2.4 · pp. 30–31

7.4. Safety Evaluations

• Safety evaluations run at model and product levels, with development, assurance, external, and red-team testing performed at risk-appropriate cadences.Assurance sets are held out, while policy evaluations cover every post-trained checkpoint released into Google product areas.Source: §7.4 · pp. 31–32

7.4.1.1 Content safety

• Text-to-text policy violation rates improved sequentially, but medical advice and harassment remained areas with particular room for improvement.Ultra and Pro showed similar safety profiles in these tests.Source: §7.4.1.1 · p. 33
• Image-to-text evaluations initially found violative captions for adversarial images, motivating dedicated multimodal mitigation and later notable improvements in Pro and Ultra.The work identifies diverse image sourcing and tooling for categorizing violative multimodal content as continuing challenges.Source: §7.4.1.1 · p. 33

7.4.1.2 Representational harms

• Many representational-harm datasets quickly saturate near 99% accuracy, motivating measures that go beyond binary gender and common stereotypes.On BBQ, the average bias score stayed close to zero, while average toxicity on Real Toxicity Prompts fluctuated around 6%.Source: §7.4.1.2 · p. 34
• Representational-harm evaluations found ungrounded inferences in image-to-text and video-to-text, without consistent evidence that Gemini targeted one group more than another.The evaluation suite uses datasets with defined desirable-versus-harmful behavior, but the authors note that further work is necessary.Source: §7.4.1.2 · pp. 34–35

7.4.1.3 Dangerous capabilities

• Dangerous-capability tests found mixed risks: models solved some entry-level cybersecurity tasks, showed mixed persuasion results, lacked most self-proliferation capability, and were generally unable to notice infrastructure opportunities without hints.The CBRN results suggested the models were unlikely to provide information leading to catastrophic harm.Source: §7.4.1.3 · p. 35

7.4.2. Gemini Advanced

• Gemini Advanced evaluations incorporate safety filtering, diverse user journeys, regional sensitivities, and adversarial testing across critical policy areas.Evaluation sets draw on red-teaming, responsibility experts, real-world data, and human-curated language-model augmentation.Source: §7.4.2 · p. 36

7.4.3.1 Model-level Red Teaming

• Model-level red teaming combines adversary simulations with structured sociotechnical testing across security, safety, privacy, and social harms.Testing includes realistic attacker scenarios and comparisons across levels of adversarial pressure.Source: §7.4.3.1 · pp. 36–37
• Early models were vulnerable to simple jailbreak and prompt-injection attacks that elicited affirmative responses involving violence, self-harm, and dangerous substances.The finding enabled mitigation in subsequent models.Source: §7.4.3.1 · p. 36
• Structured red teaming found vulnerabilities across content-policy areas, with a higher proportion of successful attacks in highly adversarial settings.The approach explicitly examines disproportionate impacts across demographic groups and uses expert input, including lived experience and medical expertise.Source: §7.4.3.1 · p. 37

7.4.3.2 Gemini Advanced

• Gemini Advanced was tested by 164 Google testers across 24 countries, with more than 1,400 submitted queries or conversations and over 100,000 aggregate safety ratings.Additional dogfooding included tens of thousands of participants and 100,000 queries or conversations in the first 14 hours.Source: §7.4.3.2 · p. 38
• Red-team and safety-evaluation findings are used iteratively to strengthen evaluations and improve model performance.The broader testing program includes priority users, power users, and external security testers for Gemini Advanced.Source: §7.4.3.2 · p. 38, §7.4.5 · p. 39

7.4.4.1 Gemini Ultra External Evaluations

• Independent external groups conducted structured evaluations, qualitative probing, and unstructured red teaming across autonomous replication, CBRN, cybersecurity, and societal risks.Groups independently designed their methodologies and reported results while testing a December 2023 Gemini API Ultra checkpoint.Source: §7.4.4.1 · p. 38

7.5. Deployment

• Approved Gemini versions receive internal model cards documenting critical performance and responsibility metrics for consistent review and external communication.External model and system cards are released through technical-report updates and enterprise documentation.Source: §7.5 · p. 39
• Online product documentation covers terms of use, model distribution and access, change control, logging, monitoring, and feedback.The listed policies include prohibited-use, service, privacy, and cloud-platform terms.Source: §7.5 · p. 39

• Gemini Ultra scores 90.0% on MMLU, surpassing human-expert performance on an exam benchmark used to measure language-model progress.Source: §8 · p. 40
• Gemini Ultra sets state-of-the-art results across several image, video, and audio benchmarks without task-specific modifications or tuning.Source: §8 · p. 40
• Gemini models support applications involving complex images, interleaved images, audio, and text, including education, problem solving, communication, summarization, extraction, and creativity.Source: §8 · p. 40
• Current LLMs still require research on hallucinations and more challenging evaluations because strong exam performance does not guarantee causal, logical, or counterfactual reasoning.Source: §8 · p. 40
• The Gemini family provides a foundation toward a large-scale modularized system with broad generalization across modalities.Source: §8 · p. 40

Gemini Leads

• Gemini’s leads covered text, multimodal vision and audio, data, fine-tuning, instruction tuning, reinforcement learning, human data, safety, evaluation, infrastructure, codebase and parallelism, ecosystem, product, and operations.The listed roles identify co-leads and leads for these areas.Source: Gemini Leads · p. 55

Gemini App Leads

• Gemini App leadership spanned core modeling, fine-tuning, factuality, instruction following, conversationality, multimodality, multilinguality, safety, coding, tool use, evaluation, serving, technical program management, and product.The section also identifies program, post-training, and overall technical leads, including Jeffrey Dean and Oriol Vinyals as equal-contribution overall technical leads.Source: Gemini App Leads · pp. 55, 69
• Contributions were organized into Lead, Core Contributor, Contributor, Program Lead, Overall Post-Training Lead, and Overall Technical Lead roles, with equal contributions within roles and randomized ordering.The section states that ordering within each role does not indicate contribution order.Source: Gemini App Leads · p. 69
• Gemini was a cross-Google effort involving Google DeepMind, Google Research, Bard/Assistant, Knowledge and Information, Core ML, Cloud, Labs, and other teams, whose support is acknowledged along with report leaders, reviewers, colleagues, and named infrastructure and engineering teams.The acknowledgments thank report preparers and reviewers and recognize teams including AlphaCode, JAX, ML SRE, ML Supercomputer, Platforms Infrastructure Engineering, and XLA Compiler.Source: Gemini App Leads · pp. 69–70
• Google colleagues supported the core Gemini team by sharing enthusiasm, providing feedback on early models, creating demo uses, and contributing across many project aspects.The acknowledgment covers people not explicitly mentioned elsewhere.Source: Gemini App Leads · p. 70

10.1. Gemini Ultra Model Card

• Gemini V1.0 comprises decoder-only Transformer variants—Nano, Pro, and Ultra—trained jointly across image, audio, video, and text data.The models support 32K context length and use efficient attention mechanisms such as multi-query attention.Source: §10.1 · p. 71
• The model card covers multimodal multilingual training data, post-training with supervised fine-tuning and RLHF, evaluation datasets, intended applications, and safety considerations.Known caveats require assessing and mitigating safety and fairness concerns before downstream deployment.Source: §10.1 · pp. 71–73

10.2. Chain-of-Thought Comparisons on MMLU benchmark

• 90.0% MMLU accuracy is achieved by Gemini Ultra with uncertainty-routed chain-of-thought using 32 samples, versus 84.0% with greedy sampling and 85.0% with chain-of-thought alone.The method selects majority votes above model-specific validation thresholds and otherwise defers to greedy sampling.Source: §10.2 · p. 74

10.3. Capabilities and Benchmarking Tasks

• The benchmarking harness spans more than 50 tasks across factuality, long context, mathematics, reasoning, summarization, multilinguality, image, video, and audio understanding.The listed evaluations include chart understanding, document and text recognition, multimodal reasoning, video captioning and question answering, and speech recognition or translation.Source: §10.3 · p. 75

10.4. Qualitative Examples

• Qualitative examples demonstrate chart reasoning, multimodal question answering, interleaved image-text generation, visual reasoning, geometry, object information seeking, humor understanding, and multilingual commonsense reasoning.The examples include recognizing plants and objects, interpreting cultural context in memes, and generating consistent dog images within a blog post.Source: §10.4 · p. 76, Figure 10 · p. 77, Figure 11 · p. 78

10.5.1. Reasoning and code generation

• Gemini follows complex website-generation instructions and converts them into functioning code for a single-file web application.The example requires query redirection, visual similarity to Google Search, an opossum image, and specified footer text.Source: §10.5.1 · p. 85, Figure 19 · p. 86

10.5.2. Mathematics: Calculus

• The appendix illustrates mathematical reasoning through calculus solutions with step-by-step explanations, multi-step calculations, and correctly calculated markdown tables.It also presents complex multimodal code generation requiring inverse graphics, mathematical transformations, and relevant code output.Source: Figure 20 · p. 87, Figure 21 · p. 88, §10.6 · p. 88

10.6.2. Video understanding and reasoning

• Gemini Ultra analyzes temporally connected video frames and provides recommendations about how actions in a soccer penalty sequence could be improved.The example complements broader reported strength on few-shot video captioning and zero-shot video question answering.Source: Figure 23 · p. 90