Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Model family

• Gemini 1.5 combines sparse and dense scaling with advances in training, distillation, and serving infrastructure to improve efficiency, reasoning, planning, multilinguality, function calling, and long-context performance.The family comprises the updated Gemini 1.5 Pro and the lighter Gemini 1.5 Flash, which is designed for efficiency with minimal quality regression.Source: §1 · p. 1

Long-context retrieval

• Both models achieve greater than 99% needle recall across text, video, and audio over multiple millions of tokens, with Pro maintaining recall when context reaches 10M tokens.Gemini 1.5 Pro reaches more than 99.7% recall at 1M tokens across all modalities and maintains this performance to 10M text tokens, 9.7M audio tokens, and 9.9M video tokens.Source: §1 · pp. 2–3, Figure 1 · p. 2

Long-context applications

• In realistic long-context tasks, Gemini 1.5 Pro outperforms competing models across modalities, including systems augmented with external retrieval.The evaluations cover retrieval and reasoning over long documents and videos, as well as long-context speech recognition, video understanding, planning, and multimodal analytics.Source: §1 · pp. 3–4

In-context learning

• With a 500-page grammar, dictionary, and approximately 400 parallel sentences, both models translate English to Kalamang at a level similar to a learner using the same materials.Adding 45 minutes of transcribed Kalamang speech also enables in-context speech-recognition learning for a language with fewer than 200 speakers.Source: §1 · pp. 3–4

Core multimodal capabilities

• Gemini 1.5 models outperform Gemini 1.0 Pro across core evaluations, with Pro winning 44/50 comparisons and Flash winning 41/50.Reported gains include Math, Science and Reasoning of +49.6% for Pro and +30.8% for Flash, alongside improvements in multilinguality, vision, documents, multimodal reasoning, code, and function calling.Source: §1 · p. 4

Overall capability results

• Gemini 1.5 Pro beats Gemini 1.0 Ultra on 35/45 benchmarks, including 18/21 vision and 16/19 text benchmarks, despite using less training compute.Gemini 1.5 Flash exceeds Ultra on 13/21 vision and 8/18 text benchmarks while being more efficient and faster to serve.Source: §1 · p. 4

• More than 10% relative improvement on average across evaluations distinguishes the May 2024 Gemini 1.5 Pro from the February version.These gains followed multiple pre-training and post-training iterations.Source: §2 · p. 4
• 58.5% to 67.7% on MATH and 41.5% to 46.2% on GPQA show substantial reasoning gains over the previous version.The cited benchmarks are MATH (Hendrycks et al., 2021b) and GPQA (Rein et al., 2023).Source: §2 · p. 4
• 52.1% to 63.9% on MathVista, 72.7% to 81.0% on InfographicVQA, and 65.1% to 72.2% on EgoSchema quantify multimodal improvements.Gemini 1.5 Pro improved on all image-understanding benchmarks and most video-understanding benchmarks.Source: §2 · pp. 4–5
• Figure 2 shows improvements across reasoning, coding, vision, and video benchmarks, while audio and translation performance remained neutral.The figure compares Gemini 1.5 Pro from May 2024 with the initial February 2024 version; for FLEURS, lower is better.Source: Figure 2 · p. 5
• Gemini 1.5 Pro achieved state-of-the-art results on AI2D, MathVista, ChartQA, DocVQA, InfographicVQA, and EgoSchema.These results extend the reported gains on multimodal benchmarks.Source: §2 · p. 5

3.1. Gemini 1.5 Pro

• Gemini 1.5 Pro is a sparse MoE Transformer whose learned routing activates only a parameter subset per input, enabling larger total capacity at constant activation.Its architecture changes support multimodal inputs and context lengths up to 10 million tokens without performance degradation, including interleaved audio, visual, text, and code.Source: §3.1 · pp. 5–6

3.2. Gemini 1.5 Flash

• Gemini 1.5 Flash is a Transformer decoder sharing Pro’s multimodal capabilities and 2M+ context, optimized for lower-latency TPU serving through parallel computation and online distillation.It is trained with higher-order preconditioned methods for improved quality.Source: §3.2 · p. 6

3.3. Serving efficiency and latency

• Over 650 characters per second makes Gemini 1.5 Flash more than 30% faster than Claude 3 Haiku for English, while Pro exceeds GPT-4 Turbo and Claude 3 models.Across all four evaluated languages, Flash is fastest; the comparison uses streaming API measurements averaged over 32 queries and reports each language’s minimum observed value.Source: §3.3 · p. 6, Table 3 · p. 6

Training Infrastructure and Dataset

• Gemini 1.5 models use multiple distributed 4096-chip Google TPUv4 pods and are pretrained on web, code, image, audio, video, multimodal, and multilingual data.Instruction tuning uses paired multimodal instructions and responses, followed by further tuning on human preference data.Source: §4 · p. 7

5. Evaluation Results

• Existing evaluations often isolate modalities or use short contexts, motivating benchmarks that measure reasoning over long mixed-modality sequences.The evaluation is organized into qualitative long-context multimodal, quantitative long-context multimodal, and quantitative core capability categories.Source: §5 · p. 7

5.1. Qualitative Examples of Multimodal Long-Context Capabilities

• Gemini 1.5 Pro handles qualitative multimodal tasks spanning a 746,152-token JAX codebase, a 732k-token Les Misérables text, a 45-minute movie, and new-language translation from reference materials.The model identifies code locations, links sketches to scenes, retrieves movie moments with second-level timestamps, and translates English to Kalamang at similar quality to a human trained on the same materials.Source: §5.1 · p. 7, Figure 3 · p. 8, Figure 4 · p. 8

5.2. Long-context Evaluations

• Gemini 1.5 Pro achieves near-perfect long-context retrieval across modalities, including 99.2% text recall at 10M tokens, successful retrieval across 10.5 hours of video, and 100% audio accuracy at 107 hours.Gemini 1.5 Flash reaches 100% text recall and >99.8% video recall at up to 2M tokens, while achieving 98.7% audio accuracy.Source: §5.2 · p. 10, §5.2.1.2 · p. 12, §5.2.1.3 · p. 13

5.2. Realistic Long-Context Evaluations

• The evaluation framework combines diagnostic probing with realistic multimodal tasks, including long-document QA, long-context speech recognition, language learning from one book, and long-context video QA.The authors note that future tests should assess prompt robustness and complex reasoning that integrates scattered facts, draws inferences, or resolves inconsistencies.Source: §5.2 · p. 10, §5.2.1.5 · pp. 15–16, §5.2.2 · p. 17

5.2.1.1 Perplexity over Long Sequences

• Gemini 1.5 models improve next-token prediction throughout tested lengths: up to 1M tokens for long documents and 10M tokens for code, with power-law behavior fitting to 1M and about 2M tokens respectively.The power-law trend deviates near 10M-token code contexts, where repeated code blocks can provide outsized benefits; this may depend on the dataset.Source: Figure 7 · p. 10, §5.2.1.1 · p. 11

5.2.1.5 Improved Diagnostics

• In multiple-needle retrieval, Gemini 1.5 Pro maintains around 70% recall to 128K tokens and >60% to 1M tokens, versus GPT-4 Turbo’s roughly 50% average recall at 128K.On MRCR, Gemini 1.5 Pro overtakes GPT-4 Turbo and Claude 3 competitors at roughly 8K–32K tokens, while both Gemini models average around 75% at 1M tokens and avoid serious degradation.Source: §5.2.1.5 · pp. 15–16, Figure 11 · p. 15, Figure 12 · p. 16

5.2.2.1 In-context language learning – learning to translate a new language from one book

• Gemini 1.5 Pro reaches 4.14 human-evaluation points for kgv→eng and 5.46 for eng→kgv, approaching the human learner’s 5.52 and 5.58 scores when learning from the full materials.In the half-book setting, Gemini 1.5 Pro outperforms GPT-4 Turbo and Claude 3 by a wide margin; eng→kgv is similar to the human learner, while kgv→eng retains a substantial qualitative gap.Source: §5.2.2.1 · pp. 18–19, Table 4 · p. 18, Table 5 · p. 18

5.2.2.2 In-context language learning - learning to transcribe speech in a new language in context

• 35.0% CER without in-context examples improves to 22.9% with text and audio context for Gemini 1.5 Pro learning Kalamang speech.The model hears the language zero-shot but initially struggles with segmentation and spelling; many-shot context improves these outputs, although grammatical errors remain, and Flash follows the same trajectory with worse scores.Source: Table 6 · p. 19, §5.2.2.2 · p. 20, Table 7 · p. 20

5.2.2.3 Scaling In-Context learning for low-resource machine translation

• +11.1/+21.4 chrF on Bemba and +9.5/+15.9 chrF on Navajo over zero-shot translation show substantial gains for 1.5 Pro/Flash as many-shot context scales.Gemini 1.5 generally improves with up to 1K or 4K examples and often surpasses GPT-4 Turbo; Flash shows more pronounced scaling, despite an initial degradation on Acholi.Source: Figure 13 · p. 20, §5.2.2.3 · p. 21

5.2.2.4 Long-document QA

• 78% of side-by-side comparisons favor full-context Gemini 1.5 Pro over Gemini 1.5 Pro with 4k-token retrieval, and 83% favor it over retrieval-augmented GPT-4 Turbo.The evaluation uses 100 questions about the full 1,462-page, 710K-token Les Misérables book and Bradley-Terry analysis of pairwise answer quality.Source: §5.2.2.4 · pp. 21, 23, Figure 14 · p. 22

5.2.2.5 Long-context Audio

• 5.5% WER is achieved by Gemini 1.5 Pro on unsegmented 15-minute videos, outperforming models that require segmentation or preprocessing.Gemini 1.0 Pro reaches 100% WER without segmentation and 7.8% with 30-second segmentation; USM reaches 8.8%, Whisper 7.3% with segmentation, and Flash 8.8%.Source: §5.2.2.5 · p. 23, Table 8 · p. 23

5.2.2.6 Long-context Video QA

• 1H-VideoQA introduces 125 five-way questions over 40–105-minute videos designed to require understanding events that sparse frame sampling may miss.The benchmark addresses the lack of existing long-video QA evaluations that differentiate models operating over different context lengths.Source: §5.2.2.6 · p. 24
• 70.2% accuracy makes Gemini 1.5 Pro state of the art on EgoSchema with 16 frames, while performance on 1H-VideoQA increases consistently as more video frames are provided.Gemini 1.5 Pro also consistently outperforms GPT-4V on 1H-VideoQA with 16 or 150 frames; Gemini 1.5 Flash is slightly behind GPT-4V with 150 frames but better with 16.Source: §5.2.2.6 · p. 25, Table 9 · p. 25

5.2.2.7 In-Context Planning

• In-context planning evaluates whether models can solve PDDL and natural-language planning tasks as additional examples are added, requiring one-shot plan generation without checking intermediate moves.The setup treats planning as both task understanding from examples and producing a complete solution in one pass.Source: §5.2.2.7 · p. 25
• Gemini 1.5 Pro outperforms other models across PDDL and natural-language few-shot planning benchmarks, using fewer examples and longer context effectively.Gemini 1.5 Flash is generally outperformed by Pro but occasionally matches GPT-4 Turbo.Source: §5.2.2.7 · pp. 25–26, Figure 16 · p. 26
• 40-shot BlocksWorld performance reaches 48% for Gemini 1.5 Pro, exceeding GPT-4 Turbo’s best 200-shot performance of 43%.In the 1-shot setting, Gemini 1.5 Pro reaches 35% and Gemini 1.5 Flash 26%, while GPT-4 Turbo is close to zero.Source: §5.2.2.7 · p. 26
• 400-shot Mini-Grid accuracy reaches 77% for Gemini 1.5 Pro, versus 38% for GPT-4 Turbo at 80 shots.Gemini 1.5 Pro reaches 28% in the 1-shot setting, while GPT-4 Turbo reaches 15%; additional context improves Pro’s performance.Source: §5.2.2.7 · p. 27
• 100-shot Calendar Scheduling accuracy reaches 52% for Gemini 1.5 Pro, compared with 36% for GPT-4 Turbo at 40 shots.Gemini 1.5 Pro reaches 33% in the 1-shot setting, while GPT-4 Turbo is under 10%; more context improves both models.Source: §5.2.2.7 · p. 27

5.2.2.8 Unstructured Multimodal Data Analytics Task

• Gemini 1.5 Pro improves all-attributes image extraction accuracy by 9% absolute, or 27% relative to GPT-4 Turbo.The evaluation processes 1024 images, using mini-batches when necessary; more images consistently improve Pro’s results, unlike GPT-4 Turbo.Source: §5.2.2.8 · p. 28

6. Core Capability Evaluations

• Gemini 1.5 Pro uniformly outperforms Gemini 1.0 Pro and often matches or surpasses Gemini 1.0 Ultra, with audio regressions on some multilingual datasets.The regressions are attributed to post-training data concentrated in five head languages, affecting YouTube, FLEURS, and Covost 2.Source: §6 · pp. 28–29

6.1.1. Math and Science

• +14.5% over Gemini 1.0 Ultra on Hendrycks MATH, +13.2% on AMC, and +5.8% on GPQA demonstrate Gemini 1.5 Pro’s gains in demanding math and science tasks.On Functional MATH, Gemini 1.5 Pro solved 81.1% of original and 64.6% of modified problems, with a 20.3% reasoning gap.Source: §6.1.1 · pp. 29, 31

6.1.2. General Reasoning

• Gemini 1.5 Pro achieves 89.2% on BigBench-Hard and 93.3% on Hellaswag, while Gemini 1.5 Flash outperforms Gemini 1.0 Pro on general reasoning.Gemini 1.0 Ultra, 1.5 Pro, and 1.5 Flash all exceed 80% on MMLU.Source: §6.1.2 · pp. 31–32

6.1.3. Code

• Gemini 1.5 Pro is the strongest Gemini coding model to date, surpassing Gemini 1.0 Ultra on HumanEval and the held-out Natural2Code benchmark.The evaluation emphasizes held-out tests because including HumanEval data in continued pretraining increased Gemini 1.0 Ultra’s score from 74.4% to 89.0%.Source: §6.1.3 · p. 32

6.1.4. Multilinguality

• Gemini 1.5 Pro improves over Gemini 1.0 Ultra on multilingual math and translation, including an almost +9% gain on MGSM and gaps above 9% and 7% for medium- and low-resource languages.Gemini 1.5 Flash matches Gemini 1.0 Ultra on WMT23 and surpasses it by > 3 on MGSM despite its smaller size.Source: §6.1.4 · p. 32

6.1.5. Function Calling

• Gemini 1.5 Pro substantially improves weighted function-calling accuracy over Gemini 1.0 Pro, aided by parallel calls and better decisions about when not to call functions.Gemini 1.5 Flash is extremely close to 1.5 Pro, while the benchmark may be near saturation for simple one-step calls.Source: §6.1.5 · p. 33, Table 12 · p. 33

6.1.6. Instruction Following

• On 1,326 long enterprise prompts, Gemini 1.5 Pro improves response accuracy by 32% over Gemini 1.0 Pro, fully follows 59% of prompts, and reaches 90% instruction-level accuracy.On 406 shorter prompts, Gemini 1.5 models follow 86–87% of instructions and fully follow 65% of prompts, similarly to Gemini 1.0 Pro.Source: §6.1.6 · p. 34, Table 13 · p. 34

6.1.7. Real-world and long-tail expert GenAI tasks

• Across expert occupational tasks, Gemini 1.5 Pro saves 26% of time in architecture, 73% in photography, and 75% in programming, with average usefulness rated 4.0 versus 2.7 for Gemini 1.0 Pro.Responses were rated as saving time across all evaluated jobs, and the study reports productivity improvements across multiple domains.Source: §6.1.7 · pp. 37–38, Figure 19 · p. 38

6.2. Core Vision Multimodal Evaluations

• Gemini 1.5 Pro consistently and often substantially improves over Gemini 1.0 Pro on image-understanding benchmarks, matching or exceeding Gemini 1.0 Ultra on all but one; Gemini 1.5 Flash shows similar improvement over Gemini 1.0 Pro.The comparison reports Average Normalized Levenshtein Similarity for DocVQA, InfographicVQA, and DUDE, and accuracy for other datasets.Source: §6.2 · p. 38, Table 18 · p. 39

6.2.1. Multimodal Reasoning

• Gemini 1.5 Pro scores 62.2% on MMMU, 63.9% on MathVista, and 69.7% on ChemicalDiagramQA, while Gemini 1.5 Flash exceeds Gemini 1.0 Ultra on all these reasoning benchmarks except MMMU.MathVista is described as a new state-of-the-art result, and ChemicalDiagramQA substantially outperforms all Gemini 1.0 models.Source: §6.2.1 · pp. 38, 40, Table 18 · p. 39

6.2.2. Charts and Document Understanding

• Gemini 1.5 Pro sets new state of the art on ChartQA, improves over Gemini 1.0 Pro by more than 20% on BetterChartQA, and is more than 24% better than Gemini 1.0 Ultra on TAT-DQA.Gemini 1.5 Flash outperforms Gemini 1.0 Ultra on 4 of 6 tested chart and document benchmarks.Source: Table 18 · p. 39, §6.2.2 · pp. 40–41

6.2.3. Natural Image Understanding

• Gemini 1.5 models remain competitive with Gemini 1.0 on TextVQA and VQAv2, perform favorably against prior benchmark results on RealWorldQA and BLINK, but retain a notable gap from human-level understanding.On the high-resolution V* Benchmark, Gemini 1.5 Pro surpasses both Gemini 1.0 Pro and Gemini 1.0 Ultra by a large margin and comes within a few points of SEAL.Source: §6.2.3 · p. 41

6.2.4. Video Understanding

• Gemini 1.5 Pro outperforms Gemini 1.0 Ultra on question answering for all tested several-minute videos, exceeds it on VATEX and VATEX ZH captioning, but underperforms on YouCook2.Gemini 1.5 Flash outperforms Gemini 1.0 Ultra on OpenEQA, whereas Gemini 1.5 Pro is slightly behind because of refusals to answer.Source: Table 19 · p. 40, §6.2.4 · p. 41

6.3. Core Audio Multimodal Evaluations

• Gemini 1.5 Pro significantly improves over specialist USM and Whisper on speech benchmarks, while Gemini 1.5 Flash also outperforms specialist models.Gemini 1.5 Pro performs similarly to Gemini 1.0 Pro on non-long-context speech tasks, and Gemini 1.0 Ultra has slight benefits while requiring more training compute and serving resources.Source: §6.3 · p. 42, Table 20 · p. 42

Evaluation setup

• The evaluation covers MATH, AIME 2024, MathOdyssey, HiddenMath, and IMO-Bench, including held-out and expert-graded problems.HiddenMath contains problems held out from training sets, while IMO-Bench tests capabilities at the International Math Olympiad level.Source: §7 · p. 43

Benchmark results

• 91.1% MATH accuracy with 256-sample candidate selection, versus 80.6% from a single sample, advances state-of-the-art performance across all evaluated mathematical benchmarks.The model uses no code execution, theorem-proving libraries, Google Search, or other tools, and the single-sample result is on par with human-expert performance.Source: §7 · p. 43, Table 21 · p. 43
• The model solves four times more AIME problems and improves substantially on MathOdyssey, HiddenMath, and IMO-Bench beyond its MATH performance.These results support generally improved mathematical performance beyond the MATH benchmark.Source: §7 · p. 43

Qualitative examples

• On an APMO problem missed by Gemini 1.5 Pro, GPT-4 Turbo, and previous Gemini models, the math-specialized model finds a different route from the official solution.Its proof assumes a ≥ b ≥ c, bounds a^2 + b + c below (a + 1)^2, and derives a contradiction with positive b and c.Source: §7 · pp. 43–44
• For an AI Mathematical Olympiad optimization problem, the model rewrites the objective as a^2 + b^2 and obtains the minimum value 800 under |a| + |b| = 40.The minimum is achieved when |a| = |b| = 20.Source: §7 · p. 44

Limitations

• The math-specialized model is currently limited to Google internal research use cases, with deployment of stronger math capabilities planned for future models.The paper states that broader deployment is hoped for soon.Source: §7 · p. 44

Model design

• Over 1 million tokens of context and inherited architectural and data-mixture optimizations give Flash-8B multimodal capability in a single-digit-billion-parameter model.The model is derived from Flash and retains its core architecture, optimizations, and data mixture refinements.Source: §8 · p. 45

Efficiency and applications

• High throughput and extremely low latency make Flash-8B suitable for affordable, timely large-scale multimodal deployments despite lower quality than Flash and 1.5 Pro.The stated use cases include massive-data labeling, high-throughput agent serving, and integration into complex multi-model workflows.Source: §8 · p. 45

Status and limitations

• Preliminary results establish competitive standing while Flash-8B remains under active development to maximize performance within its inference budget.The section frames this progress as expanding access to high-quality intelligence for billions of users.Source: §8 · p. 45

Multimodality

• 80–90% of Flash’s performance on established visual benchmarks demonstrates a minimal efficiency–capability trade-off for Flash-8B.These initial multimodal evaluations span a range of visual tasks and are reported after instruction-tuning.Source: §8 · p. 45, Table 22 · p. 46

Long context

• NLL decreases monotonically through 1M tokens for long documents and 2M for code, showing that Flash-8B uses information from the full tested context.The same evaluation sources used for Gemini 1.5 Pro and Flash support these results, and a power law relates log-loss to context length at extreme lengths.Source: §8 · p. 45, Figure 20 · p. 47

9.1. Our Process

• The safety process identifies and mitigates foreseeable societal impacts through assessments, policy criteria, safety training, red teaming, external evaluations, assurance testing, governance review, and product handover.Assurance evaluations use held-out datasets outside the model-development team and assess both policy violations and dangerous capabilities, informing release decisions.Source: §9.1 · pp. 48–49

9.2.1. Identifying risks (potential impact assessments)

• Audio and long-context capabilities increase potential benefits but may amplify representational and privacy harms and require stronger robustness against adversarial attacks.Assessments therefore extend prior Gemini work by focusing on audio and increased context length, with identified risks informing policies, evaluations, and safety training.Source: §9.2.1 · p. 50

9.2.2. Safety policies

• Safety policies prohibit harmful content across modalities, protect private information and users from attacks, and require adherence even when unsafe instructions are buried in long context.Optional policies govern use-case-dependent areas including medical, legal, and financial advice, explicit content, violence, impersonation, and endorsements.Source: §9.2.2 · pp. 50–51

9.2.3. Desiderata, aka “helpfulness”

• Helpfulness desiderata require Gemini to fulfill user goals whenever possible without violating policy and to use an objective tone when refusal is necessary.These criteria complement defining prohibited behavior by discouraging blanket refusal and biased assumptions about user intent.Source: §9.2.3 · p. 51

9.3. Training for Safety, Security, and Responsibility

• Safety training combines data filtering, conditional pre-training, monitoring, supervised fine-tuning, and reinforcement learning from human feedback, guided by automated and human evaluations.SFT and RLHF use adversarial prompts, human feedback, custom data generation, and continuous evaluation to reduce policy violations and unhelpful refusals while detect regressions.Source: §9.3 · p. 51

9.3.1. Pre-Training

• Pre-training applies strict-policy data filtering, toxicity-related control tags, and monitoring with representational-harm benchmarks and toxicity scores.These interventions are intended to structure representations for safer post-training and track harms before later alignment stages.Source: §9.3.1 · p. 51

9.4. Results on Training/Development Evaluations

• Gemini 1.5 Pro and Flash are the safest models to date in development evaluations, with large policy-violation improvements and stronger jailbreak robustness than Gemini 1.0 Ultra.They show slight quality improvements but also additional refusals and non-neutral tone; handcrafted prompt injections remain effective, whereas garbage-token attacks do not.Source: §9.4 · p. 52, §9.4.1 · p. 53

9.4.1. Policy Violations

• 62% and 43% fewer image-to-text violations were observed for Gemini 1.5 Pro and Flash, respectively, than for Gemini 1.0 Ultra.Text-to-text evaluations also found both models safer in English and multilingual settings, while audio-to-text unsafe rates were similar to or lower than text-to-text rates.Source: Table 23 · p. 52, §9.4.1 · pp. 53–54
• Long-context safety violation rates are lower overall because models sometimes fail to locate adversarial needles, but successful responses show similar violation levels to short-context tasks.The authors hypothesize that improved long-context capabilities could increase violations, and more sophisticated prompts produce higher violations.Source: §9.4.1 · p. 56, Table 28 · p. 58
• Gemini 1.5 Pro and Flash are the safest models to date, with large decreases in policy violations and increased robustness to jailbreaks relative to Gemini 1.0 Ultra.They resist gradient-based attacks using non-human-interpretable tokens but remain susceptible to attacks containing human-readable instructions.Source: Table 26 · p. 55, §9.4.1 · p. 56

9.4.2. Helpfulness

• Gemini 1.5 Pro is preferred over Gemini 1.0 Ultra on text-to-text safety-related prompts, with small gains in instruction following, effective refusals, and quality but a regression in tone.The evaluation uses side-by-side annotator preferences and separately measures instruction following, autonomy-respecting tone, and effective refusal of unsafe queries.Source: §9.4.2 · p. 58, Table 29 · p. 58

Image to Text Refusals

• Both Gemini 1.5 models improve overall response quality over Gemini 1.0 Ultra on grounded and ungrounded image-to-text refusal queries.The evaluation distinguishes ungrounded questions, which should be refused, from grounded questions containing contextual evidence that should be answered.Source: Image to Text Refusals · p. 59, Table 30 · p. 59
• 35/140% and 7/60% are the increases in Gemini 1.5 Flash’s and Pro’s refusal rates on ungrounded/grounded image queries, respectively, revealing a tradeoff between refusing unsafe guesses and answering grounded questions.Neither model is a Pareto improvement; humans preferred responses more when refusals included explanations and when non-refusals avoided hallucinated or incorrect information.Source: Image to Text Refusals · pp. 59–60, Figure 22 · p. 59

9.4.3. Security and Privacy

• Handcrafted prompt injections are much more successful than optimization-based attacks at eliciting sensitive-data exfiltration, although these results come from a simulated email/document-retrieval setup.The attack targets passwords, identification numbers, payment-card numbers, and email addresses; practical success depends on deployment details, safety filters, and retrieval-stage instructions.Source: §9.4.3 · pp. 61–62, Table 32 · p. 62
• Gemini 1.5 Pro and Flash memorize less than existing, even smaller, models, while approximately memorized content is about 14x more frequent than exact memorization for these models.They also memorize much less personal data than Gemma, and no memorized data with high sensitivity was found.Source: §9.4.3 · pp. 62–63, Figure 24 · p. 63, Figure 25 · p. 64
• 0.35% of divergent Gemini 1.5 Pro generations contained training data, versus around 2.8% for ChatGPT 3.5 and around 0.1% for LLaMA and Mistral, while longer prompts increased divergence from 35.6% to 61.5%.The model is therefore susceptible to divergence, but this attack often fails to elicit training-data emission; longer prompts nevertheless make memorized data easier to obtain.Source: §9.4.3 · pp. 63–64
• Gemini 1.5 Pro was susceptible to handcrafted prompt-injection attacks, while testing found no memorized sensitive data and audio safeguards still require improvement.Audio capabilities were broadly equivalent to other commercial or research systems, but the evaluation recommends continued safeguard improvements.Source: §9.4.3 · pp. 64–65

9.4.4. Representational Harms

• Representational-harm measurement remains challenging because highly capable models approach 100% accuracy on datasets such as BBQ, weakening their future usefulness for detecting bias.The report identifies improved evaluation and mitigation of representational harms as an important area for future work.Source: §9.4.4 · pp. 65, 68
• Gemini 1.5 generally performed similarly or better than baselines on representational harms, with exceptions including income-group accuracy gaps and Gemini 1.5 Pro speech-recognition disparities.BBQ results improved over Gemini 1.0 Ultra, while Dollar Street showed Gemini 1.5 Pro leading average and worst-group accuracy but having a larger income accuracy gap; Flash had greater speech parity than USM, whereas Pro had lesser parity.Source: §9.4.4 · pp. 65–67

9.5.1. Baseline Assurance

• Assurance evaluations found notable safety improvements, with both Gemini 1.5 Pro and Flash achieving their best performance in highest-severity policy testing across modalities.Audio and interleaved-video results were not directly comparable with Gemini 1.0 Pro because those capabilities were not released in Gemini 1.0 models.Source: §9.5.1 · p. 68

9.5.2. Dangerous Capability Evaluations

• Dangerous-capability evaluations found no vulnerability-detection improvement over Gemini 1.0 Ultra, increased performance on the easiest cyber challenges, and higher self-proliferation milestone completion without end-to-end success.Persuasion results were mixed: Gemini 1.5 Pro was not clearly better than baselines in Money Talks and was no better than humans for persuading participants of correct answers in Web of Lies.Source: §9.5.2 · pp. 69, 71

9.6. External Safety Testing

• External safety testing used independent groups across societal, cyber, and CBRN domains, and its findings exposed gaps in existing evaluation methods and safety policies.Groups supplied analyses, raw materials, and model responses; Google DeepMind then assessed issue severity and communicated concerns to modeling teams.Source: §9.6 · pp. 71–72

9.6.1. Societal risks

• External testing found representational biases and ungrounded inferences in image-to-text and video-to-text, including gender assumptions and stereotypical nationality associations.Testers also observed inappropriate optimistic moral commentary on violent or historically difficult images and videos.Source: §9.6.1 · pp. 72–73

9.6.2. Radiological and Nuclear risks

• Radiological and nuclear assessments found no major or specific risks, while generated information was mostly low or intermediate in accuracy and informativeness.External testing covered text and visual modalities, multiple actors and harm outcomes, and suggested further testing of large-scale data sifting and summarization.Source: §9.6.2 · p. 73

9.6.3. Cyber risks

• Cyber evaluations found minor capability enhancement for code obfuscation, with gains below established tools and requiring expertise in obfuscation methods plus prompt engineering.Testing isolated individual tactics because end-to-end attack automation was ineffective; evaluators focused on tasks across the cyber-attack kill chain.Source: §9.6.3 · p. 73

9.7. Product Deployment

• Product deployment uses safety filters, adjustable developer thresholds, limitation disclosures, feedback channels, and pre-launch evaluations with central or specialized governance review.These safeguards apply across products and APIs, including Google AI Studio and Cloud Vertex AI, and are reviewed for risks and accountability before deployment.Source: §9.7 · p. 73

10. Discussion

• Gemini 1.5 Pro expands the Gemini 1.0 context window from 32K to multiple millions of tokens and improves long-context performance through 10M tokens.The authors describe it as commercially surpassing Claude 3’s 200k-token ceiling across modalities.Source: §10 · p. 74
• Near-perfect multimodal needle-in-a-haystack recall enables Gemini 1.5 to retrieve and reason over large amounts of context.The evaluations include diagnostic and realistic multimodal long-context benchmarks.Source: §10 · p. 74
• Gemini 1.5 performs long-document QA on 700k-word material, long-video QA on 40- to 105-minute videos, and in-context English-to-Kalamang translation from a grammar manual alone.Kalamang has fewer than 200 speakers, and the models learn from information absent from training.Source: §10 · p. 74
• 1.5 Pro surpasses 1.0 Ultra on text capabilities despite significantly less training compute, while 1.5 Flash outperforms 1.0 Pro with greater efficiency.The authors characterize the Gemini 1.5 series as a generational performance leap over Gemini 1.0 without sacrificing multimodal core capabilities.Source: §10 · p. 74

Long-context evaluations, call-to-action

• Existing benchmarks under-test very long multimodal contexts, while manual annotation becomes increasingly costly and time-consuming as length and complexity grow.The authors recommend multiple needles-in-a-haystack diagnostics, improved automatic metrics, and new realistic multimodal benchmarks.Source: Long-context evaluations, call-to-action · p. 74

• Core contributors made significant project-wide impacts, while contributors participated partially; contributions are equal within each role and listed in randomized order.Source: §11 · p. 104
• Gemini is a cross-Google effort spanning Google DeepMind, Google Research, Bard/Assistant, Knowledge and Information, Core ML, Cloud, Labs, and other groups.Source: §11 · p. 104
• The authors thank report leaders, reviewers, colleagues, numerous Google teams, and everyone who supported Gemini through feedback, demonstrations, or collaboration.Source: §11 · p. 104

12.1. Model Card

• Gemini 1.5 Pro is a sparse MoE Transformer, whereas Gemini 1.5 Flash is a dense Transformer distilled online from Pro.The architectures build on prior MoE vision/language work and knowledge-distillation research.Source: §12.1 · p. 105
• Gemini 1.5 accepts text, images, audio up to 22 hours, and video up to two hours, producing generated text such as answers and summaries.Outputs can also compare documents or videos.Source: §12.1 · p. 105
• Gemini 1.5 targets applications requiring learning from large information volumes, more relevant in-context responses, and reasoning across modalities.Longer context supports more sophisticated reasoning across different information types.Source: §12.1 · p. 105
• Gemini 1.5 should not be deployed generally or downstream without assessing and mitigating safety and fairness concerns specific to the use case.The model card identifies this as a known caveat.Source: §12.1 · p. 105

Hardware & Software Training

• JAX, XLA, GSPMD, and ML Pathways supported efficient, automatically parallelized training of foundation models on hardware including TPUs.JAX and ML Pathways enabled a single Python process to orchestrate the entire training run, simplifying development.Source: Implementation Frameworks · p. 106

Compute Requirements and Model Stats

• Compute requirements and model statistics were not reported.Source: Implementation Frameworks · p. 106

Model initialization

• The models were trained from random initialization.Source: Implementation Frameworks · p. 106

Model Status

• The models are static models trained on an offline dataset.Source: Implementation Frameworks · p. 106

Training Dataset

• Gemini 1.5 models are trained on a variety of multimodal and multilingual data.Further details are provided in Section 4 (Training Infrastructure and Dataset).Source: Data overview · p. 106

Evaluation Dataset

• Core capability evaluations cover text, audio, and vision datasets.The evaluation details are provided in Section 5 (Evaluation).Source: Data overview · p. 106

Fine-tuning Dataset

• Gemini 1.5 Pro and Flash are fine-tuned on multimodal data containing paired instructions and corresponding desired responses.Further details are provided in Section 4 (Training Infrastructure and Dataset).Source: Data overview · p. 106

• The evaluation addresses core capabilities in text.Detailed text evaluations are covered in Section 5.Source: Evaluation Results · p. 106
• The evaluation addresses core capabilities in audio.Detailed audio evaluations are covered in Section 5.Source: Evaluation Results · p. 106
• The evaluation addresses core capabilities in vision.Detailed vision evaluations are covered in Section 5.Source: Evaluation Results · p. 106

Model Usage & Limitations

• The report notes that Gemini models retain limitations described in the Gemini 1.0 Technical Report, with long-context issues discussed in the safety, security, and responsibility section.Sensitive-use risks are likewise directed to that section.Source: Model Usage & Limitations · pp. 106–107

12.2. Further haystack results

• Gemini 1.5 Pro showed a consistent robust recall trend when retrieving 50 or 100 needles, while recall decreased more gradually than GPT-4 Turbo’s within the first 128K tokens.The task required retrieving multiple unique needles from a single long-context haystack.Source: §12.2 · p. 107, Figure 27 · p. 108

12.6.2. Intermediate Algebra (Levels 4 and 5): A Persistent Challenge

• 18.6% solve rate was achieved by Gemini 1.5 Pro on Intermediate Algebra levels 4 and 5, compared with 12.5% for GPT-4 and 20.6% for GPT4-turbo.This subdomain is described as the most difficult among seven MATH subdomains and represents 10.6% of the dataset.Source: §12.6.2 · p. 109

12.6.3. Leveraging Python and SymPy for Enhanced Performance

• 25.8% solve rate on Intermediate Algebra levels 4 and 5 was achieved by Gemini 1.5 Pro with approximately 730,000 tokens of SymPy and SciPy examples, surpassing Gemini 1.5 Pro, GPT-4, and GPT-4-turbo baselines.These results used Gemini 1.5 Pro rather than the Math-Specialized Gemini 1.5 Pro model.Source: §12.6.2 · p. 109, §12.6.3 · p. 110

12.6.5. Python Evaluation Process

• 80–110 of 528 problems were solved in the initial Python stage, followed by 20–30 in the first correction stage and 5–10 in the final stage.The process used successive error traces and prior solutions, with correctness verified using the MATH evaluation procedure.Source: §12.6.5 · pp. 110–111

12.6.9. Example problem and a Gemini 1.5 Pro solution

• 0.015624507088912548 was obtained for a constrained optimization problem, within the MATH evaluation tolerance of the exact answer 1/64, using an approximate SciPy SLSQP solution.The approach minimizes the negative objective under nonnegative bounds and the constraint a + b + c = 1.Source: §12.6.9 · p. 116

12.7. Unstructured Multimodal Data Analytics

• For unstructured multimodal analytics, models were prompted to classify each image by category, color, and semantic attribute, with individual-attribute results presented in Figure 29.The prompt required a table-only output format across anonymized classes and image identifiers.Source: §12.7 · pp. 116–117
• Gemini 1.5 Pro is shown producing solutions for unstructured multimodal data analytics tasks.Figure 29 reports model performance on these tasks, while the supplied passage identifies semantic attribute extraction accuracy.Source: §12.7 · p. 117, Figure 29 · p. 117

12.8. Planning

• The cited examples contain completed action sequences, a 13-day itinerary spanning London, Madrid, Berlin, Dublin, Oslo, and Vilnius, and a proposed Monday 16:00–16:30 meeting time.Source: §12.8 · pp. 117, 119, 120, 121, 123, 124

12.9. Productivity Impact of LLMs Across Jobs

• The section points to example tasks from a pre-school teacher, real estate photographer, and PR manager as illustrations of LLM productivity across jobs.These examples are provided in Tables 49–51, with further examples referenced from Section 6.1.7.Source: §12.9 · p. 124, Table 49 · p. 125, Table 50 · p. 126

12.10.0.1 Background: Why evaluate the long tail?

• Gemini 1.5 models significantly and strongly outperform 1.0 Pro on Expert QA in accuracy ratings, severe-inaccuracy ratings, and side-by-side rankings.The comparison holds despite Gemini 1.5 producing much longer, more detailed responses, and side-by-side rankings also account for informativeness.Source: §12.10.0.1 · p. 128

12.10.1. Results

• Expert QA evaluation asks raters to label responses as fully accurate, somewhat inaccurate, or severely inaccurate, then rank them by accuracy, completeness, and informativeness.Experts also judge whether responses are well-crafted and understandable for their target audience; the results are summarized in Tables 52 and 53.Source: §12.10.1 · p. 128, §12.10.2 · p. 129

12.11. STEM QA with Context: Data and Human Evaluation

• STEM-context QA uses 158 questions from 145 Qasper articles, providing each model the full article and evaluating response sentences for accuracy and severity of errors.The selected questions lack supplied gold answers or supporting evidence, exclude questions judged unanswerable, and come from articles averaging 5,138 tokens.Source: §12.11 · p. 130
• STEM raters classify each sentence as accurate, disputed, unsupported, inaccurate, or not confidently assessable, and separately mark qualifying issues as severe or not severe.Reported measures include the proportions of accurate, inaccurate, and severely inaccurate sentences, excluding sentences with no claim or unassessable content.Source: §12.11 · p. 130

12.12. QA for Web Search Topics

• The TREC web-search evaluation combines web, session, tasks, and health-misinformation tracks, covering informational, multi-step, real-world-task, and health-misinformation topics.Prompts convert topic descriptions into requests for detailed answers based on the models’ parametric knowledge, and raters use 7-point preference and 5-point helpfulness scales.Source: §12.12 · pp. 130–131
• On QA for web-search topics across multiple TREC datasets, Gemini 1.5 improves significantly over 1.0; Flash is comparable to Ultra and beats Pro, while Pro leads on all measures.The table reports human helpfulness ratings and model preferences, with Likert scores scaled to [0,100]; preference above 50 indicates preference over the comparison baseline, with 95% confidence intervals obtained by bootstrap.Source: §12.12 · p. 131, Table 54 · p. 132

12.13.1. BBH

• The BBH prompting procedure appends a task-specific description and few-shot examples with chain-of-thought answers, then asks the model to continue reasoning before extracting the final answer.For example, the task description can request evaluation of a random Boolean expression.Source: §12.13.1 · p. 132

12.13.2. DROP

• DROP uses three fixed demonstrations plus paragraph-specific examples, standardizes question order, instructs the model to answer only the final question, and scores the output before the first newline against ground truth.The demonstrations include reading-comprehension questions whose answers are directly extracted from the supplied paragraphs.Source: §12.13.2 · pp. 133–134

12.13.3. Hellaswag

• Hellaswag, MMLU, and AMC use benchmark-specific preambles before few-shot examples; Hellaswag and MMLU score candidate answer choices directly, while AMC setup begins with its preamble.Hellaswag scores each possible letter answer, and MMLU selects the highest-scoring choice among A–D without postprocessing.Source: §12.13.3 · p. 134, §12.13.4 · p. 134, §12.13.5 · p. 134

12.13.5. AMC; 12.13.6. MATH; 12.13.7. GSM8K

• Mathematical evaluations use benchmark-specific demonstrations and final-answer formats: four AMC examples, zero- or four-shot MATH, and eight- or eleven-shot GSM8K.AMC answers are extracted between fixed prefix and suffix strings, MATH supports symbolic and numerical equivalence checks, and GSM8K extracts the final number.Source: §12.13.5 · pp. 134–135, §12.13.6 · pp. 135–136
• The AMC setup presents worked examples and instructs the model to provide only the final solution with a letter answer, while MATH and GSM8K likewise standardize final-answer extraction.AMC examples include competition problems with expert-written solutions; GSM8K uses prompts from Cobbe et al. (2021), and MATH uses a prompt from Lewkowycz et al. (2022).Source: §12.13.5 · pp. 134–135, §12.13.7 · p. 136

12.13.8. GPQA; 12.13.9. PhysicsFinals

• GPQA prompts require chain-of-thought reasoning followed by “Final Answer: (X),” while unmatched or multiply matching options are resolved by choosing the closest option.PhysicsFinals provides no prompt and directly inputs questions to instruction-tuned models, with human experts grading responses.Source: §12.13.8 · p. 136, §12.13.9 · p. 136

12.13.10. HumanEval and Natural2Code

• HumanEval and Natural2Code modify function-completion prompts to prevent clarifying questions, then execute hidden tests in a sandbox against the returned code block.Postprocessing expects a complete code markdown block before computing pass rate.Source: §12.13.10 · pp. 136–137

12.14. Vision tasks; 12.14.2. AI2D; 12.14.3. MMMU; 12.14.4. MathVista; 12.14.5. ChartQA; 12.14.7. DocVQA; 12.14.8. DUDE

• Vision tasks are generally evaluated zero-shot with instruction-tuned models and task-specific answer-format preambles.AI2D, MMMU, MathVista, ChartQA, DocVQA, and DUDE specify concise or structured outputs, while MathVista additionally requests image descriptions and reasoning.Source: §12.14 · p. 137, §12.14.2 · p. 137, §12.14.3 · p. 138

12.14.1. V* benchmark

• Gemini 1.5 Pro outperforms GPT-4V on V* and comes within a few points of SEAL on high-resolution attribute recognition and spatial reasoning.V* uses 191 questions over small-object attributes and spatial relations, with randomized answer-option positions.Source: §12.14.1 · p. 137
• V* evaluates high-resolution image understanding on a 2246 x 1582 average-resolution subset of SA-1B, covering 115 attribute-recognition and 76 spatial-reasoning samples.Questions concern small-object properties such as color or material and relative positions, with human-crafted multiple-choice options.Source: §12.14.1 · p. 137

12.14.6. ChemicalDiagramQA

• ChemicalDiagramQA contains 76 multiple-choice questions spanning easy single diagrams, medium paper subfigures, and hard full-paper figures.Figures come from randomly selected bioRxiv papers and US patents, and questions require reasoning before an answer formatted as “Answer: <a,b,c,d>”.Source: §12.14.6 · p. 139

12.14.9. TAT-DQA … 12.14.12. VQAv2

• Visual QA prompts require terse answers grounded in images, with dataset-specific constraints for TAT-DQA, InfographicVQA, TextVQA, and VQAv2.Instructions include avoiding paraphrase, omitting full stops, limiting answers to one or two words, and answering binary questions with Yes or No.Source: §12.14.9 · p. 139, §12.14.10 · p. 140, §12.14.11 · p. 140

12.15. Blink … 12.16.5. OpenEQA

• Video benchmarks use direct short answers or multiple-choice formats, while VATEX and YouCook2 use 4-shot video captioning and OpenEQA uses zero-shot visual reasoning.ActivityNet-QA requests likely short answers, EgoSchema and Blink require formatted choices with reasoning, and OpenEQA asks for concise answers or educated guesses from video observations.Source: §12.16.1 · p. 140, §12.16.2 · p. 141, §12.16.3 · p. 141

12.16.6. Text Needle-in-a-Haystack … 12.16.8. Audio Needle-in-a-Haystack

• Needle-in-a-haystack evaluations place a secret item in text, video, or audio context and query its content using modality-specific prompts.Text evaluation appends a magic-number cue to reduce refusals on very long contexts; video inputs include timestamps and frame bytes, while audio inputs contain an embedded speech segment naming the keyword.Source: §12.16.6 · pp. 141–142, §12.16.7 · p. 142, §12.16.8 · p. 142

12.16.9. Multi-round Co-reference Resolution (MRCR) … 12.16.11. 1H-VideoQA

• Long-context and multi-round tasks provide structured context followed by constrained queries, with formatting interventions for refusals and answer extraction.MRCR supplies conversation examples and appends a task-specific instruction after Claude 2.1 frequently refused; long-document QA requires answers based only on search results, while 1H-VideoQA timestamps sampled frames and randomizes choices.Source: §12.16.9 · pp. 142–143, §12.16.10 · p. 143, §12.16.11 · p. 144

12.16.13. MTOB … 12.16.14. ASROB

• MTOB and ASROB provide linguistic resources in context, including grammar books, bilingual word lists, parallel sentences, and speech segments with transcriptions for translation and speech transcription tasks.MTOB uses disjoint 50-sentence test sets, 375 parallel training sentences, and direction-specific word lists; its prompt supplies complete word lists and parallel sentences in context rather than external similarity-based retrieval. The authors attribute the score increase for their Claude 2.1 baseline over Claude 2 in Tanzer et al. (2023) to this difference.Source: §12.16.13 · pp. 144–145, §12.16.14 · p. 145

12.16.15. Multilinguality … 12.16.16. Dolomites

• Multilinguality uses one-shot translation prompts and eight-shot chain-of-thought prompts for MGSM, while Dolomites generates structured writing outputs and uses a zero-shot language-model judge.Dolomites instructions require matching section lengths and headers, factual detail, and comparison of two candidates against a reference output.Source: §12.16.15 · pp. 145–146, §12.16.16 · pp. 146–147

12.16.16. Dolomites

• Dolomites guidance emphasizes explaining the selected output and keeping responses concise, since longer outputs are not necessarily higher quality.Conciseness is explicitly identified as a desirable output quality.Source: §12.16.16 · p. 147

12.17. More MTOB results

• Human MTOB ratings are biased because the rater recognized prior translations, knew experimental definitions and automatic scores, and favored familiar eng→kgv phrasing.For eng→kgv, absolute scores may also be inflated because the rater was non-native and non-fluent.Source: §12.17 · pp. 147–148
• Combining the word list with parallel sentences, while omitting the grammar book, matches or outperforms GPT-4 Turbo and Claude 3 given half-book context.No-book outputs use more diverse vocabulary, while half-book outputs have better grammar, suggesting that the book can dominate context and distract from the word list.Source: §12.17 · p. 148
• Full-book kgv→eng translation remains worse than the human baseline, whereas eng→kgv is comparable on average, with humans struggling more with retrieval and models with grammatical application.The results still leave room for improvement in both translation directions.Source: §12.17 · p. 148

12.17.1. Qualitative Examples

• The qualitative examples reuse held-out MTOB training examples and include a BIG-bench canary to mitigate future contamination.The examples cover translation between Kalamang and English.Source: §12.17.1 · p. 149

12.18. More BetterChartQA Details and Results

• 65.8% micro-averaged BetterChartQA performance makes Gemini 1.5 Pro strongest, followed by Gemini 1.5 Flash at 59.0%, GPT-4 Turbo at 50.3%, Gemini 1.0 Ultra at 47.9%, and Gemini 1.0 Pro at 43.0%.Both Gemini 1.5 models outperform GPT-4 Turbo; gains are especially strong for multi-figure, table, chart-in-context, and scatter-plot categories, including complex multi-series parsing and numerical extraction.Source: §12.18 · p. 151, Figure 32 · p. 154