PaperlayerView exact revision on arXiv

Source-linked AI summary

Gemma 3 Technical Report

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju-yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, Léonard Hussenot

arXiv:2503.19786v1cs.CLcs.AI

TL;DR

Gemma 3 addresses the need for lightweight open models with vision understanding, broader language coverage, and long context without reducing performance. It combines architectural changes, multimodal and multilingual training, distillation, and improved post-training, yielding broad capability gains while remaining compatible with standard hardware. The release also reports no detected personal information in memorized outputs and characterizes its overall risk impact as negligible.

  • Problem

    Lightweight open models needed broader modality, language coverage, and context length while remaining suitable for standard hardware.

  • Method

    Gemma 3 combines a multimodal vision pathway, long-context local/global attention, revised multilingual training, knowledge distillation, and a novel post-training approach.

  • Results

    Gemma 3 adds image understanding and long context while improving multilinguality and STEM-related abilities across pre-trained and instruction-tuned models.

  • Takeaways & Limitations

    The models and architectures are designed for standard hardware, extending Gemma to text, image, and code use cases.

  • Takeaways & Limitations

    The authors report no personal information in memorized outputs under their detection thresholds and expect the release to have a negligible effect on the overall risk landscape.

Abstract

from arXiv · show

We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.

1. Introduction

Gemma 3 extends the lightweight Gemma family with multimodality, longer context, broader language coverage, and improved instruction-tuned capabilities while targeting consumer hardware.

  • New capabilities: Gemma 3 adds multimodality, long context, multilinguality, and a 1B model while preserving or surpassing prior performance.The models are designed for phones, laptops, and high-end GPUs.
  • New capabilities: Images are encoded as soft tokens through a tailored SigLIP vision encoder and condensed into 256 vectors to reduce image-processing inference cost.Pan and Scan supports flexible image resolutions.
  • Architecture: 128K-token context is enabled without reducing performance by interleaving local and global attention layers with 1024-token local spans.Only global layers attend to long context, using one global layer for every five local layers.
  • Training: Pre-training uses a revised data mixture, the Gemini 2.0 tokenizer, image understanding, multilingual data, and knowledge distillation.The data mixture is revisited to improve multilingual capabilities.
  • Post-training: A novel post-training approach improves math, coding, chat, instruction following, and multilingual capabilities, with instruction-tuned models outperforming predecessors.The approach also integrates long-context and image inputs.

2. Model Architecture

Gemma 3 retains a decoder-only transformer backbone while adding a multimodal vision pathway, long-context attention design, revised multilingual training, distillation, and quantized model formats.

  • Core architecture: Gemma 3 uses a decoder-only transformer with Grouped-Query Attention, RMSNorm, and QK-norm replacing Gemma 2’s soft-capping.Most architecture elements remain similar to earlier Gemma versions.
  • Long context: Local sliding-window attention is interleaved with global attention in a 5:1 local-to-global layer pattern.The model starts with a local layer.
  • Long context: Gemma 3 supports 128K-token context except for the 1B model, which supports 32K, using different RoPE base frequencies for global and local layers.Global layers use a 1M RoPE base frequency, while local layers retain 10k.
  • Vision modality: The shared 400M SigLIP vision encoder processes 896 × 896 images and remains frozen for the 4B, 12B, and 27B models.Pan and Scan crops non-square or high-resolution images into encoder-sized windows only when necessary.
  • Pre-training: Pre-training uses 14T, 12T, 4T, and 2T tokens for the 27B, 12B, 4B, and 1B models respectively, with increased multilingual data.The tokenizer has 262k entries and is more balanced for non-English languages.
  • Pre-training: Knowledge distillation samples 256 logits per token and trains the student on the teacher distribution using cross-entropy loss.Nonsampled logits receive zero probability before renormalization.
  • Deployment: The release includes quantized checkpoints using per-channel int4, per-block int4, and switched fp8 weight representations.Quantization-aware fine-tuning typically runs for 5,000 steps.

3. Instruction-Tuning

Instruction tuning combines improved knowledge distillation with reinforcement-learning fine-tuning and carefully filtered data to improve helpfulness, reasoning, safety, and multilingual performance.

  • Post-training approach: Pre-trained checkpoints are converted into instruction-tuned models with an improved post-training approach.The approach differs from the prior Gemma recipe.
  • Post-training approach: Post-training combines knowledge distillation from a large instruction-tuned teacher with reinforcement-learning fine-tuning based on improved BOND, WARM, and WARP methods.These components form the core post-training approach.
  • Reinforcement learning: Multiple reward functions improve helpfulness, mathematics, coding, reasoning, instruction following, and multilingual abilities while minimizing harmfulness.Rewards include human-feedback models, code-execution feedback, and ground-truth math rewards.
  • Data filtering: Post-training data is filtered for personal information, unsafe or toxic outputs, mistaken self-identification, and duplicates.Data encouraging attribution, hedging, and refusals improves factuality metrics without degrading other metrics.
  • Formatting: PT models end generation with <eos>, whereas IT models end with <end_of_turn>, and each format requires its respective end token.Both formats begin with an explicitly added [BOS] token.

4. Evaluation of final models

Gemma 3 instruction-tuned models are evaluated through automated, human, and static benchmarks against prior Gemma and selected Gemini models. The 27B model performs strongly in human preference evaluation, while the authors caution against direct comparisons with external models using different settings.

  • Gemma 3 instruction-tuned models are evaluated across automated benchmarks, human evaluations, and static benchmarks such as MMLU.
  • LMSYS Chatbot Arena: Gemma 3 27B IT scores 1338 in Chatbot Arena, ranking among the top 10 models and above several much larger non-thinking open models.The compared scores are DeepSeek-V3 1318, LLaMA 3 405B 1257, and Qwen2.5-70B 1257.
  • Standard benchmarks: Direct comparisons with external models are omitted because differing evaluation settings do not guarantee fairness.The authors recommend third-party static leaderboards for fairer cross-model comparisons.

5. Ablations

The ablations examine general abilities, attention patterns, long-context scaling, distillation, and vision design choices. Gemma 3 improves across most probed categories while reducing KV-cache overhead and benefiting from shorter local windows, larger image resolutions, and Pan & Scan.

  • Pre-training ability probing: Gemma 3 improves over Gemma 2 in most probed categories, including science, code, factuality, multilinguality, reasoning, and vision.The authors note that benchmark contamination remains a risk despite decontamination techniques.
  • Local:Global attention layers: A 5:1 local-to-global attention ratio has minimal impact on perplexity compared with Gemma 2’s 1:1 ratio.The ablation compares different local-to-global layer ratios, with 1:1 used in Gemma 2 and 5:1 in Gemma 3.
  • Sliding window size: Reducing the local attention sliding window significantly does not impact perplexity.The study compares sliding-window sizes across different global-to-local ratio configurations.
  • KV cache memory: 60% memory overhead with global-only attention falls below 15% with a 1:3 ratio and 1024-token sliding windows.The comparison uses a 32k-token context and contrasts the global-only configuration with the reduced-cache configuration.
  • Long context: Gemma 3 models generalize to 128K-token context after RoPE rescaling, using a scaling factor of 8.The 4B, 12B, and 27B models are scaled from 32K-token training sequences, with global-layer RoPE base frequency increased from 10k to 1M.
  • Knowledge distillation: Longer training favors a larger distillation teacher, whereas shorter training favors a smaller teacher.The study observes this reversal when training a student with teachers of different sizes across training horizons.
  • Vision ablations: Higher-resolution vision encoders perform better, and Pan & Scan particularly helps tasks requiring text reading from images.Pan & Scan preserves image aspect ratio and resolution more closely, while image representations are reduced to 256 tokens.

6. Memorization and Privacy

The paper measures memorization and potentially personal information in Gemma 3 outputs using extraction-based tests and a broad privacy detector. Gemma 3 shows lower long-form memorization than prior models, while detected personal information is absent but the detector likely overestimates true prevalence.

  • Memorization measurement: Memorization rate is the ratio of generations matching training data to all model generations.The audit distinguishes exact and approximate memorization using extraction tests.
  • Memorization measurement: Exact memorization requires every continuation token to match the source suffix, while approximate memorization permits up to 10% edit distance.The test uses a 50-token prefix and 50-token suffix sampled across training-data corpora.
  • Memorization results: Gemma 3 models memorize long-form text at a much lower rate than prior models, with only marginal differences among the 4B, 12B, and 27B models.The 1B model memorizes less than those larger Gemma 3 models; Figure 9 uses a log-scaled y-axis.
  • Personal information: No personal information is observed in outputs classified as memorization for any Gemma 3 model.The authors interpret this as a low rate below the detector’s thresholds.
  • Personal information: The privacy detector likely overestimates potentially personal information because its high-recall rules ignore context and generate many false positives.The analysis classifies text as personal at any severity level reported by the detector.

7. Responsibility, Safety, Security

Gemma 3’s release pairs expanded capabilities with safety processes addressing harmful content, multimodal risks, and the irreversible nature of open weight releases.

  • Safety processes: Gemma 3 integrates safety mitigation during training and transparent evaluations for its new image-to-text capabilities.The approach extends internal safety processes across the development workflow.
  • Governance & assessment: The release evaluates openness against potential malicious uses that may harm individuals and institutions.The authors describe monitoring and collaboration with academic and developer communities to investigate reported misuse.
  • Scope and residual risk: The authors expect this release to have a negligible effect on the overall risk landscape because larger powerful open models are already available.They also state that safe applications require system-level, use-case-specific mitigations before sharing models.
  • Training-time safeguards: Fine-tuned models use SFT and RLHF, while pre-training data undergoes safety filtering to reduce undesirable behavior.The stated safety target includes harmful content such as personally identifiable information, hate speech, dangerous content, and sexually explicit content.
  • Safety evaluations: Baseline assurance measures policy violation rates using synthetic adversarial queries and human ratings, finding Gemma 3’s overall violation rate significantly low.The assessment focuses on whether answers violate specified safety policies.
  • Extreme-risk knowledge: Evaluations find Gemma 3 models have low knowledge in biological, radiological, nuclear, and chemical-risk domains.These assessments use closed-ended, knowledge-based questions and chemical-hazard evaluations.

8. Discussion and Conclusion

Gemma 3 adds image understanding and long context while improving multilinguality and STEM abilities, with models designed for standard hardware and performance maintained.

  • Discussion and Conclusion: Gemma 3 focuses on image understanding, long context, multilinguality, and STEM-related improvements.The models support text, image, and code use cases.
  • Discussion and Conclusion: Model sizes and architectures are designed to remain compatible with standard hardware while maintaining performance.The paper states that most architectural improvements are tailored to this hardware constraint.

Contributors (alphabetical order)

The paper acknowledges contributors listed in alphabetical order, including technical leads and additional named collaborators.

  • Contributors (alphabetical order): The contributor list presents authors and collaborators in alphabetical order.The supplied contributor passages enumerate the names across several consecutive entries.
  • Contributors (alphabetical order): A separate line identifies Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot as technical leads.These names are explicitly grouped under the technical-leads heading.

Details of pre-trained performances.

The pre-trained evaluations cover factuality, reasoning, STEM, code, multimodal understanding, multilinguality, and long-context performance across multiple benchmarks and model comparisons.

  • Factuality and common-sense: Factuality and common-sense evaluations use standard benchmarks including HellaSwag, BoolQ, PIQA, TriviaQA, ARC, WinoGrande, BBH, and DROP.The paper reports these results in Table 9 for comparison with previous model versions.
  • STEM and code: STEM performance improves consistently over pre-trained predecessors, while code improves for the 4B and 12B models but not the 27B model.The evaluated STEM and code benchmarks include MMLU, MATH, GSM8K, GPQA, MBPP, and HumanEval.
  • Image understanding: Multimodal evaluations cover captioning, document understanding, visual question answering, diagrams, charts, spatial reasoning, and counting.These evaluations are reported across the listed vision benchmarks, with Table 11 scores measured without Pan and Scan.
  • Comparison to PaliGemma 2: After multimodal fine-tuning, Gemma 3 excels on document-understanding benchmarks and outperforms the larger PaliGemma 2 variant, while results vary across VQAv2 and COCO Caption.Gemma 3 4B and 12B are also described as about 10x cheaper to transfer at the same 896 x 896 resolution.
  • Multilinguality: Multilingual evaluations use MGSM, Global-MMLU-Lite, WMT24++, FLoRes, XQuAD, ECLeKTic, IndicGenBench, and XOR QA.IndicGenBench is summarized as an average over the detailed benchmarks in Table 14.
  • Long context and IT evaluations: Long-context evaluations measure pre-trained and instruction-tuned models on RULER and MRCR at 32K and 128K sequence lengths.Additional instruction-tuned evaluations include internal and external benchmarks, with N2C and BBEH explicitly defined.
Loading 2503.19786v1…