Source-linked AI summary
Gemma: Open Models Based on Gemini Research and Technology
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu-hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, Kathleen Kenealy
TL;DR
Gemma addresses the need for openly available language models whose performance, safety, and responsible development can be evaluated across broad domains. The paper presents an open family with instruction-tuning and safety evaluations, finding state-of-the-art performance among open models and stronger safety-benchmark results than competitors.
Problem
Responsible release of open language models matters for safety, equitable access, rigorous evaluation, and future innovation.
Method
The paper presents Gemma, an openly available family of generative language models, and uses supervised fine-tuning followed by reinforcement learning from human feedback to improve downstream evaluations.
Results
Gemma advances state-of-the-art performance across broad domains, outperforms competitors on 6 standard safety benchmarks and human side-by-side evaluations, and achieves 64.3% on MMLU and 44.4% on MBPP.
Takeaways & Limitations
Gemma provides an openly available family of models alongside safety evaluations and mitigations intended to support responsible model development.
Takeaways & Limitations
The authors cannot prevent bad actors from fine-tuning Gemma for malicious intent and state that more robust mitigation strategies are required.
Abstract
from arXiv · showhide
This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models (2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of model development. We believe the responsible release of LLMs is critical for improving the safety of frontier models, and for enabling the next wave of LLM innovations.
Introduction
Gemma is an open model family offered in two sizes with pretrained and fine-tuned checkpoints. It targets broad capabilities while emphasizing evaluation, safety, and responsible deployment.
- Gemma is an openly available family of generative language models.
- The 2B model targets CPU and on-device applications, while the 7B model supports efficient GPU and TPU deployment.Each size addresses different computational constraints, applications, and developer requirements.
- Both sizes include pretrained checkpoints and versions fine-tuned for dialogue, instruction-following, helpfulness, and safety.
- Gemma advances performance relative to comparable-scale and some larger open models across automated benchmarks and human evaluation.Evaluated domains include question answering, commonsense reasoning, mathematics, science, and coding.
- Responsible release is presented as important for improving frontier-model safety, equitable access, rigorous evaluation, and future innovation.The authors also state that testing cannot cover every application and scenario, requiring use-case-specific safety testing before deployment.
- The report covers model development, broad quantitative and qualitative evaluations, and approaches to safe and responsible deployment.
Model Architecture
Gemma uses a transformer-decoder architecture with an 8192-token context and several post-Transformer design improvements. The 7B and 2B models differ in their attention mechanisms.
- Gemma’s architecture is based on a transformer decoder and uses an 8192-token context length.
- The 7B model uses multi-head attention, whereas the 2B checkpoints use multi-query attention with num_kv_heads = 1.The choice follows ablations indicating that multi-query attention works well at small scales.
- Rotary positional embeddings replace absolute positional embeddings, and embeddings are shared across inputs and outputs to reduce model size.
- GeGLU replaces the standard ReLU non-linearity with an approximated GeGLU activation function.
- RMSNorm normalizes the inputs of each transformer sub-layer, including attention and feedforward layers, to stabilize training.
Training Infrastructure
Gemma training uses TPUv5e pods with sharded model and optimizer states, coordinated through distributed software and networking approaches. The report also estimates training emissions and accounts for data-center energy.
- The 7B model trains across 16 TPUv5e pods totaling 4096 chips, while the 2B model trains across 2 pods totaling 512 chips.
- Within pods, the 7B model uses 16-way model sharding and 16-way data replication, while the 2B model uses 256-way data replication.
- Optimizer state is further sharded using techniques similar to ZeRO-3, and cross-pod data-replica reduction uses the data-center network.
- Jax’s single-controller paradigm, Pathways, GSPMD, and the MegaScale XLA compiler support orchestration and distributed training computation.
- Training carbon emissions are estimated from hourly data-center energy use and per-cell carbon-emission data, including additional energy for data-center creation and maintenance.
- Google data centers are described as carbon neutral through energy efficiency, renewable-energy purchases, and carbon offsets.
Pretraining
Gemma is pretrained on large, primarily English datasets spanning web documents, mathematics, and code. Data processing filters sensitive or harmful content, checks contamination, and stages the corpus mixture during training.
- The 2B and 7B models are trained on 3T and 6T tokens respectively, using primarily English web documents, mathematics, and code.Unlike Gemini, Gemma is not multimodal and is not trained for state-of-the-art multilingual performance.
- Gemma uses a Gemini-derived SentencePiece tokenizer with byte-level unknown-token encoding and a 256k-token vocabulary.The tokenizer splits digits and preserves extra whitespace for compatibility.
- Pretraining data is filtered with heuristics and model-based classifiers to remove harmful, low-quality, personal, and other sensitive content.
- Evaluation sets are filtered from the pretraining mixture and targeted contamination analyses check for evaluation-set leakage.
- The final data mixture is selected through ablations on both model sizes and staged to increase high-quality relevant data toward the end of training.
Instruction Tuning
Gemma’s instruction-tuning pipeline combines filtered data, supervised fine-tuning, formatting controls, and RLHF to improve helpfulness, factuality, safety, and evaluation performance. The process uses specialized data selection and preference optimization while requiring a specific conversation formatter.
- Instruction Tuning: SFT and RLHF both improve downstream automatic evaluations and human preference evaluations of model outputs.
- Supervised Fine-Tuning: Held-out prompt comparisons with a larger language model guide SFT data mixtures across instruction following, factuality, creativity, and safety.
- Filtering: Synthetic data is filtered to remove personal information, unsafe or toxic outputs, mistaken self-identification data, and duplicates.
- Filtering: Data subsets encouraging attribution, hedging, and refusals improve factuality metrics without degrading other metrics.
- Filtering: The final data mixtures and SFT recipe are selected to improve helpfulness while minimizing safety- and hallucination-related harms.
- Formatting: A dedicated formatter marks conversation roles and turn boundaries with reserved tokenizer control tokens during both training and inference.
- Formatting: Without the formatter, coherent generation remains possible but is out-of-distribution and very likely produces worse generations.
- Reinforcement Learning from Human Feedback: RLHF trains a Bradley-Terry reward function from human preference pairs and optimizes it with a novel reinforcement-learning algorithm.
Evaluation
Gemma is evaluated across automated academic benchmarks, human preference studies, safety, and memorization. Results show strong performance against comparable open models, alongside low or comparable memorization rates and explicit evaluation limitations.
- Human Preference Evaluations: 61.2% positive win rate for Gemma 7B IT and 45% for Gemma 2B IT over Mistral v0.2 7B Instruct on instruction-following prompts.On basic safety prompts, Gemma 7B IT achieved a 63.5% win rate and Gemma 2B IT achieved 60.1%.
- Evaluation Scope: The evaluation covers physical and social reasoning, question answering, coding, mathematics, commonsense reasoning, language modeling, and reading comprehension using automated and human evaluation.The report also compares Gemma with external open-source models across academic benchmarks and uses methodology aligned with Gemini or replicated from the Mistral report for selected benchmarks.
- Automated Benchmarks: Gemma 7B outperforms all same-or-smaller-scale OSS alternatives on MMLU and several larger models, including LLaMA2 13B.The benchmark authors gauge human expert performance at 89.8%, leaving room for further improvement toward Gemini and human-level performance.
- Automated Benchmarks: At least 10 points separate Gemma from other models on GSM8K and MATH, while Gemma exceeds alternate open models by at least 6 points on HumanEval.On MBPP, Gemma 7B scores 44.4% versus 41.4% for CodeLLaMA-7B.
- Memorization Evaluations: No sensitive data was memorized, while some potentially personal data was memorized at lower rates and automated detection likely overestimates such findings.The personal-data detector uses pattern matching without context, producing many false positives.
Responsible Deployment
Gemma’s responsible deployment approach balances the benefits of open access with risks from malicious and unintended uses. The paper combines safety evaluations, mitigations, transparency resources, and continued research while requiring users to test models for their own applications.
- Responsible development uses structured assessments to identify, measure, and manage foreseeable downstream societal impacts.
- Open release can advance science, innovation, access, evaluation, and beneficial applications, but must be balanced against risks of enabling harm.
- Open model weights create deployment challenges, including malicious fine-tuning and unintended toxic, discriminatory, hallucinatory, or privacy-harming behavior.
- Mitigations include filtering and measuring biases in pretraining data, standardized safety benchmarks, internal red teaming, and ethics and safety evaluations.
- The release includes model documentation and a responsible toolkit, while Google DeepMind commits to continuous mitigation research because new uses and misuses remain possible.
Discussion and Conclusion
Gemma is presented as an openly available family of generative language models for text and code, with reported gains in performance and safety. The authors also emphasize unresolved risks and the need for further research toward robust, reliable models.
- Gemma is an openly available family of generative language models for text and code that advances open-model performance, safety, and responsible development.
- The models outperform competitors on 6 standard safety benchmarks and in human side-by-side evaluations.
- Gemma improves performance across dialogue, reasoning, mathematics, and code generation, with MMLU at 64.3% and MBPP at 44.4%.
- The authors expect researchers and developers to use Gemma for new research, beneficial applications, user experiences, and functionality.
- Further research is needed to create robust, safe models that reliably perform as intended, including work on factuality and alignment.
Contributions and Acknowledgments
The paper credits a large, distributed team spanning technical, research, operational, and leadership roles. It also acknowledges broad organizational support for the Gemma effort.
- Core contributors include Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, and other named collaborators.
- The contributor list spans research and technical collaborators, including Aakanksha Chowdhery, Adam Roberts, Aditya Barua, and many others.
- The project identifies separate product-management, program-management, executive-sponsor, lead, and technical-lead roles.
- Acknowledged supporting teams include Gemini, Gemini Safety, Gemini Infrastructure, Gemini Evaluation, Google Cloud, Google Research Responsible AI, Kaggle, and Keras.
Gemma 1.0 IT results
The appendix preserves Gemma 1.0 IT results for comparison with the paper’s core Gemma 1.1 IT results. It includes side-by-side comparisons with Mistral 7B v0.2 and safety benchmark results against similarly sized open models.
- The paper’s core results concern Gemma 1.1 IT, while Gemma 1.0 IT results are retained in the appendix for comparison.
- Table 9 reports Gemma 1.0 IT win rates versus Mistral 7B v0.2 Instruct with 95% confidence intervals and win, tie, and loss breakdowns.
- Table 10 reports safety academic benchmark results for Gemma 1.0 IT models compared with similar-size open models.