PaperlayerView exact revision on arXiv

Source-linked AI summary

Gemma 2: Improving Open Language Models at a Practical Size

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, Alek Andreev

arXiv:2408.00118v3cs.CLcs.AI

TL;DR

Small models have improved mainly through longer training, but gains scale inefficiently and require enormous token counts. Gemma 2 explores richer training information through distillation, reporting stronger performance than comparable-scale open models and competitiveness with models more than twice its size, while noting unresolved limitations in factuality, robustness, reasoning, and alignment.

  • Problem

    Small models require increasingly long training for marginal gains, motivating alternatives that improve the information received at each training step.

  • Method

    Gemma 2 trains its 2B and 9B models with knowledge distillation from larger language models over large token quantities, alongside releasing a 27B model trained from scratch.

  • Results

    Gemma 2 significantly advances performance over comparable-scale open models and is competitive with some models more than twice its size across automated benchmarks and human evaluations.

  • Takeaways & Limitations

    Distillation improves small-model performance over training from scratch and can produce superior results to purely next-token training.

  • Takeaways & Limitations

    The authors identify unresolved limitations requiring further research in factuality, robustness to adversarial attacks, reasoning, and alignment.

Abstract

from arXiv · show

In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.

1. Introduction

Gemma 2 targets the limits of scaling small models by improving training information and Transformer design. It combines knowledge distillation and architectural modifications, achieving strong performance relative to comparable and larger open models while requiring use-case-specific safety testing.

  • Motivation: Small-model gains increasingly rely on longer training, with up to 15T tokens yielding less than 1–2% state-of-the-art improvement.This motivates alternatives that improve performance without solely increasing training length.
  • Training approach: Gemma 2 replaces next-token prediction with knowledge distillation for its 2B and 9B models, using a large language model as teacher.The teacher supplies a distribution over potential next tokens, providing richer training information.
  • Architecture: The architecture interleaves local and global attention layers and uses Grouped-Query Attention.These are established Transformer modifications applied in Gemma 2.
  • Results: Gemma 2 significantly advances performance relative to comparable-scale open models and competes with some models more than twice its size.Evaluations span automated benchmarks and human evaluations across domains including question answering, commonsense reasoning, mathematics, science, and coding.
  • Scope and deployment: The report covers architecture, training, pre- and post-training recipes, broad evaluations, safety, responsible deployment, limitations, and advantages.The models are released to the community, but users should conduct rigorous safety testing for their specific use cases.

2. Model Architecture

Gemma 2 retains the decoder-only Transformer while introducing deeper networks and several architectural changes. Local-global attention alternation and GQA shape its efficiency and context handling, with ablations reporting faster inference for GQA while maintaining downstream performance.

  • Core architecture: Gemma 2 uses a decoder-only Transformer with an 8192-token context length, RoPE, and an approximated GeGLU non-linearity.The architecture also uses RMSNorm around Transformer sub-layers, attention, and feedforward layers.
  • Attention: The model alternates local sliding-window and global attention in every other layer.Local attention uses a 4096-token window, while global attention spans 8192 tokens.
  • Stabilization: Gemma 2 caps logits in attention and final layers between −soft_cap and +soft_cap.The soft-cap values are 50.0 for self-attention and 30.0 for the final layer.
  • Grouped-Query Attention: Grouped-Query Attention uses num_groups = 2, with ablations showing increased inference speed while maintaining downstream performance.The comparison reports overall few performance changes between GQA and MHA across several benchmarks.

3. Pre-training

Gemma 2 is pretrained at three scales on primarily English data from web, code, and science sources. The 2B and 9B models use teacher-based knowledge distillation, while the report also describes large-scale infrastructure and carbon emissions.

  • Training data: Gemma 2 27B, 9B, and 2B models are trained on 13T, 8T, and 2T tokens, respectively.The data include web documents, code, and science articles, and the models are not multimodal or specifically trained for state-of-the-art multilingual capabilities.
  • Data processing: The pretraining data use filtering to reduce unsafe utterances, sensitive information, evaluation contamination, and sensitive-output recitation.These filtering steps are described as part of the Gemma 1 data-filtering approach.
  • Knowledge distillation: Knowledge distillation trains smaller models from a teacher’s token probabilities rather than only one-hot next-token targets.The student minimizes negative log-likelihood between teacher and student probabilities.
  • Infrastructure: Training uses TPUv4, TPUv5e, and TPUv5p configurations with data replication, model sharding, and optimizer-state sharding.The 2B, 9B, and 27B runs use 512, 4096, and 6144 chips, respectively.
  • Environmental accounting: Pretraining emissions are estimated at 1247.61 tCO2eq, calculated from TPU data-center energy use and additional data-center energy.The report states that Google data centers are carbon neutral through efficiency, renewable-energy purchases, and offsets.

4. Post-Training

Gemma 2 instruction tuning combines supervised fine-tuning, reinforcement learning from human feedback, model averaging, and filtered data. The recipe extends Gemma 1.1 with conversationally oriented reward modeling and formatting changes for multi-turn dialogue.

  • Pipeline: Post-training applies supervised fine-tuning, RLHF, and model averaging to convert pretrained models into instruction-tuned models.The final recipe and data mixtures were selected using tuned hyperparameters.
  • Model merging: Model averaging combines models produced by the pipeline with different hyperparameters.The report presents averaging as a post-training component intended to improve overall performance.
  • Supervised fine-tuning: Supervised fine-tuning uses synthetic and real prompts with predominantly teacher-generated responses, plus teacher distillation on the student distribution.This combines behavioral cloning with distillation during SFT.
  • RLHF: RLHF uses a reward model an order of magnitude larger than the policy and oriented more toward conversational, especially multi-turn, capabilities.The algorithm is similar to Gemma 1.1, but the reward model differs.
  • Data filtering: Synthetic-data filtering removes personal information, unsafe or toxic outputs, mistaken self-identification, and duplicates.Data encouraging attribution, hedging, and refusals improves factuality metrics without degrading other metrics.
  • Formatting: Gemma 2 changes the dialogue formatting schema so generations explicitly end with <end_of_turn><eos> tokens.Multi-turn continuation removes the generated <eos>, restores the user-turn control tokens, and continues the chat template.

5. Ablations

The ablations identify knowledge distillation as the main performance driver, while architecture changes offer targeted efficiency or consistency benefits. Distillation improves smaller models across sizes, GQA preserves performance with lower inference cost, and sliding-window changes enable modest speed gains.

  • Distillation: Distilling from a larger model improves performance over training a 2B model from scratch on 500B tokens.The comparison uses a 7B teacher, with 500B tokens equal to 10× the compute-optimal quantity for a 2B model.
  • Distillation: Distillation gains remain as model size increases when smaller models are trained with a fixed 7B teacher.The ablation measures perplexity on a validation set across different model sizes.
  • Attention: GQA produces few performance changes relative to MHA across four benchmarks for the 9B model.GQA is selected because it requires fewer parameters and is faster at inference time.
  • Network shape: A deeper 9B network is slightly but consistently better than a wider 9B network with the same parameter count.The consistent cross-benchmark gap supports switching to the deeper architecture.
  • Inference: Changing the local-attention sliding-window size at inference has moderate perplexity impact and can provide slight inference-speed gains.The adjustment is evaluated for the 9B model.
  • Evaluation: Gemma 2B models are slightly less robust to formatting variations than larger Gemma 2 models, while Mistral 7B is significantly less robust.Robustness is assessed using standard deviations across 12 MMLU formatting and evaluation combinations.

6. Evaluation

Gemma 2 models are evaluated across automated benchmarks, human preferences, instruction following, safety, and multi-turn conversations. Across these evaluations, they perform strongly against similarly sized and larger models, with improvements over earlier Gemma versions.

  • Evaluation setup: Gemma 2 is evaluated across automated benchmarks and human evaluations, with comparisons focused on models of similar size and permissive licenses.The evaluation uses total rather than active parameters because total memory usage often limits deployment on standard devices.
  • Pre-training evaluations: The 27B model is best in its size category and competitive with a larger model trained for longer.The comparison includes Qwen1.5 34B and LLaMA-3 70B.
  • Pre-training evaluations: The 2B and 9B models improve substantially over previous versions, with gains of up to 10% on some benchmarks for the 9B model.The 2B comparison shows significant improvement despite similar training-token counts, supporting the reported benefit of distillation under that condition.
  • Post-training evaluations: Gemma 2 instruction-tuned models set a new state of the art on the LMSYS Chatbot Arena.Blind side-by-side human ratings are used to compare models.
  • Post-training evaluations: Gemma 2 models strongly outperform open models in their parameter ranges, while Gemma 27B ranks above Llama 3 70B with Elo scores of 1218 and 1206.Gemma 9B is similar to GPT-4-0314 at Elo 1187 versus 1186.
  • Post-training evaluations: Human evaluations report improved safety, instruction-following preferences, user satisfaction, and conversation-goal achievement relative to older Gemma models.The multi-turn evaluation uses 500 scenarios, and instruction-following results improve by several percentage points across models.

7. Memorization and Privacy

Gemma 2 exhibits low exact and approximate memorization rates under the reported prompted-generation evaluation. Memorization is lower than prior or comparable-size models across data sources, with limited personal-information emission detected.

  • Evaluation method: The evaluation measures exact and approximate memorization after prompting with 50 tokens and assessing 50 generated tokens.Approximate matching uses an edit-distance threshold of 10%.
  • Verbatim memorization: Gemma 2 memorization rates are below 0.1% and significantly lower than prior models at similar sizes.The reported definition concerns induced near-copies of training examples, not arbitrary retrieval of training data.
  • Verbatim memorization: Gemma 2 memorizes more from code, wiki, and science sources, while memorizing significantly less across data sources overall.This pattern is described as similar to Gemma 1 for the relative source breakdown.
  • Approximate memorization: Approximate memorization is higher than exact memorization but remains low and is much lower than Gemma 1’s exact memorization.The increase over exact memorization is lower than in prior models and is absent in some cases.
  • Personal data: No high-severity personal data was emitted, and lower-severity personal information accounted for 0.00026% of memorized data.The automated detection tools may produce false positives because they do not account for context.

8. Responsibility, Safety, Security

Gemma 2’s responsibility approach combines training-time mitigation, evaluations, and deployment guidance. The reported assurance studies cover safety policies and capabilities relevant to extreme risks, with mixed capability findings.

  • Safety approach: Gemma 2 follows a three-pillar safety approach spanning training-time mitigation, model evaluations, and responsible deployment practices.The report also discusses broader implications, limitations, and advantages.
  • Safety approach: The report emphasizes monitoring evolving risks and understanding real-world use because open models can enable malicious applications.Examples include deepfake imagery, disinformation, and illegal or disturbing material.
  • Safety mitigation: Safety mitigation includes data filtering, supervised fine-tuning, and reinforcement learning from human feedback to steer models away from undesirable behavior.Fine-tuned models are aligned with Google’s safety policies.
  • Safety evaluations: Gemma 2’s overall violation rate is significantly lower on the listed safety policies, particularly for child-safety content.The assurance evaluation uses synthetic adversarial queries and human raters.
  • Extreme-risk evaluations: Gemma models show low knowledge in biological, radiological, nuclear, and chemical risk domains.These findings come from closed-ended knowledge-based evaluations.

Code vulnerability detection

The report evaluates Gemma 2 on code vulnerability detection, persuasion, rapport, deception, and donation tasks. Results indicate narrow strengths alongside limited or comparable capabilities in several risk-relevant settings.

  • Code vulnerability detection: Gemma 2 shows close-to-chance performance on PrimeVul, DiverseVul, and SPI vulnerability-detection datasets, while matching Gemini 1.0 Ultra on SecretPatch.The evaluation reports accuracy for these datasets.
  • Persuasion and rapport: Gemma 2 27B performs better than Gemini 1.0 models in the Charm Offensive rapport evaluation.Participants also described the model as an excellent conversationalist and often reported enjoying the interaction.
  • Persuasion and deception: In Hidden Agenda tasks, all models persuade some participants to take the targeted actions, but Gemma 2 does not differ significantly from Gemini models.The actions involve clicking a suspicious link, finding information, or running obfuscated JavaScript.
  • Donation persuasion: The report measures charitable persuasion by asking participants how much of a £20 bonus they would donate after conversing with the model.The outcome is the average indicated donation amount.
  • Persuasion and deception: Gemma 2 is significantly weaker than a human baseline at persuading participants to accept an incorrect answer.As with previous models, it is more persuasive when telling the truth than when lying.

9. Discussion and Conclusion

Gemma 2 presents distillation as effective for training open language models, while acknowledging unresolved limitations and the need for further research.

  • Distillation enables superior results over purely next-token prediction for Gemma 2 models.
  • The released models are intended to expand community access to capabilities previously seen only in large-scale language models.
  • Future research must address factuality, adversarial robustness, reasoning, and alignment.

Contributions and Acknowledgments

The report acknowledges broad contributions from core contributors, technical advisors, and technical leads.

  • Core contributors: The acknowledgments identify a group of core contributors to the Gemma 2 work.
  • Technical advisors: The acknowledgments separately recognize technical advisors.
  • Technical leads: The acknowledgments separately recognize technical leads.
Loading 2408.00118v3…