PaperlayerView exact revision on arXiv

Source-linked AI summary

2 OLMo 2 Furious

Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerquin, David Heineman, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Jake Poznanski, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, Hannaneh Hajishirzi

arXiv:2501.00656v3cs.CLcs.LG

TL;DR

Open language models are increasingly capable, but released weights alone do not provide the artifacts needed for diverse, reproducible research. OLMo 2 develops and fully releases a 7B–32B family using stability improvements, staged data curricula, and post-training methods, achieving competitive performance with fewer training FLOPs while retaining transparency.

  • Problem

    Open-weight models increasingly narrow the gap with closed systems, but weights alone are insufficient for diverse research requiring transparent development artifacts.

  • Method

    OLMo 2 combines stability-oriented architecture and infrastructure changes, two-stage pretraining with late-stage specialized data and micro-annealing, and Tülu 3-based post-training with RLVR.

  • Results

    OLMo 2 base and instruction-tuned models are competitive with comparable open-weight models and some proprietary models, while the base models require far fewer training FLOPs and retain full openness.

  • Takeaways & Limitations

    OLMo 2 provides a fully open foundation for reproducing, studying, and extending competitive language-model training and post-training research.

  • Takeaways & Limitations

    The learning-rate experiments did not cover the full range because of cost concerns, leaving the plateau boundaries insufficiently defined.

Abstract

from arXiv · show

We present OLMo 2, the next generation of our fully open language models. OLMo 2 includes a family of dense autoregressive language models at 7B, 13B and 32B scales with fully released artifacts -- model weights, full training data, training code and recipes, training logs and thousands of intermediate checkpoints. In this work, we describe our modified model architecture and training recipe, focusing on techniques for achieving better training stability and improved per-token efficiency. Our updated pretraining data mixture introduces a new, specialized data mix called Dolmino Mix 1124, which significantly improves model capabilities across many downstream task benchmarks when introduced via late-stage curriculum training (i.e. specialized data during the annealing phase of pretraining). Finally, we incorporate best practices from Tülu 3 to develop OLMo 2-Instruct, focusing on permissive data and extending our final-stage reinforcement learning with verifiable rewards (RLVR). Our OLMo 2 base models sit at the Pareto frontier of performance to training compute, often matching or outperforming open-weight only models like Llama 3.1, Qwen 2.5, and Gemma 2 while using fewer FLOPs and with fully transparent training data, code, and recipe. Our fully open OLMo 2-Instruct models are competitive with open-weight only models of comparable size and even some proprietary models like GPT-3.5 Turbo and GPT 4o Mini.

1 Introduction

OLMo 2 addresses the need for more transparent and reproducible language-model research by releasing a fully open model family and detailing the development pipeline. It targets stability, staged pretraining, post-training, and infrastructure while reporting competitive base and instruction-tuned performance.

  • Scope and motivation: OLMo 2 introduces fully open 7B, 13B, and 32B models with detailed training data, code, recipes, evaluation, and infrastructure documentation.The report describes the full model-development procedure, including training data, pretraining, post-training, and evaluation.
  • Reported performance: OLMo 2 base models are competitive with Llama 3.1, Qwen 2.5, and Gemma 2, while OLMo 2-Instruct models are competitive with open-weight and some proprietary models.The report positions these results as evidence that the released development approach produces competitive models.
  • Pretraining Stability: Training stability is a central target because instabilities and loss spikes increase costs and can harm final performance.The authors discuss techniques intended to improve stability and ensure the performance of final trained models.
  • Mid-training Recipe: Two-stage pretraining uses a later mid-training stage to infuse new knowledge and patch capability deficiencies, with micro-annealing reducing experimentation costs.The mid-training stage evaluates data sources independently before broader use.
  • Post-training Pipeline: OLMo 2-Instruct applies the Tülu 3 recipe, permissive data, and multi-stage reinforcement learning with verifiable rewards.The post-training pipeline is designed to improve chat variants built from stronger base models.
  • Infrastructure as a Research Catalyst: Infrastructure investments in monitoring and orchestration reduced failure rates and increased cluster utilization.The report treats reliable infrastructure as necessary for efficient large-scale pretraining.

2 OLMo 2 Family

OLMo 2 is a family of fully open 7B, 13B, and 32B decoder-only language models whose architecture, tokenizer, training recipe, data, and evaluation are designed for stability, efficiency, and competitive performance. Its two-stage training uses specialized mid-training data, while the released artifacts support reproduction and research.

  • Model architecture: The architecture adds RMSNorm, reordered normalization, QK-norm, z-loss regularization, and a RoPE θ of 500,000 to improve training stability.QK-norm is intended to prevent overly large attention logits, while z-loss regularization has been empirically associated with improved run stability.
  • Tokenizer: OLMo 2 uses a larger cl100k-based tokenizer while retaining earlier masking tokens for backwards compatibility with early Dolma data.The tokenizer comparison reports measurable gains, particularly on OLMES tasks, but a slight disadvantage at this model size and compute budget.
  • Base model training recipe: Base models train in two stages: long web-heavy pretraining followed by shorter mid-training that emphasizes higher-quality, curated, and synthetic data.Pretraining uses 90–95% of training FLOPs, while mid-training uses 5–10% and applies a linear learning-rate decay to zero.
  • Mid-training data: Dolmino Mix 1124 focuses mid-training on STEM references, high-quality text, and domain-specific data intended to patch deficiencies such as math-solving.For OLMo 2 7B, the authors anneal three 50B-token runs with different data orders and average the resulting models.
  • Evaluation and results: OLMo 2 models are competitive with comparable open-weight models while using fewer training FLOPs, and OLMo 2-Instruct is competitive with open-weight and some proprietary models.The authors report that gains on development metrics largely translate to an unseen evaluation suite, while noting that comparable models’ held-out status cannot be guaranteed.

3 Deep Dive: Pretraining Stability

OLMo 2 addresses training instability—especially escalating gradient and loss spikes that can cause divergence—with data, initialization, normalization, and optimizer interventions. The revised initialization and training choices improve stability across model widths, while repeated n-gram filtering reduces spike frequency on average.

  • Motivation: Training instability involved sudden loss and gradient-norm spikes, gradient-norm growth, and eventual divergence that made larger-scale training difficult.More dramatic gradient-norm spikes often preceded loss spikes, and larger models showed spikes more frequently.
  • Other interventions: Additional stability changes included RMSNorm, output-side reordered normalization, QK-norm, z-loss, embedding-exempt weight decay, and lowering AdamW ϵ from 10^-5 to 10^-8.The lower AdamW value permits larger early updates and makes the gradient norm settle faster and remain lower.
  • Data-related instability: Repeated n-gram sequences were associated with spikes, although the relationship varied across model size, data ordering, and batch occurrence.The same sequence could spike in one setting but not another, so the association was not deterministic.
  • Data-related instability: Filtering documents with 32 or more repeated n-grams and masking their loss during loading decreased spike frequency on average.The intervention was applied during data curation and training, with Figure 3 showing the effect of masking.
  • Initialization: OLMo 2 initializes every parameter from a normal distribution with mean 0 and standard deviation 0.02 instead of using OLMo-0424’s scaled initialization.The new scheme is intended to preserve activation and gradient scales across layers and transfer better across model widths.
  • Initialization: OLMo 2’s growth exponent was closer to 0 across model widths, indicating greater resistance to exploding or vanishing activations and gradients.The analysis evaluates activation and gradient 2-norm changes across layers using randomly initialized models and sampled documents.
  • Initialization: The new initialization produced no loss spikes and reduced the gradient L2-norm spike score from 0.40 to 0.03, despite slightly slower convergence.The effect persisted across model scales and token counts; other hyperparameter changes compensated for slower convergence.

4 Deep Dive: Mid-training Recipe

OLMo 2’s mid-training recipe combines learning-rate annealing with curated, specialized data mixtures to improve capabilities after pretraining. Experiments show that data composition drives substantial gains, especially on knowledge, reading, and mathematics benchmarks, while learning-rate effects are generally small or context-dependent.

  • 4.1 Learning rate annealing: Learning-rate experiments show higher rates perform better early, but lower rates eventually overtake them.The 30 ⋅10−4 setting became unstable during warm-up, while the 3 ⋅10−4 and 6 ⋅10−4 runs crossed after 200B tokens.
  • 4.1 Learning rate annealing: Annealing to zero over 50B tokens yields equivalent final training loss across four learning-rate variants, aside from a small lag for the lowest setting.This suggests higher learning rates improve mid-training by approximately offsetting worse pretraining loss.
  • 4.1 Learning rate annealing: The 3 ⋅10−4 and 6 ⋅10−4 settings show virtually no difference across nine OLMES tasks after 1T and 2T tokens.Downstream-task noise mirrors the training-loss comparison.
  • 4.1 Learning rate annealing: With 100B high-quality tokens, final OLMES scores differ by less than 0.1 points, although the higher rate is 2.8 points better on GSM8K.The authors caution that more study is needed before treating the GSM8K result as dependable.
  • 4.2 Data Curriculum: Dolmino Mix 1124: Mid-training improves average performance by 10.6 points for OLMo 2 7B and 10.3 points for 13B.Both models improve across knowledge-intensive, reading-comprehension, and mathematics benchmarks.
  • 4.3 Dolmino Mix 1124: High Quality Sources: FT7 FW2+ Math + Ins achieves +1.7 on OLMES, +5.7 on generative tasks, +1.3 on MMLU, and +19.5 on GSM*.The reported mix combines web and math sources with instruction data.
  • 4.4.1 Math Sources: DolminoSynthMath contains 28M synthetic math tokens designed to improve GSM8K and raw mathematical calculations.It includes basic arithmetic question-answer pairs, number-replaced GSM8K examples, and MIND-rewritten GSM8K data.

5 Deep Dive: Post-training Pipeline

OLMo 2-Instruct follows a permissive, multi-stage Tülu 3 post-training pipeline combining SFT, preference tuning, and RLVR. The authors report competitive performance and consistent RLVR gains across model scales.

  • Post-training Pipeline: OLMo 2-Instruct uses three training phases: supervised finetuning, preference tuning with DPO and on-policy data, and RLVR.The pipeline adapts the Tülu 3 recipe with permissive data and adjusted hyperparameters.
  • Supervised Finetuning: SFT combines high-quality existing instruction datasets with scaled PersonaHub-based synthetic data in separate mixes for 7B/13B and 1B/32B models.The primary 7B/13B mix contains 939,104 prompts, while the 1B/32B mix contains 866,138 prompts.
  • Preference Finetuning: Synthetic preference data uses GPT-4o as a judge, selecting the highest-rated completion and sampling another completion as the rejected response.Ratings cover helpfulness, truthfulness, honesty, and instruction following, then are averaged across aspects.
  • RLVR: RLVR applies PPO to GSM8K, MATH, and prompts with constraints, initializing the value function from reward models for 7B and 13B models.The 1B and 32B models instead use GRPO, which does not require a reward model.
  • Hyperparameter Selection: OLMo 2 required significantly higher learning rates than the Llama 3.1 training recipe, with final configurations selected through targeted sweeps.The reported tuning covers SFT, DPO, and reward-model stages for 7B and 13B models.
  • Results: OLMo 2-Instruct approaches Qwen 2.5 14B Instruct and surpasses Tülu 3 8B and Llama 3.1 8B Instruct, while RLVR improves evaluation metrics across scales.The reported gains occur alongside increases in the training reward signal.

6 Deep Dive: Infrastructure as a Research Catalyst

OLMo 2 relies on portable, monitored infrastructure spanning two H100 clusters, enabling distributed training across heterogeneous environments. The environmental analysis estimates substantial emissions while framing the estimate as incomplete.

  • Training Outcomes: RLVR increases training rewards and downstream evaluations such as GSM8K for the 7B model.Figure 15 also tracks KL divergence, response lengths, average suite scores, IFEval, and MATH Flex.
  • Cluster Infrastructure: OLMo 2 training used the Jupiter and Augusta clusters, with 7B and 13B variants trained partially across both through Beaker.Most 7B training occurred on Jupiter and most 13B training on Augusta.
  • Compute: Jupiter provides 1,024 H100 GPUs across 128 servers, while Augusta uses 160 nodes of A3 Mega virtual machines with eight H100 GPUs each.The clusters differ in hardware and operating environments but both supplied sufficient training throughput.
  • Workload Management: Beaker supports portability across three data centers and isolates concurrent workloads through containerization and resource allocation that minimizes noisy-neighbor effects.Workloads can typically move between locations by changing one line of code.
  • Reliability: GPU health checks, node cordoning, monitoring, and automatic rescheduling reduce interruptions and quarantine unreliable hardware before it destabilizes workloads.Beaker tests tensor multiplication before execution and can restart failed training jobs automatically.
  • Environmental Impact: 154 tCO2eq is the estimated emissions from training the latest models, calculated from power consumption and location-specific carbon intensity.The analysis uses 0.332 kg CO2 per kWh for Jupiter and 0.352 kg CO2 per kWh for Augusta.

Conclusion

OLMo 2 and OLMo 2-Instruct extend fully open language modeling to 7B, 13B, and 32B scales. The release combines competitive performance with broad reproducibility artifacts and documented development practices.

  • Conclusion: OLMo 2 and OLMo 2-Instruct are fully open 7B, 13B, and 32B parameter models trained on up to 6T tokens.The authors release training and evaluation code, datasets, checkpoints, and logs required to reproduce and extend the models.
  • Conclusion: Base and instruct models are competitive with similarly sized open-weight models including Qwen 2.5, Gemma 2, and Llama 3.1.The report frames this progress as part of building an ecosystem for studying and sharing language-model training methods.

A OLMo 2 Evaluation Framework

OLMo 2 is evaluated with standard language-model benchmarks, while OLMo 2-Instruct is assessed on diverse tasks to measure downstream adaptation. OLMES provides a unified toolkit for development and evaluation.

  • Evaluation Framework: OLMES is a unified, standardized evaluation suite and toolkit used to guide model development and assess language-model performance.Its role spans evaluation during development and final performance assessment.

A.1 Base Model Eval

OLMo 2 base models are evaluated using a standardized suite spanning multiple-choice, generative, and held-out tasks. The evaluation follows OLMES principles and makes implementations publicly available.

  • Evaluation Suite: 11 tasks evaluate OLMo base models across multiple-choice, generative, and held-out settings.The suite contains 5 multiple-choice tasks, 2 generative tasks, and 4 held-out tasks.
  • Evaluation Protocol: OLMES standardizes dataset splits, sampling, formulations, normalization, and evaluation procedures for reproducible language-model comparisons.For multiple-choice tasks, both MCF and CF formulations are evaluated, with the better-performing formulation used.
  • Generative Tasks: Generative evaluation covers factual knowledge and reading comprehension using Natural Questions, Jeopardy, SQuAD, DROP, and CoQA.Most generative tasks use 5-shot curated in-context examples, while CoQA uses prior conversation turns instead.
  • Held-out Tasks: Held-out evaluation tests admission and qualification exams, challenging reasoning, mathematics, advanced knowledge, and unseen factual knowledge.The suite includes AGIEval English, BBH, GSM8K, MMLU Pro, and TriviaQA.
  • Reproducibility: The evaluation implementations are publicly available through the OLMES repository.The repository is hosted at github.com/allenai/olmes.

A.2 Instruct Model Eval

OLMo 2-Instruct is evaluated with OLMES configurations spanning chat, reasoning, knowledge, instruction-following, code, and mathematics. The section also describes a smaller experimental model used to develop the recipe.

  • Evaluation Setup: Instruct models are evaluated using the OLMES benchmark suite and the configuration reported by Tülu 3.The evaluation follows existing practices in the literature.
  • Benchmark Coverage: The instruct suite includes chat versions of GSM8K and BBH, long-tail knowledge, misconceptions, and instruction-following tasks.The named tasks include PopQA, TruthfulQA, IFEval, and AlpacaEval 2.
  • Specialized Tasks: Code completion and competition mathematics are evaluated with OLMES answer extraction and setup.The code tasks are HumanEval and HumanEval+, while mathematics uses MATH.
  • Recipe Development: A 1B experimental model is used to test recipe development for the target 7B, 13B, and 32B model sizes.The 1B model differs from OLMo 2 7B in layers, hidden size, attention heads, and batch size.

B.1 Difficulties with OLMo 2 1B

OLMo 2 1B was used to develop the larger-model recipe, but scaling its training proved difficult. Its results indicate strong sensitivity to late-stage specialized data and limited base-model capacity.

  • Recipe Transfer: Recipe findings from OLMo 2 1B generalize to the 7B, 13B, and 32B scales, based on competitive results.The paper uses the 1B model as a development proxy for the target sizes.
  • Training Limitation: Scaling the number of training tokens for OLMo 2 1B was difficult.This is the section’s explicit training limitation for the 1B model.
  • Training: OLMo 2 1B was pretrained for 4T tokens, followed by a single 50B-token anneal on Dolmino Mix 1124.Training used 2000 warmup steps and a 4.0 ⋅10E−4 peak learning rate.
  • Base Results: OLMo 2 1B remains competitive with SmolLM 2 but lags behind smaller Gemma 2 and Qwen 2.5 base models.The comparison is reported for the main base-model evaluation suite.
  • Interpretation: The paper attributes the 1B difficulty to possible pretraining token inefficiency from limited model capacity.It also hypothesizes that smaller models may need task-specific data or distillation for challenging tasks.
  • Data-Mix Effects: +37.0% is the benefit of Dolmino Mix 1124 for the 1B model, compared with +18.7% for 7B, +15.9% for 13B, and +12.3% for 32B.The reported benefit is higher for smaller base models.
  • Data-Mix Effects: Stage 1-only OLMo 2 1B remains at random-performance levels on several multiple-choice tasks.Reported values are 25% for MMLU and ARC Challenge and 10% for MMLU Pro.
  • Instruct Results: Applying the OLMo 2-Instruct post-training recipe to 1B yields performance competitive with Qwen 2.5 and Gemma 3.This result is reported in Table 23 for comparable open-weight models.

C.2 Additional RLVR Learning Curves

Additional 13B RLVR learning curves are provided for further analysis of reinforcement-learning behavior. The curves appear in Figures 18, 19, and 20.

  • Additional RLVR Learning Curves: Additional 13B RLVR learning curves are provided in Figure 18.
  • Additional RLVR Learning Curves: Additional 13B RLVR learning curves are provided in Figure 19.
  • Additional RLVR Learning Curves: Additional 13B RLVR learning curves are provided in Figure 20.

C.3 OLMo 2-Instruct Preview Models

The initial OLMo 2-Instruct release was reclassified as a preview after a tokenizer inconsistency was found, prompting retraining. Its reported curves and evaluation results document RLVR behavior and preview-model performance.

  • Tokenizer issue: The base and instruct models used different pre-tokenization logic because the base model’s custom logic was lost during instruct training.The discrepancy involved splitting or truncating sequences and was documented through a tokenizer file diff.
  • Tokenizer issue: The authors retrained OLMo 2-Instruct models for consistency with the base models and marked existing post-trained models as previews.The preview learning curves remained available for reference.
  • RLVR evaluations: Figure 18 tracks verifiable rewards, KL divergence, response lengths, downstream evaluations, and average evaluation-suite scores for OLMo-2-1124-13B-RLVR1.The figure separates training curves in the top row from downstream evaluations in the bottom row.
  • RLVR evaluations: Figure 19 reports the corresponding training curves and downstream evaluations for OLMo-2-1124-13B-RLVR2.Solid lines identify the downstream evaluation results and average scores across evaluation suites.
  • RLVR evaluations: Figure 20 presents verifiable rewards, KL divergence, response lengths, and downstream evaluations for OLMo-2-1124-13B-Instruct.The top row contains training curves, while the bottom row contains downstream evaluations and suite averages.
  • RLVR evaluations: Preview-model evaluations include GSM8K, MATH, IFEval, and average scores across evaluation suites, with IFEval OOD values affected by minor bug fixes.The reported IFEval OOD numbers for OLMo 2 7B and 13B are not exactly comparable with earlier results.

D Additional Hyperparameters

The appendix documents models and prompts used for preference-data generation, reward-model initialization, and RLVR evaluation of OLMo 2-Instruct.

  • Preference-data pipeline: The on-policy preference-data generation models are listed separately, with samples drawn from SFT checkpoints.The appendix points to Table 25 for the model list and identifies the SFT checkpoints as an additional source.
  • Preference-data pipeline: External preference-data sources include Yi, Tülu 2, Gemma 2, GPT-4o, MPT, Mistral, and Qwen model variants.These models are enumerated as sources used in the synthetic preference pipeline.
  • RLVR configuration: Table 26 specifies the hyperparameters used to train the reward model for RLVR value-network initialization.The table documents initialization-related reward-model training settings.
  • Preference-data pipeline: Table 27 records the prompt sources for the preference-finetuning datasets.The appendix directs readers to Lambert et al. (2024) for more information about the prompt contents.
  • RLVR evaluations: The RLVR evaluation materials include GSM8K, MATH, and IFEval, alongside average scores across evaluation suites.The appendix labels these evaluations for OLMo-2-1124-13B-Instruct and related preview models.

E Annealing Data Details

The annealing-data appendix documents prompts used to generate hard math word problems and their solutions, including persona conditioning for problem generation.

  • Math-data generation: Figure 24 shows the prompt used to generate hard math word problems, with persona placeholders borrowed from Chan et al. (2024).The prompt uses persona-driven conditioning to specify the generation context.
  • Math-data generation: A separate prompt is used to generate solutions for the hard math word problems.The solution-generation prompt is presented in Figure 25.
Loading 2501.00656v3…