2 OLMo 2 Furious

Motivation

• Open-weight models alone do not expose the training recipes and development pipelines needed to support diverse research, motivating fully open releases of data, code, and recipes.Prior fully open efforts have supported studies of training dynamics, concept acquisition, and memorization, while a performance gap remains.Source: §1 · p. 3

OLMo 2 models

• OLMo 2 models trained on up to 6T tokens are competitive with Llama 3.1, Qwen 2.5, and Gemma 2, while OLMo 2-Instruct is competitive with open-weight models and some proprietary models.The report names GPT-3.5 Turbo and GPT 4o Mini as examples of proprietary models.Source: §1 · p. 3

Pretraining Stability

• The report describes techniques for reducing training instabilities and loss spikes, which can harm final model performance.The authors identify stability as critical to the performance of the final trained model.Source: §1 · p. 4

Mid-training Recipe

• Mid-training uses a later data stage to infuse knowledge and address capability deficiencies, while micro-annealing independently assesses data sources to reduce experimentation costs.The report builds on prior demonstrations of curriculum benefits during pretraining.Source: §1 · p. 4

Post-training Pipeline

• OLMo 2-Instruct applies the Tülu 3 recipe, emphasizes permissive data, and extends reinforcement learning with verifiable rewards across multiple stages.The report also examines how improved base models translate into better chat variants.Source: §1 · p. 4

Infrastructure as a Research Catalyst

• Infrastructure changes and monitoring and orchestration solutions are presented as ways to reduce failure rates and increase cluster utilization.The report details changes from OLMo-0424 that enable OLMo 2's improvements.Source: §1 · p. 4

Development Procedure

• The report describes the full development procedure, covering training data, pretraining, post-training, and evaluation, and notes related work on scaling laws and benchmark standardization.The cited projects include Bhagia et al. (2024) on scaling laws and Gu et al. (2024) on OLMES.Source: §1 · p. 4

2.1 Model Architecture

• OLMo 2 uses RMSNorm, reordered normalization, QK-norm, z-loss, and RoPE with θ increased to 500,000 from 10,000.These changes target activation normalization, attention-logit scale, run stability, and positional encoding resolution.Source: §2.1 · p. 5

2.2 Tokenizer

• The larger cl100k-based OLMo 2 tokenizer brings measurable gains, particularly on OLMES, but is slightly disadvantaged at the evaluated model size and compute budget.The authors expect its larger vocabulary to matter more at larger scales and with more training tokens.Source: §2.2 · pp. 5–6

2.3 Base Model Training Recipe

• OLMo 2 uses long web-heavy pretraining followed by shorter mid-training with higher-quality web and curated non-web data, including synthetic data to patch math capabilities.For model averaging, 7B uses three 50B-token anneals; 13B and 32B use three 100B-token runs and a fourth 300B-token run. Total training is 4.05T, 5.6T, and 6.6T tokens, respectively.Source: §2.3 · pp. 6–7

2.4 Base Model Data

• OLMo 2 Mix 1124 comprises approximately 3.9 trillion tokens, over 95% from web data, combining DCLM baseline 1.0, Dolma 1.7 subsets, and filtered StarCoder code.Code filtering removes documents from repositories with fewer than 2 GitHub stars and documents dominated by repeated words or n-grams.Source: §2.4.1 · pp. 7–8

2.4.2 Mid-training data: Dolmino Mix 1124

• Dolmino Mix 1124 follows pretraining with more restrictively quality-filtered web data and high-quality, often synthetic, domain-specific material to build math-centric skills and expose models to STEM references.The mixture is made in varying sizes for mid-training.Source: §2.3 · p. 7, §2.4.2 · p. 8

2.5 Evaluation and Results

• OLMo 2 models are competitive with the best comparable-size open-weight models while requiring far fewer training FLOPs and retaining full openness, and development gains largely carry over to unseen evaluations.The authors note they cannot guarantee that tasks treated as unseen for OLMo 2 were absent from other models’ development sets.Source: Table 6 · p. 9, §2.5 · p. 10
• OLMo 2-Instruct is competitive with the best instruction-tuned open-weight models and some popular proprietary models.The models use supervised finetuning, on-policy preference tuning, and reinforcement learning with verifiable rewards (RLVR).Source: §2.5 · p. 10

3 Deep Dive: Pretraining Stability

• Training showed loss spikes, more frequent gradient-norm spikes at larger model sizes, and gradually rising gradient norms; combined, these issues could cause divergence.More dramatic gradient-norm spikes often preceded loss spikes, and rising norms correlated with more frequent spikes.Source: §3 · p. 10, Figure 2 · p. 12

3.1 Repeated n-Grams

• Filtering documents with runs of 32 or more repeated n-grams and masking such sequences during loss computation mitigated, but did not eliminate, gradient spikes and did not affect gradual gradient-norm growth.An n-gram is any span of 1 to 13 tokens; the association between repeated sequences and spikes was not deterministic.Source: §3.1 · pp. 12–13, Figure 3 · p. 13

3.2 Model Initialization

• The new initialization produced no loss spikes and reduced gradient-norm spike score from 0.40 to 0.03, with growth exponents nearer zero across widths and better expected hyperparameter transfer.The new initialization converged slightly more slowly; the authors say they compensated through other hyperparameter settings.Source: §3.2 · p. 14, Figure 4 · p. 14, Figure 5 · p. 15

3.3.2 Reordered norm and QK-norm

• Applying RMSNorm after attention and MLP blocks and normalizing queries and keys together reduced gradient spike score from 0.108 to 0.069, though neither change worked well alone.The two changes improve both gradient-norm growth and spikiness when combined.Source: §3.3.2 · p. 15, Figure 7 · p. 16

3.3.3 Z-Loss

• Z-loss adds 10−4 ⋅log2 Z to the loss to discourage final softmax activations from growing too large, improving model stability.Z is the denominator in the softmax over the logits.Source: §3.3.3 · p. 16

3.4.1 ϵ in AdamW

• Lowering AdamW ϵ from 10−5 to 10−8 lets early updates be larger, so gradient norms settle faster and remain lower as training loss improves faster.The authors report that this trend continues in runs longer than those shown.Source: §3.4.1 · p. 17, Figure 9 · p. 17

3.4.2 Weight decay on embeddings

• Excluding embeddings from weight decay lets their norms settle in a healthy region; weight decay otherwise shrinks embeddings and increases gradient norms and spikes.The reported spike scores were 0.16 with embedding decay and 0.092 without it.Source: §3.4.2 · p. 17, Figure 10 · p. 17

4.1 Learning rate annealing

• Higher learning rates perform better early but are eventually overtaken by lower rates; annealing largely erases pretraining loss differences, while 3 ⋅10−4 and 6 ⋅10−4 show virtually no difference across nine downstream tasks.In high-quality-token mid-training, the settings’ OLMES scores are within 0.1 points, though 6 ⋅10−4 is 2.8 points better on GSM8K; the learning-rate sweep was limited by cost.Source: §4.1 · pp. 18–19, Figure 11 · p. 18

4.2 Data Curriculum: Dolmino Mix 1124

• 7B and 13B mid-training improve average performance by 10.6 and 10.3 points, respectively, with 7B surpassing the larger 13B model after pretraining.The models also improve on knowledge-intensive, reading-comprehension, and math benchmarks, including GSM8K.Source: §4.2 · p. 20, Table 9 · p. 20

4.3 Dolmino Mix 1124: High Quality Sources

• High-quality source curation combines web filters, decontaminated instruction data, screened Stack Exchange Q&A, code, and academic or reference material.For web data, the DCLM FastText filter retains about 65.6%; FineWeb Edu thresholds of over 3 and 2 retain 5.8% and 20.3%, respectively.Source: §4.3 · pp. 21–22
• The FT7 FW2+ Math + Ins mix achieves the best overall results: +1.7 OLMES, +5.7 generative tasks, +1.3 MMLU, and +19.5 GSM*.FT7 FW2+ Math performs slightly better on math tasks, motivating further work on math data that combines well with other high-quality sources.Source: §4.3 · p. 23

4.4 Dolmino Mix 1124: Math Mix

• Early web-focused mixes struggled on math, prompting investigation of human-authored and synthetic or augmented data designed to address common errors in math validation sets.The team assessed sources individually with microanneals and found their effectiveness persisted when combined with the other sources.Source: §4.4 · p. 23

4.4.1 Math Sources

• Math sources include persona-conditioned TuluMath, calculation and GSM8K-focused DolminoSynthMath, and filtered synthetic, code-derived, and existing datasets.TuluMath contains about 230M synthetic tokens; DolminoSynthMath contains 28M tokens, including basic arithmetic pairs and rewritten GSM8K examples. TinyGSM-MIND rewrites filtered, executable-code examples from Tiny-GSM.Source: §4.4.1 · pp. 23–24

4.4.2 Evaluating Math Data with Microanneals

• Microanneals assess small math-source mixtures by combining a candidate source with general data and linearly annealing the learning rate, enabling cheaper source-level evaluation.The 19 runs totaled 130B tokens, less than three 50B-token full annealing runs, and improvements were visible after fewer than 10B tokens.Source: §4.4.2 · p. 24
• GSM* rose from 28.5 pre-anneal to 63.5 with a 35/65 math/DCLM mix and 61 with a 10/90 mix.The results suggest that a small proportion of domain-specific data can help during annealing.Source: §4.4.2 · p. 25
• GSM* scores were 61 with one copy of the math data, 66 with two copies, and 65 with four copies.The experiment suggests that limited duplication can provide gains when high-quality domain-specific data is scarce.Source: §4.4.2 · p. 25
• Rewriting Tiny-GSM into natural language dramatically improved GSM* performance, whereas its code-form version degraded performance relative to the baseline.The study contrasts an inline-annotated TinyGSM form with the “Problem Solving” MIND rewritten variant; the pretraining mix contains only 2% code.Source: §4.4.2 · p. 25

4.5 Final Midtraining mix and Checkpoint Soups

• The final Dolmino Mix 1124 comprises 50B-, 100B-, and 300B-token mixes, with filtered DCLM web data at roughly 50% of each token budget.Stack Exchange Q&A and mid-training math are repeated twice in the 100B mix and four times in the 300B mix; FLAN is repeated twice and Wiki four times in the 300B mix. OLMo 2 7B uses the 50B mix, 13B uses the 100B mix, and 13B also tests a longer anneal with the 300B mix; the same procedure is followed for 32B.Source: Table 13 · p. 25, §4.5 · pp. 25–26
• Across six mid-training mixes, checkpoint soups consistently matched or outperformed the best single checkpoint trained on the same mix.The soups averaged three checkpoints annealed on different permutations of the same data mix.Source: Table 14 · p. 26, §4.5 · p. 26
• OLMo 2 7B averages three checkpoints, while OLMo 2 13B and 32B average four; the latter approach outperformed averaging only three 100B runs.For 13B and 32B, the four checkpoints comprise three trained on the 100B sample and one on the 300B sample.Source: §4.5 · p. 26

Supervised Finetuning (SFT)

• SFT retained multilingual data after removing Aya and multilingual Wildchat degraded average scores by ∼0.5 points; the 7B/13B mix contains 939,104 prompts.The 1B/32B mix filters date-cutoff instructions and keeps math examples only when the answer wins a majority vote over five completions; it contains 866,138 prompts.Source: §5 · pp. 26–27

Preference Finetuning (PreFT) with DPO

• DPO preference data used on-policy OLMo 2 SFT responses, completions from 20 models, and GPT-4o ratings to select the highest-rated response and a remaining response as rejected.The pipeline builds on UltraFeedback (Cui et al., 2023) and rates helpfulness, truthfulness, honesty, and instruction-following.Source: §5 · pp. 27–28

Reinforcement Learning with Verifiable Rewards (RLVR)

• PPO-based RLVR used preference-trained reward models and GSM8K, MATH, and constraint prompts; 13B received two further domain-specific passes after its initial GSM8K and MATH performance lagged.The 1B and 32B models instead used GRPO, which does not require a reward model.Source: §5 · p. 28, Figure 13 · p. 29

Hyperparameter selection

• OLMo 2 required significantly higher learning rates than the Llama 3.1 recipe, with optimized hyperparameters reported for the final models.Hyperparameter sweeps used early development checkpoints; configurations for the final models appear in Tables 17 and 18.Source: Table 17 · p. 27, §5 · p. 29

Evaluation of OLMo 2-Instruct

• Evaluation covered five non-code categories using the Tülu 3 non-code evaluations and recommended settings, and also included an unseen suite without code tasks.Table 15 describes the evaluation settings; unseen-suite scores are reported in Table 24.Source: Table 15 · p. 27, §5 · pp. 29–30
• OLMo 2 13B Instruct approached Qwen 2.5 14B Instruct and surpassed Tülu 3 8B and Llama 3.1 8B Instruct; RLVR improved evaluation metrics across both scales.The paper also reports that these improvements rose with the training reward signal.Source: Table 16 · p. 28, §5 · p. 30

6.1 Clusters

• OLMo 2 used Jupiter and Augusta, with Beaker enabling workloads to move between clusters; most 7B training ran on Jupiter and most 13B training on Augusta.Both clusters provided sufficient training throughput despite hardware and architectural differences.Source: §6.1 · p. 31

6.1.1 Jupiter

• Jupiter comprises 1,024 NVIDIA H100 GPUs across 128 servers, each GPU with 80GB HBM3 and 700W power.The cluster is located in Austin, Texas, and its servers use dedicated cooling cabinets; the reported data-center PUE is 1.2.Source: §6.1.1 · pp. 31–32

6.1.2 Augusta Cluster

• Augusta is a 160-node Google Cloud cluster whose A3 Mega virtual machines each contain eight NVIDIA H100 GPUs.Its physical servers are located in Council Bluffs, Iowa.Source: §6.1.2 · p. 32

6.2 Beaker

• Beaker enabled portable workloads across data centers and isolated containerized jobs, helping preserve dependencies and make experiments reproducible.Workloads can typically move locations with a one-line code change; containers capture software dependencies and runtime details for replaying results.Source: §6.2 · p. 32

6.3 Stability and Operations

• Beaker’s GPU health checks quarantine failing hosts and reschedule workloads, while node cordoning and telemetry-based alerts help identify and address cluster problems.This reduced interruptions requiring manual attention and allowed training jobs to restart automatically on working nodes.Source: §6.3 · p. 33

6.4 Maximizing hardware utilization

• PyTorch compilation improved throughput and GPU memory use, while asynchronous bookkeeping kept its training-time impact negligible and synchronized garbage collection improved distributed-training stability.The trainer disables automatic garbage collection and runs collection at regular intervals across processes; default asynchronous bookkeeping operations otherwise introduce host-device synchronization points.Source: §6.4 · pp. 33–35

6.5 Environmental Impact

• Using Austin Energy’s reported 0.332 kg CO2 per kWh (33) and Iowa’s 0.352 kg CO2 per kWh average (34) for calculations, training the latest models emitted about 154 tCO2eq and consumed about 1.1 million liters of water.The authors describe these estimates as a lower bound on the total cost of developing large models.Source: §6.5 · p. 36

Conclusion

• OLMo 2 and OLMo 2-Instruct are competitive with open-weight models in their size categories, including Qwen 2.5, Gemma 2, and Llama 3.1.The models are fully open at 7B, 13B, and 32B parameters and trained on up to 6T tokens.Source: Conclusion · p. 36
• Building the models required stable infrastructure, architecture improvements for stability, late-stage data innovations, and updated post-training techniques.The paper notes these contributions extend beyond those in the original OLMo.Source: Conclusion · p. 36
• The release includes training and evaluation code, datasets, checkpoints, and logs to support reproducibility and further research.The paper frames OLMo 2 as progress toward an open research ecosystem for sharing and understanding training methods.Source: Conclusion · p. 36

• OLMES is a unified, standardized evaluation suite and toolkit used to guide development and assess language-model performance.Source: A OLMo 2 Evaluation Framework · p. 48

A.1 Base Model Eval

• The evaluation covers 11 tasks: 5 multiple-choice, 2 generative, and 4 held-out tasks not used during model development.Table 20 lists the benchmarks and standardized splits, sample counts, and formulations.Source: §A.1 · p. 48, Table 20 · p. 48

Multiple-choice tasks

• Multiple-choice evaluations follow OLMES formatting and normalization, using both MCF and CF where applicable and the better-performing formulation.For efficiency, MMLU and held-out multiple-choice evaluations use MCF only because relevant models strongly prefer it (Biderman et al., 2024).Source: §A.1 · p. 48, Table 20 · p. 48

Generative tasks

• Generative evaluation covers factual knowledge and reading comprehension, using 5-shot examples except for CoQA, whose conversation history guides output format.Because gold answers are often incomplete, F1 is the primary metric to award partial credit.Source: §A.1 · pp. 48–49

Held-out tasks

• The held-out suite includes AGIEval English, BBH, GSM8K, MMLU-Pro, and TriviaQA, covering exams, challenging tasks, math, reasoning, and unseen factual knowledge.Prompts use existing in-context examples where available: 8-shot CoT for GSM8K, 3-shot CoT for BBH, 5-shot for MMLU-Pro, 1-shot with passage context for AGIEval English, and manually curated 5-shot examples for TriviaQA (Suzgun et al., 2022; Cobbe et al., 2021; Wang et al., 2024; Wei et al., 2023).Source: §A.1 · p. 49
• During development, GSM8K decisions used a 200-example subset, termed GSM*, rather than the full test set.This subset informed choices such as annealing mixtures; the full test set was not evaluated during development (Wei et al., 2023).Source: §A.1 · p. 49

A.2 Instruct Model Eval

• Instruct evaluation uses the OLMES benchmark suite with the configuration reported by Lambert et al. (2024).The evaluation follows existing practices in the literature.Source: §A.2 · p. 49
• The instruct task set covers chat GSM8k and BBH, long-tail knowledge PopQA, misconception TruthfulQA, and instruction-following IFEval and AlpacaEval 2.The passage cites Cobbe et al. (2021), Suzgun et al. (2022), Mallen et al. (2022), Lin et al. (2021), Zhou et al. (2023), and Dubois et al. (2024), respectively.Source: §A.2 · p. 49
• MMLU instruct evaluation uses Lambert et al. (2024)'s chain-of-thought prompt, asking models to summarize reasoning before answering.Python completion on HumanEval and HumanEval+ and competition MATH use OLMES setup and answer extraction.Source: §A.2 · p. 49

B OLMo 2 1B

• OLMo 2 1B is introduced as a useful experimentation size alongside the work's target 7B, 13B, and 32B models.The work's goal is to develop recipes for the larger target sizes.Source: B OLMo 2 1B · p. 49
• Relative to OLMo 2 7B, the 1B variant has 16 rather than 32 layers and hidden size 2048 rather than 4096.It is otherwise defined similarly to OLMo 2 7B, with further departures in attention heads and batch size.Source: B OLMo 2 1B · p. 49
• OLMo 2 1B uses 16/16 Q/KV attention heads with MHA, rather than 32/32.The passage identifies the 1B departures from the OLMo 2 7B configuration.Source: B OLMo 2 1B · p. 49
• Its batch size is 512, half the OLMo 2 7B batch size of 1024.Source: B OLMo 2 1B · p. 49

B.1 Difficulties with OLMo 2 1B

• The OLMo 2 recipe was developed using the 1B model, and its findings generalize well to 7B, 13B, and 32B, as reflected in competitive results in Table 6.Source: §B.1 · p. 49
• OLMo 2 1B was pretrained for 4 trillion tokens on OLMo 2 Mix 1124, then annealed for 50B tokens on Dolmino Mix 1124; the planned 5-trillion-token schedule was truncated.Training used 2000 warmup steps and a peak learning rate of 4.0 ⋅10E−4.Source: §B.1 · p. 50
• Table 22 reports that OLMo 2 1B is competitive with SmolLM 2 but lags behind the smaller Gemma 2 and Qwen 2.5 base models.Source: Table 22 · p. 50, §B.1 · p. 50
• The authors hypothesize that limited model capacity may hinder 1B token efficiency and that smaller models may need task-specific pretraining data or distillation to handle harder tasks.They note that OLMo 2 1B is smaller than the smallest variants of Qwen 2.5 and Gemma 2.Source: §B.1 · pp. 50–51
• Applying the OLMo 2-Instruct post-training recipe makes the 1B model highly competitive with Qwen 2.5 and Gemma 3.Table 23 presents the comparison; the supplied passage does not provide specific scores.Source: Table 23 · p. 51, §B.1 · p. 51
• Dolmino Mix 1124 improves performance by +37.0% (1B), +18.7% (7B), +15.9% (13B), and +12.3% (32B).With only Stage 1 pretraining, OLMo 2 1B remains near random on multiple-choice tasks: 25% on MMLU and ARC Challenge, and 10% on MMLU Pro.Source: §B.1 · p. 51

C.1 Additional Hyperparameters

• Appendix C.1 lists preference-data model sources in Table 25, prompt sources in Table 27, and RLVR value-network reward-model hyperparameters in Table 26.The prompt-source passage refers readers to Lambert et al. (2024) for dataset contents.Source: §C.1 · p. 52

C.2 Additional RLVR Learning Curves

• Additional 13B RLVR learning curves are shown in Figures 18, 19, and 20.Source: §C.2 · p. 52

C.3 OLMo 2-Instruct Preview Models

• The base models’ custom GPT2Tokenizer pre-tokenization logic was lost during instruct-model training, creating a tokenizer mismatch.The base models used custom sequence-splitting or truncation logic that was absent from the instruct tokenizer.Source: §C.3 · p. 52
• To restore consistency with the base models, the team decided to retrain OLMo 2-Instruct and label existing post-trained models as previews.An initial release preceded discovery of the tokenizer issue.Source: §C.3 · p. 52
• RLVR increased training rewards and downstream evaluation scores for the OLMo 2-Instruct Preview models.The 13B curves include GSM8K, IFEval, and MATH Flex evaluations; the 7B preview figure also reports average evaluation and GSM8K scores.Source: Figure 21 · p. 54, Figure 22 · p. 55
• Table 24 reports OLMo Instruct results on Lambert et al.’s (2024) unseen suite, but bug fixes make the 7B and 13B IFEval OOD results not exactly comparable.The fixes were minor and applied to both models.Source: Table 24 · p. 55

Additional Hyperparameters

• The synthetic preference pipeline uses external off-policy samples from a broad model pool in addition to on-policy samples from SFT checkpoints.The listed pool includes models from Yi, Tülu, Gemma, GPT-4o, MPT, Mistral, Qwen, Falcon, SmolLM2, Phi, and NuExtract.Source: D Additional Hyperparameters · p. 56, Table 25
• The appendix includes reward-model training hyperparameters for RLVR value-network initialization and prompt sources for preference-finetuning datasets.These are identified in Tables 26 and 27; their specific values and sources are not given in the supplied passages.Source: Table 26 · p. 56, Table 27 · p. 57
• RLVR increased OLMo-2-1124-13B-Instruct training rewards and downstream evaluation scores, including GSM8K.Figure 23 also shows KL divergence, response lengths, evaluation-suite scores, IFEval, and MATH Flex scores.Source: Figure 23 · p. 57

E Annealing Data Details

• Figure 24 shows the prompt for generating hard math word problems, whose personas are borrowed from Chan et al. (2024).The passage does not specify the personas or reproduce the prompt text.Source: Figure 24 · p. 58
• Figure 25 shows the prompt used to generate solutions for hard math word problems.The passage does not reproduce the solution-generation prompt.Source: Figure 25 · p. 58