PaperlayerView exact revision on arXiv

Source-linked AI summary

OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling

Zengzhi Wang, Fan Zhou, Xuefeng Li, Pengfei Liu

arXiv:2506.20512v1cs.CLcs.AIcs.LG

TL;DR

The paper studies why Llama and Qwen differ in reinforcement-learning behavior and tests whether mid-training can improve Llama’s RL suitability. It develops the Stable-then-Decay strategy and OctoThinker models, whose post-RL performance closes the gap with RL-friendly Qwen models. The findings also identify QA distribution and long-CoT formatting as important constraints on RL outcomes.

  • Problem

    The paper asks what makes a base language model suitable for reinforcement learning, motivated by divergent reasoning-task RL behavior between Llama and Qwen.

  • Method

    The study applies controlled mid-training interventions and introduces Stable-then-Decay: 200B constant-learning-rate tokens followed by 20B decayed-learning-rate tokens across three CoT-focused branches.

  • Results

    OctoThinker achieves strong mathematical-reasoning performance after RL and closes the performance gap with RL-friendly model families, while the first mid-training stage delivers consistent 10–20% gains.

  • Takeaways & Limitations

    High-quality reasoning-intensive corpora and appropriately formatted QA data are central to improving RL stability and effectiveness in Llama-based foundation models.

  • Takeaways & Limitations

    QA gains depend on the distribution gap with downstream tasks, and long-CoT patterns can cause excessive responses and sudden performance drops in RL-tuned models.

Abstract

from arXiv · show

Different base language model families, such as Llama and Qwen, exhibit divergent behaviors during post-training with reinforcement learning (RL), especially on reasoning-intensive tasks. What makes a base language model suitable for reinforcement learning? Gaining deeper insight into this question is essential for developing RL-scalable foundation models of the next generation. In this work, we investigate how mid-training strategies shape RL dynamics, focusing on two representative model families: Qwen and Llama. Our study reveals that (1) high-quality mathematical corpora, such as MegaMath-Web-Pro, significantly improve both base model and RL performance, while existing alternatives (e.g., FineMath-4plus) fail to do so; (2) further adding QA-style data, particularly long chain-of-thought (CoT) reasoning examples, enhances RL outcomes, and instruction data further unlocks this effect; (3) while long-CoT improves reasoning depth, it can also induce verbosity of model responses and unstability of RL training, underscoring the importance of data formatting; (4) scaling mid-training consistently leads to stronger downstream RL performance. Building on these insights, we introduce a two-stage mid-training strategy, Stable-then-Decay, in which base models are first trained on 200B tokens with a constant learning rate, followed by 20B tokens across three CoT-focused branches with learning rate decay. This yields OctoThinker, a family of models demonstrating strong RL compatibility and closing the performance gap with more RL-friendly model families, i.e., Qwen. We hope our work will help shape pre-training strategies for foundation models in the RL era. To support further research, we release our open-source models along with a curated math reasoning-intensive corpus of over 70 billion tokens (i.e., MegaMath-Web-Pro-Max).

1. Introduction

The paper asks why Qwen and Llama differ in reinforcement-learning dynamics and investigates whether targeted mid-training can make Llama more suitable for RL scaling. It finds that corpus quality, QA and instruction data, data formatting, and scaled training shape downstream RL performance.

  • Motivation: Qwen models are more amenable to RL scaling than Llama, which predicts answers prematurely and produces repetitive outputs during training.The study uses controlled mid-training interventions on Llama to investigate this divergence.
  • Findings: High-quality mathematical pre-training corpora improve RL scaling more than alternatives such as FineMath-4plus.MegaMath-Web-Pro is identified as providing significantly greater benefits for RL scaling.
  • Findings: QA-style data further improves RL outcomes, while a small amount of instruction-following data enhances RL effectiveness even more.These interventions are applied on top of a high-quality mathematical pre-training corpus.
  • Findings: Long-CoT mid-training can introduce instability into the subsequent RL phase.The paper later addresses this issue through a two-stage strategy with separate short-CoT, long-CoT, and hybrid branches.
  • Approach: The Stable-then-Decay strategy trains for 200B tokens at a constant learning rate, then branches into short-CoT, long-CoT, and hybrid mixtures with learning-rate decay.The three branches are subsequently refined through RL training.
  • Results: Across 14 mathematical reasoning benchmarks, the first mid-training stage consistently improves performance by 10–20%, and post-RL models reach parity with same-size Qwen2.5 models.Both mid-training stages produce gains, with the first stage producing the most consistent improvement.

2. Preliminaries

The paper frames mid-training as an intermediate stage used to alter data quality and distribution before post-training, then compares RL behavior across Llama and Qwen models. Llama shows abnormal response-length growth and repetition during RL, motivating mid-training interventions.

  • RL Setup: The RL experiments use verl with GRPO, MATH8K prompts, batch size 128, 16 rollouts per query, and a 4096-token maximum output.The configured learning rate is 1 × 10−6, with a sampling-to-gradient-update ratio of 2 for more stable training.
  • Evaluation: The comparison uses Llama-3.2-3B-Base and Qwen2.5-3B-Base with few-shot evaluation for base models and zero-shot evaluation for RL-tuned models.Indicator tasks include GSM8K, MATH500, OlympiadBench, and AMC23, with additional tasks used for base-model assessment.
  • Observations: Qwen’s correct-response length grows steadily during RL, whereas Llama’s average length escalates abnormally to 4,096 tokens.Figure 2 compares downstream performance and average correct-response length against corresponding base-model few-shot values.
  • Observations: Llama commonly starts with “\boxed:{}” and repeats until the output limit, unlike Qwen’s coherent reasoning responses.Qwen improves substantially over its base model across benchmarks, while Llama shows marginal gains or GSM8K regression.
  • Motivation: These divergent RL dynamics motivate mid-training interventions that adjust the pre-training trajectory and measure downstream effects on RL behavior.The paper investigates whether such interventions can make Llama more amenable to RL scaling.
  • Preliminaries: Mid-training sits between pre-training and post-training in computational and token requirements.It is used for objectives including domain expansion, long-context extension, data-quality improvement, synthetic-data use, and post-training preparation.

3. Digging Deeper: Exploring Key Factors through Controllable Mid-training

Head-to-head mid-training experiments examine data quality, QA and instruction data, formatting, and token budget as factors shaping downstream RL performance. High-quality math corpora, carefully formatted reasoning data, instruction data, and larger budgets improve RL outcomes, though long-CoT data can destabilize training.

  • Experimental factors: Systematic experiments vary math-corpus quality, QA inclusion and type, instruction data, and the pre-training token budget to study their effects on RL.These interventions connect mid-training choices with downstream RL dynamics and inform scaled-up training recipes.
  • Math web corpora: MegaMath-Web-Pro and MegaMath-Web-Pro-Max produce significant RL gains for Llama, whereas FineMath-4plus yields only marginal improvements and repetitive 4,096-token responses.The authors attribute the contrast to differences in data quality and select MegaMath-Web-Pro and MegaMath-Web-Pro-Max as their mathematical corpora.
  • QA-format data: Long-CoT data significantly improves RL performance but can trigger unstable training, sudden performance drops, and sharply increasing response lengths.Short-CoT data shows no improvement over web-only mid-training, possibly because of a distribution gap with downstream tasks.
  • Instruction-following data: Instruction-following data unlocks short-CoT gains after 200 steps and stabilizes response length, but does not prevent long-CoT training from declining overall.For long-CoT mixtures, improvements appear after 150 steps despite continued performance decline and rapid response-length growth.
  • Training stability: A complex RL prompt template produces smoother response-length growth than the simple template, although benchmark performance still deteriorates later in training.The authors therefore also apply a progressive maximum response-length scheduler to address instability.
  • Mid-training budget: Increasing the mid-training budget consistently improves downstream RL performance, even when base-model evaluations show no corresponding gains.The 70B- and 100B-token checkpoints perform comparably as base models and both significantly outperform the 20B checkpoint, while RL performance continues improving with scale.

4. OctoThinker-Base: Branching Reasoning Foundations via 2-Stage Mid-training

OctoThinker-Base uses stable-then-decay mid-training to strengthen Llama’s reasoning foundations and diversify behaviors through three data branches. Across mathematical benchmarks, the branches improve over Llama baselines, with 30% QA offering the best reported trade-off for RL-oriented preparation.

  • 4.1. Recipe for the First Stage: Building Strong Reasoning Foundations: The stable stage trains on high-quality web corpora for 200B tokens with a constant learning rate before branching.The resulting models are called OctoThinker-Base-Stable.
  • 4.2.1. Pilot Studies: Increasing QA data improves RL performance, but gains plateau beyond 30%, making 30% QA the adopted ratio.The paper attributes diminishing returns at 40% to token redundancy and reduced diversity.
  • 4.2.2. Recipe for the Decay Stage: The decay stage uses cosine learning-rate decay and branches into Long, Short, and Hybrid variants with distinct reasoning-data mixtures.The three branches respectively emphasize long reasoning, short reasoning, or a mixture of both.
  • 4.3. Evaluation on OctoThinker-Base Series: Each OctoThinker branch improves 10%-20% over the original Llama base model across 13 mathematical benchmarks and also gains over the stable-stage model.The authors note that challenging competition benchmarks still show random or poor performance before post-training.

5. OctoThinker-Zero Families: RL Scaling with Diverse Thinking Behaviors

The OctoThinker-Zero family applies RL to the branched base models, producing variants with different thinking behaviors. In 3B comparisons, the Long branch consistently outperforms original Llama and reaches performance on par with Qwen2.5.

  • 5. OctoThinker-Zero Families: RL Scaling with Diverse Thinking Behaviors: RL training produces Short-Zero, Hybrid-Zero, and Long-Zero models corresponding to the three decay-stage data mixtures.The Long branch tends to produce longer responses within a controlled range.
  • 5. OctoThinker-Zero Families: RL Scaling with Diverse Thinking Behaviors: The Long branch’s training dynamics are documented for the 1B and 3B series, alongside cross-family dynamics comparisons.The figures cover branch-level dynamics and comparisons with Llama-3.2-3B-Base and Qwen2.5-Base.
  • 5. OctoThinker-Zero Families: RL Scaling with Diverse Thinking Behaviors: OctoThinker-Long-3B consistently outperforms Llama-3.2-3B during RL and reaches performance on par with Qwen2.5-3B.The Hybrid and Short branches are marginally inferior, especially on challenging benchmarks.

6. Related Works

Related work situates OctoThinker within research on RL for reasoning and the construction of mathematical pre-training corpora. The paper contributes MegaMath-Web-Pro-Max as a large reasoning-focused corpus for mid-training.

  • Understanding RL along with Language Models: Recent studies apply large-scale RL to smaller language models for reasoning-intensive tasks, extending successes demonstrated by o1, o3, and R1.The related work describes this as a wave of follow-up research on models below 100B parameters.
  • Curation of Math Pre-training Corpora: Existing open mathematical corpora made progress but typically remained under 100B tokens, constraining scale for domain-specific mid-training.Examples include OpenWebMath, MathPile, InfiMM-Web-Math, and FineMath.
  • Curation of Math Pre-training Corpora: MegaMath-Web-Pro-Max exceeds 70B tokens, matches MegaMath-Web-Pro’s quality, powers OctoThinker’s first stage, and is released for the open-source community.

7. Conclusion

The paper argues that mid-training helps explain and reshape divergent RL behavior across Llama and Qwen. Its two-stage strategy yields OctoThinker models with strong mathematical reasoning performance and a narrower gap to RL-friendly families.

  • 7. Conclusion: The study identifies mid-training as a decisive factor in the divergent RL behavior of Llama and Qwen.High-quality, reasoning-intensive corpora are reported to improve RL stability and effectiveness.
  • 7. Conclusion: The two-stage strategy transforms Llama into a more RL-scalable foundation model, while OctoThinker performs strongly across mathematical reasoning tasks.The models close the gap with RL-friendly model families within the reported evaluation scope.

Future Work

The authors identify several directions for extending OctoThinker and clarifying how pre-training interacts with reinforcement learning.

  • Future work includes curating higher-quality mathematical corpora to further enhance mid-training.
  • The authors plan to design RL-friendly base models using open recipes without distillation from powerful long-CoT reasoning models.
  • They aim to disentangle QA format from content and extend OctoThinker with branches such as tool-integrated reasoning.

Appendix

The appendix includes prompts for scoring mathematical usefulness and refining web text, along with visualizations and labels for data mixtures, model variants, and RL dynamics.

  • Data curation prompts: Figure 15 presents a FineMath prompt for scoring text usefulness in mathematics.
  • Data curation prompts: Figure 16 presents the web-text refinement prompt used in MegaMath-Web-Pro.
  • Experimental configurations: Appendix labels identify MegaMath-QA mixtures of 10%, 20%, 30%, and 40%, alongside OctoThinker-3B-Base-Stable and RL dynamics under different QA datasets and mixing ratios.
Loading 2506.20512v1…