PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Mid-Training

Yijun Tian, Shaoyu Chen, Zhichao Xu, Yawei Wang, Jinhe Bi, Peng Han, Wei Wang

arXiv:2509.24375v1cs.CL

TL;DR

Reinforcement mid-training addresses the gap between pre-training and post-training, where reasoning efficiency, token difficulty, and use of the full token distribution remain challenges. RMT combines a dynamic reasoning budget, adaptive token sampling, and reinforcement learning with next-token prediction. It reports language-modeling gains with shorter reasoning and improved performance after subsequent post-training.

  • Problem

    Reinforcement learning during mid-training faces excessive reasoning, difficulty-imbalanced token sampling, and underuse of the low-entropy majority, limiting efficiency and token coverage on unlabeled data.

    Source: §1 · p. 2

  • Method

    RMT combines a dynamic token budget and easy-to-hard adaptive sampling with reinforcement learning on selected tokens and next-token prediction on the remaining tokens.

    Source: §1 · p. 2, §3 · p. 3, §4.1 · p. 4

  • Results

    RMT improves language modeling over RPT by an average +30.74% with RMT-R1 and +64.91% with RMT-Q3, while RMT-Q3 improves post-training +18.76% over Qwen3-14B.

    Source: §5.2 · p. 7, §5.3 · p. 7

  • Takeaways & Limitations

    The reported results support reinforcement mid-training as an intermediate stage that produces models better prepared for subsequent post-training.

    Source: §6 · p. 9

Abstract

from arXiv · show

The development of state-of-the-art large language models is commonly understood as a two-stage process involving pre-training and post-training. We point out the need for an additional intermediate stage called reinforcement mid-training with potential for strong performance gains. In this paper, we formally define the problem and identify three key challenges: (1) inefficient training due to excessive reasoning steps, (2) disregard of the imbalanced token entropy distribution, and (3) underutilization of token information. To address these challenges, we propose RMT, a framework for efficient, adaptive, and unified reinforcement mid-training with various innovative components. In particular, we first introduce a dynamic token budget mechanism that constrains unnecessary reasoning steps and mitigates model overthinking. Next, we design a curriculum-based adaptive sampling method that fosters a progressive learning trajectory from easy to hard tokens. Finally, we present a dual training strategy that combines reinforcement learning with next-token prediction, ensuring targeted learning on key tokens and full exploitation of all token information. Extensive experiments demonstrate the superiority of RMT over state-of-the-art methods, achieving up to +64.91% performance improvement with only 21% of the reasoning length in language modeling. We also show that checkpoints obtained after reinforcement mid-training can benefit the subsequent post-training, yielding up to +18.76% improvement in the mathematical domain.

1 INTRODUCTION

The paper argues that the familiar pre-training/post-training framing misses reinforcement mid-training, an intermediate stage using unlabeled pre-training data for targeted capability development. RMT addresses challenges in reasoning efficiency, token difficulty, and token coverage with a dynamic budget, adaptive sampling, and combined RL/NTP training.

1 INTRODUCTION

  • Reinforcement mid-training is presented as an intermediate phase that uses large-scale unlabeled pre-training data with targeted objectives to strengthen capabilities before post-training.Unlike post-training, it does not require extra human annotations or verified rewards.Source: §1 · p. 1
  • Existing approaches face excessive reasoning, indiscriminate sampling of difficult tokens, and omission of the low-entropy majority of tokens.The paper links these challenges to added computation, potential overload early in training, and missed learning opportunities.Source: §1 · p. 2
  • +64.91% performance improvement with only 21% of the reasoning length is reported for language modeling; RMT also improves subsequent post-training by up to +18.76%.Figure 1 describes RMT-Q3 and RMT-R1 as achieving superior performance with less reasoning length.Source: §1 · p. 2, Figure 1 · p. 2
  • RMT combines a dynamic token budget, easy-to-hard adaptive sampling, and token-selective reinforcement learning with token-inclusive next-token prediction.The paper describes the framework as plug-and-play and applicable with different backbones and reinforcement learning algorithms.Source: §1 · p. 2
  • Experiments on language modeling and continual post-training are reported to show RMT's effectiveness and efficiency relative to state-of-the-art methods.The paper describes these results as establishing a stronger foundation for post-training.Source: §1 · p. 3

2 RELATED WORK

The related-work section contrasts established pre-training and post-training paradigms with the more recent emergence of mid-training. It frames reinforcement learning in mid-training as underdeveloped, with computational cost over unlabeled corpora remaining a central challenge.

LLM Pre-Training

  • Pre-training is described as unsupervised learning, commonly via next-token prediction, that equips language models with linguistic fluency and broad world knowledge.Scaling-law research examines interactions among compute, data, and parameters.Source: §2 · p. 3

LLM Post-Training

  • Post-training refines models for specific tasks or user requirements, primarily through supervised fine-tuning and reinforcement learning.The section also notes work addressing stability and computational efficiency limits of PPO, alongside curriculum learning and combinations of SFT and RL.Source: §2 · p. 3

LLM Mid-Training

  • Mid-training is an emerging intermediate stage with computational and data demands between pre-training and post-training, intended to prepare models for post-training.The section says reinforcement learning in this stage remains underdeveloped and identifies RPT's computational cost over massive unlabeled corpora as unresolved.Source: §2 · p. 3

3 REINFORCEMENT MID-TRAINING

Reinforcement mid-training applies reinforcement learning to selected tokens in unlabeled pre-training data while using next-token prediction for the remaining tokens. The framework seeks to reduce reinforcement-learning cost while retaining concise reasoning and broad token coverage.

3 REINFORCEMENT MID-TRAINING

  • Reinforcement mid-training optimizes on large-scale unlabeled pre-training data to improve subsequent post-training, applying RL primarily to influential reasoning tokens and NTP to the rest.The formal setup partitions a sequence into disjoint RL and NTP token sets, with many more tokens assigned to NTP.Source: §3 · p. 3
  • For each RL-selected token, the policy predicts it from preceding tokens using a sampled reasoning-and-prediction response, with a matching reward based on the predicted and ground-truth tokens.For NTP tokens, the model maximizes the likelihood of each token given its preceding tokens.Source: §3 · p. 3
  • The framework identifies token difficulty using entropy, assigns a dynamic token budget, and progressively samples easier then harder tokens for RL while incorporating low-entropy tokens through NTP.RL uses length and verifiable matching rewards within a unified training objective.Source: §3 · p. 3, Figure 2 · p. 4

4 METHOD

RMT combines a dynamic reasoning-token budget, curriculum-based sampling by token difficulty, and a unified objective that pairs selective reinforcement learning with next-token prediction.

4.1 DYNAMIC TOKEN BUDGET MECHANISM

  • A training-step-dependent token budget decays as training progresses, while a length reward peaks at the target and penalizes short or overly long reasoning.Reasoning longer than twice the budget receives zero length reward.Source: §4.1 · pp. 4–5

4.2 CURRICULUM-BASED ADAPTIVE SAMPLING

  • Curriculum sampling adjusts probabilities across easy, medium, and hard entropy-based token categories, moving from easier tokens toward harder ones as training advances.The method samples a difficulty category from the current probabilities, then selects uniformly among available candidates at that level.Source: §4.2 · p. 5

4.3 TOKEN-SELECTIVE RL AND TOKEN-INCLUSIVE NTP

  • RMT combines token-selective reinforcement learning with next-token prediction, applying RL to selected tokens and NTP to most low-entropy tokens.The combined loss balances the reinforcement and NTP objectives; GRPO supplies the reinforcement-learning signal.Source: §4.3 · pp. 5–6

5 EXPERIMENTS

Across language modeling and continual post-training, RMT outperforms the reported baselines. Its experiments also examine response efficiency, component contributions, and a worked reasoning example.

5.1 EXPERIMENTAL SETUP

  • The experiments use OmniMATH for language modeling and Skywork for continual post-training, with 200 and 600 evaluation examples, respectively.Training sets contain 4,051 and 4,096 examples, respectively.Source: §5.1 · p. 6

5.2 LANGUAGE MODELING

  • RMT-R1 improves over RPT by +30.74% and RMT-Q3 by +64.91% in language modeling.RMT-Q3 consistently outperforms RMT-R1 across difficulty levels.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 CONTINUAL POST-TRAINING

  • After continual post-training on Skywork, RMT-Q3 rises from 25.17 to 64.33 accuracy and improves +18.76% over Qwen3-14B.Before post-training, it improves +8.63% over RPT.Source: Table 2 · p. 7, §5.3 · p. 7

5.4 RESPONSE LENGTH AND GENERATION TIME ANALYSIS

  • RMT-R1 and RMT-Q3 generate 188 and 186 tokens per response, respectively—21% of RPT’s 872 tokens and 12% of Qwen3-14B’s 1,577 tokens.The reported shorter responses maintain superior performance in Table 1.Source: Table 3 · p. 8, §5.4 · p. 8
  • During continual post-training, RMT starts with shorter reasoning than RPT, then lengthens responses as downstream tasks grow more complex while remaining more efficient.Generation-time curves mirror this adaptive pattern.Source: Figure 3 · p. 8, §5.4 · p. 8

5.5 ABLATION STUDY

  • Removing next-token prediction substantially harms performance; removing dynamic token budgeting or curriculum-based adaptive sampling also reduces performance and efficiency.RMT performs best with all components integrated.Source: §5.5 · p. 8, Table 4 · p. 8

5.6 CASE STUDY

  • In the OmniMath example, RMT-Q3 predicts “probability” correctly with 165 tokens, while RPT uses 1,376 tokens and predicts “same” incorrectly.The table describes RMT traces as concise and coherent and RPT traces as verbose and inconsistent.Source: §5.6 · pp. 8–9, Table 5 · p. 9

6 CONCLUSION

The paper presents Reinforcement Mid-Training as an intermediate stage before downstream post-training and proposes RMT to address its challenges. Experiments show gains in performance and efficiency, and the authors position this stage as a foundation for subsequent post-training.

6 CONCLUSION

  • Experiments show that RMT improves both performance and efficiency, producing models better prepared for subsequent post-training.Source: §6 · p. 9
  • Reinforcement Mid-Training is defined as an intermediate stage that strengthens large language models before downstream post-training.Source: §6 · p. 9
  • The paper identifies excessive overthinking, imbalanced token entropy, and underutilization of token information as three challenges in this setting.Source: §6 · p. 9
Loading 2509.24375v1…