PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Pre-Training

Qingxiu Dong, Li Dong, Yao Tang, Tianzhu Ye, Yutao Sun, Zhifang Sui, Furu Wei

arXiv:2506.08007v1cs.CL

TL;DR

Reinforcement pre-training reframes next-token prediction as a reasoning task trained with reinforcement learning and rewards verified against the corpus. It improves next-token accuracy, with gains as training compute increases, and provides a stronger foundation for further reinforcement fine-tuning.

  • Problem

    RL with verifiable rewards is often limited by scarce annotated answers, while human-feedback RL relies on costly preference data.

    Source: §1 · p. 2

  • Method

    RPT trains next-token reasoning with reinforcement learning and correctness-based rewards derived from the pre-training corpus.

    Source: §1 · p. 2

  • Results

    RPT improves next-token prediction accuracy, with performance consistently improving as training compute increases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RPT provides a stronger pre-trained foundation for reinforcement fine-tuning and enhances zero-shot performance on downstream tasks.

    Source: §1 · p. 2, §6 · p. 9

    Experiments primarily use a 14B model, a predominantly mathematical corpus, and initialization from a reasoning model.

    Source: §6 · p. 9

Abstract

from arXiv · show

In this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling paradigm for large language models and reinforcement learning (RL). Specifically, we reframe next-token prediction as a reasoning task trained using RL, where it receives verifiable rewards for correctly predicting the next token for a given context. RPT offers a scalable method to leverage vast amounts of text data for general-purpose RL, rather than relying on domain-specific annotated answers. By incentivizing the capability of next-token reasoning, RPT significantly improves the language modeling accuracy of predicting the next tokens. Moreover, RPT provides a strong pre-trained foundation for further reinforcement fine-tuning. The scaling curves show that increased training compute consistently improves the next-token prediction accuracy. The results position RPT as an effective and promising scaling paradigm to advance language model pre-training.

1 Introduction

Reinforcement pre-training (RPT) turns next-token prediction into a reasoning task trained with correctness-based reinforcement rewards. It uses unannotated text for general-purpose RL and improves next-token accuracy, with performance rising as training compute increases.

1 Introduction

  • Existing RL approaches face limits for general-purpose pre-training: human-feedback RL relies on costly preference data, while RLVR is constrained by scarce annotated answers.RLVR therefore tends to serve domain-specific fine-tuning rather than general-purpose pre-training.Source: §1 · p. 2
  • RPT trains next-token reasoning with reinforcement learning and verifiable rewards based on whether predictions match the corpus’s ground-truth next token.This turns unannotated pre-training text into data for general-purpose RL without external annotations or domain-specific reward functions.Source: §1 · p. 2
  • RPT uses rule-based next-token rewards to reduce reward-hacking risk and encourages reasoning patterns intended to promote generalization over rote memorization.The approach reuses vast unannotated text rather than requiring external annotations.Source: §1 · p. 2
  • RPT improves next-token prediction accuracy, and its performance consistently improves with increased training compute.It also provides a stronger pre-trained foundation for reinforcement fine-tuning and enhances zero-shot performance on downstream tasks.Source: §1 · p. 2

2 Preliminary

The preliminary section describes next-token prediction as a foundational language-model training objective and contrasts it with reinforcement learning using verifiable rewards.

2 Preliminary

  • Next-token prediction (NTP) trains a language model on a corpus sequence by maximizing the objective over its next-token predictions.The model parameters are denoted by θ.Source: §2 · p. 3
  • RLVR trains on labeled question-answer pairs, with a deterministic verifier rewarding generated responses against the supplied answer.The model is trained to maximize expected reward.Source: §2 · p. 3

3 Reinforcement Pre-Training

This section frames next-token prediction as a reasoning task, trains it with on-policy reinforcement learning and verifiable prefix rewards, and describes the pre-training setup and downstream evaluations.

3.1 Pre-Training Task: Next-Token Reasoning

  • Each context becomes a reasoning problem: the model generates chain-of-thought before predicting the next token, using patterns such as brainstorming, self-critique, and self-correction.The task reconstructs the pre-training corpus as a set of reasoning problems.Source: §3.1 · p. 3

3.2 Pre-Training with Reinforcement Learning

  • RPT samples G reasoning-and-prediction trajectories per context and trains on prefix-match rewards for exact byte prefixes ending at valid ground-truth token boundaries.The reward supports predictions spanning multiple tokens or involving out-of-vocabulary tokens.Source: Figure 3 · p. 4, §3.2 · p. 4

3.3 Pre-Training Setup

  • Training uses OmniMATH and filters low-entropy token positions with a 1.5B proxy model, using Deepseek-R1-Distill-Qwen-14B as the base model.The main experiment runs for 1,000 steps and samples G=8 responses per question.Source: §3.3 · pp. 4–5

3.4 Evaluation of Pretrained Models

  • The pretrained models are evaluated for next-token prediction accuracy and for continual reinforcement fine-tuning on downstream tasks.The evaluation tests language-modeling performance and scaling properties, as well as whether RPT further enhances post-training on end tasks.Source: §3.4 · p. 5

4 Experiments

Across five experiments, RPT improves next-token prediction, scales with training compute, and performs strongly on downstream tasks. The analyses also report differences between RPT’s next-token reasoning patterns and explicit problem-solving patterns.

4.1 Language Modeling

  • RPT-14B achieves higher next-token prediction accuracy across all difficulty levels than R1-Distill-Qwen-14B and matches the significantly larger R1-Distill-Qwen-32B.Table 1 covers three test splits of varying difficulty; Figure 4 shows average accuracy across difficulty levels.Source: Table 1 · p. 5, §4.1 · pp. 5–6, Figure 4 · p. 5

4.2 Scaling Properties of Reinforcement Pre-Training

  • RPT’s next-token prediction accuracy improves reliably as training compute increases across all data difficulty levels, and the fitted curves have high R2 values.The analysis fits a power-law relationship between training compute and validation-set accuracy.Source: §4.2 · p. 6, Figure 5 · p. 6

4.3 Reinforcement Fine-Tuning with RPT

  • With limited data, RPT’s stronger pre-trained foundation can transfer strengthened reasoning patterns to end tasks during RLVR fine-tuning.The comparison also reports declining reasoning ability with continued next-token prediction training on the same data, followed by only slow RLVR improvement.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Zero-Shot Performance on End Tasks

  • RPT-14B outperforms R1-Distill-Qwen-14B across benchmarks and surpasses R1-Distill-Qwen-32B under next-token prediction, with gains of 7 points on SuperGPQA and approximately 22 points on MMLU-Pro.The results cover zero-shot end tasks.Source: §4.4 · p. 7, Table 3 · p. 7

4.5 Next-Token Reasoning Pattern Analysis

  • Compared with problem-solving responses, RPT-14B next-token reasoning uses hypothesis patterns 161.8% more and deduction patterns 26.2% more, while problem solving relies more on breakdown.The analysis compares 200 sampled OmniMATH responses per model and categorizes patterns into six types.Source: §4.5 · pp. 7–8, Figure 6 · p. 8
  • An example describes RPT considering semantic context, plausible continuations, alternatives, structural cues, and token-level details rather than simply matching a pattern.The paper presents the example in Table 4.Source: §4.5 · pp. 8–9, Table 4 · p. 8

5 Related Work

Related work situates RPT alongside training-time and test-time scaling and reinforcement learning for alignment or reasoning. It distinguishes RPT by framing each next-token prediction as a reasoning task and using prediction correctness as a rule-based reward.

Scaling Paradigms of Large Language Models

  • RPT frames each next-token prediction as a reasoning task, integrating principles associated with training-time and test-time scaling.The section contrasts training-time scaling through model parameters and training data with test-time scaling through extended inference compute.Source: §5 · p. 9

Reinforcement Learning for Large Language Models

  • Unlike helpfulness-based rewards that can be hacked by repeating the target token in a rationale, RPT uses next-token correctness as a rule-based reward.The related work identifies [ZHS+24] as a relevant approach and describes the potential harm of that shortcut.Source: §5 · p. 9

6 Conclusion and Future Work

RPT reframes next-token prediction as verifiable reasoning trained with correctness-based rewards, and experiments report gains in next-token prediction and zero-shot mathematical and general reasoning. The initial exploration is bounded by its model, corpus, and initialization choices.

6 Conclusion and Future Work

  • RPT frames next-token prediction as a verifiable reasoning task using correctness-based reinforcement rewards.The paper presents it as a pre-training paradigm for large language models.Source: §6 · p. 9
  • Experiments report improved next-token prediction and zero-shot performance on mathematical and general reasoning benchmarks.RPT is also reported to provide a better starting point for further reinforcement fine-tuning.Source: §6 · p. 9
  • The experiments primarily use a 14B-parameter model and a predominantly mathematical corpus, leaving broader general-domain text for future study.The authors also identify training from a standard base language model, rather than a reasoning model, as a future investigation.Source: §6 · p. 9
  • Future directions include larger and broader corpora, more training compute, scaling laws, and combining RPT with hybrid thinking for adaptive next-token reasoning.Source: §6 · p. 9

A Design Choices of Reward

The section compares alternatives to prefix matching, including first-token matching and dense reward variants. The tested alternatives generally perform comparably to prefix matching.

A Design Choices of Reward

  • First-token matching rewards whether the prediction’s first token matches the ground-truth next token, ignoring later predicted tokens.Source: A Design Choices of Reward · p. 12
  • Dense reward gives a correct first-token prediction full reward and an incorrect one its language-model probability.Source: A Design Choices of Reward · p. 12
  • Conditional dense reward applies only when at least one of G rollouts for a prefix predicts the next token correctly.If all G rollouts are incorrect, the method instead applies a different reward scheme, such as zero reward or a uniform small penalty.Source: A Design Choices of Reward · p. 12
  • Alternative reward designs generally achieve performance comparable to prefix matching.The authors interpret this as robustness to the tested reward-signal modifications, within the scope of those variations.Source: A Design Choices of Reward · p. 12

B Hyperparameters Used for Reinforcement Pre-Training

The section specifies the reinforcement pre-training setup as exact on-policy reinforcement learning with the entropy loss coefficient set to zero. Table 5 presents the detailed hyperparameters.

B Hyperparameters Used for Reinforcement Pre-Training

C Detailed Results on End Tasks

RPT-14B is reported to outperform R1-Distill-Qwen-14B and R1-Distill-Qwen-32B on general end-task benchmarks. The section also reports that clear prompts improve initial correctness, while the experiments used the v0 template.

C Detailed Results on End Tasks

D Impact of Prompt Templates

  • The prompt-template study explores seven variants with differing instruction phrasing and context formats.Source: D Impact of Prompt Templates · p. 12
  • Clear prompts significantly improve initial correctness, and the Section 4 reinforcement pre-training experiments use the v0 template.Prompt engineering with other variants is left for future work; the authors state that it tends to improve final performance.Source: D Impact of Prompt Templates · p. 13, Table 8 · p. 13

E Keywords for Reasoning Pattern Analysis

This section identifies keyword groups for reasoning-pattern analysis and describes prompt templates for next-token reasoning.

E Keywords for Reasoning Pattern Analysis

F Case Studies

The section offers qualitative case studies of next-token reasoning by RPT-14B, with three examples collected in Table 11.

F Case Studies

Loading 2506.08007v1…