PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning on Pre-Training Data

Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, Evander Yang, Kun Li, Haoyuan Wu, Jiajia Wu, Zihao Zheng, Chenchen Zhang, Kun Shi, Kyrierl Deng, Qi Yi, Ruibin Xiong, Tingqiang Xu, Yuhao Jiang, Jianfeng Yan, Yuyuan Zeng, Guanghui Xu, Jinbao Xue, Zhijiang Xu, Zheng Fang, Shuai Li, Qibin Liu, Xiaoxue Li, Zhuoyu Li, Yangyu Tao, Fei Gao, Cheng Jiang, Bo Chao Wang, Kai Liu, Jianchen Zhu, Wai Lam, Wayyt Wang, Bo Zhou, Di Wang

arXiv:2509.19249v2cs.CLcs.AIcs.LG

TL;DR

As compute scales while high-quality text data grows finitely, conventional next-token training may favor surface memorization over deeper generalization. RLPT applies reinforcement learning to pre-training data through next-segment reasoning, reporting gains across general-domain and mathematical benchmarks, including an 8.1 absolute gain on GPQA-Diamond for Qwen3-4B-Base.

  • Problem

    Conventional next-token supervised training dominates LLM optimization, while evidence suggests it can promote surface-level memorization rather than deeper generalization.

    Source: §2.2 · p. 3

  • Method

    RLPT applies reinforcement learning to pre-training data using next-segment reasoning objectives that reward predicting the ground-truth segment from preceding context.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    3.0, 5.1, 8.1, and 6.0 absolute gains on MMLU, MMLU-Pro, GPQA-Diamond, and KOR-Bench, respectively, for Qwen3-4B-Base accompany consistent improvements across general-domain and mathematical tasks and multiple models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RLPT provides a strong foundation for subsequent RLVR, extending LLM reasoning boundaries and boosting mathematical reasoning benchmark performance.

    Source: §1 · p. 2

    The initial exact-semantic-match reward was too rigid and produced numerous false positives.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

The growing disparity between the exponential scaling of computational resources and the finite growth of high-quality text data now constrains conventional scaling approaches for large language models (LLMs). To address this challenge, we introduce Reinforcement Learning on Pre-Training data (RLPT), a new training-time scaling paradigm for optimizing LLMs. In contrast to prior approaches that scale training primarily through supervised learning, RLPT enables the policy to autonomously explore meaningful trajectories to learn from pre-training data and improve its capability through reinforcement learning (RL). While existing RL strategies such as reinforcement learning from human feedback (RLHF) and reinforcement learning with verifiable rewards (RLVR) rely on human annotation for reward construction, RLPT eliminates this dependency by deriving reward signals directly from pre-training data. Specifically, it adopts a next-segment reasoning objective, rewarding the policy for accurately predicting subsequent text segments conditioned on the preceding context. This formulation allows RL to be scaled on pre-training data, encouraging the exploration of richer trajectories across broader contexts and thereby fostering more generalizable reasoning skills. Extensive experiments on both general-domain and mathematical reasoning benchmarks across multiple models validate the effectiveness of RLPT. For example, when applied to Qwen3-4B-Base, RLPT yields absolute improvements of $3.0$, $5.1$, $8.1$, $6.0$, $6.6$, and $5.3$ on MMLU, MMLU-Pro, GPQA-Diamond, KOR-Bench, AIME24, and AIME25, respectively. The results further demonstrate favorable scaling behavior, suggesting strong potential for continued gains with more compute. In addition, RLPT provides a solid foundation, extending the reasoning boundaries of LLMs and enhancing RLVR performance.

1 Introduction

RLPT scales reinforcement learning on pre-training data by rewarding next-segment reasoning, and experiments report gains on general-domain and mathematical benchmarks, favorable compute scaling, and further improvements when RLPT underlies RLVR.

1 Introduction

  • Scaling LLMs through larger models and datasets faces increasingly costly infrastructure and inference, while the supply of high-quality web corpora is limited.These constraints motivate RLPT as an alternative training-time scaling paradigm.Source: §1 · p. 2
  • RLPT allocates training compute for autonomous reasoning-trajectory exploration, which can augment data with inferred thought processes and foster generalization while remaining close to the original policy distribution.Unlike RLHF and RLVR, the approach removes dependence on human annotation for rewards.Source: §1 · p. 2
  • RLPT derives self-supervised rewards by comparing a predicted next text segment with the ground-truth segment, using ASR to predict a complete next sentence and MSR to infer a masked middle span.Training interleaves ASR and MSR to optimize autoregressive generation and in-context understanding.Source: §1 · p. 2
  • For Qwen3-4B-Base, RLPT gains were 3.0 on MMLU, 5.1 on MMLU-Pro, 8.1 on GPQA-Diamond, 6.0 on KOR-Bench, and Pass@1 gains of 6.6 on AIME24 and 5.3 on AIME25.Comparable gains are also reported for Llama3.2-3B-Base and Qwen3-8B-Base.Source: §1 · p. 2
  • RLPT benchmark performance empirically follows a scaling law with training compute, indicating potential for continued gains as compute increases.Qualitative analysis also finds diverse reasoning strategies in RLPT trajectories, and the paper distills practical design lessons for future research.Source: §1 · p. 2
  • With Qwen3-4B-Base, RLPT as an RLVR foundation adds Pass@1 gains of 2.3 on AIME24 and 1.3 on AIME25, and Pass@8 gains of 3.7 and 2.0, respectively.The results characterize RLPT as a foundation that extends LLM reasoning capabilities and boosts mathematical reasoning benchmarks.Source: §1 · p. 2

2 Preliminary

The section reviews reinforcement learning and next-token prediction as foundations for RLPT, contrasting annotation-dependent reward training with the limits of supervised next-token prediction.

2 Preliminary

  • RL and next-token prediction are reviewed as the foundations of RLPT.The section introduces both paradigms as context for the proposed approach.Source: §2 · p. 3

2.1 Reinforcement Learning in Large Language Models

  • RLHF and RLVR use human-annotation-dependent rewards, while PPO and GRPO are common policy-gradient optimizers.RLHF uses a reward model trained on annotated preferences; RLVR checks outputs against reference answers. Despite their effectiveness, both face scalability challenges from reliance on human annotations.Source: §2.1 · p. 3

2.2 Next-Token Prediction

  • SFT under next-token prediction can promote surface-level memorization rather than RL's deeper generalization capabilities (Chu et al., 2025; Lai et al., 2025; Shenfeld et al., 2025).Pre-training and post-training based on next-token prediction remain the mainstream LLM optimization paradigm and have succeeded across diverse applications.Source: §2.2 · p. 3

3 Reinforcement Learning on Pre-Training Data

RLPT uses next-segment reasoning as a self-supervised reinforcement-learning objective, deriving rewards from unlabeled text rather than human annotations. Its data pipeline and segment-level objectives support training on diverse pre-training corpora.

3 Reinforcement Learning on Pre-Training Data

  • RLPT trains with next-segment reasoning on unlabeled text, removing reliance on human annotations and enabling reinforcement learning to scale on pre-training corpora.The subsequent text segment serves as ground truth.Source: §3 · p. 3

3.1 Data Preparation

  • The corpus combines diverse web sources with deduplication, PII masking, contamination removal, rule- and model-based filtering, and curated math QA data from Team et al. (2025b).The math QA data is curated from an annealing dataset to enhance reasoning ability.Source: §3.1 · pp. 3–4

3.2 Next-Segment Reasoning

  • For contiguous text segments, ASR predicts a target from preceding context, while MSR uses both preceding and subsequent context; training interleaves the two objectives.Segments may correspond to semantically coherent units, and MSR is particularly useful for tasks such as code completion.Source: §3.2 · p. 4
  • A generative reward model scores semantic consistency while allowing linguistic variation, checking predictions as valid prefixes of references spanning several subsequent segments.This avoids overly strict comparisons when predictions extend across multiple segments.Source: §3.2 · p. 4

The training objective of RLPT is defined as

RLPT balances ASR and MSR terms in its training objective, and its implementation uses instruction-following cold-start training and sentence-level next-segment targets.

The training objective of RLPT is defined as

  • The objective weights its MSR term by λ, which balances ASR and MSR contributions and can be adjusted for downstream applications.In the MSR term, the policy generates o conditioned on preceding context and the next segment, with reward r(o, si).Source: The training objective of RLPT is… · p. 5

3.3 Training Details

  • RLPT uses supervised fine-tuning on instruction-following data as a cold-start phase because base models need instruction-following ability to initiate next-segment reasoning.Source: §3.3 · p. 5
  • RLPT uses sentence-level segments by default; alternative segmentation strategies showed no clear improvement, so their exploration is left for future work.Sentences are extracted with NLTK, short sentences are filtered out, and each remaining sentence becomes an RL target.Source: §3.3 · p. 5

4 Experiments

Experiments report consistent general-domain and mathematical gains from RLPT, including further improvements when it initializes RLVR. Scaling and reward analyses characterize how RLPT behaves during training.

4.1 Experimental Setup

  • Training uses on-policy GRPO for next-segment reasoning, with eight sampled outputs per prompt and no KL regularization.Experiments use Llama3 and Qwen3 models; cold-start SFT runs for 3 epochs, and mathematical RLVR uses the next-segment task's hyperparameters.Source: §4.1 · p. 5

4.2 Evaluation Metric

  • Evaluation covers general-domain accuracy and mathematical Pass@1 and Pass@8 across the stated benchmarks.Mathematical evaluation samples 64 responses per prompt at temperature 0.6 and top-p 0.95, uses a 32,768-token maximum generation length, and checks correctness with Math-Verify2.Source: §4.2 · p. 5

4.3 Experimental Results

  • RLPT delivers consistent general-domain gains across the evaluated benchmarks and model families.For Qwen3-4B-Base, absolute improvements are 3.0 on MMLU, 5.1 on MMLU-Pro, 8.1 on GPQA-Diamond, 2.0 on SuperGPQA, and 6.0 on KOR-Bench. Qwen3-8B-Base improves by 1.4 on MMLU, 3.4 on MMLU-Pro, 2.0 on GPQA-Diamond, 2.3 on SuperGPQA, and 2.1 on OlympiadBench; Llama-3.2-3B-Base improves by 1.5 on MMLU-Pro, 11.6 on GPQA-Diamond, and 3.4 on SuperGPQA.Source: Table 1 · p. 6, §4.3 · p. 6
  • RLPT improves Qwen3-4B-Base mathematical reasoning on AIME24 and AIME25, and initializing RLVR from RLPT adds further gains.RLPT's Pass@1 improvements are 6.6 on AIME24 and 5.3 on AIME25; its Pass@8 improvements are 5.0 and 1.4, respectively. With RLPT initialization, RLVR gains 2.3 (AIME24) and 1.3 (AIME25) on Pass@1, and 3.7 (AIME24) and 2.0 (AIME25) on Pass@8.Source: Table 2 · p. 6, §4.3 · p. 6

4.4 Analysis

  • RLPT performance follows a power-law decay with training tokens, while RLPT-initialized RLVR improves throughout training on both Pass@1 and Pass@8.The authors describe this as indicating potential for further gains with more compute and note that gains do not come at the expense of exploration capability.Source: §4.4 · pp. 6–7, Figure 3 · p. 7
  • An illustrative RLPT trajectory abstracts context, selects and checks a continuation, explores alternatives, and backtracks when appropriate before answering.The example presents these actions as a structured sequence for next-segment reasoning.Source: Table 3 · p. 7, §4.4 · p. 7
  • A relaxed prefix reward replaces a strict semantic-match reward that proved too rigid and disrupted training.The prefix reward scores a prediction as 1 when it is a valid prefix of the ground-truth completion, accommodating uneven segment information, providing a more stable signal, and enabling longer responses with improved downstream mathematical reasoning performance.Source: §4.4 · pp. 7–8, Figure 4 · p. 8

5 Related Work

RLPT applies reinforcement learning to training-time scaling by letting models learn from pre-training corpora. Its next-segment objective distinguishes it from prior unlabeled-data RL approaches and removes the need for human annotation.

Scaling Paradigms

  • RLPT uses RL for training-time self-exploration of pre-training corpora, unlike supervised scaling and related RPT (Dong et al., 2025) and Quiet-STaR (Zelikman et al., 2024), by predicting next segments.The cited approaches also apply RL to unlabeled data for training-time scaling, but focus on next-token prediction.Source: §5 · p. 8

Reinforcement Learning in LLMs

6 Conclusion

RLPT applies reinforcement learning to pre-training data through a self-supervised next-segment reasoning objective, enabling training on large unlabeled corpora without human annotations. Experiments show gains in general-domain and mathematical reasoning, with performance scaling favorably with training compute.

  • RLPT applies reinforcement learning to pre-training data using a self-supervised next-segment reasoning objective, avoiding the need for human annotations.Source: §6 · p. 8
  • Experiments demonstrate RLPT’s effectiveness through substantial gains on both general-domain and mathematical reasoning tasks.Source: §6 · p. 8
  • RLPT performance scales favorably with training compute, suggesting potential for further gains.Source: §6 · p. 8
Loading 2509.19249v2…