PaperlayerView exact revision on arXiv

Source-linked AI summary

RLP: Reinforcement as a Pretraining Objective

Ali Hatamizadeh, Syeda Nahida Akter, Shrimai Prabhumoye, Jan Kautz, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi

arXiv:2510.01265v2cs.LGcs.AIcs.CL

TL;DR

Standard next-token pretraining does not explicitly encourage long-range reasoning, leaving reinforcement learning for post-training. RLP instead samples chain-of-thought during pretraining and rewards its information gain, improving reasoning benchmarks and scaling across architectures and model sizes.

  • Problem

    Next-token prediction pretraining does not explicitly encourage long-range reasoning or integration with world knowledge, motivating the question of whether reinforcement should enter earlier.

  • Method

    RLP treats chain-of-thought as an action before next-token prediction and uses its log-likelihood improvement over an EMA no-think baseline as a dense, verifier-free reward.

  • Results

    RLP consistently outperforms likelihood-only and reinforcement-pretraining baselines across datasets, domains, and architectures, with gains persisting and compounding after alignment.

  • Takeaways & Limitations

    Reinforcement pretraining is presented as a principled alternative to likelihood-only training for inducing reasoning during pretraining.

Abstract

from arXiv · show

The dominant paradigm for training large reasoning models starts with pre-training using next-token prediction loss on vast amounts of data. Reinforcement learning, while powerful in scaling reasoning, is introduced only as the very last phase of post-training, preceded by supervised fine-tuning. While dominant, is this an optimal way of training? In this paper, we present RLP, an information-driven reinforcement pretraining objective, that brings the core spirit of reinforcement learning -- exploration -- to the last phase of pretraining. The key idea is to treat chain-of-thought as an exploratory action, with rewards computed based on the information gain it provides for predicting future tokens. This training objective essentially encourages the model to think for itself before predicting what comes next, thus teaching an independent thinking behavior earlier in the pretraining. More concretely, the reward signal measures the increase in log-likelihood of the next token when conditioning on both context and a sampled reasoning chain, compared to conditioning on context alone. This approach yields a verifier-free dense reward signal, allowing for efficient training for the full document stream during pretraining. Specifically, RLP reframes reinforcement learning for reasoning as a pretraining objective on ordinary text, bridging the gap between next-token prediction and the emergence of useful chain-of-thought reasoning. Pretraining with RLP on Qwen3-1.7B-Base lifts the overall average across an eight-benchmark math-and-science suite by 19%. With identical post-training, the gains compound, with the largest improvements on reasoning-heavy tasks such as AIME25 and MMLU-Pro. Applying RLP to the Nemotron-Nano-12B-v2 increases the overall average from 42.81% to 61.32% and raises the average on scientific reasoning by 23%, demonstrating scalability across architectures and model sizes.

1 INTRODUCTION

RLP moves reinforcement learning into pretraining by rewarding chain-of-thought for improving next-token prediction on ordinary text. It provides dense, verifier-free signals and improves reasoning performance across benchmarks, including after identical post-training and at larger scale.

  • 1 INTRODUCTION: RLP treats chain-of-thought generation as an explicit action before next-token prediction and rewards its information gain over a no-think baseline.The reward is the increase in observed-token log-likelihood when conditioning on the sampled thought.
  • 1 INTRODUCTION: RLP supplies dense, verifier-free information-gain rewards across ordinary document streams, unlike sparse next-token or verifier-dependent objectives.The approach bridges pretraining and reasoning without requiring external verification or curated task-specific solutions.
  • 1 INTRODUCTION: The method combines reinforcement updates with likelihood training through group-relative advantages, clipped thought-token updates, and an EMA baseline.The paper also provides guarantees relating expected reward to cross-entropy reduction and a computable lower bound.
  • 1 INTRODUCTION: 35% relative improvement over a heavily trained baseline on NEMOTRON-NANO-12B-V2 uses only 0.125% of the data, supporting scalability across model families and sizes.

2 METHODOLOGY

RLP samples a chain-of-thought before each next-token prediction and rewards thoughts that increase the observed token’s log-evidence relative to an EMA no-think baseline. Training uses position-wise rewards, group-relative advantages, and clipped updates to the shared thought-policy/predictor network.

  • 2 METHODOLOGY: RLP samples a latent chain-of-thought, predicts the observed token with it, and compares that evidence against an EMA teacher without thinking.The same network generates thoughts and performs reasoned prediction, while the EMA teacher supplies the no-think counterfactual.
  • 2 METHODOLOGY: The EMA baseline is deliberately lagged and smoothed to remain informative while preventing reward collapse and reducing reward hacking.A frozen baseline would drift from the model, whereas an instantly tracking baseline would drive the log-likelihood ratio toward zero.
  • 2 METHODOLOGY: Multiple thoughts are sampled per context, group-relative advantages reduce variance, and a clipped surrogate updates only the sampled thought tokens.The reward is stop-gradient during policy optimization, and the shared model is updated through the information-gain objective rather than a standard next-token loss.
  • 2 METHODOLOGY: The reward is the reasoned scorer’s log-likelihood minus the no-think baseline’s log-likelihood for the observed next token.Positive reward occurs exactly when the sampled thought raises the model’s probability for that token; in expectation, reward equals cross-entropy reduction.
  • 2 METHODOLOGY: Position-wise rewards attach credit wherever thinking changes predictive probability, eliminating the need for a learned value function or external verifier.Rewards are computed under teacher forcing independently at each next-token position.

3 EXPERIMENTAL SETUP

The experiments evaluate RLP on QWEN3-1.7B-BASE and scale it to NEMOTRON-NANO-12B-V2 across reasoning and general-purpose corpora, with common post-training and benchmark assessment.

  • 3 EXPERIMENTAL SETUP: RLP training covers SFT-style reasoning datasets and general corpora including academic papers, math textbooks, and web-crawl data.For end-to-end pretraining evaluation, MRLP uses 1B input tokens, while rewards are applied to one randomly selected token per document.
  • 3 EXPERIMENTAL SETUP: MCPT serves as the compute-equivalent continuous-pretraining baseline using the same tokens as RLP.All models subsequently undergo the same SFT and RLVR stages to test whether pretraining gains persist after alignment.
  • 3.1 EVALUATION METRICS: Evaluation uses NeMo-Skills2 with MATH AVG and SCIENCE AVG aggregates spanning mathematical, conceptual-science, and graduate-level STEM benchmarks.Math scores average Pass@1 across four benchmarks, while science scores average greedy and Pass@1 results across three benchmarks.
  • 3.1 EVALUATION METRICS: Table 2 reports NEMOTRON-NANO-12B-V2 benchmark results, distinguishing RLP variants and models after SFT + RLVR post-training.Shaded columns identify RLP variants, and “Post” denotes the shared post-training pipeline.

4 RESULTS

RLP improves reasoning performance over maximum-likelihood and continuous-pretraining baselines, remains stronger after post-training, and scales across model architectures and sizes.

  • QWEN3-1.7B-BASE: 19% and 17%: MRLP exceeds Mbase and MCPT respectively on the QWEN3-1.7B-BASE overall average, with especially strong gains on AIME25 and MMLU-Pro.After identical SFT + RLVR, MRLP remains 8% above Mbase and 7% above MCPT overall, with a 3-point science advantage over MCPT.
  • Scaling Model Size and Architecture: 43% relative gain: NEMOTRON-NANO-12B-V2 rises from 42.81% to 61.32% at the base stage, then reaches 68.09% after SFT + RLVR.The post-trained RLP model remains above the compute-matched baseline at 65.31%.
  • Scaling Model Size and Architecture: Science Avg increases from 34.51% to 57.26% before post-training and 64.52% after post-training for NEMOTRON-NANO-12B-V2, versus 58.91% for continuous pretraining.The same scaling trend extends to QWEN3-14B-BASE, whose overall average rises from 60.66% to 65.00% after 1B RLP tokens.
  • RPT Comparison: Under token-matched training, RLP improves Math Avg by 2.12%, Science Avg by 1.19%, and Overall Avg by 1.66% over RPT.Under FLOP-matched training, RLP achieves a 20.12% relative average improvement over RPT.
  • RPT Comparison: RLP applies dense information-gain rewards at sampled positions, whereas RPT uses sparse binary rewards on assistant-filtered tokens.The comparison controls token and FLOP budgets, while RLP’s reward evaluates the predictive utility of each sampled chain of thought.

5 ABLATIONS

Ablations show that RLP gains transfer across corpus types, survive compute-matched comparisons, exceed continuous pretraining with reasoning data, and depend on rollout and completion-length choices.

  • Diverse Corpora: 7–9%: RLP improves averages across all evaluated corpus families, including SFT-style reasoning data and general-purpose web-crawl data.The strongest gains occur on Nemotron-Crossthink and Web-Crawl, supporting cross-domain transfer rather than dependence on curated reasoning data.
  • Compute-Equivalent Baselines: 5.3% gain: MRLP outperforms a compute-matched MCPT exposed to 6B tokens despite using 170M input tokens, or 35× less data.RLP also beats the same-token 170M MCPT baseline across math and science benchmarks.
  • Comparison with High-Quality Reasoning Data: 8% on Nemotron-Crossthink and 5% on DPT: MRLP exceeds MCPT even when continuous pretraining uses high-quality reasoning data.CPT remains sensitive to domain skew, whereas RLP generalizes more evenly across disciplines.
  • Hyperparameter Ablations: 42.17%: sixteen rollouts reach the best reported overall score, while 32 rollouts decrease slightly to 41.75%.Four and eight rollouts already reach 41.38% and 41.95%, respectively.
  • Hyperparameter Ablations: 42.17%: increasing completion length from 64 to 2048 raises Overall from 11.50%, while β = 10^-4 and 10^-3 underperform β = 0.The selected configuration is 16 rollouts, completion length 2048, and β = 0.

6 RELATED WORK

RLP extends next-token prediction with verifier-free reasoning rewards during pretraining and differs from prior verifier-free post-training approaches in scope and training stage.

  • Next-Token Prediction: Next-token prediction remains the dominant pretraining objective because it supports left-to-right generation and broad downstream accuracy, but RLP adds a dense reasoning reward during pretraining.The reward leverages reasoning before predicting the next token rather than replacing the next-token objective entirely.
  • Verifier-Free Rewards in Post-Training: RLP is a GRPO-style pretraining objective that operates on arbitrary text, including web-crawl data, academic papers, and SFT data.This distinguishes it from verifier-free approaches applied after supervised fine-tuning.

7 CONCLUSION

RLP reframes reasoning as a reinforcement pretraining objective that rewards chain-of-thought by its information gain for next-token prediction. Across datasets, domains, and architectures, it outperforms compute-matched baselines and scales to large hybrid models.

  • RLP rewards chain-of-thought by its information gain for next-token prediction during pretraining.
  • RLP’s gains persist and compound after alignment, establishing reinforcement pretraining as an alternative to likelihood-only training.
  • Experiments across datasets, domains, and architectures show that RLP consistently outperforms compute-matched baselines and scales efficiently to large hybrid models.

8 APPENDIX

The appendix proves that RLP’s tokenwise information-gain rewards correspond to cross-entropy improvements and aggregate into sequence-level improvement under teacher forcing. It also derives a Jensen-based lower bound and characterizes when that bound is tight.

  • Proposition 1: Proposition 1 establishes a tokenwise cross-entropy reduction identity by relating reward expectations to reasoned and baseline log-likelihoods.
  • Proposition 2: Proposition 2 derives a Jensen lower bound on the expected objective through marginalization over sampled thoughts.
  • Proposition 2: The Jensen bound is tight when the reasoned predictor’s token probability is almost surely constant across thoughts, including when thoughts are ignored or degenerate.
  • Proposition 3: Under teacher forcing, averaging expected tokenwise information-gain rewards equals the per-token sequence-level cross-entropy improvement.

9 WHY RELATIVE ADVANTAGES DO NOT REWARD BAD THOUGHTS

The appendix explains why group-relative advantages do not reward harmful thoughts in a pathological way. Even when all rewards are negative, the update reallocates probability toward relatively better thoughts and improves the expected objective in expectation.

  • The group-relative estimator is unbiased, so selecting the least-bad rollout follows the correct gradient direction rather than rewarding bad reasoning absolutely.
  • Even when every reward is negative, shifting probability from more-negative to less-negative thoughts increases the expected objective.
  • The update is zero-sum across advantages: probability rises for less-negative-than-average thoughts and falls for worse alternatives.
  • The EMA baseline suppresses artificial reward inflation because transient improvements dissipate as the slowly updated baseline catches up.
  • Numerical illustration: A numerical example shows the expected objective improving from −0.5500 to −0.5284 after one update despite initially negative rewards.
  • Numerical illustration: Later iterations show dense updates making initially less-harmful thoughts less negative and eventually helpful, with the objective improving again.

10 EXPERIMENTAL SETUP

The experiments apply RLP to base and intermediate checkpoints across diverse datasets using substantial GPU training budgets. The setup also defines a prompting format that inserts brief reasoning before continuation of the source text.

  • RLP: RLP training uses diverse datasets, base and intermediate checkpoints, 32 H100 GPUs, and budgets ranging from 170M to 10B tokens.
  • Continuous Pre-training: Continuous-pretraining baselines use the same 32-H100 hardware and token-budget range for comparisons.
  • Post-Training: Post-training uses OpenThoughts3 for supervised fine-tuning and MATH examples for reinforcement learning with verifiable rewards.
  • Prompt: The prompt asks the model to briefly think about the next content, then continue the prefix in the same notation, style, and tone.

11 EXTENDED ABLATION DETAILS

The ablations show that longer reasoning completions drive the largest gains, while rollout benefits saturate and KL regularization offers no net improvement.

  • Rollout count: 42.17% Overall is reached with 16 rollouts, while increasing to 32 lowers performance slightly to 41.75%.Four and eight rollouts reach 41.38% and 41.95%, respectively.
  • Completion length: 42.21% Overall at completion length 4096 is only marginally above 42.17% at 2048, making 2048 the preferred trade-off.The largest increase occurs from 512 to 1024 tokens, when Overall rises by 14.24 points to 38.89%.
  • Completion length: 11.50% Overall at length 64 rises to 42.17% at 2048, with Math increasing from 1.12% to 48.06%.Science increases from 21.88% to 36.29% across the same settings.
  • KL coefficient: 41.35% and 41.44% Overall with β = 10^-4 and 10^-3 underperform β = 0 and also increase memory use and step time.Some task-level improvements are offset by broader declines, so β = 0 is retained.
  • Summary: The ablation trends show smooth rollout saturation, a sharp length-driven regime change between 512 and 1024 tokens, and no net KL benefit.The corresponding taskwise breakdown appears in Table S.1.

12 ADDITIONAL ABLATIONS

Additional experiments test RLP across models, training conditions, computational budgets, and the semantics of its reward. They report scaling, early-checkpoint effectiveness, compute efficiency, stable prediction quality, and architecture robustness, while addressing self-referentiality concerns.

  • Scaling LLMs: 65.00% Overall average is achieved by QWEN3-14B-BASE with RLP, up from 60.66%, while Science average rises from 60.07% to 64.76%.The result indicates the dense verifier-free signal remains effective on a substantially larger model.
  • Early application: 50.14 Math average is reached from 21.93% when RLP is applied to a checkpoint trained on only 4T tokens, while Overall rises from 12.05% to 24.08%.Science Avg@1[4] increases from 5.69% to 11.96%.
  • EMA baseline: The EMA baseline remains stable across tested decay rates, with performance peaking at τ = 0.999 and no divergent behavior observed.The baseline is updated from student parameters, and rapid or slow updates can respectively collapse or stale the reward.
  • Efficiency: 43.36% accuracy from 170M tokens exceeds the FLOP-matched CPT baseline’s 35.60% after 6B tokens, a +7.76% margin despite RLP’s 2.25× per-step slowdown.RLP applies rollouts to one randomly sampled token per sequence, reducing rollout cost relative to applying it at every position.
  • Perplexity: Lower PPL and NLL hold on both Nemotron CrossThink and Wikitext-103, consistent with RLP optimizing next-token prediction rather than degrading ordinary-token modeling.The paper links maximizing RLP reward mathematically to minimizing cross-entropy.
  • Compute matching: 42.13% Overall Avg versus 38.04% for compute-matched MCPT[6B] shows RLP’s advantage persists under the reported FLOP comparison.The baseline processes 35× more data than RLP in the described comparison.
  • Reward semantics: RLP’s reward is grounded in the observed corpus token: increasing confidence in an incorrect continuation decreases reward rather than benefiting the objective.In expectation, the reward equals the cross-entropy reduction from conditioning on the thought.
  • Architecture and data diversity: 61.32% Overall average after RLP on Nemotron-Nano-12B-V2 rises from 42.81%, supporting transfer across a hybrid Mamba2-Transformer architecture and distinct data lineage.The paper characterizes this as evidence of architecture-agnostic scalability and robustness across experimental environments.

13 DATA BLEND EXTENDED RESULTS

RLP improves performance across structured reasoning and general-purpose corpora, including mixed-domain and open-ended data. The results support cross-domain transfer without requiring specialized reasoning traces and suggest that data diversity strengthens the learned signal.

  • Corpus-family comparison: RLP consistently improves over the base model across structured reasoning and general-purpose corpus families, including academic papers and web crawl data.The strongest improvements occur on Nemotron-CrossThink, but gains also appear on purely general corpora.
  • Data diversity: Mixed-domain and open-ended corpora improve math, science, and professional benchmarks without evidence of domain-specific overfitting or math degradation.The experiments report that diversity strengthens general reasoning performance.
  • Task-level transfer: Improvements across MATH500, GSM8K, MMLU, MMLU-Pro, GPQA, and related metrics occur for both structured SFT data and general pretraining text.Even web-scale crawl data produces competitive math and science averages.
  • Conclusion: The results support RLP as a domain-agnostic pretraining augmentation that enhances reasoning robustness and general benchmark accuracy without specialized reasoning datasets.The paper’s conclusions emphasize cross-domain transfer and amplification from diverse data.
Loading 2510.01265v2…