PaperlayerView exact revision on arXiv

Source-linked AI summary

PretrainZero: Reinforcement Active Pretraining

Xingrun Xing, Zhiyuan Fan, Jie Lou, Guoqi Li, Jiajun Zhang, Debing Zhang

arXiv:2512.03442v1cs.CL

TL;DR

Domain-specific verification and reward-model requirements constrain reinforcement learning beyond narrow post-training settings. PretrainZero actively selects informative content from pretraining data and trains reasoning policies to recover masked spans; the authors report gains across general and math benchmarks that remain after downstream RLVR.

  • Problem

    Post-training RL relies on domain-specific verifiers or reward models and human feedback, limiting the use of reinforcement learning for general reasoning on inexpensive pretraining data.

    Source: §1 · p. 2

  • Method

    PretrainZero jointly learns an active mask-generation policy and a masked-span reasoning policy on real-world pretraining data using reinforcement learning.

    Source: §1 · p. 2

  • Results

    PretrainZero improves Qwen3-4B-Base by 8.43 on MMLU-Pro, 5.96 on SuperGPQA, and 10.60 on math average during reinforcement pretraining, with improvements remaining after general RLVR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pretrained reasoning models can serve as reasoning foundations for general downstream RLVR tasks.

    Source: §1 · p. 3

    Vanilla RPT is constrained by noisy real-world data, low information density, dependence on post-training or external supervision setups, and single-token updates.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Mimicking human behavior to actively learning from general experience and achieve artificial general intelligence has always been a human dream. Recent reinforcement learning (RL) based large-thinking models demonstrate impressive expert-level abilities, i.e., software and math, but still rely heavily on verifiable rewards in specific domains, placing a significant bottleneck to extend the performance boundary of general reasoning capabilities. In this work, we propose PretrainZero, a reinforcement active learning framework built on the pretraining corpus to extend RL from domain-specific post-training to general pretraining. PretrainZero features the following characteristics: 1) Active pretraining: inspired by the active learning ability of humans, PretrainZero learns a unified reasoning policy to actively identify reasonable and informative contents from pretraining corpus, and reason to predict these contents by RL. 2) Self-supervised learning: without any verifiable labels, pretrained reward models, or supervised fine-tuning, we directly pretrain reasoners from 3 to 30B base models on the general Wikipedia corpus using RL, significantly breaking the verification data-wall for general reasoning. 3) Verification scaling: by tackling increasingly challenging masked spans, PretrainZero substantially enhances the general reasoning abilities of pretrained base models. In reinforcement pretraining, PretrainZero improves Qwen3-4B-Base for 8.43, 5.96 and 10.60 on MMLU-Pro, SuperGPQA and math average benchmarks. In post-training, the pretrained models can also serve as reasoning foundation models for downstream RLVR tasks.

1 Introduction

PretrainZero extends reinforcement learning into self-supervised pretraining by actively selecting informative content from real-world corpora. The paper reports reasoning gains during reinforcement pretraining that remain substantial after general RLVR post-training.

Motivation

  • Prior post-training RL relies on domain-specific verifiers or reward models and human feedback, while vanilla RLPT struggles with low-information-density data and noisy tokens.The paper frames these constraints as a data-wall for extending general reasoning capabilities.Source: §1 · p. 2

Approach

  • PretrainZero uses on-policy mask generation to select informative, verifiable, not-yet-mastered content, then trains masked-span prediction to reason and recover those spans.The mask-generation and prediction tasks are jointly optimized with a min–max bilevel reinforcement objective using GRPO.Source: §1 · p. 2, Figure 2 · p. 3

Evaluation

  • 8.43 on MMLU-Pro, 5.96 on SuperGPQA, and 10.60 on math average: PretrainZero improves Qwen3-4B-Base by these amounts during reinforcement pretraining.After general RLVR post-training, improvements remain 2.35 on MMLU-Pro, 3.04 on SuperGPQA, and 2.81 on math average.Source: §1 · p. 2

Contributions

  • The authors present PretrainZero as a stand-alone RLPT method trained directly on real-world pretraining data, without an SFT cold-start or reward models.They also state that its pretrained reasoning models can serve as foundations for general downstream RLVR tasks.Source: §1 · pp. 2–3

2 Preliminary

The preliminary section contrasts conventional self-supervised prediction objectives with reinforcement pretraining, which generates reasoning traces before token prediction. It also identifies limitations that make vanilla RPT unsuitable for practical pretraining settings.

2 Preliminary

  • Traditional self-supervised pretraining captures linguistic patterns and contextual dependencies, while reinforcement pretraining uses verifiable token-prediction tasks to learn reasoning.The section introduces these learning patterns as background for the methods that follow.Source: §2 · p. 3

2.1 Self-Supervised Pretraining

  • NTP predicts each token from preceding context, whereas MTP predicts masked tokens using both preceding and succeeding context.The paper treats both as self-supervised objectives and simulates them as reinforcement reasoning tasks.Source: §2.1 · p. 3

2.2 Reinforcement Pretraining

  • RPT treats a corpus token as ground truth, generates a chain-of-thought before predicting it, and applies GRPO with an exact-match verifiable reward.This makes the prediction a reasoning task rather than direct next-token prediction.Source: §2.2 · p. 4
  • Vanilla RPT faces noisy-corpus collapse, dependence on post-training or external SFT and reward-model setups, poor informative-token selection, and limited learning per step.Unlike NTP's parallel prediction of all tokens, RPT predicts one token per sample.Source: §2.2 · p. 4

3 Reinforcement Active Pretraining

The section establishes reinforcement pretraining baselines and then introduces active masking: a generator selects spans for a predictor to recover through reasoning. The baseline findings motivate masks that are informative but still predictable.

3 Reinforcement Active Pretraining

  • The baseline study uses Wikipedia with Qwen3-4B-Base to assess whether general-domain reinforcement pretraining can support a unified active task.The section first establishes an RLPT baseline before proposing the active pretraining task.Source: §3 · p. 4

3.1 Reinforcement Pretraining Baselines

  • Random NPT RLPT outperforms Random Mask RLPT, while random word-span selection does not effectively capture richer pretraining semantics.The reported comparison also notes more stable training dynamics for Random NPT RLPT.Source: §3.1 · p. 5
  • On Wikipedia, entropy-based token selection leads to training collapse and rapid reward degradation, as high entropy can reflect noise rather than learnable difficulty.When all samples in a group yield either 0 or 1 accuracy, the reward signal becomes degenerate; zero prediction accuracy also leaves the generator unrewarded to avoid noisy masks.Source: §3.1 · p. 5, §3.3 · p. 7

3.2 Active Pretraining Tasks

  • Mask generation selects verifiable word spans from pretraining text, and mask prediction uses chain-of-thought reasoning to recover each masked span.The generator can shift from explicit clues early in training toward harder, unsolved words and domains later.Source: §3.2 · p. 6

3.3 Reinforcement Active Learning

  • A shared LLM couples mask generation and prediction in a min–max objective: the generator seeks challenging masks while the predictor maximizes recovery reward.The objective is framed as active learning, with the generator probing the predictor’s weaknesses.Source: §3.3 · p. 6
  • Prediction receives an exact-match reward, while the generator is rewarded for lower prediction accuracy but receives zero reward when accuracy is zero.The zero-accuracy rule avoids rewarding masks that are not predictable for the predictor.Source: §3.3 · p. 7

4 Experimental Results

Experiments evaluate reinforcement pretraining on Wikipedia across base models and reasoning benchmarks, then test downstream RLVR and ablate data domain and mask regularization. PretrainZero outperforms Random RLPT across four models, and its gains persist after post-training.

4.1 Implementation Details

  • Experiments use 3B–30B base models, Wikipedia pretraining data, and general-domain and math-domain reasoning benchmarks; Qwen models receive no supervised fine-tuning cold start.The evaluation includes MMLU-Pro, SuperGPQA, BBEH, and six math benchmarks; SmolLM3-3B instead receives 100 steps of random RLPT before PretrainZero.Source: §4.1 · p. 7

4.2 Pretraining Results

  • Continued pretraining and supervised fine-tuning drop 16.47 and 8.09 points on Qwen3-4B, and 24.75 and 10.27 on Qwen3-8B; Random RLPT and PretrainZero improve 3.05 and 7.25, and 3.89 and 5.71, respectively.The authors attribute the supervised-training drops to limited useful information in low-quality Wikipedia passages and possible disruption of the pretrained distribution.Source: §4.2 · p. 8, Table 1 · p. 9, Table 2 · p. 9
  • PretrainZero outperforms Random RLPT by 4.20, 1.82, 3.17, and 3.16 points on Qwen3-4B, Qwen3-8B, Qwen3-30B-A3B-MoE, and SmolLM3-3B, respectively.The comparison uses general-domain reasoning benchmarks.Source: §4.2 · p. 8, Table 1 · p. 9
  • Across training, PretrainZero’s reasoning length and general- and math-benchmark performance improve, while its fixed-question MMLU-Pro reasoning length remains similar.The authors say training-time CoT growth mainly reflects improved mask-prediction capability, and RLPT improves CoT reasoning efficiency versus the base model.Source: §4.2 · pp. 8–9
  • For the same masked target, Random RLPT outputs an answer without explicit reasoning, whereas PretrainZero explores possibilities and analyzes and verifies them step by step.The passage states that this reasoning behavior may strengthen generalization because masked-span prediction is absent from downstream RL tasks.Source: §4.2 · pp. 9–10

4.3 Post-Training Results

  • After general RLVR post-training, PretrainZero retains improvements of 2.35, 3.04, and 2.81 on MMLU-Pro, SuperGPQA, and math average; the end-to-end gains versus the base model are 2.56 overall accuracy and 2.18 math average.The post-training comparison includes base, Random RLPT, and PretrainZero models with 1,000 or 2,000 RLPT steps.Source: §4.3 · p. 11, Table 3 · p. 11, Table 4 · p. 11

4.4 Ablation Studies

  • On Wikipedia versus MathPile, using general-domain Wikipedia yields better RLPT performance in the reported MMLU-Pro comparison.The authors note that curating high-quality mathematical data requires substantial expert effort and recommend general-domain data as a lower-cost acquisition option.Source: §4.4 · p. 11
  • PretrainZero, PretrainZero-OneMask, and PretrainZero-Words all train stably for 2,000 steps, with PretrainZero achieving better performance.The variants differ in repeated-span handling and whether masks are restricted to complete words.Source: §4.4 · p. 11

5 Related Works and Discussion

The discussion contrasts scalable self-supervised pretraining with reinforcement-learning approaches constrained by verification and data requirements. It positions PretrainZero against prior reinforcement pretraining and identifies unresolved questions about fully self-supervised RLPT.

Self-Supervised Pretraining for LLMs

  • Self-supervised next-token prediction scales across massive corpora, while masked prediction remains used in language, diffusion, and code models.The discussion presents reliability, scalability, and broad applicability as insights for reinforcement pretraining.Source: §5 · p. 11

Reinforcement Learning for LLMs

  • RLHF and RLVR rely on human annotation or domain-specific verification, while verifiable ground-truth requirements and environment construction limit RLVR scalability for general reasoning.RLHF reward models also require continuing human-labeled updates to avoid reward hacking.Source: §5 · p. 12

Reinforcement Pretraining

  • RPT demonstrated next-token prediction as an RLVR objective, but its reliance on synthetic OmniMath CoT data and a post-trained starting model limits practical prolonged RLPT.The passage characterizes Quiet-STaR and Fast Quiet-STaR as earlier token-level reasoning approaches.Source: §5 · p. 12

Open Questions

  • The discussion asks whether stand-alone RLPT can work on noisy real-world corpora without reward models, SFT cold starts, or supervised cross-entropy, and how to improve learning efficiency on low-information-density data.It contrasts PRT’s auxiliary objective and QA-style data with RLPT1’s reward model and SFT cold start.Source: §5 · p. 12

6 Conclusion

The conclusion presents PretrainZero as stand-alone reinforcement pretraining on real-world corpora, using active learning to select informative content. It reports improved end-task performance from Wikipedia and argues that more efficient ways to uncover latent corpus information merit exploration.

6 Conclusion

  • PretrainZero applies stand-alone reinforcement pretraining to a real-world corpus and uses active learning to explore informative, verifiable, not-yet-mastered content in noisy data.The conclusion says it surpasses continued pretraining, supervised fine-tuning, and random or entropy-based reinforcement pretraining.Source: §6 · p. 12
  • Even Wikipedia already used in base-model pretraining can improve end-task performance with reinforcement and active learning.The authors identify future potential in more efficient learning patterns for discovering latent information in pretraining corpora.Source: §6 · p. 12

Related papers

Loading 2512.03442v1…