Source-linked AI summary
Self-Improving Pretraining: using post-trained models to pretrain better models
Ellen Xiaoqing Tan, Jack Lanchantin, Shehzaad Dhuliawala, Danwei Li, Thao Nguyen, Jing Xu, Ping Yu, Ilia Kulikov, Sainbayar Sukhbaatar, Jason Weston, Xian Li, Olga Golovneva
TL;DR
Conventional training adds safety, factuality, quality, and reasoning only after pretraining, despite pretraining shaping model capabilities. This paper uses strong post-trained models to rewrite data and judge rollouts earlier, yielding gains in quality, safety, factuality, and reasoning.
Problem
Conventional training adds safety, factuality, quality, and reasoning after pretraining, although early learned patterns strongly shape models’ capabilities.
Method
The method uses a strong post-trained model to rewrite pretraining data and judge policy-model rollouts, applying reinforcement earlier in pretraining and mid-training.
Results
The approach improves pretrained-model factuality, safety, and overall generation quality, while thinking mid-training with RL yields a 3.2× average improvement across mathematical reasoning benchmarks.
Takeaways & Limitations
Reasoning capabilities benefit from being trained as native behavior earlier in the training pipeline.
Takeaways & Limitations
Self-improving pretraining is slower than standard next-token prediction, and experiments with rollouts were limited to at most 16 because of compute costs.
Abstract
from arXiv · showhide
Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model's capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite pretraining data and to judge policy model rollouts, thus using reinforcement earlier in training. In our experiments, we show this can give strong gains in quality, safety, factuality and reasoning.
Introduction
The classical pipeline adds safety, factuality, quality, and reasoning only after raw-text pretraining, despite early-learned patterns largely determining model capabilities. Self-improving pretraining addresses this by using stronger post-trained models to rewrite pretraining data and judge policy behavior during earlier training.
- Motivation: The classical paradigm pretrains on raw text and then extensively fine-tunes on curated data, layering desirable properties onto the model afterward.These properties include safety, factuality, quality, and reasoning.
- Motivation: Pretraining lacks early guidance for desirable properties even though acquired patterns largely dictate what the model can ultimately do.The introduction identifies this separation as the core weakness of the standard design.
- Self-Improving Pretraining: Self-improving pretraining uses stronger post-trained models to improve earlier stages of the training pipeline.The approach transfers information from an existing strong model to earlier-stage training of the current policy.
- Self-Improving Pretraining: The strong model informs earlier training by rewriting pretraining data to encourage desirable behaviors and judging policy behavior to reward them.These two mechanisms introduce behavior-oriented guidance before conventional post-training.
1 Self-Improving Pretraining for Safety, Factuality and Quality
Self-Improving Pretraining incorporates quality, safety, and factuality into pretraining by using a strong post-trained model to rewrite data and judge candidate generations with reinforcement learning. It improves generation quality, factuality, and safety over standard pretraining in continual and from-scratch settings.
- A strong post-trained model rewrites pretraining suffixes and judges rollouts, original suffixes, and rewrites for quality, safety, and factuality.Training begins with original and rewritten suffixes, then increasingly rewards high-quality policy rollouts as the model improves.
- The method trains sequence generation on streamed prefixes and suffixes rather than only predicting the next token.At each step, the recent N tokens form the suffix conditioned on the earlier prefix, aligning training more closely with deployment-time sequence generation.
- 86.3% win rate over baseline generations and 87.9% coherence win rate were achieved when optimizing for quality.These results come from the continued pretraining setting on standard prefixes.
- Factuality increased from 42.3 to 57.6, while quality reached an 84.0% win rate when optimizing for factuality.The paper reports wins in every individual factuality benchmark tested.
- Safety evaluations increased from 76.9 to 91.1, with a 77.7% quality win rate for unsafe prefixes when optimizing for safety.The paper reports wins in most individual safety benchmarks tested.
Pretraining from-scratch results
From-scratch Self-Improving Pretraining improves safety substantially when RF-NLL uses a post-trained suffix judge, while online DPO yields especially large gains. Rollouts become increasingly preferred as training improves, but unguided single-rollout SFT can cause model collapse.
- Safety and quality results: 32.4: RF-NLL using rollout-versus-rewrite achieves a generation-quality win rate of 32.4.NLL pretraining on rewritten suffixes improves safety but not overall quality, whereas the suffix judge promotes gains in both.
- Pretraining baselines: 76.9 to 75.5: standard next-token pretraining on RedPajama slightly lowers safety performance, while standard evaluations remain similar or slightly improve from 47.6 to 47.9.Continued pretraining on cleaner SlimPajama leaves safety evaluations essentially unchanged at 76.9 vs. 77.0, although standard evaluations drop.
- Objective ablations: Single-rollout SFT without a judge collapses, dropping performance to 2.0 on safe prefixes and 0.2 on unsafe prefixes.The resulting generations were meaningless but safe sequences of words or symbols.
- Safety and quality results: 85.0 vs. 76.9: RF-NLL with a post-trained suffix judge improves safety evaluations over the baseline.This improvement is only on par with SFT on rewrites, and neither method gives significant generation-quality gains.
- Training dynamics: Early in training, original and rewritten suffixes provide supervision more often; later, the judge increasingly selects rollouts as model quality improves.Higher RL rewards for high-quality rollouts produce a higher rollout chosen rate.
2 Thinking Mid-training: Reinforcement Learning of Interleaved Reasoning
Thinking mid-training bridges the reasoning gap between pretraining and post-training by augmenting pretraining text with interleaved thoughts, then teaching and optimizing their generation through SFT and RL. This pipeline improves both mid-training performance and the effectiveness of downstream reasoning post-training.
- Method: Thinking mid-training augments pretraining corpora with teacher-generated interleaved thoughts, then uses SFT and RL to teach and optimize reasoning during an intermediate phase.The three-step approach demonstrates thought insertion, teaches when and what to think, and improves thought generation with an LLM judge.
- Post-training results: 0.3785 average score follows the full pipeline on Llama-3.1-8B, compared with 0.1197 for direct RL post-training on the base model.The average covers GSM8K, MATH-500, AMC23, Olympiad, and GPQA-Diamond.
- Mid-training results: RL mid-training raises Llama3-8b-Base average performance to 0.1896 (9×), exceeding 0.1249 after SFT on context-augmented data.Training on raw data reaches 0.0264, while RL mid-training produces the largest reported improvement despite using much less data.
- Training dynamics: RL-mid-trained checkpoints start and remain at higher rewards than SFT models during 1,000 post-training steps, with more RL mid-training steps yielding higher rewards.RL-Midtraining rewards increase alongside generated thinking length.
- Data efficiency: 0.3785 average accuracy from RL mid-training uses 8.7B tokens, outperforming 0.3480 from SFT with 10.5B tokens.Increasing SFT tokens from 7.8B to 10.5B improves accuracy from 0.3346 to 0.3480, whereas RL mid-training achieves more with fewer tokens.
- Overall findings: On Llama-3-8B, thinking mid-training plus RL post-training yields a 3.2× improvement over the existing base-model RL post-training approach.SFT mid-training with thought-augmented data contributes a 6× improvement over the base model, and RL mid-training adds further gains.
A Additional Judge experiments · A.1 Suffix judge comparisons for quality
The appendix evaluates medium-sized post-trained models as quality judges for synthetic binary suffix comparisons. All tested models initially underperform because they favor completeness over contextual coherence, while further training addresses this failure mode.
- A.1 Suffix judge comparisons for quality: The quality-judge experiments compare medium-sized post-trained models on a synthetic binary-choice task derived from the SP validation subset.The task evaluates suffix quality using synthetic data and is summarized in Table 11.
- A.1 Suffix judge comparisons for quality: The evaluated judges are Llama3.1-8B-Instruct, Llama3.3-70B-Instruct, DeepSeek-R1-Distill-Llama-8B, and DeepSeek-R1-Distill-Llama-70B.These models were assessed on synthetic quality data from the SP validation subset.
- A.1 Suffix judge comparisons for quality: All tested models underperform on the quality task and cannot serve as judges without further fine-tuning.The comparison covers multiple model families and sizes on SlimPajama subsets.
- A.1 Suffix judge comparisons for quality: The primary failure is favoring suffixes that feel more complete instead of suffixes that are more coherent with the surrounding context.This preference makes the models unreliable for contextual suffix-quality judgments.
- A.1 Suffix judge comparisons for quality: Further training helps correct the models’ preference for completion-like suffixes over contextually coherent suffixes.The passage identifies training as the mechanism that fixes the main judging problem.
- A.1 Suffix judge comparisons for quality: Negative training data are generated by modifying continuations so they resemble the originals while introducing factual inaccuracies or unsafe content.The corruption prompt requires outputting only the modified continuation.
A.2 Suffix judge comparisons for factuality · A.2.1 Factuality Prompts: with reference
The study compares GPT-4o, GPT-OSS-120B, and Llama3.1-70B-instruct as factuality judges using with-reference and no-reference prompts on 200 SlimPajama instances. Reference suffixes improve judgment, GPT-4o performs best in manual annotation, and GPT-OSS-120B is selected for subsequent experiments.
- A.2 Suffix judge comparisons for factuality: The evaluation prompts GPT-4o, GPT-OSS-120B, and Llama3.1-70B-instruct to judge factuality on a 200-instance SlimPajama test set.The study tests five with-reference prompt versions and four no-reference versions.
- A.2 Suffix judge comparisons for factuality: With-reference prompts provide the original, typically human-written suffix to judge whether a model completion is factual given the prefix.The original suffix serves as a reference for evaluating model generations.
- A.2 Suffix judge comparisons for factuality: Manual annotation finds GPT-4o tends to provide the best factuality evaluation results among the tested post-trained models.Agreement ratios with GPT-4o are calculated for GPT-OSS-120B and Llama3.1-70B-instruct.
- A.2 Suffix judge comparisons for factuality: GPT-OSS-120B performs better as a factuality judge than Llama3.1-70B-instruct when combining manual inspection with overall metrics.GPT-OSS-120B is therefore used in subsequent experiments with the v4 with-reference prompt.
- A.2.1 Factuality Prompts: with reference: The base with-reference prompt asks the evaluator to detect hallucinations, internal inconsistencies, and implausible statements using the original text, human continuation, and model continuation.It requires reasoning about logical entailment and a final JSON label.
V1 Differences from Base: … V4 Differences from Base:
The factuality prompt variants progressively shift evaluation from reference- and coherence-based judgments toward factual correctness, tolerating plausible creative content while retaining penalties for clear falsehoods and contradictions. The without-reference variants similarly relax penalties for narrative and logical irregularities across V1–V4.
- V1 Differences from Base:: V1 uses the human continuation as the primary ground truth and evaluates whether the model continuation logically follows from both references.It emphasizes step-by-step reasoning and flags hallucinations, inconsistencies, or implausible statements relative to those references.
- V2 Differences from Base:: V2 prioritizes factual correctness over coherence with the original text, permits general world knowledge as ground truth, and defines three hallucination labels.Minor logical or coherence issues do not count, while false, self-contradictory, or implausible statements do.
- V3 Differences from Base:: V3 allows plausible invented or unverifiable terms, deemphasizes loosely connected content, and reserves hallucination judgments for clear factual errors or implausible claims.Plausible creative liberties are labeled “Possible Hallucination” unless they contradict the human continuation or known facts.
- V4 Differences from Base:: V4 removes ambiguity around coherence, style, and narrative oddities while tolerating creative, loosely grounded content and minor semantic or logical quirks.Off-topic or unusual story events are not hallucinations when plausible or creatively reasonable.
- V5 Differences from Base:: V5 is the most lenient variant: only clear falsehoods, contradictions, or impossibilities count as hallucinations, while uncertainty, imagination, and coherence gaps are generally tolerated.Plausible inventions or mild factual stretching receive at most “Possible Hallucination”.
- A.2.2 Factuality Prompts: without reference: The without-reference prompt section introduces variants that evaluate model continuations against the original text rather than a human continuation or world knowledge.The base prompt asks for step-by-step reasoning about logical entailment and a strictly JSON-formatted final label.
- V1 Differences from Base:: In the without-reference variants, V1 checks logical continuation from the original text, V2 focuses on factual correctness using world knowledge, V3 permits plausible inventions, and V4 tolerates creative narrative irregularities.Across these variants, hallucination judgments increasingly target clear factual errors rather than coherence, style, or unusual story content.
- V3 Differences from Base:: The later without-reference variants explicitly avoid penalizing minor coherence or logical issues and treat plausible creative liberties, off-topic content, and unusual events as non-hallucinatory.V3 assigns plausible invented terms or creative liberties “Possible Hallucination” only when they contradict facts.
B Synthetic data generation · B.1 Unsafe test set
The paper constructs unsafe synthetic data by applying two-stage filtering to RedPajama, then splitting the filtered samples for evaluation. Random prefix and suffix extraction can produce safe prefixes, complicating purely unsafe testing.
- B.1 Unsafe test set: Unsafe content is extracted from RedPajama using existing tags plus a quality-filtering rule for curse words or blocklisted content.The filtering procedure modifies the recommended quality-filtering rules.
- B.1 Unsafe test set: Filtered data is split into train, validation, and test sets, but random prefix and suffix extraction can yield a safe prefix.This creates a risk that evaluation samples are not purely unsafe.
C Evaluation results · C.1 Evaluation prompts
This section documents the evaluation prompts and compares five prompt variants, highlighting tradeoffs between strictness, creativity, and hallucination detection. The judge prompts for coherence, FActScore, and HaluEval are provided in Figures 16–18.
- C.1 Evaluation prompts: Judge prompts for coherence, FActScore, and HaluEval appear in Figures 16, 17, and 18, respectively.These prompts are identified as the evaluation materials for the three judges.
- C.1 Evaluation prompts: Versions v1 and v2 are overly strict, flagging too much content for rewriting.Their strictness creates a risk that valid content is unnecessarily rewritten.
- C.1 Evaluation prompts: Versions v1 and v2 risk overly generic rewrites by removing valid creative elements.The comparison attributes this risk to excessive content flagging and removal of creative material.
- C.1 Evaluation prompts: Versions v3 and v4 allow more creativity than the stricter variants.The table characterizes both versions as more permissive toward creative content.
- C.1 Evaluation prompts: Version v4 explicitly excludes style and narrative from the evaluation scope.This defines a narrower evaluation scope than versions that assess those dimensions.
- C.1 Evaluation prompts: Version v5 is potentially too lenient because it biases toward “No Hallucination”.The table identifies this bias as a limitation of the variant.
- C.1 Evaluation prompts: Version v5 may miss subtle factual errors.Its leniency is associated with reduced sensitivity to less obvious factual problems.
- C.1 Evaluation prompts: Version v5 fails to catch issues that could make rewritten text inaThe supplied passage is truncated after “ina”, so the remaining issue description cannot be recovered.
C.2 Finegrained evaluation results
This section details fine-grained evaluations of factuality, rollout scaling, judge choices, pivot-based comparisons, and continued pretraining. It also documents prompts and data-generation procedures for safety, coherence, factuality, and thinking augmentation.
- Factuality evaluation: Factuality predictions are evaluated on 200 SlimPajama test examples using GPT-4o, GPT-OSS-120B, and Llama3-70B.A separate table compares factuality-prompt agreement for Llama3.3-70B and GPT-OSS against GPT-4o.
- Prompts and data generation: The section includes procedures and prompts for unsafe-test-set construction, coherence and factuality judging, and thinking augmentation during mid-training data generation.Thinking augmentation adds missing contexts and actions so answers imitate intelligent reasoning and action-taking, with tags later replaced by a single <think>…</think> pair.
- Rollout ablations: Increasing the number of rollouts yields increasingly better factuality performance, alongside quality and safety evaluations.The rollout ablations report both overall metrics and standard task metrics across different rollout counts.
- Judge ablations: Online DPO quality is compared when GPT-OSS-120B serves as the suffix judge versus a finetuned Llama3 judge, using 8 rollouts.The comparison evaluates standard quality benchmarks.
- Pivot ablations: Pivot-based pairwise comparison is ablated for coherence and factuality, with overall, factuality-benchmark, and standard-benchmark results reported using 8 rollouts.The evaluations test whether leveraging the reference as a pivot speeds pairwise comparison.
- Continued pretraining: Continued pretraining is evaluated across quality, factuality, and safety against standard next-token prediction using the Llama Base 1.4B and Pretrain Baseline.The comparison is summarized in Table 24.