Source-linked AI summary
REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation
Haoran Que, Jiajun Shi, Ting Huang, Renming Pang, Jiaheng Liu, Ge Zhang, Wenhao Huang, Shen Yan, Wei Ye, Shikun Zhang
TL;DR
High-quality pre-training data is becoming a bottleneck, while next-token prediction leaves the reasoning connecting context to continuation implicit. REER-PT adds sparse, offline, perplexity-guided annotations to difficult but inferable continuations, yielding lower perplexity and benchmark gains in controlled 680M-parameter experiments. The evidence is limited to one model and recipe, and annotation-model choices may affect construction outcomes and code-document behavior.
Problem
High-quality training data is an increasingly important bottleneck, and conventional pre-training rarely explains why a continuation follows its context.
Method
REER-PT selects difficult, contextually inferable continuations and retains concise annotations generated and refined offline to reduce continuation perplexity while preserving source text.
Results
Perplexity reductions range from 0.42 to 7.29, while the augmented-data model gains up to 2.07 percentage points on several knowledge and reasoning benchmarks.
Takeaways & Limitations
Sparse perplexity-guided reasoning augmentation can improve continuation predictability and model performance without changing the standard next-token pre-training objective.
Takeaways & Limitations
Experiments use a 680M-parameter model and a single pre-training recipe, so behavior at larger model and data scales remains unknown; code-document insertions can disrupt program structure.
Abstract
from arXiv · showhide
As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-token prediction supervises what follows a context but leaves the intermediate reasoning behind that continuation implicit. We introduce \textbf{REER-PT}, a scalable framework that extends Reverse-Engineered Reasoning (REER) to raw pre-training data. REER-PT identifies continuations that are difficult to predict but can still be inferred from the preceding context, and inserts concise reasoning annotations that reconstruct the missing connection between context and continuation. Candidate annotations are generated and refined offline, with perplexity serving as the optimization signal. Constraints on length and target leakage filter out unhelpful or trivial annotations. This sparse transformation preserves the source text and remains compatible with standard next-token prediction, avoiding online reasoning rollouts during pre-training. We apply REER-PT to transform a source pre-training corpus into an augmented one. Across augmented-data, original-token, and selected-continuation comparisons, perplexity reductions range from 0.42 to 7.29, and only about 0.05\% of annotation 13-grams appear verbatim in the source text. We then train two 680M-parameter models with the same architecture and training configuration on the source and augmented corpora, respectively. The augmented-data model gains up to 2.07 percentage points on several knowledge and reasoning benchmarks. Together, the perplexity analysis indicates improved continuation predictability, while the controlled pre-training experiments suggest that this augmentation can improve model performance without changing the standard pre-training objective.
1 Introduction
REER-PT addresses the limits of conventional pre-training by adding concise, perplexity-guided reasoning annotations to difficult but contextually inferable continuations. The sparse augmentation preserves source text and is associated with lower perplexity and improved benchmark performance.
- High-quality training data is an increasingly important bottleneck as language-model compute scales.
- Conventional next-token prediction teaches what follows a context but rarely explains the intermediate reasoning connecting context and continuation.
- REER-PT selects difficult continuations that remain inferable from context and retains concise annotations that reduce continuation perplexity.
- Offline generation and refinement, together with length and target-leakage constraints, make corpus-scale augmentation sparse, selective, and efficient.
- Perplexity reductions range from 0.42 to 7.29 across augmented-data, original-token, and selected-continuation comparisons.
- The augmented-data model gains up to 2.07 percentage points on several knowledge and reasoning benchmarks under matched 680M-parameter training configurations.
2 Related Work
Related work improves pre-training through data selection, reasoning augmentation, and reinforcement-based training signals. These approaches operate at different granularities, from tokens and documents to segments and mid-training curricula.
- Selection-based methods prioritize corpus portions using criteria such as excess loss or estimated learnability.
- Reasoning-augmented pre-training introduces intermediate reasoning through internal rationales, latent chain-of-thought, or document-level explanations.
- Reinforcement pre-training derives rewards from next-token or next-segment prediction and from reconstructing informative masked content.
- Mid-training methods combine dynamic reasoning budgets, adaptive sampling, reinforcement learning, and next-token training.
3 Approach
REER-PT augments raw pre-training documents by locating difficult but contextually inferable continuations, then inserting concise annotations selected and refined through perplexity. The sparse transformation preserves source-token order and supports standard next-token prediction.
- Overview: REER-PT extends Reverse-Engineered Reasoning to document continuations by inserting concise book-note-style annotations between context and continuation.The method preserves the order and content of all source tokens while making implicit connections explicit.
- Annotation generation and refinement: Accepted annotations are inserted before their target continuations, and positions are discarded when no annotation improves on the no-annotation baseline.The resulting augmented corpus is fixed before pre-training and remains compatible with the standard next-token prediction objective.
- Insertion position selection: Sentence-level perplexity ranks difficult transitions, while an inferability check removes candidates unsupported by their preceding context.The procedure considers sentences in descending perplexity order and retains only meaningfully related, contextually inferable continuations.
- Insertion position selection: Each document targets K = ⌊T/1000⌋ insertion positions, using selected sentence beginnings as boundaries between local contexts c_i and continuations y_i.The context spans text between the previous selected position and the current one; the continuation begins at the selected sentence and ends before the next selected position.
- Annotation generation and refinement: The annotation model generates multiple book-note-style candidates, typically 500–1,000 words, and filters out annotations containing target leakage.Target leakage can reduce continuation perplexity by revealing the continuation rather than explaining its connection to context.
- Annotation generation and refinement: Continuation perplexity selects and refines annotations: lower PPL means the annotation makes the observed continuation easier for the PPL model to predict.Refinement is an iterative, gradient-free search over segment rewrites; the current annotation remains eligible so continuation perplexity cannot increase between steps.
4 Experiments
Experiments evaluate REER-PT through perplexity, repetition, training dynamics, and benchmark comparisons. The analyses show lower perplexity and limited copying, while pre-training gains are broad across knowledge and reasoning tasks but negative for code generation.
- Perplexity: 7.28501 lower full augmented-data PPL and 1.03078 lower original-token PPL result from optimized annotations versus no annotation.Refinement also lowers full augmented-data PPL by 0.48833 and original-token PPL by 0.42224 relative to initial annotations.
- Perplexity: 4.23539 lower selected-continuation PPL results from optimized annotations versus no annotation, with a further 1.38464 reduction versus initial annotations.The three scopes are full augmented data, unchanged original tokens, and selected continuations.
- Repetition: 0.051% mean annotation-to-source exact-overlap indicates that few annotation 13-gram occurrences exactly match source spans.Annotation self-repetition is 0.203%, compared with 0.615% for source documents.
- Pre-training setup: The raw and augmented mixtures train separate 680M-parameter models under the same architecture and training configuration.The raw mixture contains approximately 523B tokens, while the augmented mixture contains approximately 542B tokens.
- Training dynamics: The augmented-data model generally reaches lower later-training loss while following similar gradient-norm trajectories to the raw baseline.The EMA-smoothed augmented-data curve generally remains below the raw-baseline curve later in training.
- Benchmark evaluation: Positive gains span knowledge, general-reasoning, and STEM-reasoning benchmarks, while all three code-generation benchmarks decline.BBH and GPQA-Diamond each improve by 2.07 percentage points, whereas MBPP+, HumanEval+, and LiveCodeBench decline by 2.65, 1.83, and 1.79 points.
5 Conclusion
REER-PT augments raw pre-training data with sparse, concise reasoning annotations selected using continuation perplexity and constraints. The resulting augmentation improves continuation predictability and several knowledge and reasoning benchmarks in controlled 680M-parameter experiments.
- REER-PT selects difficult but context-inferable continuations and retains annotations that reduce continuation perplexity under length and target-leakage constraints.The framework preserves all source tokens and remains compatible with standard next-token prediction.
- 0.42 to 7.29: reported perplexity reductions span augmented data, original tokens, and selected continuations.Mean exact annotation-to-source 13-gram overlap is only 0.05%.
- 2.07 percentage points: the augmented-data model improves several knowledge and reasoning benchmarks over the controlled source-corpus comparison.The experiments use 680M-parameter models.
6 Limitations and Future Work
The evaluation uses one 680M-parameter model and a single pre-training recipe, leaving REER-PT behavior at larger model and data scales unknown. Data construction is also sensitive to model choices and annotation format, especially for code documents.
- A single 680M-parameter model and pre-training recipe leave behavior at larger model and data scales unknown.The limitation motivates future study of larger-scale training and annotation density.
- Different perplexity or annotation models may change perplexity estimates, insertion positions, refinement signals, inferability judgments, and annotation style.The data-construction process therefore depends on both model choices.
- Book-note-style annotations are designed primarily for natural-language documents, while natural-language insertions can disrupt code structure and coincide with code-generation regressions.The paper calls for structure-aware formats for code and other specialized domains.