PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Reasoning from Pretraining to Post-Training

Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky, Pavel Izmailov

arXiv:2607.16097v2cs.LGcs.AIcs.CL

TL;DR

It remains unclear how pretraining shapes the returns to RL and what RL changes in an inherited policy. This paper builds a controlled chess testbed spanning pretraining, supervised fine-tuning, and verifiable-reward RL, finding that pretraining loss predicts post-RL performance while RL reshapes policies differently across puzzle difficulty.

  • Problem

    It remains difficult to isolate how pretraining and RL interact because systematic compute sweeps are expensive, corpora are heterogeneous, and final-answer evaluation obscures step-level behavior.

  • Method

    The paper instantiates the standard language-model pipeline with human-game pretraining, synthetic reasoning-trace fine-tuning, and verifiable-reward RL in a controlled chess testbed.

  • Results

    Pretraining loss predicts post-RL pass@1, RL slope grows approximately linearly with log pretraining tokens, and RL amplifies preferred correct moves on easy puzzles while surfacing nearly absent correct moves on hard puzzles.

  • Takeaways & Limitations

    The results provide a quantitative account of the pretraining–RL interface and show that RL changes policy behavior heterogeneously with puzzle difficulty.

  • Takeaways & Limitations

    The findings are based on a chess testbed, with transfer evidence limited to a fixed 1B language model pretrained on math-domain text.

Abstract

from arXiv · show

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.

1 Introduction

This section introduces a controlled chess testbed that mirrors the LLM pipeline to study how pretraining shapes RL and what RL changes in reasoning policies. It reports a joint pretraining–RL scaling law, heterogeneous policy changes, and transfer of the pattern to math-domain pretraining.

  • Controlled testbed: 36 pretraining–RL combinations across 5M-to-1B-parameter models enable systematic measurement of how pretraining choices shape later RL scaling.The pipeline uses human-game pretraining, synthetic-trace SFT, and RL with verifiable rewards.
  • Joint pretraining–RL scaling law: Pretraining loss predicts post-RL pass@1, while the local RL slope grows approximately linearly with log pretraining token count.The scaling law supports evaluating compute-allocation recipes across model parameters, pretraining tokens, and RL compute.
  • Mechanism of RL policy change: RL amplifies SFT-preferred correct moves on easy puzzles but surfaces nearly absent correct moves on hard puzzles, while also reinforcing some incorrect moves.These heterogeneous policy changes accompany improved pass@1 without consistent improvement in pass@k.
  • Evidence beyond chess: 10B–200B math-domain pretraining tokens show transfer beyond chess: longer-pretrained 1B checkpoints perform better at fixed RL compute and improve faster under RL.The transferred pattern is higher fixed-budget performance together with a steeper local RL scaling slope.

2 Framework: Chess as a Testbed for Reasoning

The framework mirrors the standard language-model pipeline in a controlled chess testbed: models are pretrained on human games, fine-tuned with synthetic reasoning traces, and optimized with verifiable-reward RL on chess puzzles. Chess games are tokenized as move sequences, while puzzle solving requires selecting every correct move in a ground-truth solution line.

  • Pipeline: The testbed comprises pretraining on human games, synthetic reasoning-trace generation for supervised fine-tuning, and RL in a verifiable chess-puzzle environment.This structure mirrors the standard language-model training pipeline.
  • Chess representation: Chess games are serialized as alternating player moves, with each move encoded by four tokens for the piece, source, destination, and special-case flag.Any valid token prefix determines a unique board state.
  • Pretraining: Pretraining learns plausible move-sequence distributions from Lichess games, with corpus composition controllable by player Elo and game length through subsampling.The model uses standard autoregressive next-token prediction over tokenized games.
  • Reasoning-trace supervision: Synthetic reasoning traces serialize sampled game continuations into a tree-traversal-like structure, after which supervised fine-tuning trains the model to commit to the best continuation.Training masks opponent-move tokens and applies loss only to the reasoning trace and model moves.
  • Reinforcement learning: RL starts from the SFT policy and uses GRPO with binary reward 1 only when every executed move matches the ground-truth solution line, and 0 otherwise.A single incorrect move anywhere in the line yields no reward.
  • Data and evaluation: The chess setup uses a 54B-token Lichess pretraining corpus, 156K quality-filtered puzzles across five difficulty bins, and a 1,480-puzzle evaluation benchmark.The pretraining games are Blitz and Rapid games played on Lichess in 2022.

3 Scaling Analysis: From Pretraining to Post-Training

The analysis finds that pretraining quality and scale govern post-RL performance: lower validation loss predicts higher pass@1, while more pretraining tokens predict faster RL improvement. RL strongly improves pass@1 but offers limited or negative pass@16 gains for larger models, making pretraining comparatively more important in low-compute regimes.

  • Pretraining scaling: 11 pretraining budgets spanning 6.5 × 10^16 to 6.5 × 10^19 FLOPs cover approximately 200M to 52B tokens across 10 model sizes.The study reports IsoFLOP curves for validation loss, pass@1, and pass@16.
  • Compute allocation: RL substantially improves pass@1 across runs, with pass@1 continuing to increase as RL training continues.For fixed model size and total compute, the frontier initially favors more pretraining and less RL, indicating initialization-limited performance.
  • Compute allocation: For larger models, pass@16 remains nearly flat or slightly degrades with additional RL, so extra pretraining can outperform allocating the same compute to RL.The 20M model is an exception, showing a sharp initial pass@16 improvement followed by slower gains.
  • Predicting RL scaling: |ρ| = 0.93 to |ρ| = 0.99: lower pretraining validation loss increasingly predicts higher post-RL pass@1 as reference RL compute rises.The correlation increases as log10 Cref increases from 16 to 20, motivating Cref = 10^20 FLOPs for subsequent analysis.
  • Predicting RL scaling: Pearson r = +0.84: the RL reward-growth slope correlates positively with log10 pretraining tokens, and a joint model using log10 T and log10 N predicts it best.The joint model has the lowest RMSE and highest R2 among the tested parameterizations.
  • Compute-optimal allocation: 20% to 28%: the compute-optimal RL fraction rises from 50M to 680M models, while optimal pretraining tokens show no systematic deviation from Chinchilla allocation.The simulated and empirical frontiers closely agree.

4 Mechanism Analysis: Policy Evolution During RL Post-training

RL reshapes policies through state-dependent redistribution rather than simple sharpening: it amplifies preferred correct moves on easy puzzles, discovers tail correct moves and reinforces wrong modes on harder ones. During RL, reasoning traces improve move quality mainly through broader rather than deeper search.

  • Move-policy evolution: RL is not well explained by uniform temperature scaling of the SFT policy, motivating state-level analysis of probability-mass redistribution.The study tests a power transformation of SFT probabilities and fits global and per-state slopes.
  • Move-policy evolution: On easy puzzles, RL mostly amplifies correct moves already preferred by SFT; on harder puzzles, it surfaces near-absent correct moves while also reinforcing incorrect modes.These changes are categorized as ground-truth amplification, tail discovery, and wrong-mode amplification.
  • Move-policy evolution: RL produces limited pass@k gains because it strengthens correct modes, discovers some correct tail moves, and amplifies wrong modes on harder tasks.The passage identifies mitigating wrong-mode amplification as important for improving RL beyond pass@1.
  • Reasoning-trace evolution: Reasoning trace quality improves during RL, with better proposed moves and opponent responses and more frequent surfacing of and commitment to the ground-truth move.The analysis uses prefix trees to examine reasoning structure, move quality, and search behavior.
  • Reasoning-trace evolution: RL primarily expands search breadth rather than depth: width-to-depth ratio and branching factor increase while maximum search depth remains roughly flat.This pattern is reported for representative 20M and 50M models pretrained under matched compute.

5 Transfer to Text: A Qualitative Case Study in Math

A 1B-parameter OLMo-2 model trained on a 200B-token math-domain corpus provides early evidence that the pretraining-to-post-training scaling structure found in chess extends to modern language-model training. Across checkpoints from 10B to 200B pretraining tokens, lower pretraining loss predicts higher post-RL performance, with RL reward-curve slopes improving nearly linearly with log10 T and the fit tightening as RL compute increases.

  • Experimental setup: A 1B-parameter OLMo-2 model was pretrained on 200B tokens, comprising 70% Nemotron-CC-Math-v14 and 30% Dolma3.The experiment used a mixed math-domain corpus and a single main run with a linear learning-rate schedule and constant warmup.
  • Scaling transfer: The post-RL performance level at high RL compute is well-predicted from pretraining loss, despite differences in task format, data distribution, and training recipe.The result is presented as early evidence that the pretraining-to-post-training scaling structure extends to modern language-model training.
  • Scaling transfer: Across 1B OLMo-2 checkpoints from 10B to 200B pretraining tokens, lower pretraining loss consistently predicts higher post-RL performance Rref_T.This reproduces the predictive pattern identified in the chess setting.
  • Scaling transfer: The slope BT increases nearly linearly with log10 T, and the fit between pretraining loss and Rref_T tightens as RL compute increases.This relationship mirrors the corresponding chess results.

6 Related Work

Related work studies what verifiable-reward RL changes in pretrained reasoning policies, scaling-law predictions beyond pretraining, and compute allocation across pretraining and RL. These lines of work motivate analyzing the pretraining-to-RL interface quantitatively.

  • Reinforcement Learning for Reasoning: Prior work debates whether reasoning RL mainly amplifies existing patterns or composes pretrained skills into new ones.Some studies report improved pass@1 alongside possible reductions in large-k coverage.
  • Scaling Laws from Pretraining to Post-training: Scaling-law research predicts language-model performance from model size, data, and compute, extending beyond standard pretraining loss.Kaplan et al. identified power-law behavior, while Hoffmann et al. refined compute-optimal allocation between model size and training tokens.
  • Allocating Compute between Pretraining and RL: Research on multi-stage training examines how pretraining and RL compute should be allocated, finding that post-training gains can saturate as both increase.Related studies compare pretraining, continued pretraining, supervised fine-tuning, and RL, and also investigate early RL during pretraining.

7 Conclusions

The study establishes a joint pretraining-to-RL scaling law in chess: pretraining loss predicts post-RL performance, while pretraining data scale predicts RL improvement slope. It also identifies compute-allocation and initialization tradeoffs, and motivates methods that broaden correct-solution support beyond pass@1.

  • Scaling law: Pretraining loss predicts post-RL performance level, while pretraining data scale is closely associated with the slope of RL improvement.These findings form the study’s joint scaling law for how pretraining influences RL dynamics.
  • Compute allocation: As total compute grows, the optimal pretraining fraction tends to decrease, indicating that RL should receive an increasingly large share of compute.The resulting compute-allocation frontier implies a tradeoff between pretraining and RL.
  • Initialization: Starting RL too early from weakly pretrained checkpoints gives limited gains, suggesting that RL is initialization-dependent and requires sufficient pretraining exposure.The conclusion is specific to the reported setting.
  • Future directions: Improving RL beyond pass@1 likely requires reducing wrong-mode amplification and expanding correct-solution support, rather than only sharpening the current policy.The paper presents this as a direction for future work.

A Discussions and Limitations · B Extended Related Work · C Implementation Details

The paper uses chess as a controlled analogue of the standard LLM pipeline to study pretraining–RL dynamics, while emphasizing limits on transfer to natural language and situating the testbed within broader scaling and reasoning research.

  • A Discussions and Limitations: Chess mirrors the standard pipeline of pretraining on human data, SFT on reasoning traces, and RL with verifiable rewards.The testbed targets pipeline-level dynamics rather than chess-specific phenomena.
  • A Discussions and Limitations: Chess limits direct transfer because its vocabulary has 81 tokens, verification is exact, and reasoning is not entangled with world knowledge.These structural differences distinguish chess from natural language.
  • C Implementation Details: The supplied passages contain no substantive implementation details for the merged implementation-details section.No implementation-specific claim can be grounded in the provided passage set.
  • B Extended Related Work: Pretraining scaling-law research models language-model loss as power laws in model size, data, and compute.Kaplan et al. established these relationships, while later work refined compute-optimal allocation.
  • B Extended Related Work: Compute allocation research asks how much pretraining is needed for effective RL and how limited compute should be divided between pretraining and post-training.This frames the pretraining-to-RL interface as a multi-stage resource-allocation problem.
  • B Extended Related Work: Coverage theory connects pretraining and post-training by arguing that probability mass on high-quality responses determines whether post-training and test-time scaling can succeed.This provides a theoretical lens for allocating compute across training stages.
  • B Extended Related Work: The chess testbed complements controlled synthetic reasoning settings by more closely mirroring language-model training on human behavior data followed by RL with verifiable rewards.Chess also requires multi-step planning and supports exact verification at every move.

C.1 Datasets … D FLOP Estimation

The paper uses a controlled chess pipeline with curated, decontaminated datasets, deep decoder-only models, synthetic-trace SFT, and GRPO-based RL. It defines puzzle reasoning formats, constructs verifier-selected traces, and estimates pretraining, SFT, and RL compute with dense-model FLOP approximations.

  • C.1 Datasets: 54B tokens of Lichess Blitz and Rapid games form the pretraining corpus, while post-training retains 156K quality-filtered puzzles across five Elo-based difficulty bins.Games span balanced player Elo ratings from 800 to 3000; puzzles use Elo and popularity filters and are balanced by solution length and theme coverage.
  • C.2 Models: The model family uses untied embeddings and relatively deep Qwen-style decoder-only Transformers with grouped-query attention.The deeper designs are motivated by prior work on small language models.
  • C.3 Algorithms: SFT masks tokens outside synthetic traces and model moves, while RL optimizes the policy with GRPO using grouped sampled trajectories and normalized rewards.The GRPO formulation samples ζ1, . . . , ζG from the old policy, computes rewards, and normalizes them within each group to obtain advantages.
  • C.4 Training Configurations: Pretraining runs for one epoch per assigned token budget; SFT uses 3 epochs, learning rate 3 × 10^-4, and 524,288 effective tokens per optimizer step, while RL uses G = 8.RL also uses learning rate 10^-5, KL coefficient β = 0.001, clip ratio ϵ = 0.2, temperature 1.0, and 3072-token maximum responses.
  • C.5 Puzzle Game Example: Each chess puzzle supplies a starting position represented as a move sequence, and solving requires identifying every correct move in the continuation.The reasoning format optionally uses a <T> token, and the best solution line is unique for all puzzle games.
  • C.6 Additional Details of Synthetic Reasoning Trace Generation: Synthetic traces merge proposal-policy rollouts into a prefix tree, serialize retained root-to-leaf paths in DFS order, and append a verifier-selected continuation for supervised training.Opponent-move tokens in the selected continuation are masked because the environment produces them during inference.
  • D FLOP Estimation: Training compute follows the dense-model approximation 6NT, with pretraining cost Cpt = 6NTpt and fixed-length SFT cost Csft = 6NEsftnsftLsft.N is the active parameter count and T is the number of processed tokens; the approximation targets dense decoder-only Transformers.
  • D FLOP Estimation: RL FLOPs sum rollout generation, reference-model evaluation, and policy optimization; without a reference model, Crl is reduced to 8NTrollout.Rollout generation and reference evaluation each use one forward pass over rollout tokens, while optimization uses the standard training FLOP approximation.

E Pretraining Law Fitting … G.6 Asymptote Ceiling Fitting

Across controlled pretraining, SFT, and RL scaling analyses, pretraining tokens and validation loss predict downstream performance and RL returns, while local slope estimates depend on saturation and reference compute. The joint law validates reasonably well, but asymptote estimates are available only for smaller models whose trajectories reach saturation.

  • E Pretraining Law Fitting: 11 compute budgets spanning 6.5 × 10^16 to 6.5 × 10^19 FLOPs were evaluated across 10 model sizes and approximately 200M–52B training tokens.IsoFLOP curves show a validation-loss valley at each budget; chess-optimal allocations use more tokens per parameter than D/N = 20, with the gap narrowing as compute grows.
  • F SFT Performance Comparisons: Stronger pretrained checkpoints consistently improve fixed-size SFT performance, including settings with and without synthetic reasoning traces.Performance is evaluated across four model sizes and pretraining compute, with additional pass@4 and pass@8 analyses.
  • G Joint Pretraining–RL Scaling Law: The joint law predicts post-RL reward from model size, pretraining tokens, validation loss, and RL compute, with pretraining loss serving as a direct or configuration-derived input.The fitted reference reward depends monotonically on pretraining loss, while the RL slope uses a joint model over tokens and size.
  • G.1 Interpretation of the Local RL Scaling Fit: Local RL slopes are smallest near the initial or saturated regimes and largest at intermediate performance, making early or near-plateau estimates less predictable.The local slope is a trajectory-dependent approximation rather than a global RL scaling parameter.
  • G.2 Parameterizations: The RL slope is driven primarily by pretraining tokens, while model size provides a weaker correction; stronger models on easy benchmarks can show low slopes because they are already near saturation.On B3–B4, log-scale tokens correlate with slope at Spearman 0.84, and the joint fit has a 2× larger coefficient on tokens than on model size.
  • G.3 Leave-one-out Fitting Validation: RMSE 0.0194 in reward was achieved for strict Chinchilla-L leave-one-out validation, while replacing observed loss with predicted loss increased RMSE by 0.0099.The increase represents about 49% of strict LOO RMSE, with residual error concentrated more in predicting RL slope than reference reward.
  • G.3.1 Leave-one-model-size-out validation: Predictions beyond 680M parameters should be treated as extrapolations at least as uncertain as the 680M-held-out estimate, although additional 32M and 410M tests showed low prediction errors.The LMSO procedure refits the loss surface and joint law after removing each held-out model size.

G.7 Extrapolating the Compute-Optimal Frontier

The fitted scaling law enables compute-optimal frontier estimation by allocating total compute across model size, pretraining, and RL. The resulting frontier predicts successive takeovers by larger models and a shift toward proportionally more RL at higher budgets, within the supported empirical regime.

  • Per-size frontier: A training recipe is specified by model size N, pretraining tokens T, and RL compute C_RL, allowing hypothetical recipes to be scored without running them.For fixed N and total budget C, the method maximizes predicted reward over the split between pretraining and RL.
  • Per-size frontier: The per-size optimum is found by sweeping T, setting C_RL = C − 6NT − C_SFT, and refining a coarse grid search around its best region.The coarse search evaluates approximately 400 feasible allocations before local refinement.
  • Validity: The reported frontier is optimal only over the evaluated candidate set and within the empirical support of the fitted scaling laws, not under unrestricted extrapolation.The refinement reduces discretization error but does not guarantee continuous-domain global optimality.
  • Global frontier: The global frontier forms successive takeovers in which progressively larger models become compute-optimal, while the optimal RL share records how total compute should be divided.The continuous-N optimum is computed with the Nelder–Mead simplex method after coarse-grid initialization and warm starts.
  • Global frontier: At lower compute budgets, additional pretraining is more valuable, whereas higher budgets warrant a proportionally larger RL allocation.This allocation trend is summarized by the frontier’s RL share ρ_RL = C_RL/C.

G.8 Limitations … H.3 Policy Categorization

The paper qualifies its fitted scaling law and then defines move-space diagnostics to characterize how RL changes SFT policies. These analyses find that RL is not fully captured by simple sharpening, with stage-dependent sharpening, heterogeneous updates, and broader search behavior.

  • G.8 Limitations: The fitted law is a local approximation and should not be interpreted as evidence that RL improvement is unbounded.It assumes RL reward is approximately linear in log10 CRL over the measured compute range.
  • G.8 Limitations: Strict extrapolation compounds prediction error from the Chinchilla loss surface with residual error in the f and g maps.Observed-L leave-one-out evaluation reduces the latter error, but practical extrapolation must predict L from (N, T).
  • G.8 Limitations: The RL slope map g(N, T) is materially noisier than the loss-based map f(L), reflecting unexplained run-level variation.Possible sources include optimizer details, SFT/RL data differences, reward-model variation, and other omitted factors.
  • G.8 Limitations: The frontier analysis is best viewed as a diagnostic for allocation trends near the observed training distribution, not as evidence that fitted exponents extrapolate arbitrarily far.Data are densest near the observed model sizes, token counts, and RL-compute range; the reward surface is also flat near its peak, while the ε-optimal band shifts upward with compute.
  • H Move Policy Evolution: Move-space policy metrics are well-defined from token-space training because valid token prefixes correspond to legal board states and moves have token serializations.For post-training policies, trace-conditioned move policies are marginalized over reasoning traces using K = 128 Monte Carlo rollouts.
  • H.2 Fitting Power-Sharpening Transformations: Power-sharpening diagnostics test whether RL transforms the SFT marginal policy by increasing relative probability on high-probability moves.The fitted α⋆ and centered-logit slope β quantify sharpening, while residual divergence or low R2 indicates changes beyond uniform sharpening.
  • H.2 Fitting Power-Sharpening Transformations: On the 50M model pretrained with 4.6B tokens, both α⋆ and β tend to increase during RL despite substantial sample-level heterogeneity.Table 12 reports α estimates and Table 13 reports β estimates across RL stages and states.
  • H.3 Policy Categorization: Policy-update categories distinguish ground-truth amplification, tail discovery, top-k correction, and ground-truth regression by changes in rank, top-k membership, and probability.In a hard puzzle, RL promotes a nearly absent correct move to the top while also retaining significant mass on a competing wrong move.

H.4 CoT Evolution Analysis … I.1 Implementation Details

The paper analyzes how structured chain-of-thought search evolves during RL, finding broader but shallow search, improved candidate selection, and concentrated move policies. It also documents entropy changes, limited recovery of deep continuations, and a controlled 1B math-model setup varying only pretraining tokens.

  • H.4 CoT Evolution Analysis: Each reasoning rollout is reconstructed as a prefix tree rooted at the puzzle state, enabling analysis of search structure and reasoning quality.Nodes correspond to moves in the explicit move sequence.
  • H.4 CoT Evolution Analysis: The study characterizes traces using search shape, Stockfish-based move quality, and related structural metrics including node count, maximum depth D, branching factor, and |L|/D.The normalized move-quality score is (r −1)/(n −1), with lower values indicating better moves.
  • H.4 CoT Evolution Analysis: Search traces primarily expand breadth rather than depth, providing finer-grained behavioral information than reasoning-token counts alone.The analysis compares representative 20M and 50M RL runs pretrained under matched compute.
  • H.4 CoT Evolution Analysis: RL makes models more likely to mention the ground-truth move, commit to their best considered candidate, and revisit previously explored lines.These traces become less aligned with strict depth-first-search serialization.
  • H.5 Entropy: Policy entropy drops from 1.370 →0.705, whereas CoT token entropy decreases from 0.536 →0.492.RL therefore concentrates move selection more strongly than token-level variability, with harder B3–B5 puzzles retaining higher entropy.
  • I Olmo Experiment Additional Details: RL improves coverage of ground-truth continuations at depths k = 1, 2, 3, 4, but gains decay sharply with depth.Even at step 1000 for the 50M model, few rollouts recover the deepest continuations described in the passage.
  • I.1 Implementation Details: The math experiment pretrains a fixed-architecture 1B-parameter OLMo-2 model, varies only pretraining tokens across checkpoints, then applies convergence annealing, SFT, and GRPO RL.Tables 15 and 16 summarize the implementation setup.

I.2 Additional Results

Additional results report fitting outcomes on downstream benchmarks and specify the training hyperparameters used across pretraining, fine-tuning, and reinforcement learning.

  • Fig. 27 reports fitting results on downstream benchmarks.
  • Pretraining uses a warmup–stable–decay schedule, while RL uses GRPO with 8 rollouts per prompt at temperature 1.0.Intermediate stable-phase checkpoints define pretraining-scale anchors, and each anchor is annealed before supervised fine-tuning and RL.
Loading 2607.16097v2…