PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models

Charlie Zhang, Graham Neubig, Xiang Yue

arXiv:2512.07783v1cs.CL

TL;DR

It remains unclear whether RL extends reasoning beyond pre-training because existing pipelines obscure prior knowledge and mid-training effects. Using controlled synthetic tasks, parseable traces, and manipulated training stages, the paper finds that RL gains require pre-training headroom and edge-of-competence data, while contextual transfer, mid-training, and process rewards improve supported reasoning outcomes.

  • Problem

    Opaque pre-training corpora and underexamined mid-training obscure whether RL extends reasoning beyond capabilities acquired during pre-training.

  • Method

    The paper uses controllable synthetic reasoning tasks, parseable processes, process-level evaluation, and systematic manipulation of pre-, mid-, and post-training distributions.

  • Results

    RL produces true pass@128 gains when pre-training leaves headroom and RL data target the model’s edge of competence; contextual transfer, mid-training, and process rewards also improve supported outcomes.

  • Takeaways & Limitations

    Reasoning training benefits from allocating complementary roles to pre-training, mid-training, edge-of-competence RL data, and process-aware rewards.

Abstract

from arXiv · show

Recent reinforcement learning (RL) techniques have yielded impressive reasoning improvements in language models, yet it remains unclear whether post-training truly extends a model's reasoning ability beyond what it acquires during pre-training. A central challenge is the lack of control in modern training pipelines: large-scale pre-training corpora are opaque, mid-training is often underexamined, and RL objectives interact with unknown prior knowledge in complex ways. To resolve this ambiguity, we develop a fully controlled experimental framework that isolates the causal contributions of pre-training, mid-training, and RL-based post-training. Our approach employs synthetic reasoning tasks with explicit atomic operations, parseable step-by-step reasoning traces, and systematic manipulation of training distributions. We evaluate models along two axes: extrapolative generalization to more complex compositions and contextual generalization across surface contexts. Using this framework, we reconcile competing views on RL's effectiveness. We show that: 1) RL produces true capability gains (pass@128) only when pre-training leaves sufficient headroom and when RL data target the model's edge of competence, tasks at the boundary that are difficult but not yet out of reach. 2) Contextual generalization requires minimal yet sufficient pre-training exposure, after which RL can reliably transfer. 3) Mid-training significantly enhances performance under fixed compute compared with RL only, demonstrating its central but underexplored role in training pipelines. 4) Process-level rewards reduce reward hacking and improve reasoning fidelity. Together, these results clarify the interplay between pre-training, mid-training, and RL, offering a foundation for understanding and improving reasoning LM training strategies.

1 Introduction

The paper addresses whether RL extends reasoning beyond pre-training by using a controlled framework that separates training stages and evaluates extrapolative and contextual generalization. It finds that RL gains depend on pre-training headroom and edge-of-competence data, while mid-training and process rewards strengthen outcomes.

  • Prior work disagrees on whether post-training refines existing reasoning or produces gains beyond pre-training.
  • Opaque pre-training corpora make it difficult to identify learned reasoning primitives and isolate post-training’s causal effect.
  • The study uses controllable synthetic tasks, parseable reasoning processes, and manipulated training distributions to disentangle pre-training, mid-training, and RL.
  • Evaluation measures extrapolative depth generalization and contextual breadth generalization across novel surface contexts.
  • RL produces genuine gains when pre-training leaves headroom and RL data target the model’s edge of competence rather than tasks that are too easy or too hard.
  • ≥1% contextual exposure can provide a sufficient seed for RL to reinforce strong cross-context generalization.
  • Mid-training strengthens in-domain and out-of-domain performance under fixed compute, while process rewards mitigate reward hacking and improve reasoning fidelity.

2 Preliminaries

The paper builds a synthetic reasoning testbed with controllable structure, context, training distributions, and process-level evaluation. Its setup separates training stages and measures both depth-wise and context-wise generalization.

  • Controllable Synthetic Reasoning Dataset: The framework uses dependency graphs, contextual rendering, and process-verified evaluation to isolate training-stage effects.
  • Controllable Synthetic Reasoning Dataset: Dependency graphs encode variables and dependencies, with a designated answer node defining the final answer.
  • Controllable Synthetic Reasoning Dataset: Contextual templates render equivalent graph structures into varied natural-language problems.
  • Controllable Synthetic Reasoning Dataset: Separate pre-, mid-, and post-training distributions provide contamination-free control over structure and context.
  • Generalization Axes: Extrapolative generalization tests increasing reasoning depth, whereas contextual generalization tests transfer across novel surface contexts.
  • Evaluation: Process-verified evaluation checks predicted dependency graphs, dependencies, values, reasoning steps, and final answers against ground truth.
  • Training Setup: The 100M-parameter models use disjoint synthetic-data splits for pre-training, mid-training, and post-training; pre-training emphasizes op=2-10 primitives while retaining headroom.

3 When Does Post-Training Incentivize Reasoning Beyond the Base Model?

The study isolates when RL extends reasoning beyond a base model by fixing pre-training and varying post-training difficulty and coverage. RL improves pass@128 on harder tasks when data target the model’s edge of competence, but mainly sharpens in-domain skills otherwise.

  • Experimental Setup: The experiment fixes pre-training on ID problems and varies RL across ID, mixed, edge, and hard difficulty ranges.
  • Results: For ID tasks, RL improves pass@1 but not pass@128, indicating sharper existing abilities rather than capability extension.
  • Results: RL improves pass@128 on OOD tasks when trained on edge-of-competence data at op=11-14.
  • Interpretation: RL produces true pass@128 gains only when pre-training leaves headroom and RL data are neither too easy nor too hard.
  • Interpretation: The results reconcile conflicting reports by distinguishing well-covered standard tasks from synthetic tasks with limited pre-training coverage.
  • Practical Guidance: The paper recommends selecting tasks where the model fails at pass@1 but succeeds at pass@k, then refreshing the pool as competence expands.

4 How Does Pre-training Exposure Shape Post-Training Generalization?

The experiments test how pre-training exposure to reasoning primitives affects RL-based contextual generalization. RL transfers to context B only with minimal pre-training exposure, while greater exposure supports novel structures on harder compositions.

  • The study varies pre-training exposure to context B while holding the RL recipe fixed to measure its effect on post-training generalization.The pre-training data include context A and long-tailed context B examples; RL uses a 50% context A and 50% context B mixture.
  • 0% or 0.1% context B exposure causes RL to fail to transfer, whereas 1% exposure substantially improves generalization through op=20.
  • RL reinforces contextual transfer when the base model already contains the relevant primitives, but cannot create transfer without minimal pre-training exposure.
  • As complexity rises from op=2-10 to op=11-20, models generate more novel reasoning structures, especially with sufficient context B pre-training.
  • The results motivate seeding broad long-tail atomic primitives during pre-training so RL can amplify them on difficult tasks.

5 How Does Mid-Training Interact with Post-Training?

This section examines how mid-training and RL share a fixed compute budget. Mid-training strengthens performance near the training distribution, while heavier RL improves the hardest unseen tasks.

  • The study compares mid-training and RL under an equivalent fixed compute budget by varying their allocation ratio.The allocation is T_mid = (1 − β) · T and T_RL = β · T, with configurations ranging from full mid-training to full RL.
  • Light-RL achieves the best pass@1 on OOD-edge tasks, where full mid-training and light RL outperform heavier or full RL allocations.
  • Heavy-RL improves pass@1 and pass@128 on OOD-hard tasks, while Light-RL improves best pass@1 on OOD-edge tasks.
  • Mid-training bridges pre- and post-training distributions, strengthens reasoning priors, and substantially improves generalization under fixed compute.
  • Compute should be allocated according to the target: emphasize mid-training for similar-task reliability and retain RL for harder out-of-distribution generalization.

6 Mitigating Reward Hacking via Process Supervision in Outcome Rewards

The section replaces outcome-only rewards with composite rewards that also verify intermediate reasoning steps. Process-aware supervision improves accuracy and structural faithfulness while mitigating reward hacking.

  • Outcome-only rewards can produce correct final answers through invalid reasoning chains, motivating process-level verification.
  • The composite reward combines a sparse outcome reward with a dense process-verification reward, balanced by α.R_out rewards a correct final answer, while R_pv reflects the correctness of each reasoning step.
  • Process verification improves pass@1 by 4–5% across extrapolative op=15-20 settings.
  • A moderate mix of 0.2 R_out + 0.8 R_pv best balances outcome accuracy and reasoning consistency, while strict verification further improves results.
  • Process-aware rewards shift models away from shortcut exploitation by reducing structural errors and reinforcing correct intermediate steps.

7 Related Work

Related work disagrees on whether RL extends reasoning beyond base-model capabilities. The paper argues that these views can coexist because RL outcomes depend on existing reasoning primitives and post-training data.

  • One line of work views RL as a capability refiner whose advantage over base models diminishes as pass@k increases.
  • Another line reports that RL can improve synthesized reasoning tasks and induce novel strategies for complex problem families.
  • The paper reconciles these perspectives by making RL generalization conditional on the base model’s reasoning primitives and the post-training data.
  • Mid-training is presented as a possible explanation for differing RL responsiveness because it aligns supervision with post-training distributions.

8 Conclusion

The study disentangles how pre-training and post-training jointly shape reasoning capabilities, finding that post-training yields genuine improvements only when pre-training establishes key reasoning primitives.

  • Controlled synthetic tasks and process-level evaluations isolate how training stages influence reasoning generalization.The framework supports causal analysis of stage-specific contributions.
  • Genuine post-training improvements arise only when pre-training establishes key reasoning primitives.This condition explains when RL can extend reasoning rather than merely refine existing abilities.
  • The findings guide data curricula, reward-function design, and compute allocation across training stages.These applications follow from the study’s clarified account of reasoning development.

A.1 Data Generation Framework

The data-generation framework formalizes controllable reasoning instances through graph structure, abstraction, contextual rendering, and generation procedures.

  • The framework defines graph-level formalism for reasoning instances.It specifies the underlying structural representation.
  • It separates structure from numeric and linguistic instantiations.This enables controlled variation of reasoning problems while preserving their underlying structure.
  • It includes contextual rendering, concrete generation, and deduplication procedures.These steps map formal instances into usable natural-language data.

A.1.1 Graph-Level Formalism

Each reasoning instance is represented as a DAG whose operation count controls compositional complexity and depth-generalization difficulty.

  • Each reasoning instance is grounded in a directed acyclic graph.Nodes represent latent quantities and edges encode functional dependencies.
  • The designated query node yields the ground-truth answer from numeric assignments to leaf nodes.Evaluation proceeds through the graph’s dependency structure.
  • The query node is selected as the final generated numeric node or the distinguished unknown parameter in reverse generation.These rules identify the target quantity for the two generation modes.
  • The operation count op(G) measures structural complexity.It lower-bounds the minimal compositional reasoning-chain length and is varied to study depth generalization.

A.1.2 Abstract and Instance Parameters

The framework independently varies symbolic structure, concrete instantiation, linguistic context, training stage distributions, and parsed reasoning traces to study depth and contextual transfer.

  • Abstract and Instance Parameters: Abstract parameters define variables, graph edges, and node operations independently of particular numbers or entities.They specify the symbolic reasoning structure.
  • Abstract and Instance Parameters: Instance parameters assign leaf values and bind variables to context-specific surface forms.Different instantiations preserve the same abstract graph while changing concrete numbers and entities.
  • Abstract and Instance Parameters: Implicit reasoning separates directly verbalized dependencies from graph dependencies omitted from the wording.This probes whether models recover unspoken relations.
  • Process Representation: Step-by-step solutions follow the symbolic graph’s topological order and expose intermediate derivations.This supports process-level evaluation of reasoning traces.
  • Training and Evaluation Design: Varying op(G) and the template produces a two-dimensional testbed for depth scaling and context transfer.Different regions of this space define pre-training, mid-training, and post-training distributions.
  • Process Representation: Parsed solutions become dependency graphs with inferred values and extracted final answers.The parser uses named quantities, dependencies, and arithmetic expressions to align traces with graph-level reasoning.
  • Generalization Results: Shared reasoning primitives support contextual transfer, while moderate context-B exposure improves learning after sparse exposure plateaus.Context-B reward improves with 10–100% exposure, and hard-problem pre-training leaves RL most effective when prior competence is partial.
  • Compute Allocation: Mid-training plus RL complements either stage alone across compute budgets.The best pass@128 allocation depends on compute: roughly 80% RL with limited resources, versus full RL with more compute.
Loading 2512.07783v1…