PaperlayerView exact revision on arXiv

Source-linked AI summary

Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining

Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, Eran Malach

arXiv:2504.07912v2cs.LG

TL;DR

The paper asks how pretraining data composition interacts with RL fine-tuning, a relationship obscured by undocumented data in existing base models. It trains models from scratch on open mixtures and applies multiple RL methods across scales, finding that RL amplifies one pretraining mode, with scale-dependent output preferences and transfer from simpler to harder tasks.

  • Problem

    Existing studies lack control over pretraining data, leaving the mechanisms linking pretraining composition, RL algorithms, hyperparameters, and model scale insufficiently understood.

  • Method

    The authors pretrain models from scratch on curated mixtures of open datasets, then compare PPO, GRPO, and Expert Iteration across model scales using mathematical question-answering tasks.

  • Results

    RL fine-tuning amplifies one mode from the pretraining mixture while collapsing others; the amplified mode depends on model scale, and simpler-task post-training improves performance on harder mathematical datasets.

  • Takeaways & Limitations

    Controlled small-scale proxies can reveal how RL shapes language-model behavior, including output-distribution concentration, scale-dependent bias, and cross-task transfer.

  • Takeaways & Limitations

    The study leaves extension to more complicated mixtures, including multilingual data and scale-specific optimal pretraining mixtures, for future work.

Abstract

from arXiv · show

Reinforcement learning (RL)-based fine-tuning has become a crucial step in post-training language models for advanced mathematical reasoning and coding. Following the success of frontier reasoning models, recent work has demonstrated that RL fine-tuning consistently improves performance, even in smaller-scale models; however, the underlying mechanisms driving these improvements are not well-understood. Understanding the effects of RL fine-tuning requires disentangling its interaction with pretraining data composition, hyperparameters, and model scale, but such problems are exacerbated by the lack of transparency regarding the training data used in many existing models. In this work, we present a systematic end-to-end study of RL fine-tuning for mathematical reasoning by training models entirely from scratch on different mixtures of fully open datasets. We investigate the effects of various RL fine-tuning algorithms (PPO, GRPO, and Expert Iteration) across models of different scales. Our study reveals that RL algorithms consistently converge towards a dominant output distribution, amplifying patterns in the pretraining data. We also find that models of different scales trained on the same data mixture will converge to distinct output distributions, suggesting that there are scale-dependent biases in model generalization. Moreover, we find that RL post-training on simpler questions can lead to performance gains on harder ones, indicating that certain reasoning capabilities generalize across tasks. Our findings show that small-scale proxies in controlled settings can elicit interesting insights regarding the role of RL in shaping language model behavior.

1 Introduction

The paper studies RL fine-tuning under controlled pretraining conditions to clarify how data composition, algorithms, hyperparameters, and model scale shape model behavior. It finds that RL often concentrates outputs into one pretraining distribution, with performance gains, scale-dependent preferences, and positive transfer to unseen evaluations.

  • 1 Introduction: The study pretrains models from scratch on mixtures of fully open datasets, enabling controlled analysis of how pretraining and RL jointly shape behavior.The experiments track quantitative and qualitative generation changes across training stages.
  • 1 Introduction: RL fine-tuning consistently drives outputs toward a single pretraining distribution, often improving pass@1 accuracy while reducing diversity.The preferred distribution is typically the one with the strongest base-model accuracy, although occasional failures occur.
  • 1 Introduction: Smaller models favor simpler, code-like formats, whereas larger models shift toward natural-language outputs.This indicates that the preferred output distribution depends on model scale.
  • 1 Introduction: RL fine-tuning improves performance on evaluation datasets not seen during post-training.This provides evidence of positive transfer from RL fine-tuning.

2 Experimental Setup

The experiments use 150M- and 1B-parameter decoder-only models pretrained from scratch on open mathematical and synthetic instruction datasets. These models are then fine-tuned with PPO, GRPO, or Expert Iteration on mathematical question-answering tasks and evaluated across GSM8K, MATH-500, and AIME.

  • 2 Experimental Setup: The study trains 150M- and 1B-parameter decoder-only language models using the OLMo codebase.The models use widths of 768 and 2048 and depths of 12 and 16 layers, respectively.
  • 2 Experimental Setup: Pretraining combines FineMath-3+, Algebraic-Stack from Proof-Pile-2, TinyGSM, OpenMathInstruct1, and OpenMathInstruct2 in varied ratios.Question-answer datasets are concatenated as prompt-answer text without chat templates or special formatting.
  • 2 Experimental Setup: The pretraining optimizer uses AdamW with learning rate 0.001, weight decay 0.1, 5,000 warmup steps, and cosine decay to 10% of peak learning rate.
  • 2 Experimental Setup: RL fine-tuning uses PPO and GRPO with verifiable rewards, while Expert Iteration generates 64 samples per problem and supervised-fine-tunes on correct, deduplicated generations.
  • 2 Experimental Setup: Models are fine-tuned on GSM8K training questions, evaluated on GSM8K test questions, and subsequently tested on MATH-500 and AIME.

3 RL on Models Pretrained from Scratch with Different Mixtures

Across controlled PPO experiments, models rapidly concentrate on one pretrained output format, often improving pass@1 while reducing generation diversity. The favored format depends on pretraining mixture and model scale, and similar concentration appears with other RL algorithms.

  • 3.1 RL converges to favour one distribution in the mixture: RL fine-tuning rapidly concentrates generations in the format of a single pretraining distribution, suppressing alternatives.This pattern is observed across pretraining mixtures and is accompanied by improved pass@1 in many cases.
  • 3.1 RL converges to favour one distribution in the mixture: Approximately 5% majority@64 improvement accompanies a decline in pass@64 accuracy toward the end of training.The decline is consistent with reduced generation diversity after RL fine-tuning.
  • 3.1 RL converges to favour one distribution in the mixture: A higher KL coefficient preserves some OpenMathInstruct2-style generations while producing comparable final pass@1 accuracy and stable pass@64.The model still prefers TinyGSM-style outputs under the higher KL setting.
  • 3.2 RL doesn’t always favor the most performant, nor the most common distribution: In one mixture, TinyGSM becomes dominant despite only 28% initial generations versus 62% for OpenMathInstruct1; doubling OpenMathInstruct1 instead makes it dominant and lowers final performance.The latter behavior is identified as a failure mode, although most 150M runs select the initially best-performing distribution.
  • 3.3 How does performance within one distribution improve during RL?: Repeating TinyGSM 1, 2, 4, or 8 times improves post-fine-tuning pass@1, pass@64, and majority@64, with the highest proportion yielding the best final performance.The highest-TinyGSM mixture also shows the largest performance gain from fine-tuning.
  • 3.4 The effect of scale: larger models prefer different distributions: 150M models generally favor TinyGSM-style outputs, whereas 1B models favor OpenMathInstruct2-style natural-language responses on the same mixtures.The authors characterize this as a scale-dependent bias in behavior.
  • RL algorithm comparisons: GRPO and Expert Iteration show analogous preference for one distribution, while GRPO is less stable and often briefly collapses before recovering.GRPO performance is similar to PPO overall but slightly worse as TinyGSM pretraining data increases.
  • Supporting theory: A reward-weighted mixture-of-policies analysis is consistent with RL converging toward the strategy that maximizes reward.The analysis describes RL as reweighting policies according to rewards from the original mixture.

4 Transfer to other evaluation datasets

The study evaluates whether RL improvements transfer beyond GSM8K, using MATH-500 and AIME after fine-tuning on GSM8K. Transfer is observed on some harder-task metrics but is limited for AIME pass@1 and majority@64.

  • 4 Transfer to other evaluation datasets: Models fine-tuned on GSM8K are evaluated on MATH-500 and AIME to test performance on datasets not used during fine-tuning.The evaluation targets possible improvements in error rates and broader reasoning capabilities.
  • 4 Transfer to other evaluation datasets: AIME 2022–2024 shows little to no improvement in pass@1 and majority@64, while pass@64 improves across pretrained models.These results provide a mixed transfer pattern from GSM8K fine-tuning.

5 Discussion and Conclusion

The study finds that RL fine-tuning amplifies one pretraining mode, with the favored mode depending on model scale and amplification depending on KL-penalty settings. RL on simpler mathematical data also improves performance on harder datasets, while controlled small-scale studies expose open questions about optimal mixtures and broader data settings.

  • The study uses models pretrained from scratch on open data mixtures, addressing confounding from undisclosed pretraining datasets in existing work.
  • RL fine-tuning amplifies one mode from the pretraining mixture while suppressing the others.
  • The amplified mode depends on model scale, while amplification strength depends on the KL-penalty coefficient.
  • RL on GSM8K improves performance on MATH and, to a lesser extent, AIME.
  • Small-scale proxies can provide insights into RL fine-tuning in language models.
  • Open questions include extending the results to multilingual or more complex mixtures and identifying scale-dependent optimal pretraining mixtures.

B Dataset and Evaluation Details

The evaluation identifies distinct output formats associated with the instruction datasets and applies format-specific procedures before parsing final answers. Accuracy is reported with pass@1, pass@64, and majority@64.

  • Dataset formats: TinyGSM solutions use Python code inside simple math problem(), ending with return result.
  • Dataset formats: OpenMathInstruct1 wraps code in <llm-code> tags, includes parsed outputs, and may provide a final boxed answer.
  • Dataset formats: OpenMathInstruct2 uses natural-language responses with a final boxed answer.
  • Evaluation: Code outputs are executed and extracted differently by format, while final answers from code or natural language are parsed with Math-Verify.
  • Evaluation: Pass@1 uses one greedy generation, pass@64 checks whether any of 64 sampled generations is correct, and majority@64 evaluates the most frequent sampled answer.

C Additional Experimental Details

The experiments use specified PPO, GRPO, and Expert Iteration configurations, including controlled KL-coefficient and learning-rate variations.

  • PPO and GRPO use OpenRLHF implementations with default configurations, while KL coefficients are additionally set to 0 or 0.01.PPO applies a token-level KL penalty to the reward; GRPO applies it in the loss with the non-negative k3 estimator.
  • Expert Iteration uses a separate hyperparameter configuration with k = 64 samples generated per problem before correctness filtering.
  • Peak learning rates were swept over [5 × 10^-6, 1 × 10^-5, 1 × 10^-4, 0.001], with only marginal 1-2% gains beyond 1 × 10^-4 in EI’s first iteration.

D.1 Mixtures with OpenMathInstruct1 and OpenMathInstruct2

Across additional 150M mixtures, PPO generally converges toward TinyGSM-formatted outputs, while KL settings have limited effect except at high KL.

  • 150M models trained with TinyGSM plus OpenMathInstruct1 or OpenMathInstruct2 converge to TinyGSM-format code across mixtures, except with a high KL coefficient.
  • KL coefficient 0 produces similarly performant results to the default setting 0.001 across all mixtures.
  • 5-10% improvement in majority@64 occurs during PPO training, while pass@64 does not improve from initialization during fine-tuning.

E Additional Mixtures - 1B Models

Additional 1B experiments show scale-dependent output preferences and generally stronger or more stable behavior than corresponding 150M models, while GRPO remains less stable than PPO.

  • A 1B model pretrained on TinyGSM and 4× OpenMathInstruct1 amplifies the OpenMathInstruct1 code format and achieves higher final accuracy than the corresponding 150M model.
  • GRPO generally prefers one distribution like PPO but is less stable, with performance collapses that may recover or remain unrecovered.
  • GRPO performance is very similar to PPO overall but slightly worse as the amount of TinyGSM in pretraining increases.
  • A 1B model pretrained on TinyGSM and OpenMathInstruct2 amplifies natural-language solutions even though TinyGSM is more accurate at initialization.
  • 1B models trained on individual data subsets outperform corresponding 150M models but do not match the final accuracy of models trained on mixed datasets.

F.2 Expert Iteration

Expert Iteration produces slower shifts toward dominant formats and lower final performance than PPO, with only modest accuracy or preference changes across model scales.

  • Expert Iteration generates k = 64 candidates per GSM8K problem, filters correct de-duplicated generations, and uses them for iterative supervised fine-tuning.
  • Expert Iteration underperforms PPO and does not produce a strong preference for any dataset format, although TinyGSM preference increases modestly over time.
  • Accuracy after three EI iterations remains below 45% from an 8× TinyGSM base model that reaches 60% GSM8K test accuracy after PPO.
  • On 1B models, two EI iterations yield marginal accuracy improvements and a modest shift toward OpenMathInstruct or natural-language answers.
  • The slower shift toward a dominant format is hypothesized to reflect repeated fine-tuning from a fixed base model rather than more online updates.

G Confidence-Based Metrics

RL fine-tuning increases confidence in the output formats that become dominant, with generation probabilities tracking output proportions and continuing to rise after format stabilization.

  • Output-prefix probabilities closely follow output proportions during RL fine-tuning, with a smoother trajectory and narrowing error bars.The tracked prefixes represent TinyGSM- and OMI1-style generations.
  • GRPO reproduces PPO’s preference for TinyGSM in one setting and OpenMathInstruct1 in another, followed by a later performance collapse.The figures also show an initial performance increase before collapse.
  • GRPO shows near-identical trends to PPO, except pass@1 accuracy is slightly worse as the quantity of TinyGSM increases.
  • Average generation probabilities increase throughout training even after output format largely stabilizes, indicating growing confidence within the dominant distribution.

H.1 Qualitative Analysis on MATH-500 Generations

The qualitative analysis attributes most pretrained-model errors to misunderstanding problems or flawed algebraic and logical deductions. Fine-tuning improves harder-task performance, but gains vary across evaluations and are strongest in some OpenMathInstruct mixtures.

  • H.1 Qualitative Analysis on MATH-500 Generations: Across most pretrained models, the dominant error sources are misinterpreting questions and making flawed algebraic or logical deductions.
  • H.1 Qualitative Analysis on MATH-500 Generations: Fine-tuning gains on MATH-500 appear to reflect improved problem comprehension and reasoning, rather than arithmetic accuracy alone.The analysis also discusses format-level refinements.
  • H.1 Qualitative Analysis on MATH-500 Generations: PPO on GSM8K produces minimal to no improvement in AIME 2022–2024 pass@1 and majority@64 performance.
  • H.1 Qualitative Analysis on MATH-500 Generations: Broader AIME 1983–2024 evaluation shows more substantial gains in pass@1 and majority@64, while pass@64 improves across both AIME subsets.
  • H.1 Qualitative Analysis on MATH-500 Generations: Models pretrained with OpenMathInstruct mixtures achieve the largest improvements after post-training on the reported AIME evaluations.
  • H.1 Qualitative Analysis on MATH-500 Generations: Compared with GSM8K fine-tuning, the alternative evaluation setting yields smaller MATH-500 improvements and similar AIME results, with significant gains only for pass@64.
Loading 2504.07912v2…