PaperlayerView exact revision on arXiv

Source-linked AI summary

GRPO Training Dynamics for Small Language Models

Rajat Ghosh, Vaishnavi Bhargava, Henry Wong, Aryan Singhal, Debojyoti Dutta

arXiv:2609.39321v1cs.LGcs.AI

TL;DR

GRPO training dynamics on small language models remain poorly understood, despite their relevance to resource-constrained post-training. This study characterizes GRPO across model families and reasoning tasks and uses mechanistic diagnostics to guide tuning. GRPO improves mathematical reasoning, while MCQ and coding require targeted refinement, with code responding better than MCQ.

  • Problem

    GRPO training dynamics remain poorly understood for SLMs, limiting reliable adoption and reproducibility in resource-constrained environments.

    Source: §1 · p. 1

  • Method

    The study characterizes GRPO across three SLM families and reasoning datasets, using weight-displacement and activation-shift diagnostics to guide targeted tuning.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    GRPO enhances mathematical reasoning, while MCQ and coding tasks require targeted hyperparameter refinement; code responds well, but MCQ performance shows limited improvement.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Evaluation and training signals can diagnose bottlenecks such as reward–accuracy misalignment and guide corrective hyperparameter changes.

    Source: §6 · p. 9

    OpenCoder outputs from DeepSeek-R1-Qwen-1.5B often have inconsistent reasoning and answer tags and verbose traces that overflow the token budget.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Group Relative Policy Optimization (GRPO) has emerged as a memory-efficient reinforcement fine-tuning (RFT) technique for reasoning-intensive tasks. How- ever, GRPO training dynamics on small language models (SLMs) remain poorly understood, limiting its reliable adoption and reproducibility in open and resource- constrained environments. In this work, we present a systematic study of GRPO fine-tuning for SLMs ranging from 1.5B to 7B parameters under a practical single- node 8xA100 compute budget. Our study spans multiple model families and reasoning domains, including mathematics, coding, and multiple-choice question answering (MCQ) in science. Across these settings, we analyze how group size affects policy convergence, training stability, and downstream benchmark per- formance. We further characterize tensor-level update dynamics during GRPO training and investigate whether the choice of LoRA target modules and layers can improve the performance of GRPO-tuned models. While our initial GRPO-tuned models outperform their base counterparts on approximately 80% of mathematical benchmark evaluations, they demonstrate limited capability on MCQ and code reasoning tasks. Guided by our mechanistic evaluations, we refined our LoRA and reward-shaping configurations to improve performance in latter domains. These findings provide practical guidance for GRPO training for SLMs.

1 Introduction

GRPO offers memory-efficient reinforcement learning for reasoning tasks, but its training dynamics and mechanistic behavior remain poorly understood for small language models under tight compute budgets. The paper studies GRPO dynamics across reasoning tasks and uses mechanistic evaluation to guide targeted performance tuning.

1 Introduction

  • GRPO training dynamics remain poorly understood for SLMs under tight compute budgets, with limited analysis of convergence, reward saturation, and compute–performance trade-offs.The paper also identifies mechanistic interpretability as a critical gap.Source: §1 · p. 1
  • SLMs face post-training challenges including mode collapse, catastrophic forgetting, and reward hacking under strong GRPO optimization pressure.The passage links these risks to narrowed output distributions and discarded pretrained behaviors.Source: §1 · p. 2
  • The paper characterizes GRPO training dynamics across three SLM families and GSM8K, ARC-Challenge, and OpenCoder.These datasets cover mathematics, closed-book question answering, and code generation.Source: §1 · p. 2
  • Mechanistic evaluation guides hyperparameter tuning for targeted GRPO performance improvement.The paper presents this as a contribution alongside its empirical characterization.Source: §1 · p. 2
  • The paper open-sources its GRPO post-training pipeline.This is presented as a reproducibility contribution.Source: §1 · p. 2

2 Preliminaries

GRPO replaces a learned critic with reward baselines estimated from sampled outputs within each prompt group. The paper frames training design as bilevel optimization and identifies diagnostics for tuning its parameters.

2 Preliminaries

  • GRPO replaces a learned critic with intra-group baselines estimated directly from sampled rollouts for each prompt.Without the learned value-function stabilizer used by methods such as PPO, GRPO can be more sensitive to reward scaling, clipping, KL regularization, and group size.Source: §2 · p. 2
  • The training-design problem is framed as inner GRPO training at fixed λ and outer refinement of λ to maximize target performance under behavioral and model-level constraints.The constraints cover behavioral, output-level, activation-level, and weight-level properties.Source: §2 · p. 2
  • The design space λ includes optimization settings, LoRA choices, data mixture, and reward weights.It includes learning rate, clipping threshold, KL coefficient, group and batch sizes, LoRA modules, layers, rank and scaling, and reward weights.Source: §2 · p. 2
  • Weight displacement and activation shift can diagnose training behavior and guide targeted hyperparameter refinement.The paper presents mechanistic evaluation-guided tuning as an alternative to potentially cost-prohibitive black-box search.Source: §2 · p. 2
  • For each prompt, GRPO samples G outputs with rewards and normalizes each output's advantage relative to its group's reward mean and standard deviation.The policy is updated using a clipped objective with a KL penalty against a fixed reference policy.Source: §2 · pp. 2–3

3 Experimental design

The experiments address whether GRPO-only post-training improves SLM reasoning and whether mechanistic feedback can improve training performance. They span three model families and reasoning datasets, compare two group sizes, and use repeated-seed monitoring and out-of-distribution benchmarks.

3 Experimental design

  • The study asks whether GRPO-only post-training improves SLM reasoning and whether mechanistic feedback can improve training performance.These are the two research questions framing the experiments.Source: §3 · p. 3
  • Experiments use DeepSeek-R1-Qwen-1.5B, Nemotron4B, and DeepSeek-7B on GSM8K, ARC, and OpenCoder, comparing two GRPO group sizes across model-dataset combinations.The datasets evaluate mathematics, closed-book scientific reasoning, and code generation, respectively.Source: §3 · p. 3
  • Runs use two independent random seeds, track four evaluation-time and four training-time metrics, and evaluate out of distribution on additional task-specific benchmarks.Reward functions for sparse rewards and experiment hyperparameters are specified in the appendices.Source: §3 · p. 3

4 Results

GRPO’s out-of-distribution results vary by reasoning domain: improvements are most frequent on math benchmarks, less frequent on MCQ and code benchmarks, and depend on the model and training configuration. Mechanistic diagnostics guide refinements that improve some outcomes, with trade-offs across models and metrics.

4.1 GRPO Training Dynamics on GSM8K

  • On GSM8K, DeepSeek-R1-Qwen-1.5B with G = 16 reaches the highest accuracy despite DeepSeek-7B’s higher reward, alongside higher entropy, lower KL, longer completions, and persistent clipping.Most accuracy and reward gains occur within about 500 steps; the group-reward coefficient of variation stabilizes but remains non-zero.Source: §4.1 · pp. 3–4, Figure 1 · p. 4
  • 58 of 72 math model–benchmark–metric configurations improve over base models, but gains vary by model, benchmark, and group size.Every math benchmark metric has at least one improvement; DeepSeek-R1-Qwen-1.5B often benefits from G = 16, while several DeepSeek-7B and Nemotron-4B results are comparable or better under G = 4.Source: Table 1 · p. 4, §4.1 · p. 5

4.2 GRPO Training Dynamics on ARC

  • On ARC, Nemotron-4B performs best, followed by DeepSeek-7B and DeepSeek-R1-Qwen-1.5B on both reward and accuracy; raising G from 4 to 16 improves both for DeepSeek-R1-Qwen-1.5B, but yields limited gains for the stronger models.KL remains near zero for DeepSeek-R1-Qwen-1.5B and increases for DeepSeek-7B and Nemotron-4B, especially under G = 16.Source: §4.2 · pp. 5–6
  • 28 of 72 MCQ configurations improve over base models, with gains highly heterogeneous across model families and evaluation settings.DeepSeek-R1-Qwen-1.5B improves 16 of 24 benchmark metrics; larger group size is generally beneficial, though effectiveness remains configuration-dependent.Source: §4.2 · p. 6, Table 2 · p. 6
  • On MCQ, refinements guided by mechanistic metrics improve GRPO-tuned performance, including increased KL regularization for Nemotron-4B and a higher clipping coefficient for DeepSeek-7B.The Nemotron-4B treatments also reduce LoRA layers or modules, alone or in combination.Source: §4.2 · p. 6, Table 3 · p. 7

4.3 GRPO Training Dynamics for OpenCoder

  • On OpenCoder, performance and reward rise rapidly before diminishing returns, while the 7B model’s performance drop with increased KL is reported as indicating possible mode collapse.Entropy generally decreases and KL remains bounded under these settings; completion lengths differ substantially by model.Source: §4.3 · pp. 7–8, Figure 3 · p. 7
  • 17 of 36 code configurations improve: DeepSeek-R1-Qwen-1.5B gains pass@1 +4.86% on MBPP, while DeepSeek-7B improves Codeforces pass@1 from 0.76 to 2.60 at G = 8.Nemotron-4B generally regresses on MBPP and APPS.Source: §4.3 · p. 8, Table 4 · p. 8
  • On coding benchmarks, Nemotron-4B’s reduced-layer and correctness-reward treatments recover regressions, while reduced modules yield pass@1 of 33.92 on MBPP (+3.7%) and 11.83 on APPS (+34.6%).Pass@5 does not show similar gains; DeepSeek-7B’s reduced-layer treatment can stabilize transfer to closer benchmarks while reducing gains on Codeforces.Source: §4.3 · pp. 8–9, Table 5 · p. 9

5 Related Work

Prior RLHF work emphasizes the stabilizing role of learned value functions, while GRPO stability remains an active research challenge. Mechanistic evaluation has emerged as a complementary aid for diagnosing and tuning these actor-only methods.

5 Related Work

  • Prior RLHF studies emphasize that a parameterized value function plays a critical role in stabilizing training and improving overall performance, while GRPO stability remains an active research area.Prior RLHF research has primarily analyzed optimization stability, reward modeling, and policy dynamics in PPO-based pipelines [30] [32].Source: §5 · p. 9
  • GRPO stability and implementation remain challenging despite the introduction of several algorithm variants.The cited work describes optimization stability and performance as active research areas [50, 52, 24].Source: §5 · p. 9
  • Mechanistic evaluation is relevant to GRPO because downstream performance and stability are sensitive to many hyperparameters and lack a traditional baseline estimator.Recent work explores mechanistic evaluation as a complementary signal for hyperparameter optimization and diagnostics in large-scale post-training pipelines [35] [27] [37].Source: §5 · p. 9

6 Conclusion

The study finds that GRPO improves mathematical reasoning for small language models, while code and MCQ tasks need targeted refinement. It also reports non-uniform LoRA contributions and proposes diagnostic signals and tuning guidance for resource-constrained GRPO adoption.

6 Conclusion

  • GRPO enhances SLM mathematical reasoning, while code and MCQ tasks require targeted hyperparameter refinement; code responds well, but MCQ gains remain limited.The authors suggest MCQ gains may be limited by the knowledge-intensive nature of these tasks.Source: §6 · p. 9
  • Layer- and module-level contributions in LoRA-based GRPO are non-uniform, and weight-displacement-guided module selection improves optimization efficiency and downstream generalization over uniform adaptation.Source: §6 · p. 9
  • The proposed diagnostics combine four evaluation-time outcome metrics with training-time process metrics to identify bottlenecks and guide corrective actions.When reward CV is low, the authors suggest increasing temperature to improve sample diversity.Source: §6 · p. 9
  • The authors offer a blueprint for reliable GRPO adoption in resource-constrained environments and identify scaling mechanistic interventions as future work.They argue that SLM post-training should move beyond behavioral metrics toward understanding internal representation dynamics.Source: §6 · p. 10

A.1 GRPO Formulation

At each GRPO step, the policy samples a group of completions for a prompt and assigns each completion a scalar reward. The formulation then identifies group-relative advantages and policy ratios, penalizes deviation from a fixed reference policy, and describes the inner training dynamics.

A.1 GRPO Formulation

  • For a prompt drawn from the training distribution, the current policy samples a group of completions at GRPO step k.Each completion is represented as a sequence of tokens.Source: §A.1 · p. 14
  • Each sampled completion receives a scalar reward, and the formulation identifies its group-relative advantage and policy ratio.Source: §A.1 · p. 14
  • The objective penalizes deviation from a fixed reference policy and defines inner GRPO training dynamics over steps k = 0 to T(λ) − 1.Source: §A.1 · p. 14

A.2 Mechanistic Feedback and Evaluation

The section uses weight displacement and activation shift to characterize GRPO’s mechanistic changes. Updates are structured by task, model, module, and layer, with activation shifts concentrated near the output.

A.2 Mechanistic Feedback and Evaluation

  • The section pairs normalized weight displacement across parameter groups with activation shift as metrics for assessing GRPO training health.A parameter group can be a layer–module pair, such as an attention or MLP projection.Source: §A.2 · pp. 14–15
  • For LoRA adaptation, the effective weight includes a low-rank update scaled by the LoRA rank and scaling parameter.The displacement metric is adjusted for this adapted-weight form.Source: §A.2 · p. 14
  • GRPO updates concentrate in attention and MLP projections, with little layer-normalization movement; GSM8K induces larger displacement than ARC or OpenCoder.The pattern varies across model families, with DeepSeek-R1-Qwen-1.5B showing broad, intense GSM8K changes and DeepSeek-LLM-7B more moderate changes.Source: Figure 4 · p. 15, §A.2 · p. 17
  • Across six model–dataset combinations, hidden-state shifts stay small through most layers and rise sharply in the final 2–3 layers.The comparison covers AI2-ARC and MBPP test prompts across three base models.Source: Figure 5 · p. 16

A.3 Reward Shaping Scores

Table 6 identifies the reward-shaping components used in GRPO training.

A.3 Reward Shaping Scores

  • Table 6 lists the reward-shaping components used in GRPO training.The supplied passage identifies the table’s subject but does not specify its components.Source: Table 6 · p. 18

A.4 Training Hyperparameters

This block gives a training-configuration reference, then presents qualitative GRPO-versus-base examples across math, science QA, and coding. It also describes potential societal benefits and risks of lower-resource reasoning-model fine-tuning.

A.4 Training Hyperparameters

  • Table 7 presents GRPO training configurations across GSM8K, OpenCoder, and ARC-Challenge.The supplied passage identifies the table’s scope, not the configuration values.Source: Table 7 · p. 18

A.5.1 GSM8K Dataset

  • GSM8K examples show GRPO correcting an arithmetic slip, repairing malformed final-answer formatting, and replacing a vacillating reasoning trace with a shorter committed calculation.In the self-doubt example, the base model reaches the correct intermediate answer but ultimately chooses the wrong value.Source: §A.5.1 · pp. 19–20, Table 8 · p. 19, Table 9 · p. 19

A.5.2 ARC Dataset

  • ARC examples show GRPO resolving rotation–revolution confusion and improving functional analogy and climatic inference compared with the base model.The examples contrast axial rotation with orbital period, transport function with surface association, and climate implications with water-presence reasoning.Source: §A.5.2 · pp. 21–23, Table 11 · p. 21, Table 13 · p. 22

A.5.3 MBPP

  • In MBPP examples, GRPO produces a correct generator, an executable solution after base-model thinking overflow, and a successful evaluation where the base model loops incorrectly.The examples compare GRPO and base-model outputs on individual tasks.Source: §A.5.3 · pp. 23–24, Table 14 · p. 23, Table 15 · p. 23

A.5.4 APPS-Intro

  • APPS-Intro examples show GRPO solving bitwise and modulo requirements, correcting an LCM approach, and producing working code where the base model overthinks.One example uses a structured rationale and a Python solution using Counter.Source: §A.5.4 · pp. 25–27, Table 17 · p. 25, Table 18 · p. 26

A.6 Societal Impact

  • The societal-impact discussion cites resource-constrained model development and local-device reasoning as potential benefits, alongside misuse risks and hard-to-audit errors from unstable models.The stated risks include specialized phishing, exploit generation, misinformation, and confidently incorrect or biased outputs in agentic workflows.Source: §A.6 · pp. 27–28

Related papers

Loading 2609.39321v1…