PaperlayerView exact revision on arXiv

Source-linked AI summary

GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning

Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, Yong Wang

arXiv:2504.02546v4cs.LGcs.AI

TL;DR

The paper addresses the computational cost and complexity of reinforcement fine-tuning methods for reasoning models. It proposes Group Policy Gradient, which directly optimizes the policy-gradient objective with minimal components and no distributional constraints. Experiments and analyses report efficient, strong performance, while evaluation excludes extremely large models because of computational-budget constraints.

  • Problem

    PPO-based reinforcement fine-tuning can require substantial resources because it integrates critic and reference models, motivating simpler reasoning-training methods.

  • Method

    GPG directly optimizes the original policy-gradient objective while omitting surrogate losses, critic and reference models, and distributional constraints.

  • Results

    GPG consistently outperforms GRPO across controlled tasks, while analyses address reward and gradient-estimation bias and group-size trade-offs.

  • Takeaways & Limitations

    The results support GPG as a simpler, more resource-efficient framework for training language models with complex reasoning capabilities.

  • Takeaways & Limitations

    Because of computational-budget constraints, the method is not evaluated on extremely large models; its formulation also uses a final reward signal rather than intermediate-step rewards.

Abstract

from arXiv · show

Reinforcement Learning (RL) can directly enhance the reasoning capabilities of large language models without extensive reliance on Supervised Fine-Tuning (SFT). In this work, we revisit the traditional Policy Gradient (PG) mechanism and propose a minimalist RL approach termed Group Policy Gradient (GPG). Unlike conventional methods, GPG directly optimize the original RL objective, thus obviating the need for surrogate loss functions. By eliminating the critic and reference models, avoiding KL divergence constraints, and addressing the advantage and gradient estimation bias, our approach significantly simplifies the training process compared to Group Relative Policy Optimization (GRPO). Our approach achieves superior performance without relying on auxiliary techniques or adjustments. As illustrated in Figure 1, extensive experiments demonstrate that our method not only reduces computational costs but also consistently outperforms GRPO across various unimodal and multimodal tasks. Our code is available at https://github.com/AMAP-ML/GPG.

1 INTRODUCTION

GPG revisits policy-gradient reinforcement learning to simplify reasoning-model training while addressing computational cost and estimation bias. The paper reports strong results across unimodal and multimodal reasoning tasks.

  • Motivation: PPO improves reasoning but requires critic and reference models, increasing training complexity and computational demands.GRPO removes the critic and uses normalized rewards within sample groups, while remaining part of the broader simplification trend.
  • Motivation: Policy gradients offer a natural streamlined foundation because pretrained and supervised-fine-tuned LLMs already possess strong representations.The paper frames value baselines and additional trajectory sampling as common ways to mitigate policy-gradient variance.
  • Contributions: GPG’s motivation is to retain minimal reinforcement-learning components while improving scalability through simpler training.The contribution summary explicitly connects component removal and the absence of distributional constraints with potential scalability advantages.
  • Contributions: GPG directly optimizes the policy-gradient objective, eliminating surrogate losses, critic and reference models, and distributional constraints.The method also analyzes reward and gradient-estimation bias, introducing accurate gradient estimation and thresholded resampling for low valid-sample ratios.
  • Results: GPG achieves state-of-the-art results across diverse unimodal and multimodal visual reasoning tasks.Figure 1 describes gains over SOTA baselines on mathematical benchmarks and stronger performance than other RL methods in multimodal settings.

2 METHOD

GPG revisits policy-gradient training for reasoning by directly optimizing the original objective while removing value and reference models and avoiding distributional constraints. It uses group-level rewards and gradient corrections to reduce estimation variance and bias, with experiments reporting strong mathematical, unimodal, and multimodal reasoning results.

  • Preliminary and Task Formulation: Policy-gradient methods optimize expected return by adjusting policy parameters through gradient ascent, with advantage estimation used to reduce variance.The advantage compares action values with state values; one-step estimation is reported as sufficient for model reasoning and computationally efficient.
  • Group Policy Gradient: GPG samples responses, obtains final rewards, and normalizes rewards within each group to stabilize policy-gradient estimation without a value model.The method is designed for settings where intermediate-step rewards are difficult to construct and final rewards are available.
  • Group Policy Gradient: GPG directly optimizes the policy-gradient objective, eliminating surrogate loss, critic and reference models, and distributional constraints.These design choices are presented as simplifying training and improving potential scalability compared with methods built around additional components.
  • Group Policy Gradient: AGE addresses invalid all-correct or all-incorrect group samples by correcting the gradient estimate, while thresholding and resampling mitigate high variance when valid samples are scarce.The threshold is defined as βth = 1/αth for the proportion of valid samples.
  • Group Policy Gradient: Reward normalization in GRPO improves the average score by 1.4% over Fnorm=1, α=1, with within-group reward standard deviations ranging from 0.10 to 0.35 and α from 1.5 to 4.0.The authors connect this variation to a gradient-correction effect and motivate explicit analysis of reward normalization.

3 EXPERIMENTS

Experiments evaluate GPG across unimodal and multimodal tasks under controlled settings, consistently finding higher performance than GRPO and other baselines. Ablations examine implementation choices, group size, and distribution constraints, while the study does not evaluate extremely large models.

  • Experimental setup: GPG consistently outperforms GRPO across all evaluated tasks under controlled experimental settings.The authors retain GRPO hyperparameters despite noting they may be suboptimal for GPG.
  • Experimental setup: The experiments use GPG with a 1.5B distilled SFT model and a 7B base model across mathematical and multimodal benchmarks.The implementation follows original code bases and evaluates both unimodal and multimodal scenarios.
  • Unimodal task evaluation: 55.7% average accuracy is achieved by GPG-RS1 among evaluated 1.5B distilled models.GPG-RS1 and GPG-RS3 also exceed Open-RS on AMC23, with scores of 77.5% and 80.0% versus 67.5% and 70.0%.
  • Unimodal task evaluation: 57.7% average score is achieved by GPG-7B, exceeding Oat-Zero-7B by an average of 6.3%.GPG-7B reaches 82.5% on AMC23 and 39.0% on Minerva, exceeding SimpleRL-Zero-7B by 22.3% and 11.4%.
  • Multimodal task evaluation: GPG surpasses GRPO across multimodal tasks, including visual reasoning, geometry reasoning, classification, and reasoning grounding.Reported gains include 51.33% versus 47.48% on GEOQA, 89.0% average classification performance with a 7.1-point gain, and 76.15% versus 59.47% on CV-Bench.
  • Ablation study and discussion: Increasing group size from 2 to 16 progressively improves most metrics, while group size 8 is selected as a cost-performance tradeoff.An ablation also finds that adding a distribution constraint negatively impacts performance.
  • Impact and limitation discussion: The study reports a computational-budget limitation: extremely large models are not evaluated.

4 RELATED WORK

Related work covers methods for stepwise reasoning in language and multimodal models, alongside the evolution of policy-gradient reinforcement learning methods. It also notes that additional related work is deferred to Section C.

  • Large Model Reasoning: Large and multimodal language models increasingly use stepwise reasoning methods such as CoT, Tree-of-Thought, and Monte Carlo Tree Search.
  • Reinforcement Learning: Policy-gradient research progressed from REINFORCE toward methods such as TRPO and PPO that address high-variance or unstable policy optimization.
  • Related work: Further related work is discussed in Section C.

5 CONCLUSION

GPG incorporates group-based decision dynamics into standard policy gradient to simplify reinforcement fine-tuning. It reduces computational overhead without sacrificing reasoning quality.

  • GPG integrates group-based decision dynamics into the standard policy-gradient method.
  • The method simplifies training and significantly reduces computational overhead.
  • GPG provides a more efficient framework for training language models capable of complex reasoning.

6 REPRODUCIBILITY STATEMENT

The authors support empirical reproducibility through detailed experimental descriptions and publicly available implementation foundations. Their unimodal and multimodal setups build on established open-source repositories.

  • The paper provides comprehensive descriptions of experimental setups, datasets, and corresponding benchmarks.These details are provided in Section 3.1.
  • Unimodal experiments use the VERL, Open-r1, and Open-rs code repositories.
  • Multimodal experiments adopt the VisualThinker-R1-Zero framework as an implementation base.

A ANALYSIS OF DISTRIBUTED GRADIENT AVERAGING WITH INVALID SAMPLES

The analysis formalizes distributed gradient averaging when samples can be invalid. It contrasts conventional averaging over all assigned samples with the theoretically correct average over valid samples only.

  • Problem Formulation: A batch of B samples is evenly distributed across N GPUs, each processing K = B/N samples.
  • Problem Formulation: On GPU i, Mi initial samples produce zero gradients, while the remaining K − Mi samples produce valid gradients.
  • Problem Formulation: Mtotal = ∑_{i=1}^N Mi counts invalid samples, and S = B − Mtotal denotes effective valid samples.
  • True Gradient Computation: The valid gradient sum for GPU i is defined using gradients from its valid samples.
  • True Gradient Computation: The theoretically correct gradient averages over valid samples only, unlike the conventional estimate.
  • Conventional Approach Derivation: Conventional distributed averaging uses each GPU’s local mean based on K assigned samples rather than valid samples.

B.1 EXPERIMENT SETTINGS

The experiments use established unimodal and multimodal codebases and specify concrete training configurations. Settings include model-specific training steps, accuracy-based rewards, and optimizer parameters.

  • B.1 EXPERIMENT SETTINGS: The 7B training setup uses 144 prompts per global batch, 8 responses per prompt, and accuracy-based rewards.
  • B.1 EXPERIMENT SETTINGS: The 7B model is optimized with AdamW at a constant learning rate of 1× 10−6 and weight decay 0.1.
  • B.1 EXPERIMENT SETTINGS: The 7B setup trains for 1100 steps on 48 NPUs, with βth set to 0.6.
  • B.1 EXPERIMENT SETTINGS: Unimodal reasoning experiments use Open-r1 and Open-rs repositories with the DeepSeek-R1-Distill-Qwen-1.5B model.
  • B.1 EXPERIMENT SETTINGS: Multimodal experiments build on VisualThinker-R1-Zero, R1-V, and Visual-RFT across visual, geometric, and image-perception tasks.

B.2 MORE ABLATION EXPERIMENT RESULTS

The ablations examine reward normalization, group size, RL-method components, and robustness across random seeds. They identify normalization and gradient-bias corrections as important design factors for GPG.

  • General evaluation: Table 12 evaluates GPG on MMLU and C-Eval, extending the ablation appendix beyond mathematical reasoning benchmarks.
  • Reward normalization: Table 13 evaluates reward normalization for Qwen2.5 Math 7B, comparing within-batch and within-group normalization.
  • RL-method comparison: Tables 14 and 15 compare RL-method components and report mathematical-reasoning results for Qwen2.5-Math-7B.
  • Method components: Group B corrects reward normalization but performs worse, which the authors attribute to gradient bias; Group C mitigates this with α-scaling.

B.3 EVALUATION ON GENERAL BENCHMARKS

The paper tests whether GPG’s reasoning gains preserve general capabilities and evaluates its broader zero-shot behavior. GPG improves both general benchmarks and remains ahead of GRPO on code generation and general question answering.

  • General benchmarks: MMLU and C-Eval are unrelated to the reasoning datasets used for training, providing a general-capability evaluation beyond specialized reasoning tasks.
  • General benchmarks: GPG improves MMLU by +0.22 and C-Eval by +0.38 under identical OPENCOMPASS settings.
  • Zero-shot generalization: GPG continues to outperform GRPO in zero-shot code generation and general QA tasks.
  • Robustness: The appendix includes mean and standard-deviation results over four random seeds, where GPG consistently shows clear advantages over other baselines.
  • Evaluation setup: The reasoning setup uses system-prompt instructions that encourage intermediate reasoning steps and task-dependent accuracy, formatting, or IoU rewards.

B.5 ZERO-SHOT EVALUATION ON GENERAL QA AND CODING TASK

The appendix presents GPG–GRPO comparisons and a mathematical case study under the specified DeepSeek-R1-Distill-Qwen-1.5B setup. It also summarizes PPO and GRPO mechanisms relevant to interpreting those comparisons.

  • Comparison study: Figure 3 compares GPG and GRPO using training loss, rewards, and completion length on DeepSeek-R1-Distill-Qwen-1.5B.
  • Case example: The case study presents a step-by-step assistant solution whose ground-truth answer is 480.
  • Mathematical reasoning: Figure 4 compares GPG and GRPO on mathematical reasoning using Open-rs training data and an AIME24 test case.
  • Related methods: PPO constrains policy updates through clipping, with the clip interval controlling update magnitude and limiting excessive policy deviations.
  • Related methods: GRPO samples multiple outputs and defines each token advantage from its response group’s relative reward, while its objective includes clipping and KL regularization.
Loading 2504.02546v4…