GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning

Introduction

• GPG achieves state-of-the-art results across unimodal and multimodal visual tasks, with substantial mathematical-benchmark gains and robust multimodal generalization.Figure 1 presents the comparison across unimodal reasoning benchmarks and multimodal reasoning; the introduction reports state-of-the-art results across these task types.Source: Figure 1 · p. 1, §1 · p. 3
• PPO’s critic and reference models increase training complexity and computational demands, motivating simpler RL methods; ReMax (Li et al., 2024) and GRPO remove the critic.ReMax introduces a baseline, while GRPO uses normalized rewards within a sample group.Source: §1 · p. 2
• Because LLM policies already have pretrained and SFT representations, the introduction argues that removing unnecessary RL components is important for scalability despite policy-gradient variance.It identifies value-function baselines and sampling more trajectories as common ways to mitigate that variance.Source: §1 · p. 2
• GPG directly optimizes the original RL objective instead of a surrogate loss, while removing critic and reference models and distributional constraints.These design choices are presented as simplifying the method and offering potential scalability advantages.Source: §1 · p. 2
• GPG analyzes reward bias in existing advantage functions and proposes AGE for gradient-estimate bias, plus thresholding and resampling when valid samples are scarce.The threshold maintains a minimum partition of valid samples to mitigate potentially high gradient-estimation variance when the valid-sample proportion is excessively small.Source: §1 · p. 2

2.1 PRELIMINARY AND TASK FORMULATION

• For reasoning tasks, one-step advantage estimation is presented as sufficiently effective and computationally efficient compared with the more sophisticated GAE approach.The task formulation samples an autoregressive response from a question and prompt, then optimizes the policy using a final reward.Source: §2.1 · p. 3

2.2 GROUP POLICY GRADIENT

• GPG normalizes rewards using each group’s mean to reduce variance, stabilize learning, and eliminate the need for a value model.For math reasoning, the passage notes that rewards are typically well-defined, with correct answers receiving 1.0 and incorrect answers 0.0.Source: §2.2 · pp. 3–4
• 47.8% average score was achieved by GPG with accurate gradient estimation (AGE) on the reported math reasoning results.The passage attributes this result to AGE; Table 1 is identified as reporting the math reasoning results on Qwen2.5-Math-7B.Source: Table 1 · p. 4, §2.2 · p. 5
• Reward normalization improved average score by 1.4% over Fnorm=1, α=1; its within-group reward standard deviation ranged from 0.10 to 0.35 while α ranged from 1.5 to 4.0.The passage attributes this improvement to reward normalization’s within-group standard-deviation mechanism, which has a gradient-correction effect.Source: Table 1 · p. 4, §2.2 · p. 5
• Removing reward bias with Fnorm=1 yielded 43.9%, which did not clearly outperform GRPO at 43.7%.The authors argue that reward bias departs from the original problem formulation and seek to avoid surrogate objectives or bias.Source: §2.2 · p. 4
• AGE excludes all-correct or all-wrong group examples with zero gradients and corrects the batch average by the variable factor α = B/(B−M).The method proposes accumulating valid samples into a subsequent batch when their proportion falls below threshold βth, to reduce variance.Source: §2.2 · pp. 4–5

3.1 EXPERIMENTAL SETUP

• Experiments span mathematics, visual, geometry, classification, and grounding tasks, using controlled settings that closely follow GRPO hyperparameters and the original code base.Unimodal evaluations include five mathematics benchmarks; multimodal evaluations use corresponding test sets for the task datasets.Source: §3 · p. 6, §3.1 · p. 6

3.2 UNIMODAL TASK EVALUATION

• GPG-RS1 achieves 55.7% average accuracy; GPG-RS1 and GPG-RS3 score 77.5% and 80.0% on AMC23, surpassing Open-RS at 67.5% and 70.0%.The models also score 87.6% and 85.0% on MATH-500, and 50.5% and 52.4% on OlympiadBench, respectively.Source: §3.2 · pp. 6–7, Table 3 · p. 7
• GPG-7B scores 57.7% on average, exceeding Oat-Zero-7B by 6.3% on average and SimpleRL-Zero-7B by 22.3% on AMC23 and 11.4% on Minerva.Its leading scores are 82.5% on AMC23 and 39.0% on Minerva.Source: Table 4 · p. 7, §3.2 · p. 7

3.3 MULTIMODAL TASK EVALUTAION

• GPG scores 51.33% on GEOQA, 3.85 percentage points above GRPO, and averages 89.0% across four classification datasets, 7.1 points above GRPO.GPG improves across all four classification datasets.Source: Table 6 · p. 7, Table 5 · p. 7, §3.3 · p. 8
• GPG scores 76.15% on CV-Bench, 16.68 points above GRPO, and improves every LISA grounding metric by over 14.0 points versus GRPO.The grounding evaluation uses Qwen2-VL-2B.Source: Table 7 · p. 7, Table 8 · p. 7, §3.3 · p. 8

3.4 ABLATION STUDY AND DISCUSSION

• The case study depicts GPG reaching the correct solution with more comprehensive and accurate reasoning, while GRPO makes formula-analysis errors and answers incorrectly.Figure 3 presents real-time training metrics to illustrate GPG’s effectiveness.Source: §3.4 · p. 8
• GPG outperforms DAPO with accuracy reward alone and lower training and data costs; adding a distribution constraint reduces performance.Both methods were trained on the same dataset for 1100 steps; GPG avoids DAPO’s dynamic-sampling inefficiencies. The paper also reports that GPG’s loss omits the CLIP term and KL divergence, and that group size 8 balances training cost and performance as larger groups improve average performance.Source: §3.4 · pp. 8–9, Table 9 · p. 9

3.5 IMPACT AND LIMITATION DISCUSSION

• The study does not evaluate GPG on extremely large models because of computational-budget constraints.The authors frame scalable reinforcement learning as important to augmenting model reasoning capabilities.Source: §3.5 · p. 9

Large Model Reasoning

• LLM reasoning methods include Chain-of-Thought (Wei et al., 2022; Kojima et al., 2022; Ye et al., 2025), Tree-of-Thought (Yao et al., 2023), search, and complex SFT datasets (Muennighoff et al., 2025).Search includes Monte Carlo Tree Search (Feng et al., 2024; Xin et al., 2024).Source: §4 · p. 9
• DeepSeek-R1 (Guo et al., 2025) uses large-scale RL with format-specific and result-oriented rewards to elicit complex CoT, while its methodology is applied to MLLMs for visual reasoning (Zhang et al., 2025; Liu et al., 2025b; Yuan et al., 2026b; Ji et al., 2026b; Wang et al., 2026).MLLMs process modalities in a unified LLM vocabulary representation space and show strong vision-understanding performance (Wu et al., 2024; Liu et al., 2023; Chen et al., 2024).Source: §4 · p. 9

Reinforcement Learning

• Policy-gradient RL progressed from high-variance REINFORCE through TRPO’s constrained updates (Schulman et al., 2015) to PPO’s clipped objective (Schulman et al., 2017), but conservative updates or heuristic clipping can limit exploration.PPO is widely used in language-model alignment and robotic control; later work has sought to improve its algorithm or clarify its implementation.Source: §4 · p. 9

• GPG incorporates group-based decision dynamics into standard policy gradients to address challenges in reinforcement fine-tuning approaches such as PPO and GRPO.Source: §5 · p. 10
• GPG simplifies training and reduces computational overhead without sacrificing reasoning quality.Source: §5 · p. 10
• The approach provides a more efficient framework for training advanced LLMs capable of complex reasoning, supporting resource-effective and scalable AI systems.Source: §5 · p. 10

6 REPRODUCIBILITY STATEMENT

• The paper documents datasets and benchmarks in Section 3.1 and training configurations, hyperparameters, evaluation protocols, and framework adaptations in Appendix B.1.These materials cover experimental setup and training details.Source: §6 · p. 10
• For replication, the implementation adapts public repositories: VERL (Sheng et al., 2024), Open-r1, and Open-rs (Dang & Ngo, 2025) for unimodal tasks.The passage also identifies VisualThinker-R1-Zero, R1-V, and Visual-RFT (Liu et al., 2025b) for multimodal tasks.Source: §6 · p. 10
• For multimodal tasks, the implementation adapts VisualThinker-R1-Zero, R1-V, and Visual-RFT (Liu et al., 2025b).The authors say these adaptations facilitate replication by the research community.Source: §6 · p. 10
• The authors make the full implementation and training and evaluation scripts publicly accessible through an anonymous repository, preserving the double-blind review policy.The passage states this enables reproduction of all reported results without revealing the authors’ identities.Source: §6 · p. 10

A.1 PROBLEM FORMULATION

• The setup distributes B samples evenly across N GPUs, so each GPU processes K = B/N samples, of which Mi are invalid and the remaining K − Mi are valid.Invalid samples produce zero gradients.Source: §A.1 · p. 15
• For GPU i, the formulation denotes the gradient of its j-th valid sample as gi,j and introduces a valid-gradient sum for that GPU.The supplied passage introduces this sum but does not show its formula.Source: §A.1 · p. 15
• The theoretically correct gradient averages over only the S valid samples, where S equals the batch size B minus the total invalid samples Mtotal.Mtotal is the sum of invalid samples across the N GPUs.Source: §A.1 · p. 15
• The conventional approach computes each GPU’s local mean using its assigned sample count K, including invalid samples that produce zero gradients.Global averaging of those local means is the conventional estimate described in the derivation.Source: §A.1 · p. 15

B.1 EXPERIMENT SETTINGS

• The 7B setting follows the dataset of and Algorithm 1, using accuracy-based rewards for 1,100 training steps.Training uses VERL with a global batch size of 144 prompts and 8 responses per prompt, AdamW at a constant learning rate of 1×10−6, weight decay 0.1, βth = 0.6, and 48 NPUs.Source: §B.1 · p. 16
• Unimodal evaluation uses Open-r1 and Open-rs (Dang & Ngo, 2025); DeepSeek-R1-Distill-Qwen-1.5B yields GPG-RS1 and GPG-RS3 after 100 and 50 global steps.The models are trained on the open-s1 and open-rs datasets, respectively, as reported in the repository.Source: §B.1 · p. 16
• Multimodal evaluation uses VisualThinker-R1-Zero, R1-V, and Visual-RFT for CV-Bench, GEOQA, image classification, and reasoning grounding.The frameworks cover visual reasoning, geometric reasoning, and image perception; classification datasets include Nilsback & Zisserman (2008), Parkhi et al. (2012), Maji et al. (2013), and Krause et al. (2013), while reasoning grounding uses Lai et al. (2024).Source: §B.1 · p. 16

B.2 MORE ABLATION EXPERIMENT RESULTS

• Reward normalization within a group performs better than normalization within a batch.The experiment studies reward normalization using Qwen2.5 Math 7B.Source: §B.2 · p. 16, Table 13 · p. 17
• Group A removes KL and uses PG loss; B corrects reward normalization but degrades, C’s α-scaling mitigates gradient bias, and D’s valid-sample threshold reduces variance.The threshold imposes a minimum valid-sample proportion.Source: §B.2 · p. 17, Table 14 · p. 17, Table 15 · p. 17
• Across multiple runs, GPG shows clear advantages over other baselines in mean and standard deviation, consistent with the trends in Table 4.The section reports these multi-run statistics in Table 16.Source: §B.2 · p. 17

B.3 EVALUATION ON GENERAL BENCHMARKS

• MMLU covers 57 subjects and C-Eval spans 52 disciplines; both are general benchmarks unrelated to the reasoning datasets used in training.The evaluation was intended to check whether specialized-reasoning gains came at the cost of general capabilities.Source: §B.3 · p. 17
• GPG training improved DeepSeek-R1-distill-Qwen-1.5B by +0.22 on MMLU and +0.38 on C-Eval under identical OPENCOMPASS settings.The evaluation used the base model and the same model after GPG training.Source: §B.3 · p. 17, Table 17 · p. 19, Table 18 · p. 20
• In zero-shot code-generation and general-QA evaluations, GPG outperformed GRPO.The results are reported in Table 19 for Qwen-1.5B; the supplied passage gives no numerical values.Source: §B.3 · p. 17, Table 19 · p. 21

B.4 PROMPT AND REWARD FUNCTION

• Reasoning prompts instruct the model to generate intermediate reasoning steps to facilitate reasoning capabilities.An example is attributed to Liu et al. (2025b).Source: §B.4 · p. 18
• The example prompt encloses reasoning in <think> tags and the answer in <answer> tags.It also instructs the assistant to solve the question and then provide the answer.Source: §B.4 · p. 18
• Most tasks use accuracy and formatting rewards: each grants 1.0 for a ground-truth-consistent output or the specified <think></think> <answer></answer> format, respectively.The Qwen 7B setting uses only the accuracy reward.Source: §B.4 · p. 18
• For grounding tasks, the reward uses IoU scores calculated from model-generated bounding boxes, consistent with Visual-RFT (Liu et al., 2025b).The Qwen 7B setting is an exception in which only the accuracy reward is used.Source: §B.4 · pp. 18–19

B.5 ZERO-SHOT EVALUATION ON GENERAL QA AND CODING TASK

• Figure 3 compares GPG and GRPO training loss, rewards, and completion length using DeepSeek-R1-Distill-Qwen-1.5B.The caption says the experiments use the same model as Table 3, but the supplied passages do not provide curve values or identify a winner.Source: Figure 3 · p. 21

B.6 CASE EXAMPLE

• Figure 4 presents a GPG–GRPO mathematical-reasoning test case from AIME24 using DeepSeek-R1-Distill-Qwen-1.5B trained on Open-rs.The text introduces the case study but supplies no outcome details.Source: Figure 4 · p. 21, §B.6 · p. 21

C MORE RELATED WORK

• PPO (Schulman et al., 2017) addresses TRPO’s (Schulman et al., 2015) optimization instability with a clipped surrogate objective and restricts policy ratios to [1 − ϵ, 1 + ϵ].The passages describe a probability ratio between updated and previous policies and characterize clipping as controlling policy-update magnitude.Source: C MORE RELATED WORK · pp. 21–22
• PPO combines clipped policy-gradient, value-function-loss, and entropy-regularization terms, with coefficients balancing optimization, value estimation, and exploration.The account says first-order gradient clipping replaces TRPO’s computationally intensive second-order KL constraints; Hsu et al. (2020) established surrogate-objective monotonicity guarantees.Source: C MORE RELATED WORK · p. 22
• GRPO (Shao et al., 2024) removes explicit value-function approximation by estimating token-level advantages from response rewards relative to their group average.For each question, it samples multiple candidate responses and compares their rewards, aligning with reward models trained on pairwise response rankings.Source: C MORE RELATED WORK · p. 22
• GRPO combines a clipped surrogate objective with explicit KL regularization, using a regularization coefficient β.The passages describe token-level probability ratios and an unbiased KL estimator.Source: C MORE RELATED WORK · pp. 22–23