PaperlayerView exact revision on arXiv

Source-linked AI summary

Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation

Zhiqi Yu, Zhangquan Chen, Mengting Liu, Heye Zhang, Liangqiong Qu

arXiv:2602.05548v3cs.LGcs.AI

TL;DR

GRPO’s implicit advantage symmetry limits exploration of unsampled correct paths and leaves its difficulty focus poorly matched to evolving training dynamics. The paper introduces A-GRAE, which asymmetrically modulates exploration and shifts training from simpler to harder samples; across seven benchmarks, it consistently improves GRPO and its variants.

  • Problem

    GRPO faces limited exploration of novel correct solutions and inadequate adaptation of training focus to problem difficulty.

  • Method

    A-GRAE breaks GRAE’s symmetry by asymmetrically encouraging trajectory exploration and dynamically transitioning sample emphasis from easy to hard.

  • Results

    Across seven benchmarks, A-GRAE consistently improves GRPO and its variants across LLM and MLLM settings.

  • Takeaways & Limitations

    Asymmetric exploration and evolving difficulty prioritization provide a supported alternative to GRAE’s symmetric update design within GRPO.

  • Takeaways & Limitations

    The method uses a normalization convention that requires no extra rescaling when the success rate is 0 or 1 because the GRPO advantage is zero.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR), particularly GRPO, has become the standard for eliciting LLM reasoning. However, its efficiency in exploration and difficulty adaptation remains an open challenge. In this work, we argue that these bottlenecks stem from an implicit advantage symmetry inherent in Group Relative Advantage Estimation (GRAE). This symmetry induces two critical limitations: (i) at the group level, strict symmetry in weights between correct and incorrect trajectories leaves unsampled action logits unchanged, thereby hindering exploration of novel correct solution. (ii) at the sample level, the algorithm implicitly prioritizes medium-difficulty samples, remaining agnostic to the non-stationary demands of difficulty focus. Through controlled experiments, we reveal that this symmetric property is sub-optimal, yielding two pivotal insights: (i) asymmetrically suppressing the advantages of correct trajectories encourages essential exploration. (ii) learning efficiency is maximized by a curriculum-like transition-prioritizing simpler samples initially before gradually shifting to complex ones. Motivated by these findings, we propose Asymmetric GRAE (A-GRAE), which dynamically modulates exploration incentives and sample-difficulty focus. Experiments across seven benchmarks demonstrate that A-GRAE consistently improves GRPO and its variants across both LLMs and MLLMs.

1. Introduction

The paper identifies implicit advantage symmetry in GRAE as a source of GRPO’s exploration and difficulty-adaptation limitations, then proposes A-GRAE to break that symmetry. A-GRAE combines asymmetric exploration with curriculum-like sample prioritization and improves GRPO variants across seven benchmarks.

  • Motivation: GRPO has been widely adopted for reasoning, but prior work identifies capability-boundary shrinkage and inadequate focus on problem difficulty as fundamental limitations.The difficulty-agnostic reward mechanism can cause overfitting on simpler tasks or insufficient learning on challenging ones.
  • Implicit symmetry: GRAE’s implicit advantage symmetry assigns equal update weights to correct and incorrect trajectories, restricting exploration of unsampled potentially optimal paths.At the sample level, it also implicitly prioritizes medium-difficulty instances and fails to adapt focus as training dynamics evolve.
  • Design principles: Controlled interventions show that asymmetrically suppressing correct-trajectory weights fosters exploration, while prioritizing simpler samples before harder ones improves learning efficiency.These interventions deliberately break GRAE’s symmetry at both group and sample levels.
  • Proposed method: A-GRAE dynamically encourages trajectory exploration and shifts sample emphasis from simpler to more complex instances as the model’s capability evolves.The framework instantiates both design principles within GRPO.
  • Evaluation: A-GRAE consistently enhances GRPO and representative variants across natural-language and vision-language reasoning settings, with significant gains in accuracy and Pass@k.The evaluations span seven diverse benchmarks and commonly used LLMs and VLMs.

2. Preliminary

The preliminary formulation treats foundation models as policies that generate grouped candidate responses for each query. Rule-based rewards and group-relative advantages then support GRPO’s critic-free optimization.

  • Notation: The framework represents an LLM or MLLM with current and prior policies, πθ and πθold, and samples G candidate responses for each question.For each question q, the prior policy generates the candidate response set.
  • Notation: A rule-based verifier assigns each question-response pair a scalar reward, using binary accuracy by default: correct responses receive 1 and incorrect responses receive 0.The response index i ranges over the candidates for each question.
  • GRPO: GRPO removes PPO’s critic model and estimates baselines from group-level reward scores while optimizing the target policy against sampled responses.For each query, responses are sampled from the prior policy πθold.
  • GRAE: GRAE computes each candidate’s advantage from the group of rewards, with the group-relative normalization involving the reward standard deviation.The supplied formulation identifies the reward group and its standard-deviation normalization.

3. Implicit Advantage Symmetry in GRPO

The paper analyzes GRPO as advantage-weighted SFT and formalizes two symmetry effects: sampled trajectories are updated according to their signs, while unsampled trajectories receive no gradient; sample updates favor intermediate difficulty.

  • Reweighting view: GRPO’s policy-gradient update can be viewed as a reweighting variant of supervised fine-tuning, with the advantage providing the dominant reweighting term.The advantage is shared across all tokens in a response.
  • Group level: Correct and incorrect sampled trajectories receive strictly equivalent policy-update weights under GRAE, creating the paper’s group-level advantage symmetry.The positive and negative trajectory sets partition the sampled response set.
  • Group level: Correct sampled responses receive positive logit updates that increase their probability, whereas incorrect sampled responses receive negative updates that suppress known errors.These are the positive- and negative-advantage cases in the group-level analysis.
  • Group level: Unsampled trajectories receive a strictly zero gradient, so a low-probability correct path remains static unless it is sampled.This leaves GRPO without an intrinsic active exploration mechanism for unsampled correct trajectories.
  • Sample level: GRAE’s total update magnitude is largest for medium-difficulty samples, with p = 0.5 dominating policy updates.Here p is the empirical success rate, and higher p denotes lower difficulty.
  • Sample level: Samples with p = 0.75 and p = 0.25 receive identical importance weights, while evolving sample distributions make this symmetry poorly matched to training dynamics.As the model evolves, simpler instances become more frequent and difficult instances decline, risking trivial-data overfitting and insufficient training on challenging scenarios.

4. Deconstructing the Implicit Advantage Symmetry of GRAE

The experiments break GRAE’s symmetry at group and sample levels to test its effects on exploration and difficulty focus. Results indicate that suppressing correct-trajectory advantages supports exploration, while learning efficiency improves when focus shifts from easier to harder samples over training.

  • Experimental Setup: Control Experiment I scales correct-trajectory advantages up or down to break group-level symmetry, using GRPO as the symmetric control.Positive-Dominant uses A*pos = β·Apos, whereas Negative-Dominant uses A*pos = Apos/β; β = 10.
  • Experimental Setup: Control Experiment II rescales advantages by sample success rate to create Hard-Focused and Easy-Focused training variants.Hard-Focused uses 1/√p, while Easy-Focused shifts attention toward simpler queries; γ = 0.5 normalizes the intervention.
  • Group-Level Symmetry: GRPO improves Pass@1 over the base model but can fall below the base model at Pass@256 on AMC23 and MATH, indicating limited discovery beyond the base model’s sampling support.The reported pattern suggests improved sampling probability for known correct paths without expanding the model’s intrinsic reasoning boundaries.
  • Group-Level Symmetry: Amplifying correct-path advantages causes entropy collapse and hurts performance at larger sampling budgets, whereas suppressing them consistently outperforms GRPO across the three datasets.The Negative-Dominant group’s entropy increases over training, while Positive-Dominant entropy declines sharply; suppression therefore preserves greater sampling diversity.
  • Sample-Level Symmetry: Difficulty reweighting has no universal advantage: Hard-Focused performs best on AIME2025, while Easy-Focused marginally leads on AMC23 and MATH at Pass@1.The preferred difficulty focus depends on benchmark difficulty rather than remaining fixed throughout training.
  • Sample-Level Symmetry: Easy-Focused converges fastest early, but Hard-Focused continues improving later and eventually performs better, supporting a curriculum from simpler to harder samples.The authors conclude that static difficulty reweighting is insufficient because optimal sample utility is phase-dependent.

5. Asymmetric GRAE

A-GRAE dynamically adjusts exploration and sample-difficulty focus to address limitations attributed to GRPO’s symmetric advantage weighting. Across text-only and multimodal benchmarks, it consistently improves reasoning performance and preserves generalization.

  • Method: A-GRAE uses batch-wise mean reward as a training-state proxy and shifts sample-level attention from easy to hard questions as proficiency improves.The hard-focused component gains weight as sampling success increases, while the easy-focused component diminishes.
  • Method: At the group level, A-GRAE attenuates correct-response advantages to encourage early exploration while preserving later training stability.The refined advantages can be incorporated into the GRPO objective or other GRPO variants.
  • Experimental Setup: A-GRAE is evaluated on seven benchmarks spanning text-only mathematics and multimodal mathematics and medicine, using LLMs and VLMs.The experiments include AIME2025, AMC23, MATH, Geo3k, MathVision, MathVerse, and HuatuoGPT-Vision.
  • Main Results: A-GRAE consistently improves Pass@1 and Pass@k when integrated with GRPO variants, while outperforming W-REINFORCE and GRPO-LEAD on accuracy and diversity.The reported results attribute these gains to mitigating capability-boundary shrinkage while enhancing reasoning accuracy.
  • Main Results: In multimodal tasks, A-GRAE improves both in-distribution and out-of-distribution performance, indicating enhanced sampling efficiency alongside preserved generalization.The paper describes these findings as evidence of applicability across diverse domains and tasks.
  • Analysis: Ablations indicate that sample-level asymmetry mainly improves Pass@1, group-level asymmetry more strongly improves Pass@k, and the full framework achieves the largest overall gains.Training dynamics further show that A-GRAE mitigates entropy collapse relative to GRPO.

6. Conclusion

The paper identifies advantage symmetry in GRPO as restricting exploration and difficulty adaptation, then proposes A-GRAE to dynamically address both limitations. Evaluations across seven benchmarks report improved performance over GRPO and its variants.

  • Conclusion: A-GRAE dynamically modulates exploration incentives and prioritizes samples according to their evolving utility.The method is motivated by the finding that standard GRAE restricts exploration and fails to adapt difficulty focus during learning.
  • Conclusion: The paper attributes GRPO’s exploration and difficulty-adaptation limitations to an overlooked advantage symmetry in standard GRAE.The conclusion connects the symmetry to restricted exploration and inadequate adaptation of difficulty focus.
  • Conclusion: Across seven benchmarks, A-GRAE consistently improves GRPO and its variants on both LLMs and MLLMs.The reported evaluation spans multiple model families and benchmarks.
  • Related Work: GRPO and REINFORCE++ estimate advantages without the critic model used by PPO’s GAE framework.These methods replace the critic with lightweight batch-averaged or group-relative reward baselines.
  • Related Work: RLVR research debates whether reinforcement learning expands reasoning capabilities or mainly improves sampling efficiency while reducing diversity and exploration.The paper positions its advantage-estimation analysis within this ongoing exploration–exploitation debate.

B.2. Proof of Equation 5

The proof establishes that GRAE assigns equal total absolute advantage magnitude to correct and incorrect sampled trajectories. It then formulates GRPO in behavior space to analyze how these weights affect sampled and unsampled trajectory logits.

  • Proof: The sum of absolute advantages for correct trajectories equals that for incorrect trajectories.This is the group-level advantage symmetry proved from the zero-sum standardized advantages.
  • Proof: Correct trajectories have positive advantages, while incorrect trajectories have negative advantages under binary verifiable rewards.Correct rewards exceed the group mean and incorrect rewards fall below it.
  • Behavior Space and Set Definitions: The behavior space contains all possible trajectories, while the sampled group and unsampled set partition those trajectories.The notation distinguishes generated trajectories from the much larger set not explored in the current iteration.
  • Optimization Objective: The objective maximizes expected return over the sampled group using logits and a softmax probability distribution over behavior space.The derivation applies the chain rule and the derivative of log-softmax to obtain trajectory-level updates.
  • Gradient Derivation: The resulting gradient separates a sampled-trajectory advantage term from a contribution represented by the constant C.An indicator isolates whether a trajectory belongs to the sampled group.

B.4. Proof of Theorem 2

Theorem 2 relates total update magnitude to sample success probability, showing symmetric importance for equally distant easy and hard samples. The proof also analyzes how suppressing correct-path advantages changes trajectory probabilities.

  • Proof of Theorem 2: For binary rewards, GRAE defines each advantage as Ai = (ri −µ)/σ and partitions absolute advantages into successful and failed subsets.The theorem uses empirical success probability p and the group mean and standard deviation.
  • Theorem 2: Update magnitude peaks at p = 0.5, while samples at p = 0.75 and p = 0.25 receive identical importance weights.Easy and hard samples with equal deviation from p = 0.5 are treated symmetrically.
  • Probability Updates: Suppressing correct-path advantages yields C < 0 and strictly increases the sampling probability of sampled correct responses.The positive advantage and additional −Cπbi term raise the correct response’s logit.
  • Probability Updates: Overconfident negative responses can have increased logits under the modified dynamics, potentially displacing correct responses later in training.This contrasts with GRPO, where C = 0 directly penalizes incorrect responses.
  • Probability Updates: Under GRPO, unsampled trajectories have zero gradient and unchanged logits, whereas a negative-dominant group increases their probabilities.The latter provides an exploration incentive for discovering previously unsampled correct trajectories.

C.1. Dataset Introduction

The evaluation covers mathematical reasoning, multimodal mathematical reasoning, and medical imaging reasoning benchmarks. Experiments use fixed generation and optimization settings while A-GRAE is controlled by a single scaling parameter.

  • Mathematical Reasoning: The mathematical reasoning evaluation uses MATH, AMC23, and AIME 2025 to assess logical and long-chain problem solving.These benchmarks span competition-level mathematics, high-school contests, and advanced examination problems.
  • Multi-modal Mathematical Reasoning: The multimodal evaluation uses Geo3K, MathVerse, and MathVision for visual mathematical reasoning.The datasets require combining visual diagrams or contexts with mathematical problem solving.
  • Medical Imaging Reasoning: HuatuoGPT-Vision evaluates medical-domain expertise and visual grounding across curated clinical visual-question-answering data.Its training set contains 600 MRI image-question pairs, with evaluation conducted on the benchmark described in the passage.
  • Experimental Settings: Training uses temperature 1.0, learning rate 1e-6, and maximum completion length 2048, while evaluation uses temperature 0.6, top-p 0.95, and 4096 new tokens.The reported averages use 16 runs for AIME2025 and AMC23 and 4 runs for other benchmarks.
  • Method Settings: A-GRAE requires one hyperparameter, the scaling parameter α, set to 1 for Math and 0.5 for Geo3K and Medical datasets.The smaller multimodal value is motivated by their observed higher propensity for training collapse.

D.1. Experimental Results of Control Experiment using Llama-3.2-3B-Instruct

On Llama-3.2-3B-Instruct, breaking group-level symmetry improves reasoning performance, while difficulty re-weighting has no universally optimal strategy. At k=256, all evaluated methods underperform the base model.

  • Suppressing positive advantages at the group level significantly enhances reasoning performance, particularly on Pass@k.The evaluated Pass@k values use k = {1, 2, 4, 8, 16, 32, 64, 128, 256}.
  • No universally optimal difficulty re-weighting strategy exists at the sample level.
  • At k=256, all evaluated methods underperform relative to the base model.
  • The Negative-Dominant group remains the most effective approach for preserving the base model’s fundamental capabilities.

D.3. Training Collapse of Negative-Dominant Group

Repeated training reveals stochastic instability in the Negative-Dominant group: one run remains stable, while another collapses after approximately step 78. The collapse produces rising unsolved-question counts and substantial performance degradation.

  • The Negative-Dominant group exhibits striking stochastic instability across identical-hyperparameter training runs.One trial remains relatively stable, while another undergoes catastrophic collapse after approximately step 78.
  • After approximately step 78, the collapse instance shows high-amplitude oscillations and a persistent upward trend in fully unsolved questions.The number of fully unsolved questions eventually peaks at over 110.
  • Without proper regularization, the Negative-Dominant objective can lead to divergent optimization in which erroneous trajectories reinforce themselves.
  • Balancing negative trajectory weights is necessary to ensure consistent convergence.

D.5. Comparative Analysis with Control Groups

Comparisons with control groups show that ASS matches Negative-Dominant performance while avoiding its observed collapse cases, and DDAS improves early difficulty adaptation while remaining competitive as k increases.

  • ASS is comparable with the Negative-Dominant group across all Pass@k metrics.
  • In 10 independent training runs, Negative-Dominant experienced collapse in three trials, whereas ASS had no failure cases.
  • ASS maintains training stability while facilitating model exploration.
  • Figure 10 tracks training-set entropy, test-set actor entropy loss, and greedy decoding accuracy on the Math test set.
  • DDAS achieves substantial Pass@1 gains across all datasets compared with fixed difficulty preference.
  • As k increases, DDAS remains competitive with Hard-Focused.
Loading 2602.05548v3…