Source-linked AI summary
Spurious Advantage Hidden in GRPO
Jiamian Wang, Samyadeep Basu, Koustava Goswami, Tong Yu, Zhiqiang Tao
TL;DR
GRPO’s within-group weighting can give high magnitude to correct guesses, not only reasoned solutions, in bounded or effectively bounded settings. The paper proposes SIGNBALANCE, which removes composition dependence while preserving verifier signs and balancing classes; it matches GRPO on open-answer math and improves bounded-answer math and search-agent performance.
Problem
GRPO’s within-group advantage does not distinguish correct reasoning from correct guessing, creating a spurious component in bounded-answer tasks, bounded sub-cases, and multi-turn search trajectories.
Method
SIGNBALANCE uses composition-independent magnitude, preserves each rollout’s verifier sign, and restores zero-mean balance through stop-gradient per-class rescaling.
Results
SIGNBALANCE matches GRPO on open-answer math, improves bounded-answer math at 0.5B and 3B, and gives the best Avg on multi-turn QA.
Takeaways & Limitations
The results support revisiting GRPO’s learning signal when correct answers can arise without reasoning, across bounded-answer math and multi-turn search settings.
Takeaways & Limitations
Validation covers math reasoning and multi-turn search agents but does not yet extend to more general settings such as selecting tools from a finite library.
Abstract
from arXiv · showhide
Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bounded sub-cases; and search agents whose budget opens many paths to the same answer. In all three, this misleads the policy toward guess-like behaviors. We propose SIGNBALANCE, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling. Across math and search agent benchmarks at different scales, SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents. Code will be released.
1 Introduction
GRPO’s within-group advantage can reward correct guesses as strongly as reasoned solutions, creating a spurious advantage across bounded and effectively bounded tasks. SIGNBALANCE removes this composition dependence while preserving verifier direction and requiring no added inference cost.
- Problem: GRPO assigns the same magnitude to correct rollouts whether they result from reasoning or guessing, allowing guess-derived signal into the policy gradient.The paper calls this component the spurious advantage and links it to learning lucky-guess trajectories.
- Problem: Bounded-answer tasks expose the issue because random selection can reach the correct option, while open-answer corpora and search trajectories can contain similarly finite or redundant answer paths.The paper identifies bounded benchmarks, bounded sub-cases in open-answer data, and multi-turn search agents as three structural cases.
- Method: SIGNBALANCE makes rollout magnitude independent of within-group composition, retains the verifier sign, and restores batch-level zero-mean balance through stop-gradient per-class rescaling.It is parameter-free, leaves the PPO-style surrogate unchanged, and adds no external model or inference-time cost.
- Scope: The work unifies three settings where GRPO’s learning signal can contain reasoning-independent components and evaluates the proposed estimator across math reasoning and search-agent tasks at multiple scales.The stated contribution emphasizes bounded-answer tasks, hidden bounded cases, and multi-turn search agents.
2 Related Work
Related work characterizes GRPO’s adaptive weighting and surrounding optimization mechanics. This paper studies the complementary regime where within-group composition includes reasoning-independent success and proposes composition-independent magnitudes.
- GRPO and Its Variants: GRPO research has refined clipping, sampling, normalization, aggregation, and advantage shaping around the original estimator.These studies address the surrounding optimization mechanics rather than the specific spurious component analyzed here.
- Advantage Estimators: Mroueh’s analysis shows that GRPO gives rare-success rollouts large positive weight, which is useful when rare success reflects a rare reasoning trajectory.The paper adopts this closed-form weighting as the starting point for its analysis.
- Advantage Estimators: This work complements prior estimator analyses by identifying three structural cases where composition contains a reasoning-independent component and decoupling magnitude from that composition.The proposed variant is designed for settings where within-group composition no longer reliably reflects reasoning.
3 Method
This section shows that GRPO’s within-group normalization can amplify correct rollouts reached by guessing, especially when answer surfaces or search paths are effectively bounded. SIGNBALANCE removes this composition dependence while restoring batch-level force balance.
- GRPO estimator: GRPO assigns each rollout a magnitude determined solely by the within-group counts of correct and wrong rollouts.The rollout content and policy confidence do not enter the magnitude calculation.
- Spurious advantage: Rare-correct groups receive disproportionately large gradient weights, even when correctness may result from chance rather than reasoning.For G = 16, n+ = 1 yields |Â+| ≈ 3.87, compared with 1 for n+ = 8.
- Spurious advantage: The spurious advantage combines reasoning-driven correctness with a non-reasoning component whenever the probability of guessing the correct answer is non-negligible.It is small when pg ≈ 0, as in open-answer mathematical reasoning, and becomes sizeable when pg is larger.
- Bounded answer sets: Finite candidate sets create this issue directly: random guessing succeeds with probability about 1/k and receives the same magnitude as reasoned correctness.The same analysis applies to bounded-output classification and closed-vocabulary question answering.
- Open-answer and search settings: Bounded subcases cover 55.95% of MATH-7.5K, while search agents can produce many distinct trajectories that reach the same correct final answer under outcome-only reward.In both settings, guessed or detoured rollouts can receive the same high magnitude as genuinely reasoned rollouts.
- SIGNBALANCE: SIGNBALANCE uses a composition-free global magnitude, preserves verifier signs, and restores batch-level zero-mean balance through per-class force rescaling.It is a parameter-free drop-in replacement that adds no external model or inference-time cost and excludes the spurious component from gradient assignment.
4 Experiments
Experiments evaluate SIGNBALANCE against GRPO and related methods across 0.5B and 3B math settings, plus search-agent QA. SIGNBALANCE shows its strongest gains on bounded-answer and search-agent benchmarks, while preserving competitive open-answer performance.
- 4.1 Experimental Settings: Experiments use shared backbones, training data, optimization, and evaluation pipelines, controlling only the advantage-estimation mechanism.The 0.5B and 3B math comparisons include multiple RL baselines; the search-agent comparison likewise fixes retrieval and inference parameters.
- 4.2 Main Results: SIGNBALANCE improves bounded-answer math accuracy, reaching 71.88 on SAT-Math versus GRPO’s 65.62 and 35.43 on AQuA versus 29.53.The reported gains are +6.26 on SAT-Math and +5.90 on AQuA.
- 4.2 Main Results: SIGNBALANCE brings clear improvements over the untrained baseline on open-answer benchmarks while matching GRPO’s open-answer performance overall.The strongest reported improvements occur on bounded-answer benchmarks rather than open-answer math.
- 4.3 Empirical Analysis and Discussion: At 3B, SIGNBALANCE achieves the highest Avg-8 of 43.78, with per-benchmark leadership increasing from 4/8 at 0.5B to 7/8 at 3B.The absolute Avg-8 gap over GRPO narrows from +2.37 to +0.98, while leadership broadens across benchmarks.
- 4.3 Empirical Analysis and Discussion: Guessing creates a measurable MATH-7.5K baseline: emitting “2” scores 2.69%, while uniform sampling over the top 10 strings reaches 20.6%.The bounded-answer fraction falls from 69.2% at Level 1 to 45.9% at Level 5, concentrating the spurious advantage on easier problems.
- 4.3 Empirical Analysis and Discussion: SIGNBALANCE reaches the highest search-agent Avg-6 at 37.80, exceeding StepSearch by 1.36 and Search-R1 by 1.80, and leads on 5 of 6 benchmarks.Its largest gain over Search-R1 is 7.62 on 2WikiMultiHopQA; StepSearch leads MuSiQue.
- 4.5 Ablation on the Advantage Estimation: Removing per-rollout count dependence improves bounded-answer results but can reduce MATH-500 accuracy by 4.80, motivating data-adaptive force balance.The ablation reports +6.26 on SAT-Math and +3.62 on AMC for Step 2, alongside the MATH-500 trade-off.
- 4.3 Empirical Analysis and Discussion: Across AQuA, SAT-Math, MMLU-math, and AMC, training trajectories separate within approximately 100 steps and SIGNBALANCE remains above GRPO thereafter.This supports a sustained advantage rather than a result caused by a single favorable checkpoint.
5 Conclusion
The paper identifies a spurious advantage in GRPO when correct answers can arise without reasoning, and proposes SIGNBALANCE to decouple rollout magnitude from group composition. SIGNBALANCE matches GRPO on open-answer math, improves bounded-answer math at 0.5B and 3B, and achieves the best Avg on multi-turn QA.
- GRPO’s count-dependent magnitude can amplify correct answers produced without reasoning, encouraging guess-like behaviors.
- The spurious advantage becomes substantial in bounded-answer tasks and multi-turn search agents, while bounded cases can also occur in open-answer training sets.
- SIGNBALANCE decouples per-rollout magnitude from within-group composition through a one-line advantage variant.
- SIGNBALANCE matches GRPO on open-answer math, beats it on bounded-answer math at 0.5B and 3B, and gives the best Avg on multi-turn QA.
Limitations
The validation covers math reasoning and multi-turn search-agent settings across model scales, but not more general settings or quantitative changes in reasoning ability and uncertainty behavior.
- SIGNBALANCE has been validated on math reasoning and multi-turn search-agent settings, but not yet in more general settings.
- The study has not quantitatively measured growth in reasoning confidence or uncertainty behavior across problem difficulty levels.
A Open-answer mathematical reasoning: the small-spurious-advantage case
For open-answer mathematical reasoning, the probability of reaching the correct answer without reasoning is expected to approach zero, so the spurious advantage is small but measurable in MATH-7.5K. In this regime, GRPO’s rare-correct weighting can reflect genuinely rare reasoning success.
- Open-answer math typically has a large answer surface and a single correct surface form, making non-reasoning success a ≪1 event.
- As pg → 0, per-prompt success probability becomes approximately pr, and within-group correct counts are driven by reasoning success.
- GRPO’s rare-correct magnitude is large when pr is small, meaning reasoning success is rare for the prompt.
- When pg = 0, the closed-form weighting has no spurious advantage because the non-reasoning term contributes nothing.
- In MATH-7.5K, pg is not zero and the residual spurious advantage is measurable, though small relative to bounded-answer cases.
B | ˆAi| as a per-rollout learning weight
The paper interprets |Â_i| as a multiplicative per-rollout learning weight: it determines how strongly a rollout’s log-probability is pushed, while the remaining gradient factor depends on the rollout and policy.
- Ignoring the KL term and clipping, the GRPO policy gradient factorizes into an advantage magnitude and a rollout-specific log-probability gradient.
- For a fixed rollout, |Â_i| acts as a multiplicative scalar controlling how strongly its log-probability is pushed.
- The rollout-specific gradient factor depends on the rollout’s tokens and current policy, but not on the advantage estimator choice.
- This per-rollout-weight interpretation follows the standard RL-for-LLM convention and matches Mroueh’s n−/n+ analysis under binary rewards.
- The spurious-advantage analysis concerns how per-rollout weights are composed across success channels, not absolute gradient norms or exact effects under specific PPO clipping realizations.
C Justification of the MATH-7.5K answer-shape categorization
The MATH-7.5K categorization measures answer-surface structure rather than problem difficulty, using 11 mutually exclusive and exhaustive syntactic categories over 7,399 parsable answers. Five categories have measurable random-guess hit rates and cover 4,140 problems, or 55.95% of the dataset.
- Source set: The analysis uses 7,399 parsable gold answers from the 7,500-problem MATH training split.The remaining 101 problems are dropped because their official solutions lack a parsable boxed answer.
- Category: Each gold answer is assigned to one of 11 mutually exclusive and exhaustive categories based only on answer-string shape.For example, the same numeric form receives the same category regardless of whether the underlying problem concerns geometry or arithmetic.
- Table columns: The table reports category counts and shares relative to the 7,399-answer source set.The category counts sum to 7,399, while the shares sum to 100%.
- Guess hit rate: Guess hit rate estimates the probability that a uniformly random answer of a category’s surface shape exactly matches the gold answer.Finite surfaces use 1/S, while open-ended surfaces are treated as having zero hit rate.
- Bounded: The bounded flag is set when guess hit rate is at least 0.5%, identifying categories with measurable non-reasoning success.The rule flags 5 of 11 categories, covering 4,140 problems or 55.95% of MATH-7.5K; int_small alone has a 4.76% guess rate.
D Implementation details
SIGNBALANCE uses a fixed global scale and a stop-gradient per-class rescaling rather than making rollout magnitude depend on within-group composition.
- Advantage estimator: SIGNBALANCE sets positive and negative rollout advantages to 1 and −sg[n+/n−], respectively.The ratio provides the per-class force-balance rescaling.
- Scaling: The method uses global scale c = 1 without an additional hyperparameter.This preserves the estimator’s simple parameterization.
- Evaluation procedure: Rollouts are generated with vLLM, and the best checkpoint per run is selected by Avg-8.These are the reported rollout and checkpoint-selection settings.
E Additional empirical diagnostics
Additional diagnostics show that bounded-answer benchmarks contain substantial random-guess signal, exhibit less stable GRPO evaluation, and obtain smaller, noisier gains than open-answer benchmarks.
- Checkpoint stability: The bounded-answer benchmarks have mean checkpoint standard deviation 2.69 versus 0.92 for open-answer benchmarks, a 2.91× ratio.The statistic is computed across the last 30 training checkpoints.
- Checkpoint stability: GRPO’s evaluation moves about three times more between late-stage checkpoints on bounded-answer benchmarks than on open-answer benchmarks.The passage presents this as consistent with a noisier per-update gradient from the spurious component.
- Performance gain by regime: GRPO’s gains are +7.17 on MATH-500 and +2.70 on GSM8K, compared with −0.24 on MMLU-math, +2.26 on AQuA, and +5.28 on SAT-Math.The gains use the stated regime-specific correction for bounded-answer benchmarks.
- Performance gain by regime: Bounded-answer gains are visibly smaller and noisier than open-answer gains.The passage relates this pattern to dilution of the GRPO update by the spurious component.
- Estimator comparison: The standard GRPO estimator computes per-rollout magnitude from within-group standardization, while SIGNBALANCE replaces that composition-dependent design with force-balanced rescaling.The pseudocode identifies GRPO’s dependence on (n+, n−) and SIGNBALANCE’s stop-gradient ratio and zero-mean balance.
G Reasoning trace comparisons on bounded-answer benchmarks
Trace comparisons on bounded-answer tasks show cases where GRPO performs correct mathematical reasoning but selects the wrong answer letter, whereas SIGNBALANCE selects the correct letter in the matched example.
- Setup: The comparison uses two disagreement cases: one from MMLU-math and one from SAT-Math.The traces compare policies trained with GRPO and SIGNBALANCE on bounded-answer problems.
- MMLU-math: The matched SIGNBALANCE trace reaches the correct letter C after the same factor derivation.The passage reports the correct answer as C for the SIGNBALANCE output.
- SAT-Math: In the SAT-Math example, the gold answer is A, while the GRPO output is described as producing a closely reasoned but incorrect letter selection.The prompt asks for the percentage expression corresponding to b favorable votes among 435 members.
- Interpretation: The interpretation links these errors to reinforcement for correct letters regardless of whether the letter was reached through reasoning or guessing.This mechanism is presented as allowing a letter-selection shortcut that does not faithfully follow the reasoning.