F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Daniil Plyusov; Alexey Gorbatovski; Boris Shaposhnikov; Viacheslav Sinii; Alexey Malakhov; Daria Korotyshova; Daniil Gavrilov

Source-linked AI summary

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daria Korotyshova, Daniil Gavrilov

arXiv:2602.06717v2Feb 6, 2026cs.LGcs.AI

TL;DR

Finite RLVR groups can reinforce sampled correct solutions while missing rare correct trajectories and shrinking unsampled-correct mass. The paper derives these effects, proposes F-GRPO’s difficulty-aware scaling, and reports improved math and OOD pass@256 across GRPO, DAPO, and CISPO without extra rollout cost.

Problem
Finite group sampling can create active updates that miss rare-correct behavior and redistribute probability away from unsampled correct outcomes.
Method
The paper derives non-monotonic tail-miss probabilities, analyzes categorical redistribution, and proposes F-GRPO as difficulty-aware scaling for group-relative objectives.
Results
F-GRPO consistently improves math and OOD pass@256 across GRPO, DAPO, and CISPO at fixed N=8 without additional rollout cost.
Takeaways & Limitations
Difficulty-aware scaling can preserve broader correct-solution coverage while retaining the benefits of finite-group RLVR updates.
Takeaways & Limitations
The categorical concentration-zone boundaries depend on the initial distribution and should not be interpreted as quantitative predictions for LLM training.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, computational limits often rule out very large groups, so training proceeds with finite rollout sets that can reinforce only the correct behavior they expose. At practical group sizes, updates can miss rare-correct trajectories while still containing mixed rewards, concentrating probability on more common sampled solutions. We derive the probability of such prompt-local tail-miss events as a function of group size, showing non-monotonic behavior, and in the categorical abstraction characterize how unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware scaling coefficient, inspired by Focal loss, that down-weights updates on high-success sampled groups. Empirically, categorical simulation illustrates the same effect in the categorical setting, Maze provides a single-solution test, and LLM experiments include a representative GRPO group-size sweep together with fixed-$N$ transfer across GRPO, DAPO, and CISPO. On Qwen2.5-7B at $N{=}8$, our method improves average math pass@256 from 64.1 $\rightarrow$ 70.3 (GRPO), 69.3 $\rightarrow$ 72.5 (DAPO), and 73.2 $\rightarrow$ 76.8 (CISPO); OOD pass@256 also improves in all three cases, without increasing group size or computational cost.

1 Introduction

RLVR can sharpen accessible solutions while reducing diversity, making finite group size a critical design choice. The paper analyzes this trade-off and proposes F-GRPO, which improves high-budget evaluation while preserving low-budget accuracy across methods.

RLVR can sharpen the output distribution toward solutions already accessible to the base model, with potential underperformance at large sampling budgets.
Finite group size is a critical design choice because prior work gives conflicting guidance on small, moderate, and large groups.
F-GRPO at N=8 recovers pass@256 while maintaining pass@1, using 4× less compute than N=32.
Theoretical analysis shows active updates can omit rare-correct subsets, with tail-miss probability varying non-monotonically with group size.
Categorical analysis shows unsampled-correct mass can decrease even when total correct mass increases.
F-GRPO applies difficulty-aware advantage scaling across GRPO, DAPO, and CISPO, improving math and OOD pass@256 at fixed N=8 without additional rollout cost.

2 Preliminaries

The paper formalizes RLVR as outcome-level reward learning from finite groups of sampled rollouts. It reviews group-relative optimization and a categorical policy abstraction for analyzing how sampled updates redistribute probability mass.

RLVR samples complete trajectories for each prompt and assigns rewards based only on final correctness.
GRPO computes group-relative advantages from multiple rollouts, using a clipped surrogate objective with token-level importance ratios.
When all sampled rewards are identical, GRPO produces zero learning signal because the reward standard deviation is zero.
The categorical framework partitions a finite action space into correct and incorrect actions and tracks sampled and unsampled probability mass.
The categorical analysis uses one-step surrogate updates and distinguishes prompt-level success probability from categorical positive mass.

3 Finite-Sampling Bias in Group-Relative RLVR

Finite group sampling creates two prompt-local risks: active updates may miss rare correct regions, and redistribution may reduce unsampled-correct mass. These effects explain why intermediate group sizes can improve accuracy while narrowing coverage.

Finite groups can produce active updates that omit low-mass correct regions, while local redistribution can move probability away from unsampled correct outcomes.
3.1 Tail-miss probability and the group size trade-off: A target correct subset is rare when its probability is small relative to the prompt’s overall success probability.
3.1 Tail-miss probability and the group size trade-off: The tail-miss event combines an active group with zero samples from the target correct subset.
3.1 Tail-miss probability and the group size trade-off: Tail-miss probability is non-monotonic in group size because activity increases with N while coverage of the target subset also improves with N.
Categorical updates can increase total correct mass while decreasing unsampled-correct mass within the correct set.
The redistribution mechanism combines direct downward drift on unsampled-correct mass with normalization coupling that transfers probability toward sampled-correct actions.
Computational cost and the need for active groups constrain how far group size can be increased, leaving intermediate-N risk in practice.

4 F-GRPO: Focal weighting for Group-Relative Policy Optimization

F-GRPO uses the observable empirical success rate to scale each prompt’s group-relative update, reducing the contribution of high-success groups while preserving compatibility with multiple optimizers.

Observable success signal: F-GRPO uses the empirical success rate bµpos(x) = X/N as an observable summary of how much correct behavior a sampled group exposes.Small nonzero values indicate sparsely observed correct behavior, while large values indicate many positive rollouts support the update.
Focal weighting: The Focal weight decreases with empirical success, attenuating high-success groups more strongly than mixed groups with sparse positive samples.For γ > 0, the weight approaches zero as empirical success approaches one.
Integration with group-relative methods: F-GRPO multiplies the group-level advantage without changing reward values, clipping bounds, or importance-weight construction.This makes the weighting compatible with objectives such as DAPO and CISPO, producing F-DAPO and F-CISPO variants.
Focal weighting: The weight is applied uniformly to all rollouts from the same prompt as a single scalar g(x) ∈ [0, 1].The modification requires no additional network, and γ is the only new hyperparameter.

5 Experiments & Results

Experiments validate the proposed mechanism across categorical simulation, single-solution Maze, group-size sweeps, and fixed-N transfers across RLVR optimizers. Focal weighting improves pass@256 broadly while requiring no additional rollout cost, although categorical concentration-zone boundaries are not quantitative predictions for LLM training.

Categorical simulation: Categorical simulation shows total correct mass can increase while retained positive mass collapses at intermediate group sizes.Small and large groups retain more positive mass, whereas the intermediate concentration zone combines rapid Qpos growth with severe Mret decline.
Categorical simulation: N=131,072 maintains Mret ≈1 throughout training, while γ=1 improves retention particularly in the concentration zone.The result is consistent with a predicted tail-miss probability below 10−3 for a non-anchor correct action under the initial distribution.
Scope: The categorical concentration-zone boundaries depend on the initial distribution and should not be interpreted as quantitative predictions for LLM training.The supported conclusion is qualitative: intermediate group sizes can yield the lowest retained positive mass.
Single-solution Maze: F-GRPO improves final Maze pass@1 for every tested group size, from 74.4–93.6 versus 65.6–75.8 for GRPO.With γ=0.5, pass@256 also reaches 75.9–96.5 versus 67.3–76.9, and the setting has exactly one correct sequence per prompt.
Group-size regimes: GRPO pass@256 varies non-monotonically with group size: intermediate sizes improve math pass@1 but reduce pass@256, with strongest degradation around N=8.At N=32, pass@256 partially recovers while math pass@1 continues improving; the sweep probes representative regimes rather than exhaustively mapping N.
Group-size regimes: At N=8, F-GRPO matches GRPO at N=32 on pass@256 using 4× fewer rollouts: 70.3 versus 70.1 on math and 63.3 versus 61.7 on OOD.Pass@1 has a modest in-domain trade-off but improves on OOD tasks.
Cross-optimizer transfer: Across nine method-model combinations at N=8, Focal weighting improves math and OOD pass@256 by average +3.9 and +4.1, respectively.Math pass@1 is preserved or improved in 9/9 cases, while OOD pass@1 improves in 7/9 cases.

6 Related Work

Prior work debates how RLVR changes solution diversity and how rollout group size should be chosen. This paper frames finite-sampling effects, including rare-mode undercoverage and success-based reweighting, as complementary concerns.

RLVR may improve pass@1 while degrading pass@k for large k, consistent with concentration onto fewer solutions.
The optimal rollout count remains debated, with arguments favoring both N=2 for efficiency and large groups for coverage.
This work complements prior group-size analyses by studying finite-sampling effects in group-relative methods, including rare-mode undercoverage within prompts.
Related approaches address difficulty-aware reweighting, rare-trajectory rewards, entropy regulation, and token-level concentration.

7 Conclusion

The paper connects finite-group sampling to prompt-local tail misses and redistribution within the correct set, then proposes Focal weighting as a fixed-N mitigation. Across simulations and experiments, it improves pass@256 while generally preserving or improving pass@1 without extra rollout cost.

Finite-group sampling can produce active rare-mode misses at intermediate N, while small and large groups are protected by inactivity and coverage, respectively.
Unsampled-correct mass can shrink even as total correct mass grows, concentrating probability on sampled-correct actions.
Categorical simulation, single-solution Maze, and a representative GRPO sweep provide empirical tests of the proposed finite-sampling picture.
Focal weighting improves pass@256 across GRPO, DAPO, and CISPO while preserving or improving pass@1 in most reported settings, especially on OOD tasks.
Focal weighting regulates which prompts contribute most strongly to group-relative updates.
The evaluated objectives include GRPO, DAPO’s asymmetric clipping, and CISPO’s direct importance-weight clipping.
The total-correct-mass analysis decomposes change into sampled-action effects and an unsampled coupling whose magnitude decreases with rollout size.

D Proof of Lemma 3.1

The proof partitions rollouts into rare-correct, remaining-correct, and incorrect regions, then derives the conditional probability that an active group misses the rare-correct subset. Its competing activity and coverage factors produce non-monotonic dependence on group size.

Each rollout belongs to the target correct subset E, the remaining correct region, or the incorrect region.
The probability that no rollout samples E is (1 − τ)^N.
A group is inactive when all rollouts are correct outside E or all are incorrect, with probabilities (µpos − τ)^N and (1 − µpos)^N.
Figure 3 plots conditional tail-miss probability against N across µpos values and rare-correct mass fractions, with stars marking peaks.
The proof uses softmax first-order identities and subset-mass balance to connect categorical updates with changes in correct-region probability.
For unsampled correct actions, the derivation sets S = U ∩ P and uses their zero reward to obtain the corresponding mass change.

G Detailed Term Analysis for Proposition 3.2

The term analysis explains how positive-reward batches can reduce unsampled-correct mass through direct baseline drift and normalization coupling. These effects can coexist with an increase in total correct mass.

The direct drift term −SR Upos,2 lowers unsampled-correct mass when the batch reward mean is positive.
Normalization coupling combines sampled-correct gains, sampled-incorrect losses, and baseline-driven changes among unsampled actions.
For high-success prompts with positive-reward batches, concentrated sampled-correct mass can further drain unsampled-correct mass through normalization.
The total correct mass can nevertheless remain positive because sampled-correct promotion and sampled-incorrect demotion outweigh the unsampled coupling.

H Monotonicity of Sampled Distinct Mass Conditioned on X

Conditioned on observing k correct rollouts, expected distinct sampled-correct mass is non-decreasing in k, while expected sampled-incorrect mass is non-increasing. Under standard RLVR rewards, expected sampled reward is therefore non-decreasing in the observed correct count.

Setup: The appendix defines X as the number of correct rollouts and A and B as distinct sampled correct and incorrect sets.Their corresponding sampled masses provide trajectory-level analogues of the categorical quantities.
Conditional Distributions: Conditioned on X = k, correct and incorrect rollouts are independently sampled from their respective conditional distributions.The correct draws follow qpos over C, while the N − k incorrect draws follow qneg over the complement.
Monotonicity: As X increases, expected distinct sampled-correct mass is non-decreasing and expected distinct sampled-incorrect mass is non-increasing.The proof uses the probability that a rollout appears at least once among the corresponding conditional draws.
Reward Corollary: Under Rc > Rw and Rw ≤ 0, expected sampled reward is non-decreasing in the observed correct count.The positive-mass term rises with k, while the non-positive incorrect-mass term also contributes non-decreasingly as incorrect mass falls.
Focal Weight: The Focal weight g(x) = (1 − bµpos(x))^γ attenuates high-success groups more strongly, changing the relative weighting of active updates.Figure 4 displays scaled GRPO advantage magnitudes for correct and incorrect rollouts against success probability.

J Categorical Simulation Details

The categorical simulation uses a large action space with rare non-anchor correct actions to test tail-miss behavior across group sizes. It predicts and observes concentration when intermediate groups omit rare correct mass, while sufficiently large groups preserve it.

Setup: The simulation uses a softmax policy over 128,000 actions, with 10,000 correct actions rewarded +1 and 118,000 incorrect actions rewarded −1.Initialization gives total correct mass Qpos ≈0.63 and a designated anchor correct action probability of approximately 4.7 × 10^-4.
Tail-Miss Analysis: The tail-miss probability rises for small N, plateaus near 1 at intermediate N, and declines toward zero only for N ≳215.For a typical non-anchor correct action, τE ≈ 6.3 × 10^-5.
Simulation Results: At N = 217 = 131,072, the tail-miss probability is below 10^-3, predicting preservation of probability mass on non-anchor correct actions.This was the only configuration maintaining Mret ≈1 throughout training.
Tail-Miss Analysis: Intermediate group sizes maximize the probability that a correct action is unsampled while the batch contains mixed rewards.This non-monotonic behavior explains the concentration zone.
Training Procedure: Each training step samples N actions, computes group-relative advantages, and updates logits; Focal weighting scales the objective by g = (1 − bµpos)^γ.The sweep covers N ∈ {2, 4, …, 131,072} and γ ∈ {0, 1} for 1,000 steps over four seeds.

K Maze Experiment Details

The Maze experiment tests whether evaluation coverage can degrade even when every prompt has one correct sequence, then evaluates F-GRPO across group sizes and training dynamics. F-GRPO improves final coverage and reduces GRPO’s early large-K degradation.

Environment: Maze uses procedurally generated 17×17 mazes, with 1M training mazes and 256 held-out mazes, using an approximately 3M-parameter decoder-only Transformer.The model emits symbolic navigation actions followed by a termination token.
Training Procedure: Training applies SFT initialization followed by on-policy RL with N ∈ {4, 8, 16, 32, 64, 128} and reports final results after 20K RL steps.The final Maze table reports held-out-set percentages for GRPO and F-GRPO with γ = 0.5 or 1.0.
Final Results: F-GRPO improves final pass@1 and pass@K while reducing the early training-time drop in pass@K across group sizes.The reported diagnostic uses −log(pass@K), where lower is better.
Evaluation: Each prompt has one target action sequence, so pass@K measures the probability that at least one of K samples reaches that sequence.This isolates target-sequence coverage from within-prompt alternative-solution diversity.
Training Dynamics: GRPO often shows an early evaluation pass@K drop followed by recovery, consistent with reduced target-sequence coverage from finite-group updates.This dynamics evidence complements the final pass@K versus group-size comparison.