PaperlayerView exact revision on arXiv

Source-linked AI summary

Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning

Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, Dimitris Papailiopoulos

arXiv:2508.09726v1cs.CLcs.LG

TL;DR

RLVR training can inflate reasoning length even when longer chains are not more accurate. GFPO samples larger response groups and selectively trains on concise or token-efficient outputs, reducing GRPO’s length inflation across benchmarks while preserving accuracy; Adaptive Difficulty GFPO offers a stronger efficiency–accuracy trade-off with a modest hard-problem accuracy cost.

  • Problem

    RLVR-trained models can produce unnecessarily long reasoning chains, while token-level normalization may still amplify rewards for long correct responses and drive rapid length growth.

  • Method

    GFPO samples larger response groups, scores and filters candidates, then trains only on the top-k responses selected for brevity or token efficiency.

  • Results

    Across AIME 24/25, GPQA, Omni-MATH, and LiveCodeBench, length-optimized GFPO reduces GRPO’s length inflation by 46.1%–71% while maintaining accuracy; token-efficiency optimization achieves 70.9%–84.6% reductions.

  • Takeaways & Limitations

    GFPO preserves GRPO-level reasoning ability while improving reasoning efficiency, with Token Efficiency GFPO delivering the strongest length reductions and Adaptive Difficulty GFPO improving the efficiency–accuracy trade-off.

  • Takeaways & Limitations

    Adaptive Difficulty GFPO incurs a modest accuracy drop on hard problems, where filtering can discard beneficial longer responses; increasing group size can address this limitation.

Abstract

from arXiv · show

Large language models trained with reinforcement learning with verifiable rewards tend to trade accuracy for length--inflating response lengths to achieve gains in accuracy. While longer answers may be warranted for harder problems, many tokens are merely "filler": repetitive, verbose text that makes no real progress. We introduce GFPO (Group Filtered Policy Optimization), which curbs this length explosion by sampling larger groups per problem during training and filtering responses to train on based on two key metrics: (1) response length and (2) token efficiency: reward per token ratio. By sampling more at training time, we teach models to think less at inference time. On the Phi-4-reasoning model, GFPO cuts GRPO's length inflation by 46-71% across challenging STEM and coding benchmarks (AIME 24/25, GPQA, Omni-MATH, LiveCodeBench) while maintaining accuracy. Optimizing for reward per token further increases reductions in length inflation to 71-85%. We also propose Adaptive Difficulty GFPO, which dynamically allocates more training resources to harder problems based on real-time difficulty estimates, improving the balance between computational efficiency and accuracy especially on difficult questions. GFPO demonstrates that increased training-time compute directly translates to reduced test-time compute--a simple yet effective trade-off for efficient reasoning.

1 Introduction

RLVR training can inflate reasoning length without improving correctness. GFPO addresses this by filtering larger response groups for brevity or token efficiency while preserving accuracy and shifting compute toward training.

  • On AIME 25, longer responses are more likely to be wrong than shorter ones in 72% of questions producing both correct and incorrect responses.
  • GFPO samples larger candidate groups and trains only on the top-k responses retained by a target metric, initially selecting the shortest chains.
  • 46.1% on AIME 25, 59.8% on AIME 24, 57.3% on GPQA, 71% on Omni-MATH, and 57% on LiveCodeBench are GFPO reductions in GRPO length inflation while maintaining accuracy.
  • 70.9% on AIME 25, 84.6% on AIME 24, 79.7% on GPQA, 82.6% on Omni-MATH, and 79.7% on LiveCodeBench are the reductions from token-efficiency optimization.
  • Adaptive Difficulty GFPO dynamically adjusts retained responses using question-difficulty estimates, allocating more training signal to harder questions.
  • GFPO shifts computation from inference to training, using additional candidate responses to produce shorter reasoning chains during deployment.

2 Preliminaries

GRPO simplifies PPO by estimating advantages from multiple sampled responses rather than a value model. Its scalar reward makes jointly optimizing accuracy and brevity difficult and can produce length inflation.

  • GRPO samples multiple responses per question and uses their average reward as the baseline for a clipped surrogate objective.
  • GRPO eliminates PPO’s value model for baseline-advantage estimation while retaining a similar clipped surrogate objective.
  • Token-level loss normalization is the default in several open-source RL libraries and is used in the experiments.
  • A single scalar reward makes it difficult for GRPO to jointly optimize desirable attributes such as brevity and accuracy.

3 Group Filtered Policy Optimization

GFPO broadens the response pool, filters candidates by a chosen attribute, and computes policy updates from the retained subset. Its adaptive variant allocates larger retained groups to harder questions.

  • Group Filtered Policy Optimization: GFPO samples a large response group, scores and sorts candidates, and selects a top-k retained subset for policy training.
  • Group Filtered Policy Optimization: Rejected responses receive zero advantage, while retained responses are normalized using rewards within the selected subset.
  • Filtering metrics: Response-length filtering encourages brevity, whereas token-efficiency filtering retains longer responses when proportionately higher rewards justify them.
  • Filtering metrics: GFPO can incorporate other scoring metrics, including factuality, diversity, and external quality scores.
  • Adaptive Difficulty GFPO: Adaptive Difficulty GFPO dynamically changes k using lightweight estimates of question difficulty to allocate more training signal to harder questions.
  • Adaptive Difficulty GFPO: Easy questions receive fewer retained responses, while hard and very hard questions receive more under adaptive difficulty sampling.
  • Adaptive Difficulty GFPO: Adaptive Difficulty GFPO reduces verbosity on easy examples while preserving accuracy on harder prompts by retaining more reasoning chains.

4 Setup

The experiments fine-tune Phi-4-reasoning with GFPO under a matched GRPO training setup and evaluate accuracy, response length, and excess length reduction across reasoning benchmarks.

  • Models and baseline: Phi-4-reasoning is the SFT baseline, while Phi-4-reasoning-plus is the GRPO-trained baseline used for comparison.
  • Training setup: RL training uses 72k math problems, but the 100-step batch-64 configuration exposes the model to 6.4k problems, matching Phi-4-reasoning-plus.
  • Reward: The training reward combines a length-aware binary accuracy reward with an n-gram repetition penalty, but its length penalty alone does not prevent GRPO inflation.
  • Training setup: GFPO increases group size G to add training-time exposure while keeping retained group size k ≤ 8 for comparison with GRPO.
  • Evaluation: Evaluation covers AIME 25, AIME 24, GPQA, Omni-MATH, and LiveCodeBench using pass@1 accuracy and response-length measures.
  • Evaluation: The study tests whether GFPO and GRPO accuracy differences are meaningful with a paired Wilcoxon signed-rank test.

5 Results

GFPO reduces response-length inflation while preserving reasoning accuracy, with larger sampled groups, reward-per-token filtering, and difficulty-aware retention improving the efficiency–accuracy trade-off.

  • GFPO variants reduce excess response length without statistically significant accuracy drops across multiple benchmarks.Shortest 8/16 reduces excess length by 23.8% on AIME 25, 33% on AIME 24, 23.7% on GPQA, and 31.5% on Omni-MATH.
  • Sampling more responses improves length reductions: increasing group sizes from 16 to 24 yields additional excess-length reductions across AIME 25, AIME 24, GPQA, and Omni-MATH.For Shortest 8/16 to 8/24, the additional reductions are 30.6%, 19.7%, 28.5%, and 20.4%, respectively.
  • The retained-response fraction k/G controls length pressure, with retaining 25–33% of responses observed as optimal before diminishing gains.Shortest 6/24 slightly outperforms Shortest 4/16 despite the same 25% retention fraction, while reducing 8/24 to 4/24 yields only 4.1% additional average excess-length reduction.
  • 5.4 Reinforcing Token Efficiency: Token Efficiency GFPO ranks responses by reward per token and produces the largest excess-length reductions across all evaluated tasks.Reductions reach 70.9% on AIME 25, 84.6% on AIME 24, 79.7% on GPQA, 82.6% on Omni-MATH, and 79.7% on LiveCodeBench.
  • 5.4 Reinforcing Token Efficiency: Token Efficiency GFPO achieves stronger length savings with higher training variance and minor, non-statistically significant accuracy degradations.The reported variance is attributed to conflicting reward and penalty signals on token segments shared by long correct and incorrect responses.
  • 5.5 Adaptive Difficulty GFPO: Adaptive Difficulty GFPO allocates retained responses according to question difficulty and outperforms matched Shortest-k GFPO on four of five benchmarks.It achieves stronger reductions on AIME 25, AIME 24, and GPQA, while Shortest 6/16 is stronger on Omni-MATH; it also attains the highest GPQA accuracy at 70.8%.
  • GFPO mitigates out-of-distribution length inflation, including on LiveCodeBench where it reduces excess length while some variants modestly improve accuracy.Token Efficiency GFPO reduces excess length by 79.7%, while Shortest 8/24 reaches 59.2% accuracy versus 57.7% for SFT and 56.7% for GRPO.
  • Across four of five benchmarks, at least one GFPO variant is Pareto-superior to GRPO on accuracy and response length.On AIME 25, GRPO has slightly higher accuracy, but GFPO variants remain Pareto-optimal through shorter responses; accuracy differences are not statistically significant across tasks.

6 Analysis

GFPO shortens reasoning across difficulty levels while largely preserving accuracy, with token efficiency producing the strongest reductions and adaptive filtering helping on difficult problems.

  • Length Reductions: GFPO reduces response lengths across easy, medium, hard, and very hard questions.Token Efficiency GFPO delivers the largest reductions on easy, medium, and hard questions, while Shortest 8/24 is strongest on very hard questions.
  • Length Reductions: 121.6% excess length reduction on easy questions makes Token Efficiency GFPO shorter than SFT while improving accuracy.Its reductions decline to 79.1% on medium, 63.5% on hard, and 56.5% on very hard questions as longer reasoning becomes more reward-justified.
  • Accuracy by Difficulty: Adaptive Difficulty GFPO matches or exceeds GRPO accuracy on easy, medium, and very hard questions while reducing excess length by 47%.It reaches 90.2% versus 88.4% on medium questions and 27% versus 26.6% on very hard questions.
  • Accuracy by Difficulty: 73.4% hard-problem accuracy lets Shortest 8/24 match GRPO when sampling increases from 16 to 24 responses per question.Larger groups help retain concise, high-quality chains on hard problems.
  • Accuracy by Length: 12k–16k tokens is the hard-problem accuracy sweet spot, while accuracy declines consistently with increasing length.In the longest bin, GFPO variants outperform GRPO at 66.7% versus 52.1% on Hard and 20.3% versus 17.2% on Very Hard.
  • Accuracy by Length: 9× higher prompt difficulty in GFPO’s ≤5k-token bin accompanies only minor accuracy dips, such as 100% to 97%.GFPO also reduces outputs of at least 20k tokens from 32% to 22%, indicating shorter responses can solve harder prompts.
  • Run-to-Run Variation: GFPO variants compress the long tail while retaining distinct accuracy-variability profiles across repeated AIME 25 runs.Token Efficiency has the narrowest spread, Shortest 8/24 closely matches GRPO’s central mass, and Adaptive Difficulty is slightly bimodal.
  • Accuracy–Length Trade-off: GFPO reduces the share of AIME 25 responses at least 15k tokens from GRPO’s 46.8% to 32.7% for Token Efficiency.The corresponding accuracy share of at least 70% decreases modestly for GFPO variants.

7 Related Work

Related work controls reasoning length through loss normalization, reward penalties, inference-time intervention, and rejection sampling; GFPO performs selective filtering within RL updates.

  • GRPO Loss Modifications: Dr. GRPO and DAPO modify GRPO loss normalization to improve token efficiency and training stability.Standard GRPO normalizes within each response, whereas Dr. GRPO uses maximum batch response length and DAPO uses total token count.
  • GRPO Loss Modifications: Token-level normalization can penalize long incorrect chains while boosting long correct ones, driving greater verbosity in strong reasoning models.GFPO instead modifies the advantage through selective learning.
  • Length-Aware Penalties: Length-aware reward penalties discourage long chains through token caps, adaptive penalties, or inverse scaling.GFPO contrasts with these approaches by shaping learning through rejection rather than complex reward engineering.
  • Inference-time Interventions: Inference-time methods control reasoning length without retraining, including voting over the shortest samples and budget forcing.These interventions differ from GFPO’s training-time approach.
  • Rejection Sampling Methods: Prior rejection-sampling methods use multiple solutions for post-training, whereas GFPO integrates rejection sampling directly into the RL update.GFPO filters by length or token efficiency and can adjust filtering according to question difficulty.

8 Conclusion

GFPO extends GRPO by sampling more responses and selectively learning from outputs aligned with brevity or token efficiency. Across reasoning benchmarks, it preserves GRPO-level ability while improving efficiency, with adaptive sampling especially useful on hard problems.

  • Conclusion: GFPO samples more responses and selectively learns from those aligned with brevity or token efficiency.The rejection step acts as implicit reward shaping without complex reward engineering.
  • Conclusion: Across diverse reasoning benchmarks, length-optimized GFPO preserves GRPO-level reasoning ability while substantially improving reasoning efficiency.Token Efficiency GFPO provides the strongest overall length reductions.
  • Conclusion: Adaptive Difficulty GFPO applies test-time scaling only when needed, achieving the best efficiency–accuracy trade-offs on the hardest problems.The paper frames increased training-time compute as producing lasting inference-time efficiency gains.

A Qualitative Examples

The qualitative examples contrast GRPO's repetitive or unfocused reasoning with GFPO variants that reach the same conclusions using shorter or more directed responses. The examples illustrate reduced recomputation, rapid structural recognition, and clearer problem-solving plans.

  • Token Efficiency GFPO: GRPO recomputes and checks the perpendicular-bisector equation five times, whereas Token Efficiency GFPO computes it once.Only three GRPO instances are shown, but the comparison highlights repeated checking versus a single computation.
  • Shortest GFPO: The Shortest 8/24 GFPO response recognizes the medial triangle and nine-point circle quickly, while GRPO becomes bogged down in arc bookkeeping.Both responses arrive at the same conclusion, but the GFPO response avoids the longer bookkeeping process.
  • Adaptive Difficulty GFPO: Adaptive Difficulty GFPO defines one clear approach, while GRPO cycles through factoring, substitutions, symmetry, and vague rotation ideas without reducing uncertainty.The example contrasts a committed solution plan with several abandoned alternatives.

A.1 GPQA Qualitative Examples.

The GPQA examples contrast GRPO’s repeated re-derivations and uncertainty with GFPO responses that reach the same reasoning steps more directly. Adaptive Difficulty GFPO computes the harmonic-oscillator frequencies once, while other GFPO variants reduce repeated derivations.

  • The GRPO response repeatedly rehashes whether the reaction is in the lab frame or center-of-mass frame, while Token Efficiency GFPO makes the assumption once.
  • The harmonic-oscillator derivation concludes that the potential is separable in Cartesian coordinates and corresponds to independent oscillators.
  • Adaptive Difficulty GFPO calculates the harmonic-oscillator frequencies once, whereas the GRPO response recalculates them multiple times.
  • The GRPO response circles back and re-derives velocity addition multiple times, while Shortest 8/24 GFPO performs it once with a small recheck.

B Response Length and Accuracy for Hard and Very Hard Problems

The section reports response lengths and accuracies for hard and very hard problems by dividing responses into four length bins. It compares these distributions across reinforcement-learning methods.

  • Responses to hard and very hard problems are divided into short, mid-low, mid-high, and long length quartiles for average-length comparison.
  • Accuracy is reported across the same four response-length quartiles for hard and very hard problems.
Loading 2508.09726v1…