Source-linked AI summary
WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning
Gagan Mundada, Zihan Huang, Rohan Surana, Sheldon Yu, Jennifer Yuntong Zhang, Xintong Li, Tong Yu, Lina Yao, Jingbo Shang, Julian McAuley, Junda Wu
TL;DR
GRPO can produce inefficiently long reasoning because terminal correctness does not indicate when to continue or stop, while global length penalties are difficult to calibrate. WS-GRPO trains a preference model from outcome correctness and compares consecutive prefixes to create correctness-aware pseudo-rewards. The method substantially shortens rollouts while remaining competitive with baselines, though its performance varies across mathematical reasoning tasks and models.
Problem
GRPO lacks reliable signals for assigning credit across reasoning steps, while length penalties and outcome-only supervision do not distinguish necessary reasoning from redundant continuation.
Method
WS-GRPO trains a trajectory-level preference model from outcome correctness and converts consecutive-prefix preference margins into pseudo-rewards combined with terminal correctness in GRPO.
Results
WS-GRPO substantially shortens rollouts while remaining competitive with baselines, yielding more concise and reliable solutions.
Takeaways & Limitations
WS-GRPO provides outcome-derived continue/stop guidance that reduces redundant deliberation without relying on process annotations.
Takeaways & Limitations
Performance varies across mathematical reasoning tasks and models, including accuracy decreases and substantial length increases in some DeepMath settings.
Abstract
from arXiv · showhide
Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.
1. Introduction
GRPO can encourage overly long reasoning, while global length penalties and fine-grained supervision are difficult to use. WS-GRPO derives correctness-aware prefix guidance from outcome-only preferences to reduce redundant continuation while preserving necessary reasoning.
- GRPO may favor extended deliberation, increasing computation and overthinking when only terminal correctness is supervised.
- Global length penalties are difficult to calibrate because longer trajectories can reflect problem difficulty and valid reasoning.Penalizing tokens may suppress necessary deliberation on hard problems as well as redundant continuation on easy ones.
- Outcome correctness alone does not identify when continued generation is useful, so necessary reasoning and redundant continuation can receive equivalent supervision.
- WS-GRPO compares consecutive partial trajectories with a preference model trained from trajectory-level correctness to estimate the marginal value of continuation.Its two-stage procedure learns trajectory preferences, then converts preference margins into prefix-level pseudo-rewards combined with terminal correctness in GRPO.
- Theoretical results address preference-model consistency, preference-error robustness, and high-probability generalization.
- Benchmark evaluations show substantially shorter rollouts while remaining competitive with strong baselines and producing more concise, reliable solutions.
2. Preliminaries
Weak supervision represents unavailable fine-grained labels through pairwise preferences and learns posterior preference probabilities. GRPO then uses group-normalized advantages with PPO-style clipping and KL regularization for policy optimization.
- Weakly supervised learning uses inexact or implicit supervision when dense instance-level labels are unavailable.
- A preference model maps ordered pairs to posterior probabilities that one trajectory or item is preferred to another.The model is trained from weak pairwise labels indicating which member of each pair is preferred.
- Bradley-Terry-type negative log-likelihood provides the comparative training objective for aligning the preference model with weak supervision.
- Comparative signals can map coarse supervision to latent cardinal scores for fine-grained assessment when absolute labels are unavailable or ambiguous.
- Group-Relative Policy Optimization (GRPO): GRPO computes group-relative advantages from scalar trajectory returns produced by the policy.
- Group-Relative Policy Optimization (GRPO): The GRPO objective combines PPO-style probability-ratio clipping with KL regularization.
3. Weakly-Supervised-Group-Relative Preference Optimization
WS-GRPO addresses GRPO’s tendency toward overly long reasoning under sparse outcome supervision by learning prefix-level continuation guidance from trajectory-level correctness. It combines preference-derived step signals with terminal rewards and provides theoretical guarantees on estimation, robustness, and generalization.
- Method: WS-GRPO estimates the marginal value of continuing a trajectory by comparing partial trajectories with a preference model trained from final-answer outcomes.This replaces global length penalties with correctness-aware guidance intended to suppress low-utility continuation while preserving necessary deliberation.
- Phase I: Preference Learning: Phase I trains a preference model from ordered pairs of correct and incorrect trajectories, using outcome correctness as weak supervision.Pairs include both orientations, supporting symmetric preference learning.
- Phase II: Policy Optimization: Phase II uses the preference model to generate auxiliary step-level rewards and combines them with binary terminal rewards in the GRPO objective.The resulting objective retains outcome information while incorporating learned reasoning-quality signals.
- Theoretical Analysis: The preference model is intended to recover a reliable trajectory-quality signal from outcome-only supervision through empirical-risk and uniform-convergence analysis.Theorem 3.1 connects preference-model error to sample size and the preference class’s VC-dimension.
- Theoretical Analysis: WS-GRPO’s guarantees bound policy degradation from preference errors and its generalization error under bounded trajectory length and hypothesis complexity.The generalization characterization uses dmax = max(d, dP), with logarithmic factors hidden by ˜O.
4. Experiments
Experiments compare WS-GRPO with GRPO and Dr.GRPO across four reasoning benchmarks and multiple instruction-tuned model families. WS-GRPO generally achieves large efficiency gains with modest accuracy trade-offs, but effectiveness varies by task structure and architecture.
- Setup: Experiments cover ARC, CommonsenseQA, DeepMath, and GSM8K using Qwen and Llama instruction-tuned models.The benchmarks span science, commonsense, mathematical, and grade-school word-problem reasoning.
- Overall Results: WS-GRPO reduces response length and reasoning steps by 50 to 90 percent across most configurations, with modest accuracy trade-offs.The reported pattern is consistent with additional Llama results, though effectiveness varies by dataset structure and model architecture.
- Structured Reasoning: On ARC, Qwen models retain competitive accuracy while reducing length by 86-93% and reasoning steps by 75-83%.Reported Qwen accuracies are 87.9% and 88.6% versus 90.4% and 93.0% for the compared baselines.
- Structured Reasoning: CommonsenseQA results are model-dependent: Qwen2.5-7B-Instruct reaches 83.7% accuracy with 47% fewer steps, whereas Qwen3-4B-Instruct becomes longer and more verbose.The Qwen3-4B-Instruct configuration reports +187% length and +116% steps with 76.8% accuracy.
- Mathematical Reasoning: Mathematical tasks show larger and less consistent trade-offs, including accuracy decreases of 7.5% and 1.6% on GSM8K and model-specific changes on DeepMath.DeepMath ranges from a 97% length increase for Qwen2.5-7B to a 90% reduction for Qwen3-4B-Instruct.
5. Analysis of Reasoning Efficiency
The analysis evaluates preference behavior, efficiency metrics, and training dynamics. WS-GRPO’s learned signals favor shorter trajectories on average and produce higher early accuracy per reasoning step, while efficiency remains task- and training-sensitive.
- Preference Model Efficacy: The preference-model trajectory set mostly contains 3 to 7 reasoning steps while retaining length diversity.GSM8K trajectories are labeled correct or incorrect by final-answer accuracy.
- Preference Model Efficacy: The complementary preference-score gap increases with step count, with a noticeable rise after step 3.This indicates greater separation between correct and incorrect trajectories as more reasoning context accumulates.
- Preference Model Efficacy: Average combined reward decreases with trajectory length, giving the preference model an implicit bias toward shorter trajectories.The analysis identifies this bias as potentially useful for discouraging redundant or looping reasoning.
- Training Dynamics: Step-efficiency is defined as validation Pass@1 accuracy divided by average reasoning steps.The metric is tracked at regular validation checkpoints rather than only at final test evaluation.
- Training Dynamics: Across datasets, WS-GRPO rapidly reaches competitive accuracy while reducing reasoning steps, producing higher accuracy per reasoning step early in training.GRPO and DRGRPO generally improve accuracy more slowly while relying on increasingly long reasoning chains.
- Training Dynamics: WS-GRPO maintains stable efficiency during training, whereas baseline methods frequently behave nonmonotonically.The reported dynamics reflect changing trade-offs between accuracy gains and reasoning cost.
6. Related Works
Related work targets efficient reasoning through length-aware objectives, early exits, process signals, and weak supervision. WS-GRPO instead derives correctness-aware step rewards from outcome supervision.
- Group-Relative Policy Optimization: GRPO variants add length correction, branch-level process signals, or token- and sequence-level advantages within the same framework.
- Weak Supervision: Weakly supervised methods convert outcome signals into approximate process guidance without requiring dense human step labels.
- Weak Supervision: Figure 3 analyzes trajectory lengths, absolute score differences over steps, and average combined reward as a function of trajectory length.
- Efficient Reasoning: Figure 4 measures validation step-efficiency as Pass@1 divided by average reasoning steps, with higher values indicating greater accuracy per step.
- Efficient Reasoning: Efficient-reasoning methods optimize accuracy and length using budgets, progressive limits, reward shaping, bias control, and early-exit training.
7. Conclusion
WS-GRPO addresses overly long GRPO rollouts under outcome-only supervision by converting terminal correctness into prefix-level continuation guidance. The paper reports shorter rollouts while preserving competitive benchmark performance and producing more concise, reliable solutions.
- 7. Conclusion: WS-GRPO converts terminal correctness into correctness-aware guidance over partial trajectories using a trajectory-level preference model and consecutive-prefix comparisons.
- 7. Conclusion: WS-GRPO substantially shortens rollouts while remaining competitive with baselines, yielding more concise and reliable solutions.
A.1. Detailed Proofs
The proofs establish consistency of the weakly supervised preference model, robustness of WS-GRPO to preference errors, and generalization bounds under stated boundedness and regularity assumptions. The analysis tracks preference-model error, trajectory length, model capacities, and confidence parameters.
- Preference Model Consistency: The preference model is analyzed under trajectory-level supervision using empirical-risk convergence and a finite VC-dimension class.
- Preference Model Consistency: The proof uses binary cross-entropy loss, Rademacher complexity, symmetrization, uniform convergence, and strong convexity to relate empirical and population risks.
- Policy Robustness to Preference Errors: The robustness analysis bounds policy-performance degradation from preference errors through reward decomposition and GRPO group-normalized advantages.
- Policy Robustness to Preference Errors: The robustness bound assumes bounded preference outputs, a maximum trajectory length Tmax, group standard deviations bounded below by σmin, and uniformly bounded policy outputs.
- WS-GRPO Generalization Bound: The generalization proof combines policy and preference-reward error sources using Rademacher and concentration bounds with a union bound.
- WS-GRPO Generalization Bound: The resulting generalization error retains the standard statistical learning rate, while preference-specific terms add complexity controlled by λ and model capacities.
A.2. Dataset Details
The paper evaluates validation step-efficiency for Llama models and reports dataset splits alongside accuracy and rollout-efficiency metrics for comparing training methods.
- Validation step-efficiency is defined as Pass@1 accuracy divided by average reasoning steps, with higher values indicating greater accuracy per step.
- Table 2 documents the training, validation, and testing splits for the ARC and CommonsenseQA datasets.
- Table 3 compares GRPO, DRGRPO, and WS-GRPO using test-set Pass@1 accuracy, average reasoning steps, and mean completion length in tokens.
A.3. Length Penalty
The method combines stepwise preference rewards with explicit length control and normalization, while evaluating whether preference models can rank partial reasoning paths by eventual correctness.
- Length control: A length penalty discourages trajectories shorter than 3 steps or longer than 6 steps, using α = 0.1 outside that range.
- Reward normalization: Step-wise rewards are normalized by trajectory length, while mean preference probability summarizes preference across steps for fair comparison.
- Policy optimization: Policy optimization uses eight rollouts per prompt, λ = 0.1 to mix preference and correctness rewards, and α = 0.1 for length control.
- Preference-model evaluation: The fine-grained evaluation scores prefixes of correct and incorrect trajectories, weighting correctly ordered prefix pairs inversely by step index.
- Preference-model evaluation: The prefix-ranking metric increases with trajectory length, indicating that longer trajectories provide more discriminative signal for stepwise reward assignment.