Source-linked AI summary
Self-Hinting Language Models Enhance Reinforcement Learning
Baohao Liao, Hanze Dong, Xinxing Xu, Christof Monz, Jiang Bian
TL;DR
Sparse terminal rewards can make GRPO stall because identical group outcomes collapse relative advantages and remove learning signals on hard prompts. SAGE injects privileged, self-generated hints during on-policy training to reshape rollout diversity without changing the reward, then deploys the no-hint policy. Across the reported benchmarks and models, SAGE consistently outperforms GRPO, including gains of +2.0 on Llama-3.2 and +1.3 on Qwen3.
Problem
Under sparse terminal rewards, GRPO can produce identical rollout rewards on hard prompts, collapsing relative advantages and making the policy-gradient estimate zero.
Method
SAGE uses privileged hints derived from reference solutions to condition on-policy training rollouts, while preserving the task reward and removing hints at test time.
Results
SAGE consistently achieves the highest average performance across three base models, improving over GRPO by +2.0 on Llama-3.2 and +1.3 on Qwen3.
Takeaways & Limitations
Online self-hinting remains better calibrated to the learner than fixed hints, and SAGE improves sample efficiency and final accuracy on challenging reasoning benchmarks.
Takeaways & Limitations
SAGE has the highest training cost among the compared reinforcement-learning methods because it may generate hints across multiple strength levels for complex prompts.
Abstract
from arXiv · showhide
Group Relative Policy Optimization (GRPO) has recently emerged as a practical recipe for aligning large language models with verifiable objectives. However, under sparse terminal rewards, GRPO often stalls because rollouts within a group frequently receive identical rewards, causing relative advantages to collapse and updates to vanish. We propose self-hint aligned GRPO with privileged supervision (SAGE), an on-policy reinforcement learning framework that injects privileged hints during training to reshape the rollout distribution under the same terminal verifier reward. For each prompt $x$, the model samples a compact hint $h$ (e.g., a plan or decomposition) and then generates a solution $τ$ conditioned on $(x,h)$. Crucially, the task reward $R(x,τ)$ is unchanged; hints only increase within-group outcome diversity under finite sampling, preventing GRPO advantages from collapsing under sparse rewards. At test time, we set $h=\varnothing$ and deploy the no-hint policy without any privileged information. Moreover, sampling diverse self-hints serves as an adaptive curriculum that tracks the learner's bottlenecks more effectively than fixed hints from an initial policy or a stronger external model. Experiments over 6 benchmarks with 3 LLMs show that SAGE consistently outperforms GRPO, on average +2.0 on Llama-3.2-3B-Instruct, +1.2 on Qwen2.5-7B-Instruct and +1.3 on Qwen3-4B-Instruct. The code is available at https://github.com/BaohaoLiao/SAGE.
1. Introduction
GRPO can stall on hard prompts when sparse terminal rewards make all rollouts in a group identical, collapsing advantages and eliminating updates. SAGE uses privileged hints during training to reshape rollout outcomes while preserving the task reward and removing hints at test time.
- Motivation: GRPO may produce identically rewarded rollout groups on difficult prompts, causing centered advantages and minibatch policy-gradient estimates to become zero.This is a finite-sample failure: the expected objective need not be flat, but the estimator provides no learning signal.
- Motivation: Existing remedies can bias training toward easier prompts or introduce context and distribution mismatch.These approaches include skipping degenerate updates, adaptive sampling, curriculum scheduling, and externally generated candidates.
- SAGE: SAGE conditions training rollouts on a lossy-compressed reference solution while keeping the task reward unchanged and deploying the no-hint policy at test time.Hints reshape the rollout distribution to increase mixed outcomes within finite groups.
- SAGE: SAGE schedules hint strength when within-group rewards collapse and periodically refreshes self-hints to maintain calibration to the learner.The framework uses policy-dependent hinting as an automatic curriculum and avoids relying on fixed hints alone.
2. RL with Privileged Hinting
Under sparse rewards, hard prompts often yield no positive rollout and therefore no GRPO learning signal. Privileged hints alter the rollout context and are scheduled so that informative, non-degenerate groups become more likely while deployment remains hint-free.
- GRPO stall: Vanilla GRPO stalls on hard prompts when finite groups contain no positive samples and within-group advantages collapse.The intervention changes the rollout distribution while preserving the original reward definition.
- GRPO stall: 42?
- Privileged hinting: Hints are lossy compressions of reference trajectories appended to prompts as additional context during training.Hint strength ℓ controls how much information about the reference trajectory is provided, with ℓ = 0 representing no hint.
- Policy-dependent scheduling: A policy-dependent scheduler activates positive hint strength only when a probe group collapses.With hints, the effective success rate can rise so that non-degenerate groups become common and training receives updates.
- Training and deployment: Training samples rollouts from the hint-conditioned policy, computes GRPO advantages, and sets ℓ = 0 with h = ∅ at test time.The resulting deployment policy is the no-hint policy.
3. Analysis
The analysis characterizes standardized GRPO as a gated update process whose gate opens when a rollout group contains mixed outcomes. SAGE targets this gate by increasing hard-prompt success probabilities and adapting hint distributions to the current policy.
- Gated updates: GRPO prompt updates are effectively gated by whether a rollout group contains mixed outcomes.The prompt-level update magnitude is dominated by the event s > 0, while the advantage energy collapses when s = 0.
- Gate probability: Pr[s > 0 | x, h] = 1 − (1 − pθ(x, h))^G − pθ(x, h)^G, and it is approximately Gpθ(x, h) when pθ(x, h) ≪ 1.Thus, sparse success probabilities make non-degenerate groups rare.
- Hint calibration: SAGE should choose hint strength to move hard prompts out of the Gpθ ≪ 1 regime without pushing pθ close to 1.Both very low and near-certain success rates can reduce the frequency of mixed-outcome groups.
- Hint calibration: The calibrating hint distribution depends on the current policy, so online updates to the hint distribution can reduce gate mismatch during training.A fixed hint distribution generally cannot remain near-optimal as θ changes.
- Hint sampling: At a fixed mean success rate, additional randomness across hints can reduce the expected frequency of non-degenerate groups.The analysis motivates sampling one hint realization per prompt per epoch and allocating compute to rollout groups or strength scheduling.
- Empirical validation: Online self-hinting, rather than training without hints, steadily improves performance on extremely hard prompts across six benchmarks.The figure compares hint levels and training dynamics using 32 rollouts per prompt per step.
- Implications: The proposed operational strategy is on-policy conditioning, collapse-triggered hint scheduling, and online self-hint refreshes.These components are presented as the practical implementation of the gate-opening analysis.
4. Design of SAGE
SAGE keeps hint-conditioned rollouts on-policy by including the hint in both sampling and log-probability evaluation, while online hinting and policy-dependent scheduling adapt training to the learner’s current needs. Its no-positives trigger increases hint strength when GRPO groups collapse, and deployment removes hints.
- On-policy hint conditioning: Including h in both rollout conditioning and log-probability evaluation preserves an on-policy objective; dropping h from the log-probability creates an off-policy-like mismatch.The mismatch is reported as less stable under sparse rewards.
- Online self-hinting: Online hints are refreshed from the current policy, unlike fixed hints initialized once or external teacher hints, to remain calibrated during training.The evaluated variants distinguish fixed privileged, online privileged, and external teacher hints.
- Policy-dependent scheduling: Hint strength ℓ ranges from 0 to L, with ℓ=0 representing deployment without hints; policy-dependent scheduling raises ℓ when recent rollouts provide insufficient learning signal.SAGE-LIGHT uses an epoch-level success threshold, whereas SAGE probes strengths and selects a rollout once the probe group contains a positive sample or reaches maximum strength.
- Deployment: At deployment, SAGE sets ℓ=0, h=∅, and runs the no-hint policy πθ(· | x).Hints are used during training to reshape rollout sampling, not as privileged information at test time.
- Policy-dependent scheduling: The no-positives trigger increases hint strength only for prompts whose probe group has no positive rollouts, directly targeting collapsed GRPO advantages and vanishing policy-gradient estimates.This local trigger is more reactive than epoch-level scheduling but requires additional probe rollouts.
5. Empirical Results
Across three LLMs and in- and out-of-distribution benchmarks, SAGE consistently achieves the strongest average performance while preserving reinforcement-learning behavior. Its gains over GRPO are linked to using more hard prompts, while adaptive and online self-hinting improve performance at additional computational cost.
- Main results: SAGE achieves the highest average performance across three base models, improving by +6.1 on Llama-3.2, +4.5 on Qwen2.5, and +4.2 on Qwen3.These results aggregate performance across the reported benchmarks.
- SAGE vs. SFT: SAGE preserves reinforcement-learning characteristics, whereas SFT underperforms even the base model because it tends to overfit training data.SAGE selectively sharpens the model distribution toward correct trajectories.
- SAGE vs. GRPO: SAGE uses more hard prompts than GRPO, yielding 10% more prompt utilization and a +2.0 gain on Llama-3.2, while Qwen3 gains +1.3 with 0.3% more prompts.Hard prompts that provide no GRPO signal contribute to the larger gains on weaker models.
- Baseline comparisons: SAGE avoids the instability of LUFFY and the limited exploration of Scaf-GRPO while consistently outperforming both methods.LUFFY can exhibit excessive entropy, oscillatory response lengths, or low initial rewards; Scaf-GRPO has the lowest entropy.
- Efficiency: SAGE-LIGHT achieves slightly lower accuracy but requires 53% of SAGE's training cost, while full SAGE incurs additional latency from on-the-fly hint generation.SAGE can sample hints across multiple levels for highly complex prompts.
- Generalization: SAGE and SAGE-LIGHT consistently achieve the best and second-best accuracy, respectively, on out-of-distribution benchmarks.The pattern is similar to that observed on in-distribution tasks.
- Hinting ablations: Online self-hinting outperforms offline self-hinting with multiple hints by 2.0 points, while multiple offline hints improve standard offline self-hinting by +0.9.The ablation varies hint diversity across training and compares fixed versus continually refreshed hints.
6. Related Work
Prior work addresses GRPO’s sparse-reward degeneracy mainly by reshaping data collection or adding guidance, but these strategies can bias training or create context and distribution mismatch.
- Data resampling and external guidance: Data resampling and filtering are widely used to address collapsed groupwise advantages in online RL for LLMs.Common approaches skip degenerate groups and resample or upweight prompts, improving efficiency while biasing training toward prompts with non-trivial success probability.
- Data resampling and external guidance: Adaptive sampling and curriculum schedules allocate more rollouts to difficult prompts to mitigate sparse-reward failures.These methods reshape the training distribution by directing additional sampling toward challenging prompts.
- Data resampling and external guidance: Offline data and externally generated candidates provide guidance for bootstrapping learning but can introduce context or distribution mismatch.Examples include candidates generated by stronger models and other offline guidance strategies.
- Privileged hinting and SAGE: Heuristic batch surgery augments rollout batches with hinted trajectories after detecting failure, mixing hinted and unhinted contexts within a group.This context mixing can blur the interpretation of groupwise comparisons.
7. Conclusion
The paper identifies finite-sample GRPO degeneracy under sparse binary rewards and proposes SAGE to reshape rollouts with privileged hints while preserving the reward and removing hints at inference.
- Conclusion: SAGE addresses GRPO’s finite-sample collapse when identical sparse rewards make standardized advantages and minibatch gradients vanish.The failure occurs on hard prompts with identical group rewards, especially all-zero groups.
- Conclusion: SAGE injects reference-solution-derived hints during training to shift the rollout distribution without changing the original reward definition.A policy-dependent schedule gates hint strength based on detected group collapse.
- Conclusion: Inference uses the no-hint policy, and experiments report improvements across tasks.The method therefore applies privileged hinting during training but removes it at deployment.
Impact Statements
The work is intended to improve RL training for LLMs on verifiable tasks, while its use for harmful verifiable objectives presents a misuse risk.
- Impact Statements: SAGE can reduce training cost and improve RL stability for LLMs on verifiable tasks.This is the stated intended impact of the work.
- Impact Statements: A stated risk is misuse to optimize harmful verifiable objectives.The risk concerns applying the method to objectives that can produce harmful outcomes.
A. Illustration of Privileged Hinting
The illustration shows how training-only privileged hints guide a model toward the correct algebraic path for a verifiable divisibility problem without changing the verifier or terminal reward. Progressive hints move the model from a failed representation toward cancellation and divisor enumeration, yielding the correct answer.
- Prompt: The case study asks for the sum of integer bases b > 9 for which 17b divides 97b.The example uses a verifiable divisibility question over base-b numerals.
- Reference solution: The reference solution rewrites 17b and 97b as b + 7 and 9b + 7, reducing divisibility to b + 7 dividing 56.This converts the base representation into a simple linear divisibility condition.
- Privileged hints: Privileged hints are used only during training and progressively guide the model from a key concept to intermediate algebraic steps.The levels range from rewriting the numerals and simplifying the condition to canceling the b-term and enumerating divisors.
- Representative model behaviors: Without a hint, the model may mis-expand 97b and receive reward 0 after an incorrect terminal decision.The failure illustrates how an incorrect representation path can prevent a successful rollout.
- Representative model behaviors: Stronger hints steer the model to the cancellation 9b + 7 − 9(b + 7) = −56, after which it can recover bases 21 and 49 and sum 70.The correct path uses divisor enumeration after the cancellation step.
- Operational role of privileged hinting: Hints reshape the rollout distribution so useful trajectories become more likely, increasing mixed outcomes and preventing degenerate GRPO advantages from collapsing.The verifier reward remains unchanged, while finite-group sampling is more likely to include an informative trajectory.
C.1. Training settings
The experiments use OpenRLHF for SFT and reproduce LUFFY and Scaf-GRPO with open-source implementations. GRPO shares SAGE’s training settings.
- SFT uses OpenRLHF with a learning rate of 5e-5, batch size 64, 10% warmup, and 3 epochs.
- GRPO uses the same training settings as SAGE.
- LUFFY is reproduced with batch size 128 and PPO minibatch size 64, fixed across reinforcement-learning methods.
- Scaf-GRPO is reproduced using its open-source implementation.
C.2. Evaluation settings
The appendix specifies evaluation settings and formalizes when GRPO’s standardized signal disappears. Degenerate groups have identical rewards, zero variance, and zero advantage energy.
- Evaluation settings: 8192-token responses, temperature 0.6, and top-p 0.95 are used for Table 1; other evaluations use 2048-token responses.
- Proof scope: The appendix supplies detailed proofs for Section 3.1 and additional analysis.
- Standardized GRPO: s^2 = 0 exactly when all binary rollout rewards are identical, meaning the group is either all-zero or all-one.
- Standardized GRPO: The probability of a non-degenerate group is 1 − (1 − p)^G − p^G.
- Standardized GRPO: For G ≥ 2, this probability is maximized at p = 1/2 and approximates Gp when p is small.
D.4. Proof of Proposition 3.3
The proof characterizes how hint-conditioned success probabilities determine GRPO signal energy and identifies calibrated difficulty as optimal. It also shows that fixed hint distributions can become mismatched as the policy changes.
- Optimal hint distributions: The hint objective is linear in the hint distribution, so an optimal distribution concentrates mass on hints maximizing vθ(h).
- Optimal hint distributions: Because vθ(h) is symmetric and strictly concave around pθ(x,h) = 1/2, calibrating hints make success probability as close as possible to 1/2.
- Adaptive hinting: Policy updates change pθ(x,h), so calibrating hints typically change during training; online hint updates can reduce gate-mismatch.
- Hint randomness: At fixed mean success probability, additional variability in pθ(x,h) across hints decreases expected gate-opening frequency and signal energy.
- Sparse-success regime: When p is much smaller than 1, the leading contribution to the group-opening probability scales as Gp, while the p^G term is exponentially smaller in G.
- Non-standardized energy: For non-standardized mean-centered updates, the expected energy is proportional to p(1 − p) and is uniquely maximized at p = 1/2.