Source-linked AI summary
Learning to Hint for Reinforcement Learning
Yu Xia, Canwen Xu, Zhewei Yao, Julian McAuley, Yuxiong He
TL;DR
GRPO can suffer advantage collapse when rollout groups receive uniform rewards, and existing hints are often fixed or fail to transfer to the no-hint policy. HiLL jointly trains an online hinter and reasoner, using hint reliance to reward hints that create informative groups and transfer better; experiments report consistent gains over GRPO and prior hint-based baselines, with added latency.
Problem
GRPO yields no learning signal when all sampled rollouts receive the same binary reward, while existing hints may be fixed and may not improve the no-hint policy.
Method
HiLL jointly trains a hinter and reasoner, generating hints from current incorrect rollouts and using hint reliance to define a transfer-weighted hinter reward.
Results
HiLL consistently outperforms standard GRPO and hint-based baselines across benchmarks, achieving the highest average in-distribution math accuracy on both evaluated backbones.
Takeaways & Limitations
Adaptive, transfer-aware hint learning produces signals more likely to improve the original no-hint policy than fixed hinting alone.
Takeaways & Limitations
On Llama-3.2-3B-Instruct, HiLL averages roughly 3.8× GRPO’s per-step wall-clock time because it adds hint generation, re-sampling, reliance estimation, and a hinter update.
Abstract
from arXiv · showhide
Group Relative Policy Optimization (GRPO) is widely used for reinforcement learning with verifiable rewards, but it often suffers from advantage collapse: when all rollouts in a group receive the same reward, the group yields zero relative advantage and thus no learning signal. For example, if a question is too hard for the reasoner, all sampled rollouts can be incorrect and receive zero reward. Recent work addresses this issue by adding hints or auxiliary scaffolds to such hard questions so that the reasoner produces mixed outcomes and recovers a non-zero update. However, existing hints are usually fixed rather than adapted to the current reasoner, and a hint that creates learning signal under the hinted input does not necessarily improve the no-hint policy used at test time. To this end, we propose Hint Learning for Reinforcement Learning (HiLL), a framework that jointly trains a hinter policy and a reasoner policy during RL. For each hard question, the hinter generates hints online conditioned on the current reasoner's incorrect rollout, allowing hint generation to adapt to the reasoner's evolving errors. We further introduce hint reliance, which measures how strongly correct hinted trajectories depend on the hint. We derive a transferability result showing that lower hint reliance implies stronger transfer from hinted success to no-hint success, and we use this result to define a transfer-weighted reward for training the hinter. Therefore, HiLL favors hints that not only recover informative GRPO groups, but also produce signals that are more likely to improve the original no-hint policy. Experiments across multiple benchmarks show that HiLL consistently outperforms GRPO and prior hint-based baselines, demonstrating the value of adaptive and transfer-aware hint learning for RL. The code is available at https://github.com/Andree-9/HiLL.
1 Introduction
HiLL addresses GRPO advantage collapse by jointly learning an adaptive hinter and reasoner, while selecting hints for both informative group outcomes and transfer to no-hint reasoning. It trains the hinter online on the reasoner’s evolving failures and uses hint reliance to weight transfer.
- Motivation: GRPO produces no policy gradient when all rollouts in a group receive the same binary reward.This failure occurs for both all-correct and all-incorrect groups.
- Motivation: Existing hint-based methods add scaffolds to hard questions but typically use fixed hints and primarily reward mixed-outcome groups.Such hints may make problems easier without teaching behavior useful for the original no-hint input.
- HiLL: HiLL jointly trains a hinter and reasoner, generating hints online from each question, an incorrect rollout, and the reference solution.This conditions hint generation on the reasoner’s evolving errors rather than static offline hints.
- HiLL: Hint reliance measures dependence of correct hinted trajectories on the hint, and lower reliance implies stronger transfer to no-hint success.HiLL uses this relationship to construct a transfer-weighted reward for the hinter.
- HiLL: HiLL turns hinting into online co-training over an evolving distribution of reasoner failures.Previously hard questions can become solvable while newly difficult questions define the current capability boundary.
- Results: HiLL consistently outperforms standard GRPO and hint-based baselines on comprehensive benchmarks.The reported contribution emphasizes adaptive and transfer-aware hint learning.
2 Related Work
Prior work addresses GRPO’s uninformative groups through sampling, filtering, reshaping, and input scaffolding. HiLL belongs to the hint-based direction while adding online adaptation and transfer awareness.
- Sampling and filtering: Sampling methods allocate more rollouts toward questions near the model’s capability boundary to recover rare successes under fixed compute.Examples include variance-minimizing allocation, adaptive importance-based scheduling, and knapsack-style optimization.
- HiLL: HiLL’s framework generates candidate hints from failed rollouts, re-samples under each hint, and scores hints by signal creation and transfer.The best hinted group updates the reasoner, while all candidate hints form the hinter’s GRPO group.
- Sampling and filtering: Filtering and reshaping methods skip, clip, downsample, or recalibrate degenerate groups to reduce wasted computation or extract signal.Related approaches include entropy-guided advantage shaping, virtual reward calibration, and curriculum scheduling.
- Privileged hinting: Privileged hinting modifies training inputs with solution prefixes, self-generated hints, teacher hints, progressive hints, or scaffolded reasoning steps.These methods aim to induce correct rollouts on hard questions.
3 Preliminaries
The preliminaries define verifiable-reward reasoning and explain why GRPO can lose its learning signal when rollout groups are uniform. Hard questions are especially likely to produce all-incorrect groups.
- Verifiable rewards: A reasoner samples trajectories for a question, and a verifier assigns each trajectory a binary correctness reward.The per-question success probability is the probability that a sampled trajectory receives reward 1.
- GRPO: GRPO samples G trajectories for one question and computes normalized advantages within that group.The method removes the value critic and relies on relative outcomes among sampled rollouts.
- Advantage collapse: When all G rollouts receive the same binary reward, every relative advantage vanishes and the question produces no gradient.Mixed correct and incorrect outcomes are required for a non-zero within-group learning signal.
- Advantage collapse: Hard questions with small success probability are most likely to yield degenerate groups and produce no learning signal.For small p, the probability of a mixed group is approximately Gp.
4 HiLL: Hint Learning for Reinforcement Learning
HiLL recovers learning signals from all-incorrect GRPO groups by generating and evaluating hints online. It selects hints that create mixed outcomes while favoring correct trajectories likely to transfer to the no-hint setting.
- Framework: HiLL intervenes on all-incorrect groups by generating candidate hints, resampling hinted trajectories, and replacing degenerate groups when a positive-reward hint is found.The reasoner is then updated on the selected original or hinted inputs, while the hinter receives its own GRPO update.
- Failure-Conditioned Hint Generation: The hinter conditions online on the question, an incorrect rollout, and a reference solution, exposing the reasoner’s current error mode for targeted hint generation.This lets hint generation adapt as the reasoner’s failure distribution changes during co-training.
- Signal Creation: A hint creates useful GRPO signal when its hinted rollout group contains both correct and incorrect trajectories.The non-degenerate probability s(ˆph; G) vanishes when all hinted rollouts are correct or all are incorrect and is maximized near balanced outcomes.
- Hint Reliance: Hint reliance measures how much more likely a correct trajectory is under the hinted input than under the original question.It is defined as ρ(τ; q, h) = log πθ(τ | q+h) − log πθ(τ | q), with practical estimates normalized by trajectory length.
- Signal Transfer: The transfer bound states that no-hint success is at least hinted success discounted by exp(−ρc), so lower average reliance gives a stronger transfer guarantee.The bound decomposes reliance into a success-probability log-ratio and a non-negative KL divergence between correct-trajectory distributions.
- Transfer-Weighted Hinter Reward: HiLL trains the hinter with a transfer-weighted reward that multiplies signal creation by a weight that stays at 1 for nonpositive reliance and decays as reliance increases.Invalid hints receive Rfail, while hints with all-correct or all-incorrect hinted groups receive zero signal-creation reward and need no reliance computation.
5 Experiments
HiLL is evaluated against GRPO and hint-based baselines on math and generalization benchmarks, with analyses of signal recovery, transfer weighting, hint style, and computational cost.
- Experimental Setup: HiLL is evaluated with Llama-3.2-3B-Instruct and Qwen2.5-7B-Instruct reasoners, jointly trained with a Qwen3-4B-Instruct hinter.Both policies are trained jointly via GRPO.
- Experimental Setup: The training set contains 15k prompts from OpenR1-Math-220k spanning a wide difficulty range without pass-rate filtering.The prompts include ground-truth answers and reference solutions.
- Experimental Setup: Evaluation covers six in-distribution math benchmarks and GPQA-diamond and MMLU-Pro for generalization, while the hinter is unused at test time.Reasoners are evaluated with temperature 0.6, top-p 0.95, and maximum response length 8,192.
- Main Results: HiLL achieves the highest average accuracy on in-distribution math tasks for both reasoner backbones and also improves out-of-distribution benchmarks over baselines.It outperforms standard GRPO, Scaf-GRPO, and SAGE.
- Signal Recovery and Transfer: Both HiLL variants reduce all-incorrect groups, while transfer weighting keeps hint reliance lower and is associated with stronger no-hint transfer.The comparison tracks all-incorrect ratio and hint reliance over training.
- Signal Recovery and Transfer: A smaller transfer temperature increases signal transfer but reduces signal creation, whereas a larger temperature creates more signal with lower transferability.T=0.3 is reported as a balance between signal creation and signal transfer; all tested values outperform HiLLw/o TW.
- Hint Analysis: Transfer weighting produces shorter, more conceptual hints, while HiLLw/o TW more often sets up algebra directly.HiLL hints suggest high-level strategies without carrying out the computation.
- Computational Cost: HiLL averages roughly 3.8× GRPO’s per-step wall-clock time on Llama-3.2-3B-Instruct and roughly 2.6× on Qwen2.5-7B-Instruct.The overhead is triggered only for all-incorrect groups and therefore scales with their frequency.
6 Conclusion
HiLL addresses two limitations of hint-based approaches to GRPO advantage collapse through adaptive, transfer-aware hint learning. Across eight benchmarks and two backbone LLMs, it consistently outperforms GRPO and prior hint-based baselines while producing shorter, more conceptual hints.
- HiLL jointly trains a hinter and reasoner, generating hints online from the reasoner’s current failures rather than relying on fixed or externally generated hints.This keeps hint generation calibrated as the reasoner improves.
- Hint reliance measures whether hinted success is likely to transfer to the no-hint policy, motivating a transfer-weighted reward for the hinter.The reward favors correct hinted trajectories that remain plausible without the hint.
- HiLL consistently outperforms GRPO and prior hint-based baselines across eight benchmarks and two backbone LLMs.
- Transfer weighting keeps hint reliance low throughout training and naturally produces shorter, more conceptual hints.
A Proof of Proposition 1
The proposition relates hinted and original trajectory distributions through their success probabilities and per-trajectory hint reliance. Its proof decomposes each distribution into success probability and the corresponding conditional distribution over correct trajectories, then uses the resulting identity.
- For a question-hint pair, Ph(τ) and P(τ) denote trajectory distributions under hinted and original inputs, with success probabilities ph and p.
- The proposition is stated for the case ph > 0.
- The proof defines S as the set of correct trajectories and expresses each trajectory distribution as its success probability multiplied by its conditional distribution given success.
- The per-trajectory hint-reliance expression follows by substituting these factorizations into the hinted-to-original trajectory relationship.
B Prompts for Hinter and Reasoner
HiLL’s prompts instruct the hinter to generate a minimal, conceptual hint from the original question, the reasoner’s failed attempt, and a hidden ground-truth solution. Separate prompts specify concise hint formatting and step-by-step reasoner outputs.
- Reasoner prompt: The reasoner’s hinted input appends the hint after the original question without explicit bridging prose such as “Here is a hint to help you:”.
- Prompt templates: The hinter is framed as a Pedagogical Hint Generator for a Mathematical Reasoner tasked with generating a minimal hint.
- Hint generation: Hint generation receives the original question, the reasoner’s failed attempt, and the ground-truth solution for reference while keeping that solution hidden from the reasoner.
- Hint generation: The hinter should identify the failure or misconception and provide a conceptual pointer without revealing the final answer, substituted numerical values, or computed intermediate steps.
- Output format: Generated hints are limited to one to three sentences, while the reasoner is asked to reason step by step and place its final answer within \boxed{}.