Source-linked AI summary
Teach to Learn: Hint Annealing for Self-improving LLM Reasoning
Zile Wang, Zijian Li, Haodong Wang, Jian Liu, Qianli Liu, Lucas Muli, Blaze Chen, Song Guo
TL;DR
Difficult queries can yield only incorrect rollouts, leaving GRPO without a learning signal, while hint-based recovery can favor assisted solving over the unassisted reasoning used at evaluation. HATCH learns to generate and use its own hints, regulating their influence with online weighting and gradient projection. Across three model scales, it reports gains over state-of-the-art methods on mathematical reasoning benchmarks.
Problem
Difficult queries can produce only incorrect rollouts, leaving GRPO without reward contrast, while hint-based methods recover signal under assisted conditions that may favor hinted solving over original-query solving.
Source: §1 · pp. 1–2Method
HATCH is an online single-policy self-teaching framework that learns from generating and using its own hints, with online weighting and gradient projection to regulate auxiliary learning.
Source: §1 · p. 2Results
HATCH outperforms state-of-the-art methods by 1.02 pp on Llama-3.2-1B-Instruct, 2.84 pp on Qwen3-1.7B, and 4.32 pp on Qwen3-8B across nine mathematical reasoning benchmarks.
Source: §1 · p. 2Takeaways & Limitations
Experiments across three model scales show stronger unassisted mathematical reasoning and suggest that models can improve reasoning by learning to provide effective guidance for themselves.
Source: §6 · pp. 9–10Evaluation is limited to mathematical reasoning with verifiable rewards and reference, and experiments use textual inputs and hints; interactive-task extension requires context-suited verification and reference.
Source: §5.4 · p. 9
Abstract
from arXiv · showhide
Group Relative Policy Optimization (GRPO) improves language-model reasoning by comparing verified rewards among multiple solution rollouts for each query. However, difficult training queries can yield only incorrect rollouts, leaving GRPO with no reward contrast or learning signal. Prior hint-based methods construct auxiliary hints from solution evidence and use them to re-solve failed queries, recovering learning signal. Yet the resulting trajectories are typically treated as ordinary solution trajectories despite being generated under an assisted condition unavailable at evaluation. We discover hinted reward shift: recovered reward contrast can concentrate policy updates on hinted trajectories, limiting improvement without hints. This also creates a trade-off: increasing hinted trajectories can accelerate early learning but intensify reward shift later. To address this problem, we propose HATCH (Hint-Annealed Self-Teaching), an online single-policy framework that learns from both generating and using its own hints to improve reasoning without assistance. To mitigate hinted reward shift, we introduce online weighting to anneal the contribution of hinted trajectories. However, learning to generate hints can conflict with improving query solving. We therefore use gradient projection to remove the opposing component of hint-generation updates. Together, these designs support self-improvement by enabling the policy to create learning opportunities for itself and turn them into stronger reasoning without hints. We evaluate our method on mathematical reasoning benchmarks and outperform state-of-the-art methods by 1.02 pp on Llama-3.2-1B-Instruct, 2.84 pp on Qwen3-1.7B, and 4.32 pp on Qwen3-8B.
1 Introduction
Difficult queries can leave GRPO without reward contrast, while hint-based recovery risks favoring assisted solving over the original-query reasoning used at evaluation. HATCH combines self-generated hints, online weighting, and gradient projection to learn from hints while shifting toward unassisted solving.
1 Introduction
- When all sampled rollouts for a difficult query are incorrect, GRPO receives identical rewards and therefore no relative advantages or learning signal for that query.GRPO uses within-group reward differences to reinforce stronger responses and suppress weaker ones.Source: §1 · p. 1
- Prior work allocates or filters rollouts using existing reward contrast, while hint-based methods recover contrast by re-solving difficult queries under assistance that can shift learning away from original-query solving.The latter can make successful solution paths easier to discover, but the assisted condition is unavailable at evaluation.Source: §1 · pp. 1–2
- Recovered hinted contrast can favor hinted trajectories over original-query trajectories, creating hinted reward shift and a trade-off between early gains and sustained unassisted improvement.Hinted and original solving can induce substantially different gradient directions.Source: §1 · p. 2
- HATCH jointly learns hint generation and query solving, using online weighting to regulate hinted trajectories and gradient projection to remove hint-generation updates that oppose solution updates.Its teach-to-learn process uses self-generated hints to turn failures into learnable re-solves and learns from both solutions and teaching effectiveness.Source: §1 · p. 2
- 1.02 pp on Llama-3.2-1B-Instruct, 2.84 pp on Qwen3-1.7B, and 4.32 pp on Qwen3-8B over state-of-the-art methods across nine mathematical reasoning benchmarks.The reported gains span three model scales.Source: §1 · p. 2
2 Related Work
Related work addresses difficult-query learning either by making better use of original-rollout reward contrast or by recovering contrast through auxiliary hints. HATCH is positioned as complementary to both lines, focusing on learning to provide guidance that improves unassisted reasoning.
2 Related Work
- Rollout-allocation, filtering, signal-shaping, and curriculum methods improve computation or updates using reward contrast exposed by original rollouts.These approaches include varying rollout budgets, retaining informative groups, and scheduling task difficulty.Source: §2 · p. 3
- HATCH is complementary to methods that improve use of existing original-rollout signal because it targets queries that remain uninformative under those rollouts.It studies whether learning to provide guidance can improve a policy’s unassisted reasoning.Source: §2 · p. 3
- Hint-based methods recover reward contrast by re-solving difficult queries with auxiliary contexts, including scaffolds, oracle prefixes, and generated or learned hints.Some work also studies keeping external guidance compatible with online policy updates.Source: §2 · p. 3
3 Observation
GRPO uses verified rewards across solution rollouts to create relative learning signals, but failed queries can provide no reward contrast. Hint-based re-solving recovers contrast, yet can shift updates toward hinted solving and create a trade-off between early gains and later original-query improvement.
3.1 Preliminaries
- GRPO verifies sampled responses, compares their rewards within each query group to compute relative advantages, and assigns those advantages to response tokens for its clipped objective.Source: §3.1 · p. 3
- Hint-based methods re-solve groups with no correct original rollouts under generated hints, while prior work constructs hints from solution abstractions, intermediate reasoning, or a separate hinter.Source: §3.1 · pp. 3–4
3.2 Hinted Trade-off Problem
- Hinted trajectories retain comparable or larger mean absolute advantages while often producing nearly orthogonal gradients, so recovered contrast can favor hinted solving without ensuring gains on original queries.The paper calls this tendency hinted reward shift.Source: §3.2 · p. 4
- Multiple hints yield faster early accuracy gains but later deteriorate, whereas one hint improves more slowly; hinted feedback should recede as original-query learning signal emerges.The trade-off reflects changing availability of reward contrast from original rollouts.Source: §3.2 · p. 4
4 Method
HATCH is an online single-policy framework that builds hint-generation and hinted-solving objectives from failed queries, then coordinates their influence with online weighting and gradient projection. Training combines native solving, hint generation, and hinted solving, while inference uses only the original query.
4 Method
- HATCH combines native solving, hint generation, and hinted solving in one policy, coordinating their contributions through online weighting and gradient projection.The three trajectories are jointly used to update a shared policy.Source: §4 · p. 4, Figure 3 · p. 5
4.1 Online self-hinting for valuable trajectories
- For initially failed queries, the policy generates multiple hints, re-solves with them, and uses the outcomes to train both hinted solving and hint generation.All M hinted-solve groups are retained, and hint-generation rewards reflect rollout accuracy gain and the successful trajectory’s log-probability difference across conditions.Source: §4.1 · pp. 4–5
4.2 Online Weighting of Hinted Trajectories
- Online weighting reduces hinted-solve feedback as reward contrast emerges in original rollouts, letting hints supplement learning when that contrast is scarce.The weight is based on smoothed proportions of unsolved groups and groups with reward contrast.Source: §4.2 · pp. 5–6
4.3 Gradient Projection for Hint Generation
- Because hint-generation gradients can oppose solution-side gradients, projection removes only their opposing component while preserving the solution-side direction.The hint-generation and solution-side gradients fluctuate around 90° and often exceed it.Source: §4.3 · p. 6
4.4 Hint Annealed Self Teaching
- The training update combines the three objectives after weighting hinted advantages and applying gradient projection; at inference, the policy receives only the original query.Algorithm 1 summarizes a training update that uses the resulting gradient to update the policy.Source: §4.4 · pp. 6–7
5 Experiments
Across three model scales, HATCH achieves the strongest original-query results, with training dynamics and ablations examining its hint use, joint hint learning, and gradient projection. The evaluation focuses on mathematical reasoning, and the authors identify interactive and multimodal tasks as future extensions.
5.1 Experimental Setup
- Training uses 10,000 NuminaMath problems, GRPO with eight rollouts per query, and three model scales; evaluation reports pass@1 on nine benchmarks, averaged over three runs.Checkpoints are selected by no-hint validation accuracy; baselines include GRPO, QuESTA, and HiLL.Source: §5.1 · p. 7
5.2 Main Results
- 1.70 pp, 3.56 pp, and 6.18 pp: HATCH improves original-query pass@1 over GRPO on Llama-3.2-1B-Instruct, Qwen3-1.7B, and Qwen3-8B, respectively.It has the highest macro average at every scale and beats both hint-based baselines; on Qwen3-8B it scores best on every benchmark.Source: §5.2 · p. 8
- On Qwen3-8B, HATCH exceeds 50% accuracy around step 200 and reaches approximately 53% near step 280, while the other methods remain below 49%.Up to approximately 49% best-so-far accuracy, it reaches each level with fewer cumulative GPU-hours than HiLL; two- and four-group settings reach approximately 51% by step 220, versus about 44% for one group.Source: §5.2 · p. 8, Figure 5 · p. 8
5.3 Further Analysis
- Online Hints reaches 50.16%, 3.60 pp below HATCH, while fixed offline and external hints reach 48.07% and 48.80%, respectively.These ablations support learning hint construction in addition to using hints as auxiliary inputs; Table 2 reports the ablations on Qwen3-8B.Source: §5.3 · p. 8, Table 2 · p. 8
- Fixed hinted-trajectory weighting reaches 44.96%, versus 48.25% with linear decay and 51.25% with staged exponential decay; data-driven annealing performs better than these schedules.The authors report that γ = 5 best balances useful early feedback against sustained hinted reward shift as original reward contrast emerges.Source: §5.3 · pp. 8–9, Table 2 · p. 8, Figure 6 · p. 9
- Removing gradient projection reduces accuracy to 51.93%, and reverse projection reaches 49.24%; projection preserves the shared update in the solution direction.The authors attribute the gain to preserving directionally compatible hint-generation learning rather than letting it counteract original-query solving.Source: Table 2 · p. 8, §5.3 · p. 9, Figure 6 · p. 9
- Across training, HATCH reduces solve-none groups faster and increases solve-partial and solve-all groups relative to the other methods.The group-state dynamics indicate stronger unassisted solving.Source: §5.3 · p. 9, Figure 7 · p. 9
5.4 Limitations and Future Work
- Evaluation is limited to mathematical reasoning with verifiable rewards and reference; interactive tasks require context-suited verification and reference, and multimodal self-teaching remains future work.The experiments use textual inputs and hints.Source: §5.4 · p. 9
6 Conclusion
HATCH addresses hinted reward shift by jointly learning hint generation and query solving, and experiments show stronger unassisted mathematical reasoning across three model scales. The authors conclude that models can improve reasoning by learning to provide effective guidance for themselves.
6 Conclusion
- HATCH jointly learns hint generation and query solving within one policy, using online weighting and gradient projection to regulate auxiliary learning.The approach addresses hinted reward shift, where recovered signal can favor hinted solving without corresponding gains without hints.Source: §6 · p. 9
- Across three model scales, experiments show stronger unassisted mathematical reasoning.The authors suggest that models can improve reasoning by learning to provide effective guidance for themselves.Source: §6 · pp. 9–10
- Recovered signal can favor hinted solving without corresponding gains in solving without hints, motivating HATCH’s regulation of auxiliary learning.The conclusion names this pattern hinted reward shift.Source: §6 · p. 9