Source-linked AI summary
Nudging the Boundaries of LLM Reasoning
Justin Chih-Yao Chen, Becky Xiangyu Peng, Prafulla Kumar Choubey, Kung-Hsiang Huang, Jiaxin Zhang, Mohit Bansal, Chien-Sheng Wu
TL;DR
Online RL methods such as GRPO cannot learn from problems the base policy cannot solve, leaving the model’s reasoning upper limit unchanged. NuRL selectively injects self-generated hints for zero-pass-rate problems and regenerates rollouts, achieving consistent gains across six benchmarks and three models while complementing test-time scaling. The paper finds that abstract, high-level hints applied selectively are most effective.
Problem
Online RL cannot obtain learning signals from problems that are unsolvable under the base policy, limiting improvement of the model’s upper reasoning bound.
Method
NuRL generates hints from question-conditioned Chain-of-Thought reasoning and injects them when all G base-policy rollouts fail, then trains on regenerated trajectories.
Results
NuRL consistently outperforms GRPO across six benchmarks and three models, with average improvements of 1.62% for Llama, 1.75% for OctoThinker, and 0.79% for Qwen.
Takeaways & Limitations
NuRL expands the fraction of solvable problems and the model’s comfort zone, while remaining complementary to test-time scaling and favoring abstract, high-level hints.
Takeaways & Limitations
Because NuRL optimizes final-answer correctness without directly supervising intermediate reasoning, its outputs may still contain hallucinations or misinformation.
Abstract
from arXiv · showhide
Current online reinforcement learning (RL) algorithms like GRPO share a key limitation in LLM reasoning: they cannot learn from problems that are "unsolvable" to the model. In other words, they can only improve performance on problems where the model is capable of exploring the correct answer. Consequently, the model's "upper limit" remains unchanged after RL training, even though the likelihood of solving easier, solvable problems may increase. These hard samples cannot contribute to training, as no rollouts yield rewards and thus no gradients are produced. To unlock learning from these hard samples, we propose NuRL, a "nudging" method that aims to push the upper bound of LLM reasoning using self-generated hints, i.e., abstract cues that help reduce the problem difficulty for the model. Given a question and its gold answer, the model generates a CoT and then produces a hint containing the core knowledge needed to solve the problem. During training, we generate G rollouts from the base policy and use the pass rate to decide whether the hint should be injected. For hard samples with a 0% pass rate, we inject the hint and regenerate a new batch of trajectories. This yields two benefits: (1) the hint boosts pass rates (from 0% to non-zero), thereby introducing training signals for previously unsolvable samples, and (2) the hints are self-generated, avoiding distributional shift and do not rely on external models. NuRL achieves consistent improvements across 6 benchmarks and 3 models, while remaining complementary to test-time scaling. Notably, NuRL can raise the model's upper limit, whereas GRPO leaves pass@1024 unchanged from the base model. Furthermore, we present a systematic study of what makes an effective hint and when hints are most useful. Interestingly, the best hints are abstract and high-level, and are most beneficial when applied necessarily and after GRPO has converged.
1 INTRODUCTION
Current online RL cannot learn from problems unsolvable under the base policy, leaving the model’s upper limit unchanged. NuRL addresses this by selectively injecting self-generated hints into training, improving performance across benchmarks and models while expanding solvable problems.
- 1 INTRODUCTION: Online RL cannot learn from problems that produce no correct rollout, so pass@k upper limits remain unchanged despite gains on solvable problems.When extensive exploration fails to reach the correct answer, no meaningful learning signal is obtained.
- 1 INTRODUCTION: NuRL explores self- or teacher-generated abstract cues, partial steps, explanations, and even gold answers to expand the model’s comfort zone.The central motivation is to provide guidance for difficult problems that the model cannot solve unaided.
- 1 INTRODUCTION: NuRL generates hints from question-conditioned Chain-of-Thought reasoning and injects them only when G base-policy rollouts have a 0% pass rate.The model regenerates trajectories conditioned on the appended hint, turning previously unlearnable examples into learnable ones.
- 1 INTRODUCTION: NuRL improves average performance over GRPO by 1.62% on Llama, 1.75% on OctoThinker, and 0.79% on Qwen across six benchmarks.With an external teacher model for hint generation, the improvement can reach 3.44%.
- 1 INTRODUCTION: NuRL remains complementary to test-time scaling, improving 9.4% with 16-way Self-Consistency compared with 7.8% for GRPO.The method’s analysis also finds that high-level, abstract hints are more useful than hints exposing more of the answer.
2 RELATED WORK
Related work frames RLVR as a successful approach for improving reasoning with verifiable rewards, while questioning whether RL discovers abilities or mainly sharpens existing behaviors. Other work combines offline evidence or demonstrations with online RL when the base policy yields no positive rewards.
- 2 RELATED WORK: RLVR uses rule-based verification, often checking whether a model answer matches the gold answer, and includes methods such as PPO, DPO, GRPO, and DAPO.These approaches use verifiable rewards to improve LLM reasoning.
- 2 RELATED WORK: The distribution-sharpening view holds that RL mainly amplifies high-reward behaviors already present in the model rather than discovering new reasoning abilities.This debate motivates methods that address whether RL can extend capability boundaries.
- 2 RELATED WORK: When base-policy rollouts produce no positive rewards, replay buffers, expert demonstrations, and hybrid offline-online methods provide alternative training signals.Recent approaches also separate data generation from policy updates or mix offline and online training to improve generalization.
3 METHODOLOGY
NuRL augments online GRPO with self-generated abstract hints so hard problems that yield no successful rollouts can produce informative training signals. It collects hints offline, injects them selectively when all initial rollouts fail, and trains without hints at inference.
- GRPO Framework: GRPO optimizes a clipped policy objective using group-normalized advantages computed from rule-based outcome rewards.The reward is 1 when the prediction is equivalent to the ground truth and 0 otherwise.
- Offline Hint Collection: Hints are abstract problem-specific cues collected offline to reduce task difficulty without directly revealing solution details.The method focuses on self-generated abstract cues rather than more answer-revealing partial steps, explanations, or ground-truth answers.
- Offline Hint Collection: NuRL generates a ground-truth-conditioned Chain-of-Thought, then abstracts it into a high-level hint paired with each training question.The first model output explains why the answer is correct; a second prompt converts that explanation into a less specific cue.
- Online Rollout Augmentation: NuRL activates hint-guided regeneration when all G initial rollouts fail, turning uniformly zero rewards into non-zero training signals.Only G −1 regenerated rollouts receive the hint, while one remains hint-free to preserve learning signal.
- Inference: Hints are used only during training, with the intended effect of helping the model internalize reasoning patterns that broaden its set of solvable problems.At test time, the model receives only the question.
4 EXPERIMENTAL SETUP
The experiments evaluate NuRL across three distinct language models and six diverse benchmarks using established supervised and reinforcement-learning baselines. Results are averaged across repeated runs with aligned evaluation settings, and Table 1 compares methods across models and tasks.
- Models and Datasets: The benchmark suite spans MATH 500, MATH Hard, AIME, GPQA, MMLU-Pro, and Date Understanding across mathematics, science, and general reasoning.Training data are sampled from 7.5k math and 2.5k science examples in Open-R1’s Mixture-of-Thought dataset.
- Models and Datasets: Table 1 compares NuRL with baselines across three models and six benchmarks, including a reference condition using teacher-generated hints.The evaluated models are Llama3.2-3B-Instruct, OctoThinker-3B-Hybrid-Zero, and Qwen3-4B-Instruct-2507.
- Baselines: Baselines include zero-shot and few-shot prompting, rejection-sampling fine-tuning, and reinforcement-learning methods such as ReFT and GRPO.These comparisons cover prompting, supervised fine-tuning, and reinforcement-learning approaches.
- Implementation Details: Evaluation uses pass@1, repeated runs, rule-based answer verification, and matched token limits and inference temperature across baselines.Most results average 16 runs, while MMLU-Pro averages 3 runs over its 12k samples; inference temperature is fixed at 0.7.
5 RESULTS AND ANALYSIS
NuRL consistently improves over GRPO across benchmarks and models, while targeted, abstract hints are most effective for difficult problems. Its gains extend to test-time scaling and higher pass@k when the base model’s upper bound is limited.
- Main Results: NuRL improves over GRPO by +1.62 points on Llama, +1.75% on OctoThinker, and +0.79% on Qwen across six benchmarks.These gains occur despite NuRL using fewer default rollouts than GRPO.
- Test-Time Scaling: NuRL yields larger Self-Consistency gains than GRPO: +8.0% versus +7.6% on Llama, +9.4% versus +7.8% on OctoThinker, and 1.2% versus +1.1% on Qwen.
- Hint Abstraction: More direct hints reduce performance, whereas abstract cues that avoid revealing solution details achieve the highest accuracy.Partial steps help somewhat, while explicit answers severely harm generalization through shortcut-like reward hacking.
- Hint Timing and Triggering: Hints work best when introduced after GRPO stabilizes and only when all rollouts fail, rather than from the beginning or uniformly.Table 2 compares these hint-application strategies on MATH 500 and GPQA with Llama3.2-3B-Instruct.
- Upper-Bound Performance: NuRL raises pass@k on lower-upper-bound tasks such as Date Understanding and GPQA, while neither NuRL nor GRPO improves MATH 500 pass@1024 from its 96.4% base value.For Date Understanding and GPQA, the base pass@1024 values are 85.4% and 67.2%, and GRPO provides little or no improvement.
- Solvable Problems: Self-generated hints increase the solvable training fraction by approximately 4% and raise it from 66% to 70% as training progresses.A problem is solvable when at least one rollout produces a correct answer.
6 CONCLUSION
NuRL uses self-guided hints to extend LLM reasoning capabilities, consistently outperforming strong baselines and expanding the model’s solvable problem range.
- NuRL consistently outperforms strong baselines with self-generated hints and achieves further gains with hints from a stronger external model.
ETHICS STATEMENT
NuRL optimizes final-answer correctness without directly supervising intermediate reasoning, so its outputs may still contain hallucinations or misinformation.
- Because intermediate reasoning is not directly supervised, NuRL outputs may still produce hallucinations or misinformation.
REPRODUCIBILITY STATEMENT
The authors support reproducibility by providing code, implementation details, prompts, and publicly available datasets.
- Code, implementation details, and prompts are provided to support replication of NuRL findings.
- All datasets used in the study are publicly available.
B IMPLEMENTATION DETAILS
NuRL uses a two-stage procedure: GRPO runs until convergence, then hint-based training is applied to selected samples with aligned baseline training steps.
- NuRL applies hint-based training after GRPO convergence, defined by plateaued reward and validation accuracy after 10 training steps.
- GRPO baselines also receive a second training stage without hints so total training steps match NuRL.
- Stage 1 and Stage 2 configurations and hyperparameters are documented in Tables 3 and 4.
- Stage 2 generates 8 rollouts per question and discards examples whose rollouts are all correct to improve efficiency.
C DATASET STATISTICS AND LICENSES
The study documents dataset usage and illustrates how NuRL hints are generated and structured. The examples emphasize concise, high-level guidance rather than detailed calculations or final answers.
- C DATASET STATISTICS AND LICENSES: All datasets are English and used consistently with their intended purposes, with sample sizes and licenses reported in Table 5.
- Hint generation: Hints are generated by first eliciting a step-by-step explanation conditioned on the gold answer, then converting it into an abstract hint.
- Hint generation: The hint-generation prompt requests concise, high-level core knowledge while excluding detailed steps, calculations, and the final answer.
- Worked examples: The examples show that solving the target problem involves identifying relevant contributors, computing totals, and forming the requested percentage.
F ADDITIONAL EVALUATIONS ON HARDER BENCHMARKS
NuRL remains effective on substantially harder reasoning benchmarks, improving performance over both zero-shot evaluation and GRPO on the reported Qwen results.
- F ADDITIONAL EVALUATIONS ON HARDER BENCHMARKS: 42.85% on Qwen is +4.02% over zero-shot and +1.11% over GRPO across seven harder MathArena and OlymMATH benchmarks.The same evaluation reports only 2.03% average zero-shot accuracy for Llama and 38.83% for Qwen, underscoring the benchmarks’ difficulty.
G QUALITATIVE EXAMPLES OF GRPO AND NURL OUTPUTS
The qualitative comparison indicates that NuRL changes performance without requiring hints during inference. Its outputs retain a format and style similar to GRPO while solving more training problems.
- G QUALITATIVE EXAMPLES OF GRPO AND NURL OUTPUTS: NuRL-trained Qwen3-4B models do not generate hint tokens at test time, so their output format and style remain similar to GRPO models.
- G QUALITATIVE EXAMPLES OF GRPO AND NURL OUTPUTS: The reported performance difference is attributed to NuRL solving more training problems rather than to a visible test-time hinting format.