Source-linked AI summary
KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance
Linhao Yu, Tianmeng Yang, Siyu Ding, Renren Jin, Naibin Gu, Xiangzhao Hao, Shuaiyi Nie, Deyi Xiong, Weichong Yin, Yu Sun, Hua Wu
TL;DR
RLVR is limited by sparse rewards on hard reasoning problems, while existing hinting methods often add redundant or inconsistent guidance. KnowRL decomposes hints into atomic knowledge points and uses interaction-aware subset selection for minimal-sufficient training guidance. Across eight benchmarks, the resulting model outperforms strong baselines and reaches state-of-the-art performance at the 1.5B scale.
Problem
RLVR suffers from reward sparsity on difficult samples, and existing hinting strategies often expand guidance quantity while overlooking redundancy and structure.
Method
KnowRL decomposes guidance into atomic knowledge points and uses Constrained Subset Search to curate compact, interaction-aware subsets for RL training.
Results
KnowRL-Nemotron-1.5B consistently achieves the strongest overall performance across eight benchmarks, reaching 70.08 average accuracy without inference KP hints and 73.47 with CSS-selected KPs.
Takeaways & Limitations
The results support compact, structured guidance as a practical scaling principle for sparse-reward RL and motivate extending robust KP selection to broader reasoning domains.
Takeaways & Limitations
LOO-based pruning can overgeneralize from single-KP ablations because removing multiple individually non-essential KPs together may reduce performance.
Abstract
from arXiv · showhide
RLVR improves reasoning in large language models, but its effectiveness is often limited by severe reward sparsity on hard problems. Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, yet they typically scale guidance by adding more tokens, which introduce redundancy, inconsistency, and extra training overhead. We propose \textbf{KnowRL} (Knowledge-Guided Reinforcement Learning), an RL training framework that treats hint design as a minimal-sufficient guidance problem. During RL training, KnowRL decomposes guidance into atomic knowledge points (KPs) and uses Constrained Subset Search (CSS) to construct compact, interaction-aware subsets for training. We further identify a pruning interaction paradox -- removing one KP may help while removing multiple such KPs can hurt -- and explicitly optimize for robust subset curation under this dependency structure. We train KnowRL-Nemotron-1.5B from OpenMath-Nemotron-1.5B. Across eight reasoning benchmarks at the 1.5B scale, KnowRL-Nemotron-1.5B consistently outperforms strong RL and hinting baselines. Without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 average accuracy, already surpassing Nemotron-1.5B by +9.63 points; with selected KPs, performance improves to 74.16, establishing a new state of the art at this scale. The model, curated training data, and code are publicly available at https://github.com/Hasuer/KnowRL.
1 Introduction
RLVR struggles with sparse rewards on difficult reasoning problems, while existing hint-based methods often add excessive or loosely structured guidance. KnowRL reframes hinting as selecting minimal, coherent knowledge units and uses interaction-aware subset selection during RL training.
- RLVR yields sparse rewards on difficult samples because uniformly incorrect rollouts provide zero advantage under group-based optimization.
- Existing hint-based methods treat stronger guidance as longer prefixes or richer abstractions, overlooking guidance redundancy.
- Performance shows a critical-segment effect: a short key hint segment produces a sharp accuracy jump, followed by diminishing gains.
- KnowRL decomposes guidance into atomic knowledge points and identifies minimal subsets sufficient to unlock reward learning.
- Constrained Subset Search prunes and globally searches candidate KPs, injecting minimal subsets only for harder samples during training.
- KnowRL integrates compact KP subsets through difficulty-aware prompt injection and reports state-of-the-art results across benchmarks with reduced hint length and computational overhead.
2 Related Work
Prior hint-based RL methods use fixed or adaptive solution prefixes, hybrid SFT–RL pipelines, or abstraction-based guidance. These approaches improve control or conceptuality but may depend on strong teachers or carefully designed templates, and vague abstractions can lack actionable signals.
- Solution-Prefix Hints: Fixed-ratio methods prepend a predetermined proportion of solution prefixes, with POPE adding token-level importance-based prefix selection.
- Adaptive Solution-Based Hints: Adaptive solution-based methods vary hint length or granularity using task difficulty, reward signals, semantic steps, or offline difficulty priors.
- Hybrid SFT–RL Pipelines: Hybrid SFT–RL approaches combine offline prefixes with online continuations or progressively reduce hint length during training.
- Abstraction-Based Hints: Abstraction-based hints provide high-level concepts, principles, or reasoning patterns instead of partial solutions, often generated by stronger teacher models.
- Abstraction-Based Hints: Strong teacher dependence and vague abstractions can limit abstraction-based hints by increasing design requirements or failing to provide actionable signals for difficult reasoning.
3 KnowRL
KnowRL constructs compact, problem-specific knowledge-point hints and selects them with interaction-aware search to address inconsistencies caused by naive guidance and pruning. CSS achieves a strong accuracy–compactness tradeoff, while selected KPs yield larger and more consistent gains than full-KP injection across difficulty levels.
- Framework: KnowRL constructs candidate KPs, removes leakage and redundancy, and injects compact subsets only when training problems need guidance.The workflow uses curated KP hints for RL training rather than uniformly adding guidance to every problem.
- Framework: Raw KP construction raises average accuracy from 60.46 to 61.03 while using 5.86 KPs per problem, motivating compact subset selection.The evaluation covers eight benchmarks on OpenMath-Nemotron-1.5B.
- Problem-wise KP Subset Selection: Max-Score searches only no-KP, full-KP, and single-removal configurations, so problems benefiting from smaller subsets can receive suboptimal guidance.Its coarse configuration space can mismatch the problem’s preferred number of KPs.
- Problem-wise KP Subset Selection: Single-KP pruning can overgeneralize because jointly removing individually non-essential KPs may trigger conflicts or larger-than-expected performance drops.The interaction paradox arises when KPs are mutually dependent or implicitly disambiguate one another.
- Constrained Subset Search (CSS): CSS prunes non-degrading candidates, searches all subsets of the remaining candidates, and achieves 63.90 eight-task average accuracy with 2.57 KPs per problem.This constrained search avoids exhaustive evaluation of all 2^n subsets while preserving a compact candidate space.
- Results: CSS-selected KPs produce larger, more consistent gains across difficulty buckets, whereas full-KP injection can regress on some instances; random KPs perform substantially worse than CSS and CBRS.These comparisons support interaction-aware selection rather than simply increasing the number of hints.
4 Experiments
KnowRL evaluates compact KP-guided RL training across eight reasoning benchmarks, with CSS-selected guidance improving performance even without KP hints at inference. Training analysis indicates that KP guidance reduces reward sparsity and improves policy quality rather than merely supplying test-time scaffolding.
- Overall performance: KnowRL-Nemotron-1.5B reaches 70.08 average accuracy without KP hints, surpassing Nemotron-1.5B by +9.63 points and JustRL by +1.50.With selected KPs, performance reaches 73.46 with CBRS and 73.47 with CSS.
- Overall performance: Under CSS selection, KnowRL-Nemotron-1.5B improves over Nemotron-1.5B by +15.11 on AIME25, +12.98 on HMMT25, and +15.49 on CMIMC25 without KP hints.The gains are reported on challenging competition-style reasoning benchmarks.
- Overall performance: KnowRL also reaches 96.20 on MATH-500, 82.44 on OlyBench, and 95.70 on AMC23, indicating gains across diverse reasoning distributions.The passage contrasts these broader evaluation results with performance that is not confined to one benchmark type.
- Training-data analysis: The backbone has 41.21% zero-correct queries and 22.40% mean accuracy, while KnowRL training without inference KPs reduces zero-correct queries to 13.00% and raises mean accuracy to 64.30%.The all-correct bucket increases to 34.28%, a +32.93pp change.
- Training-data analysis: Adding KP hints at inference raises the all-correct bucket to 51.07% and average accuracy to 77.04%, while midrange correct-count buckets shrink.The reported distribution is consistent with minimal sufficient knowledge resolving partial successes into consistent correctness.
5 Comparison of KP Selection Strategies
The comparison examines CSS and CBRS under matched training budgets, finding that CSS provides more stable optimization and stronger generalization with comparable KP counts. Its prune-then-search procedure outperforms consensus-based selection on the reported checkpoints.
- Training accuracy: CSS maintains higher training accuracy through most of optimization and converges to a slightly higher final accuracy than CBRS.Both strategies improve rapidly during the first 200 steps, but CSS retains a persistent advantage.
- Clip ratio: CBRS has a higher mid-to-late-training clip ratio, whereas CSS maintains a smoother trajectory associated with more controlled policy refinement.The passage characterizes CBRS as inducing more aggressive policy updates.
- Performance: 65.00 vs. 64.68 at the earlier checkpoint and 66.46 vs. 65.72 at step 900 show CSS generalizing better than CBRS under different training budgets.The comparison uses matched training budgets and eight reasoning benchmarks.
- Selection mechanism: CSS first prunes low-value candidates and then performs broader constrained enumeration, while CBRS aggregates consensus from a more limited candidate pool.The broader search can identify strong lower-frequency KP combinations that consensus may miss.
6 Conclusion
KnowRL frames RLVR hinting as minimal-sufficient guidance: atomic knowledge points are curated into compact, interaction-aware subsets through CSS. Across eight math benchmarks, the framework improves optimization stability and generalization while achieving a new 1.5B-scale state of the art.
- Framework: KnowRL decomposes hints into atomic knowledge points and selects robust subsets for RLVR training.The framework explicitly treats hint design as a minimal-sufficient guidance problem.
- Selection: CSS handles inter-KP interactions and consistently outperforms alternatives while keeping hint sets compact.The conclusion links CSS to the critical-segment phenomenon and robust subset selection.
- Results: Across eight math reasoning benchmarks under matched budgets, KnowRL improves optimization stability and generalization and achieves a new 1.5B-scale state of the art.The conclusion presents compact, structured guidance as a practical scaling principle for sparse-reward RL.
A Visualization of the Critical-segment Effect.
The controlled prefix-ratio study varies the fraction of a reference solution appended as a hint across 100 QuestA training instances while holding other decoding and evaluation settings fixed. It visualizes whether accuracy changes linearly as prefix length increases.
- Study design: The study evaluates 100 randomly selected QuestA training instances with reference-solution prefixes ranging from 0% to 90%.Each prefix is appended under the ## Hint header while other decoding and evaluation settings remain fixed.
- Interpretation: The figure tests whether accuracy increases linearly as the injected solution prefix becomes longer.The passage describes the prefix-ratio manipulation but does not state the figure’s observed outcome.
B Entropy Annealing Analysis
Entropy annealing tightens clipping during training to accelerate the shift from exploration to exploitation. Under the same training budget, the annealed setting outperforms both the non-annealed variant and JustRL across eight benchmarks.
- Training strategy: After 2,590 steps, clip_high decreases from 0.28 to 0.26, producing a tighter clipping regime.The change is intended to accelerate entropy reduction during training.
- Training strategy: The tighter clipping regime accelerates entropy reduction and encourages an earlier shift from exploration to exploitation.This helps the model reach stronger performance within fewer optimization steps.
- Ablation result: Under the same 2,960-step budget, entropy annealing achieves better overall scores than the non-annealed variant and JustRL across eight evaluation benchmarks.The comparison uses a control that keeps clip_high=0.28 throughout training.
C Prompts
The appendix provides concrete prompts for the knowledge-point curation pipeline and an augmented-prompt data example. Figures 8–10 document the prompts and example format for these stages.
- C.1 Prompts for KP Curation Pipeline: The latter two stages introduced in Section 3.1 are raw knowledge-point extraction and leakage verification.These stages are part of the knowledge-point curation pipeline.
- C.1 Prompts for KP Curation Pipeline: Figure 8 shows the prompt used for extracting raw knowledge points.The figure provides a concrete prompt example for this curation stage.
- C.1 Prompts for KP Curation Pipeline: Figure 9 shows the prompt used for leakage verification with an augmented hint.This prompt corresponds to the second curation stage described in the section.
- C.1 Prompts for KP Curation Pipeline: The section presents detailed prompt examples for both raw knowledge-point extraction and leakage verification.These examples are shown in Figures 8 and 9, respectively.
- C.2 Example Augmented Prompt: The appendix also presents a concrete data example of the augmented prompt format.The example is provided in Figure 10.
- C.2 Example Augmented Prompt: Figure 10 contains the example augmented prompt with a partial-solution hint.It illustrates how the augmented prompt format is instantiated in data.
D Effect of Tolerance Threshold
The tolerance threshold δ controls the trade-off between candidate overlap, selected-KP compactness, and performance. Among the tested values, δ = 1/32 provides the strongest reported balance.
- Threshold selection: δ = 1/32 achieves the highest average accuracy, 63.90, while keeping the average selected-KP count moderate at 2.57.This setting provides the best overall balance among the tested thresholds.
- Threshold selection: With δ = 0/32, near-optimal candidate intersections across runs are often too small, making selection brittle.The passage contrasts this setting with larger tolerances that increase overlap.
- Threshold selection: With δ = 2/32, candidate overlap rises to around 60%, but the selected KP set becomes much larger.The larger tolerance improves overlap at the cost of compactness.
- Related analyses: Figure 6 compares entropy trajectories and validation performance with and without entropy annealing under the 24k setting.Its caption states that annealing yields faster entropy reduction and consistently better validation performance.
- Threshold selection: Table 5 summarizes the effect of δ on offline performance and KP compactness, identifying δ = 1/32 as the best balance.The table compares average accuracy with the average number of selected KPs.
- Related analyses: Figure 7 visualizes the critical-segment effect across prefix ratios on 50 training instances.The figure focuses on how the effect varies with prefix ratio.