Source-linked AI summary
Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs
Wanli Yang, Hongyu Zang, Junwei Zhang, Wenjie Shi, Du Su, Jingang Wang, Xueqi Cheng, Fei Sun
TL;DR
The paper asks whether reinforcement learning can improve direct recall of parametric knowledge beyond reasoning. It studies binary-reward RL in controlled zero-shot, one-hop, closed-book QA without chain-of-thought, finding broad gains that mainly redistribute existing knowledge into reliable generations.
Problem
Whether reinforcement learning can improve direct recall of parametric knowledge, which remains unreliable despite facts being encoded in LLM parameters.
Method
The study trains RL with binary correctness rewards in non-CoT factual QA, using fact-level train-test deduplication and comparing against training- and inference-time baselines.
Results
~27% average relative gains across three LLM families and three factual QA benchmarks, with RL outperforming both training-time and inference-time baselines.
Takeaways & Limitations
RL unlocks latent parametric knowledge by moving correct answers from the low-probability tail into reliable greedy generations rather than primarily acquiring new facts.
Takeaways & Limitations
Factual RL requires datasets with non-trivial initial accuracy; near-zero accuracy and sparse exact-match rewards can collapse gains.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has achieved remarkable success in LLM reasoning, but whether it can also improve direct recall of parametric knowledge remains an open question. We study this question in a controlled zero-shot, one-hop, closed-book QA setting with no chain-of-thought, training only on binary correctness rewards and applying fact-level train-test deduplication to ensure gains reflect improved recall rather than reasoning or memorization. Across three model families and multiple factual QA benchmarks, RL yields ~27% average relative gains, surpassing both training- and inference-time baselines alike. Mechanistically, RL primarily redistributes probability mass over existing knowledge rather than acquiring new facts, moving correct answers from the low-probability tail into reliable greedy generations. Our data-attribution study reveals that the hardest examples are the most informative: those whose answers never appear in 128 pre-RL samples (only ~18% of training data) drive ~83% of the gain, since rare correct rollouts still emerge during training and get reinforced. Together, these findings broaden the role of RL beyond reasoning, repositioning it as a tool for unlocking rather than acquiring latent parametric knowledge.
1 Introduction
The paper tests whether reinforcement learning can improve direct recall of parametric knowledge without reasoning or fact-level memorization. Across models and benchmarks, RL improves recall, redistributes probability toward latent answers, and derives much of its signal from rarely recalled facts.
- Motivation and contributions: RL improves direct factual recall in zero-shot, one-hop, closed-book QA using binary outcome rewards and fact-level train-test deduplication.The setting excludes explicit reasoning and prevents held-out gains from training-query fact overlap.
- Motivation and contributions: ~27% average relative gains across three LLM families and three factual QA benchmarks exceed 53% on Natural Questions across all models.Gains transfer across datasets, scale to models up to 72B, and persist across RL algorithms.
- Comparative results: RL outperforms supervised fine-tuning, DPO, rejection fine-tuning, majority voting, and chain-of-thought prompting under matched conditions.The comparison attributes RL’s advantage to the combination of on-policy exploration and contrastive feedback.
- Which facts RL repairs: Repair rates rise with pre-RL accessibility: answers appearing 9–16 times in 128 samples are repaired at ~52%, while highly accessible answers reach ~84%.Even answers absent from all 128 pre-RL samples are repaired at 6–13%, indicating deeply suppressed knowledge can remain recoverable.
- How RL changes recall: RL primarily pulls existing correct answers from the low-probability tail into reliably top-ranked positions rather than generating new facts.Pass@k analysis tracks this redistribution across sampling budgets up to k = 256.
- Which examples drive learning: ~18% of training examples whose answers never appear in 128 pre-RL samples recover ~83% of the full-data RL gain.Rare correct rollouts still emerge during training, then are reinforced and amplified.
2 Problem Formulation and Experimental Setup
The study formulates factual RL as closed-book answer generation without chain-of-thought and evaluates it across diverse models and benchmarks. Training uses GRPO with binary semantic-correctness rewards, while standardized decoding and judge verification support the experimental comparisons.
- 2.1 Problem Formulation: RL for Factual Recall: The setup is zero-shot, one-hop, closed-book factual QA with concise final answers and no intermediate reasoning steps.Correctness is represented by a binary indicator over the generated answer.
- 2.2 RL Training: GRPO contrasts rewards within rollout groups, avoiding a separate value network for outcome-based factual RL.Binary factual correctness is verified semantically rather than by strict exact matching.
- 2.3 Experimental Setup: Experiments use Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and OLMo-2-7B-Instruct from three model families.The evaluated benchmarks are Natural Questions, TriviaQA, PopQA, and SimpleQA.
- 2.3 Experimental Setup: The evaluation covers factual QA datasets spanning common trivia, long-tail entities, and challenging frontier questions.
- 2.3 Experimental Setup: Greedy decoding is the default evaluation strategy, while analytical sampling uses the same hyperparameters as RL training rollouts.
- 2.3 Experimental Setup: Qwen2.5-72B-Instruct serves as the unified local judge for training rewards and test evaluation.A reliability analysis compares its decisions with human annotations and frontier closed-source models.
3 RL Reliably Improves Direct Factual Recall
Across QA benchmarks, models, scales, architectures, and algorithms, outcome-based RL consistently improves direct factual recall more than the evaluated training- and inference-time alternatives. The gains remain robust under cross-dataset transfer and are associated with on-policy exploration plus contrastive reward signals.
- Main results: About 10% average absolute improvement, including around 15 percentage points on NQ, makes RL the strongest method across TriviaQA, NQ, and PopQA.SimpleQA is the sole exception, where all methods fail to yield meaningful improvements.
- RL’s advantage: SFT overfits training data, DPO remains flat under static preference pairs, and RFT produces smaller gains because positive-only feedback is insufficient.RL instead combines active exploration with advantage-based reward signals to improve test recall.
- Inference-time comparison: RL outperforms majority voting and CoT across the evaluated datasets and models, while voting provides marginal gains and CoT improvements are inconsistent.Voting aggregates 32 independently sampled base-model answers.
- Robustness: Comparable performance with PPO shows that the observed improvement is not specific to GRPO.The comparison uses identical reward and hyperparameter configurations.
- Robustness: Fact-level deduplication preserves robust out-of-distribution recall improvements when training and evaluation use different QA datasets.The cross-dataset setting removes overlapping facts between source training data and target test queries.
- Robustness: Approximately 15% absolute accuracy gains on NQ extend to larger dense models and a Mixture-of-Experts architecture.The broader evaluation covers models up to Qwen2.5-72B and Qwen3-30B-A3B-Instruct.
4 RL Reshapes Access to Latent Parametric Knowledge
RL reshapes factual recall by preferentially repairing failed queries whose correct answers were already more accessible in stochastic pre-RL sampling. It also shifts correct answers toward lower-budget recall, including some answers absent from 128 pre-RL samples.
- Accessibility-stratified repair: Repair rates rise sharply with pre-RL accessibility: answers appearing 9–16 times achieve around 52% repair, while ≥64 appearances exceed 62% and peak above 93%.These rates are measured on held-out queries that initially failed under greedy decoding.
- Accessibility-stratified repair: 6%–13% of zero-accessibility queries become correct under post-RL greedy decoding even though their answers never appeared in 128 pre-RL samples.The corresponding test queries are held out from RL training.
- Sampling-budget recall: Post-RL models outperform pre-RL models at pass@k budgets up to k ≤64, with k ∈{1, 2} often matching pre-RL performance at k ∈{16, 32}.This indicates improved accessibility under stochastic sampling, not only more frequent greedy top-ranking.
- Interpretation: Together, repair and pass@k results show that RL promotes recoverable but suppressed latent knowledge into more accessible recall regimes.The analysis treats pre-RL accessibility as a practical output-distribution proxy rather than proof that a fact is stored or absent.
5 Lower-Accessibility Examples Contribute More to RL Gains
Low-accessibility training examples provide the strongest learning signal for RL’s factual-recall gains. Rare correct rollouts from apparently inaccessible examples can emerge during training and then be reinforced.
- Attribution setup: The data-attribution study partitions training examples into inaccessible@128, near-inaccessible, partially accessible, and highly accessible groups using 128 pre-RL samples.Subset comparisons are size-balanced with fixed RL hyperparameters, and efficacy is measured as the fraction of full-data gain recovered.
- Single-subset results: Inaccessible@128 facts provide the strongest single-subset signal despite appearing to offer prohibitively sparse rewards.This contradicts the expectation that partially accessible examples would dominate.
- Mechanism: Repeated rollouts steadily increase rewards for 0/128 examples, showing that finite-sample inaccessibility does not imply absent parametric knowledge.The reward dynamics are reported for Qwen as a representative case.
- Combined subsets: Combining partially accessible and inaccessible@128 data recovers the full-data RL gain on average, with a recovery fraction of 1.00.This pairwise combination yields the highest overall gain among the compared subsets.
6 Discussion: When Factual RL Works, and What It Means for RL
Factual RL works when the dataset provides enough initial correct responses for non-sparse rewards, while its strongest signals can still come from low-accessibility examples. The broader interpretation is that RL redistributes probability toward existing parametric knowledge rather than injecting facts.
- Conditions for successful factual RL: Factual RL requires non-trivial initial accuracy: highly accessible facts suffer reward saturation, while near-zero initial accuracy creates extremely sparse rewards.Strict exact-match rewards similarly collapse gains by increasing reward sparsity.
- Data curation: Low-accessibility examples should receive priority in factual-RL data curation because rare correct responses can surface during repeated rollouts and be amplified.The recommendation is specifically relative to examples of intermediate difficulty.
- Implications: RL broadens outcome-based reinforcement learning beyond reasoning by improving factual recall without chain-of-thought or knowledge injection.The paper characterizes the mechanism as shifting suppressed correct answers from the low-probability tail toward reliable generation.
7 Related Work
Prior work has studied parametric knowledge recall, RL for reasoning, and RL in knowledge-intensive QA, but direct closed-book factual recall remains less explored.
- Behavioral and representation-level studies indicate that LLMs may encode more factual knowledge than direct prompting reveals.
- RL research has primarily improved mathematical reasoning and code generation, with on-policy exploration proposed as one explanation.
- Knowledge-intensive QA research applies RL to retrieval, answer generation, instruction following, and hallucination mitigation across open- and closed-book settings.
8 Conclusion
The paper argues that RL improves factual recall by making latent knowledge more accessible, not by explicit reasoning or injected knowledge. It evaluates this claim with outcome-based RL and controlled factual QA procedures.
- RL improves factual recall by reinforcing latent knowledge into reliable direct recall rather than injecting new knowledge.
- The direct-recall setting uses zero-shot, one-hop, closed-book QA with concise answers and no intermediate reasoning.
- GRPO estimates rollout advantages from relative rewards within groups, avoiding a separate value network.
- Responses receive binary correctness rewards, with semantic verification allowing valid answer phrasings beyond exact string matches.
- The experiments use unified hyperparameters across models and datasets, including a 1 × 10^-6 learning rate, batch size 128, and 8 epochs.
- Evaluation spans NQ, TriviaQA, PopQA, and SimpleQA, with held-out partitions used for testing.
C.2 Data Deduplication Pipeline
The evaluation removes test queries sharing underlying facts with training instances, using embedding retrieval followed by LLM semantic verification. Judge reliability is checked against human annotations and frontier models.
- The deduplication pipeline removes any test query targeting the same underlying fact as a training instance.
- Embedding retrieval identifies top-10 training-query candidates for each test sample when cosine similarity exceeds 0.8.
- LLM verification determines whether candidate questions ask about the exact same underlying fact before removal.
- The judge assigns a strict binary score of 1.0 or 0.0 for semantic equivalence between predicted and target answers.
- Qwen2.5-72B achieves 92.00% average agreement, exceeding GPT-5 at 91.00% and Gemini 2.5 Flash at 84.00%.
F Extended Training Dynamics Across All Models
Across OLMo, Llama, and Qwen, RL produces sustained held-out improvements while SFT, DPO, and RFT show distinct failure modes. Majority voting provides no meaningful complementary accuracy gain.
- Extended Training Dynamics Across All Models: SFT consistently overfits across all models, raising training accuracy while producing negligible held-out improvement.
- Extended Training Dynamics Across All Models: Figure 10 compares training and test accuracy across six equally spaced fractions of each method’s training progress.
- Extended Training Dynamics Across All Models: RL consistently delivers robust, large, and sustained test-set improvements across OLMo, Llama, and Qwen.
- Extended Training Dynamics Across All Models: Majority voting at sampling budgets k ∈ {8, 16, 32, 64} has no meaningful effect on accuracy across tested settings.
H Extended Cross-Dataset Transfer Results
Across model families and knowledge domains, RL consistently improves zero-shot factual recall transfer under fact-level deduplication, with SimpleQA as the main exception. Repair rates also increase with pre-RL accessibility, while RL can recover facts absent from initial samples.
- Cross-dataset transfer: RL yields significant relative accuracy improvements across almost all cross-dataset pairs for Llama and OLMo, extending the transfer pattern beyond Qwen.The evaluation trains on one source QA dataset and tests on an out-of-distribution target dataset after fact-level deduplication.
- Cross-dataset transfer: SimpleQA is the consistent exception because its difficulty exceeds the evaluated models’ inherent factual capacity, limiting transfer gains.
- Repair rates: Post-RL repair rates rise monotonically with pre-RL accessibility across datasets and model architectures.
- Repair rates: RL also recovers zero-accessibility facts that never appeared within the initial finite sampling budget, elevating them into greedy outputs.
J Extended Results of Pass@k Scaling
Extended pass@k results show that RL’s recall gains are largest at low and medium sampling budgets and diminish at high budgets. Across model families, training on initially inaccessible examples reveals that rare correct rollouts can emerge and be amplified during RL.
- Pass@k scaling: Post-RL models outperform pre-RL models at sampling budgets k ≤64, while the gap narrows as k approaches 256 across benchmarks.The curves extend pass@k evaluation to TriviaQA and PopQA across all evaluated model architectures.
- Pass@k scaling: Convergence at k = 256 indicates that RL primarily shifts existing knowledge from the probability tail into lower-budget recall regimes rather than injecting novel facts.
- Training dynamics: For inaccessible@128 examples, average reward starts near zero but rises steadily throughout RL training across OLMo, Llama, and Qwen.This pattern confirms that finite-sample inaccessibility does not imply the absence of knowledge.
- Training dynamics: On-policy exploration eventually samples deeply suppressed facts, after which their reward signals are captured and amplified.