Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs

1 Introduction

• Reasoning models outperform instruct counterparts by 13 and 24 percentage points on PopQA and MedConceptsQA, respectively, motivating hierarchical navigation rather than direct recall.The comparison is observational; publicly released variants may combine multiple post-training stages, so it does not isolate RL.Source: §1 · p. 1
• Structured prompting cuts the MedConceptsQA gap between DeepSeek-V3 and DeepSeek-R1 from 24pp to 7pp, suggesting instruct models possess knowledge but lack a reliable navigation policy.The figure contrasts direct recall with hierarchical traversal; the structured prompt explicitly guides traversal before answering.Source: §1 · p. 2, Figure 1 · p. 2
• RL on unseen facts improves recall of held-out, frequent but previously non-extractable facts, while the gain mostly comes from iterated on-policy exploration.Across objectives, improvements track more frequent refreshes of self-generated rollouts; contrastive GRPO adds little once iteration is present.Source: §1 · p. 2
• The reasoning–instruct traversal gap widens with retrieval depth, indicating that reasoning models’ advantage grows with retrieval complexity.The paper evaluates path recall using IPC-CAR and DBO-CAR datasets and a path-matching score.Source: §1 · p. 2
• Layerwise analysis finds declarative fact representations remain similar across base and reasoning models while query representations diverge, linking post-training differences to question processing.The analysis compares matched queries with declarative statements of the same facts.Source: §1 · p. 3
• Distilled models may imitate self-correction without abandoning incorrect hypotheses, and often underperform reasoning models on knowledge recall.The authors report that this behavior leaves distilled models without the trial-and-error exploration needed to navigate the hierarchy.Source: §1 · p. 3

2 Structured Prompting Reveals Latent Knowledge in Instruct Models

• Across flat factual associations and formally organized code systems, structured prompting shrinks the instruct–reasoning accuracy gap.Response distributions and reasoning traces also support the interpretation that instruct models have the knowledge but lack a navigation policy.Source: §2 · p. 3

2.1 Experimental Setup

• The evaluation covers eight instruct–reasoning pairs from five model families and relational, taxonomic, and ontological factual-recall benchmarks.Benchmarks include PopQA, IPC-Lookup, and MedConceptsQA, in multiple-choice and open-ended formats.Source: §2.1 · p. 4
• The study compares direct answers, free-form chain-of-thought, and structured prompts that require traversal of the relevant knowledge hierarchy.Structured prompting tests whether explicit hierarchical recall reduces the instruct–reasoning performance gap.Source: §2.1 · p. 4
• Majority-voted accuracy is prioritized over pass@k to measure reliable knowledge access rather than the knowledge boundary.Sampling uses temperature 0.8, top-p 0.7, and up to 10,000 tokens across three independent runs.Source: §2.1 · p. 4

2.2 Structured Prompting Narrows the Gap Between Instruct and Reasoning Models on Knowledge Retrieval

• 23.7 pp to 7.5 pp: structured prompting shrinks the DeepSeek-V3–DeepSeek-R1 gap on MedConceptsQA; PopQA gaps also narrow for Kimi-K2.5 and Qwen2.5-32B.The PopQA gaps decrease from 12.7 pp to 3.3 pp for Kimi-K2.5, and from 13.4 pp to 6.5 pp for Qwen2.5-32B.Source: §2.2 · p. 4

2.3 Instruct Models Already Have Parametric Knowledge But Needs Better Access

• On MedConceptsQA, structured prompting shifts Kimi-K2.5-Instruct from 29% all-incorrect to 6%, while Kimi-K2.5-Think remains near its ceiling.For Kimi-K2.5-Instruct, all-correct responses increase from 60% to 78%.Source: §2.3 · p. 4, Figure 2 · p. 5
• In response analyses, instruct models retrieve correct answers under structured prompting by explicitly traversing code or concept hierarchies, while reasoning models do so spontaneously.The authors interpret this contrast as instruct models lacking the navigation policy that reasoning models internalize.Source: §2.3 · p. 5

3 Isolating RL’s Effect on Unlocking Inaccessible Knowledge Through Controlled Injection

• The controlled design trains on unseen, non-extractable facts and tests on a disjoint set of frequent but non-extractable facts.Relation-disjoint splits help test whether gains extend beyond exposure to the training facts.Source: §3 · p. 5, §3.1 · p. 5

3.2 RL Unlocks Latent Knowledge Better Than SFT

• RL raises held-out recall from 0% to 10.1% on Qwen3-8B-Instruct and from 0% to 7.05% on Qwen3-30B-A3B-Instruct, versus 0.96% and 0.64% with SFT.Training and test facts are relation-disjoint, and test facts are not used during fine-tuning.Source: §3.2 · p. 5

3.3 On-Policy Exploration Drives the Gain

• The setup compares objectives that vary self-generated data, rollout refresh frequency, and whether incorrect rollouts provide a negative gradient while holding data and hyperparameters fixed.Figure 3 plots accuracy points with horizontal 95% Wilson confidence intervals.Source: Figure 3 · p. 5, §3.3 · p. 6
• Iterated RFT reaches 9.62% held-out recall versus 2.40% for Static RFT, while Online GRPO reaches 10.1% and is statistically indistinguishable from Iterated RFT.Within GRPO, recall rises from 3.37% to 7.69% to 10.1% as rollouts become fresher.Source: §3.3 · p. 6
• Using the model’s own correct rollouts is necessary but insufficient: recall rises only from 0.96% to 2.40% without iteration.The authors conclude that iterated on-policy exploration accounts for most of the gain; GRPO’s contrastive advantage contributes little once iteration is present.Source: §3.3 · p. 6

4 Reasoning Improves Hierarchical Navigation Under Greater Retrieval Complexity

• The IPC-CAR and DBO-CAR datasets stratify questions by retrieval depth to test whether navigation performance changes with traversal complexity.The reported results show superior hierarchical traversal by reasoning models as depth increases.Source: §4 · p. 6

4.1 Experimental Setup

• IPC-CAR asks for the nearest common ancestor of two leaf codes, with retrieval depth defined as the total edges along both upward paths.DBO-CAR is an ontology-grounded counterpart testing CamelCase concepts across 768 DBpedia classes.Source: §4.1 · pp. 6–7

4.2 Reasoning Models Show Superior Path Recall When Traversal Depth Increases

• At depth 2, V3 leads R1 in PathMatch, 0.865 to 0.725; R1 overtakes from depth 4 onward, with a +0.107 gap at depth 6.Final-answer accuracy shows the same directional crossover, but PathMatch captures the depth-dependent widening; DBO-CAR shows a similar pattern.Source: Figure 5 · p. 7, §4.2 · pp. 7–8

4.1 Experimental Setup

• PathMatch averages ancestor-set F1 with common-subsequence path-order fidelity.The metric is intended to capture both set-level accuracy and order fidelity in hierarchical recall.Source: §4.1 · p. 7

5.1.1 Experimental Setup

• The analysis sampled 100 ICD9PROC question–answer probes, extracted final-token hidden states across layers, and compared cosine similarity across models and within question–answer pairs.Source: §5.1.1 · p. 8

5.1.2 Instruct and Reasoning Models Reshape Question Processing, Not Stored Knowledge

• Reasoning-model Q–A similarity fell from 0.878 at layer 0 to 0.161 at layer 63, and question representations diverged more than answers across post-trained models.Source: §5.1.2 · pp. 8–9
• Instruct and reasoning variants remained close to base (mean Q/A similarity 0.966/0.976 for instruct), unlike distilled (0.808/0.892); question representations diverged more than answers.Source: §5.1.2 · p. 9

5.2 Distillation Captures Surface Patterns Without Navigation

• On MedConceptsQA, the distilled Qwen2.5-32B outperformed base but remained below instruct and reasoning, even under structured prompting.Source: §5.2 · p. 9, Table 2 · p. 9
• Among 40 wrong distilled responses, 80% simulated self-correction without abandoning the initial hypothesis; the other 20% backtracked among answers, consistent with surface imitation without exploration.Source: §5.2 · p. 9

• The paper concludes that RL improves access to knowledge rather than what models store, reframing instruct–reasoning gaps on knowledge-intensive tasks as an access bottleneck.Source: §6 · p. 9
• Structured prompting recovers much of the instruct–reasoning retrieval gap, offering a low-cost, training-free way to guide models through knowledge already stored in their parameters.Source: §6 · p. 9
• Knowledge-injection results support access to latent facts, while performance gains with more frequent rollout refreshes point to iterated on-policy exploration as a main driver.Source: §6 · p. 9
• The reasoning advantage widens with retrieval depth; layerwise analysis finds query representations change more than factual representations, while distilled models imitate self-correction without exploratory behavior.Source: §6 · p. 9

• Co-occurrence counts are only a proxy for whether data appeared during pretraining, not a definitive measure of exposure.Source: Limitations · p. 10
• The controlled knowledge-injection experiment cannot fully account for differences between instruct and reasoning model pairs because their training recipes are closed.Source: Limitations · p. 10
• The exposure-proxy uncertainty and closed-recipe differences constrain how decisively the experiments can isolate RL’s effects on knowledge access.Source: Limitations · p. 10

Ethical considerations

• Improved benchmark performance should not be taken as evidence of reliable real-world factuality; better access may also retrieve incorrect or outdated facts.Source: Ethical considerations · p. 10

A.1 The Alignment Tax and Factual Degradation

• Prior work reports factual degradation from alignment, but this study finds reasoning models outperform base models on structured knowledge recall, suggesting the alignment tax may not affect all retrieval forms uniformly.Source: §A.1 · p. 14

A.2 Retrieval-augmented generation (RAG)

• Unlike RAG and tool-using systems that add external evidence, this work tests access to knowledge already stored in parameters, separating it from external retrieval pipelines.Source: §A.2 · p. 14

A.3 RL and Prompting for Reasoning

• Unlike prior work using RL and prompting to improve reasoning, prompting serves here as a diagnostic: hierarchy-only instructions test access to existing parametric memory.Source: §A.3 · p. 14

A.4 Reasoning for Knowledge Retrieval

• The study uses frequency-stratified, relation-disjoint experiments to separate access from exposure, and compares SFT, RL, and distillation across structured and unstructured retrieval.Source: §A.4 · pp. 14–15

A.5 Knowledge Storage versus Knowledge Access

• The access-versus-storage question is examined through prompting, response distributions, injection experiments, layerwise analysis, and distillation failure modes; Table 3 defines recurring terminology.Source: §A.5 · p. 15, §B.1 · p. 15

B.2 Zero-Shot Prompt Templates

• The prompt formats are direct QA with a single answer letter, free-form chain-of-thought, and structured traversal through relevant facts or codes.Source: §B.2 · p. 15

B.3 Evaluation Protocol for Structured Prompting Experiments

• PopQA uses exact match, IPC and MedConceptsQA use multiple-choice accuracy, and all model pairs are evaluated zero-shot; only 5,000 PopQA questions are sampled.Source: §B.3 · p. 15

B.4.1 IPC-Lookup

• IPC-Lookup questions pair patent codes with four descriptions, using adjacent-node distractors and excluding descriptions that reveal the code.Answers are randomly assigned to options, and two annotators cross-check the manually constructed questions.Source: §B.4.1 · p. 15

B.4.2 IPC-CAR

• IPC-CAR probes upward traversal through the taxonomy with 100 questions spanning depths 2–6, checked by two annotators.Distractors come from neighboring nodes, and answer options are randomly permuted.Source: §B.4.2 · p. 16

B.4.3 DBO-CAR

• DBO-CAR uses 500 questions across depths 2–6, following the IPC-CAR construction procedure.Source: §B.4.3 · p. 16

B.5 Controlled Knowledge Injection through RL

• For controlled RL experiments, questions the instruct model missed across three runs formed relation-disjoint low-frequency training and high-frequency held-out sets.The sets contained 731 training questions and 208 held-out questions; SFT and GRPO training used the same setup, with results reported at convergence.Source: §B.5 · p. 17, Table 4 · p. 18

B.6 Layerwise Similarity Metric: Formal Definition

• Layerwise representation similarity is measured with cosine similarity across probe representations from base and post-trained models, including question and answer components.The comparisons include both inter-model and within-model pairs.Source: §B.6 · p. 18

C.1 Extended Model Performance Across Prompt Types

• Across PopQA, IPC, and MedConceptsQA, the instruct–reasoning accuracy gap consistently narrows as prompt complexity increases.Source: §C.1 · p. 18

• On MedConceptsQA, DeepSeek-V3 and Qwen3-235B-A22B-Instruct fail under direct QA and CoT but succeed with structured prompting and explicit hierarchical traversal.Their reasoning counterparts reach the correct answer without structural guidance, consistent with internalizing the traversal policy supplied externally by the structured prompt.Source: §C.2 · p. 18
• On PopQA and IPC, instruct models recover only under structured prompting, while their reasoning counterparts succeed directly.For direct QA, reasoning-model internal reasoning is reported because their outputs contain only short final answers.Source: §C.2 · p. 19

C.3 Additional Depth-Stratified Results on DBO-CAR

• DBpedia path matching shifts from a DeepSeek-V3 lead at depth 2 to a DeepSeek-R1 lead at depth 6, while final-answer accuracy saturates across both models.The PathMatch gap is −0.083 at depth 2 and +0.054 at depth 6; path matching reveals the depth-dependent divergence that final-answer accuracy does not.Source: §C.3 · p. 19

C.4 Additional Layerwise Cosine Similarity Results

• Across Qwen2.5-32B variants, question representations diverge more than answer representations, and the distilled variant diverges most from the base model.The analysis partitions 64 layers into early, middle, and late bands; the Mistral family is also examined on ICD9PROC vocabulary.Source: §C.4 · p. 19

C.5 Failure Modes of Distilled Models Under Structured Prompting

• Distilled models underperform reasoning models across datasets and can fail to match instruct or base variants even with structured prompting.Observed failure patterns include surface-level self-correction that stays anchored to an initial hypothesis and backtracking that ends at the wrong option.Source: §C.5 · p. 19

D.1 Artifacts License and Intended Use

• The evaluated artifacts cover factual-recall tasks across real-world entity relations, medical codes, patent codes, and DBpedia ontology classes, all in English.The paper reports use of these artifacts and tools for research on language-model evaluation and analysis.Source: §D.1 · pp. 19–20

D.2 Model Size and Computational Budget

• The model suite ranges from 8B models to 1T-parameter mixture-of-experts models, with about 100 GPU hours used across inference, dataset construction, and RL/SFT experiments.Exact GPU type and hours are unavailable because training used serverless distributed GPU resources.Source: §D.2 · p. 20

• AI assistants were used for literature search and grammar and clarity improvements; the authors reviewed and verified everything.Source: §D.3 · p. 20
• The structured-navigation prompt directs MCQ models through hierarchical code recall and option elimination, while PopQA models identify a fact type, recall relevant knowledge, and answer.The supplied prompt captions also identify direct-QA and CoT templates and a prompt for generating IPC-CAR data.Source: Figure 15 · p. 22, Figure 16 · p. 22, Figure 17 · p. 23
• The appendix presents reasoning trajectories across prompt templates for model pairs on PopQA, IPC, and MedConceptsQA, including examples of structured-prompt answers.Source: Figure 20 · p. 27, Figure 21 · p. 28, Figure 22 · p. 29
• The Qwen2.5-32B tables compare layerwise question and answer similarities and model accuracy across datasets and prompt types.The failure-mode table categorizes 40 incorrect responses from a distilled model under structured prompting.Source: Table 9 · p. 31, Table 10 · p. 31, Table 11 · p. 31