Weakness Localization Is Not Enough: State-Aware Recursive Knowledge Improvement for Large Language Models

1 Introduction

• The same localized weakness can reflect unavailable knowledge, unreliable access, or accessible knowledge that is ineffective for downstream use, and these states can favor different interventions.Factual exposure, recall-focused training, and application-oriented examples address different bottlenecks within the same knowledge region.Source: §1 · p. 2
• An observed weakness may persist after an update even as its underlying state changes, so the appropriate intervention can change before the weakness disappears.Continuing the previous intervention may then fail to address the new bottleneck.Source: §1 · p. 2
• RKI aligns diagnosis, target knowledge, and intervention records in graph-structured units, then estimates states, selects interventions, and re-evaluates after each update.The loop revises both weakness and state profiles before selecting the next intervention.Source: §1 · p. 2
• 18.41 points over the strongest baseline on average across HOTPOTEVAL and SAFEINTERNALEVAL.RKI achieves the best performance on both benchmarks.Source: §1 · p. 2
• 5.17 points over Weakness Only under the same weakness information, while 57.1% of units with persistent application weakness show a different recognition–recall pattern after an update.These analyses show that states add guidance beyond weakness localization and can change while the observed weakness persists.Source: §1 · p. 2

2.1 Knowledge Incorporation for Large Language Models

• Knowledge incorporation spans domain adaptation, training-data construction, parameter editing, and output post-processing, but often retains predefined update strategies (Gururangan et al., 2020; Cheng et al., 2024b;a).These approaches typically make limited use of evaluation feedback to adapt subsequent knowledge updates.Source: §2.1 · p. 3

2.2 Evaluation-Guided Targeted Model Improvement

• Evaluation-guided methods target knowledge gaps at increasingly fine diagnostic granularity, yet localization may still leave intervention choice underdetermined (Chen et al., 2025; Wei et al., 2026).Other examples connect agent-based detection or hierarchical and rubric-level evaluation to targeted improvement (Cheng et al., 2024c; Zeng et al., 2025; Gupta et al., 2026).Source: §2.2 · p. 3

2.3 Knowledge States in Large Language Models

• Prior work distinguishes knowledge acquisition, recall accessibility, expression, and downstream use; RKI uses estimated states to select interventions and re-estimates them after updates (Kazemnejad et al. (2023); Gekhman et al. (2025); Calderon et al. (2026)).Earlier studies primarily examine these distinctions as properties to measure or diagnose, with limited attention to their role in intervention selection.Source: §2.3 · p. 3

3 Knowledge States for Knowledge Improvement

• RKI uses shared graph-structured knowledge units to align diagnostic evaluation with intervention-data generation across rounds.The units provide a common basis for the observations used to localize weaknesses and for targeted intervention.Source: §3 · p. 3

3.1 Operational Knowledge State

• The operational knowledge state is a model-level profile estimated from diagnostic evidence that captures knowledge availability, reliable access, and effective downstream use.It complements the weakness profile, which identifies deficient knowledge units, by characterizing the broader condition relevant to addressing them.Source: §3.1 · p. 4

3.2 Knowledge Intervention and State Transition

• An update can change the underlying state while a weakness persists, so RKI re-evaluates the model and re-estimates its state before choosing the next intervention.An intervention suited to the current state may no longer fit the state after updating.Source: §3.2 · pp. 4–5

3.3 Knowledge-State Aliasing

• When one weakness is compatible with states having disjoint optimal interventions, no rule conditioned only on that weakness can be optimal for both states.Operational state therefore supplies intervention-relevant information beyond the weakness profile.Source: §3.3 · p. 5

4 Recursive Knowledge Improvement

• RKI alternates diagnostic evaluation, state estimation, targeted intervention, and re-evaluation as a sequential, state-aware improvement process.Source: §4 · p. 5

4.1 Framework Overview

• At each round, diagnostic evidence and intervention history estimate the operational state, while localized weaknesses identify intervention targets and the state guides strategy.Source: §4.1 · pp. 5–6

4.2 Graph-Structured Knowledge Units

• Graph-structured knowledge units provide a shared basis for probes, intervention-data generation, and re-evaluation across rounds.Source: §4.2 · p. 6

4.3 Operational Knowledge-State Estimation

• RKI estimates availability, accessibility, and usability using acquisition, recognition, recall, understanding, reasoning, and application evidence, alongside intervention history.Source: §4.3 · p. 6

4.4 State-Aware Intervention and Recursive Re-estimation

• The target unit identifies what to improve, while its state guides strategy: candidate interventions address exposure, reliable retrieval, or downstream use, and are followed by updated diagnosis.Source: §4.4 · pp. 6–7

5.1 Experimental Setup

• Experiments use HOTPOTEVAL and SAFEINTERNALEVAL source corpora, a shared Qwen3-4B-Instruct-2507 backbone, and matched baseline settings.Source: §5.1 · p. 7

5.2 Knowledge-State Aliasing

• Among recall-failure instances, 37.8% still pass recognition and 62.2% fail it; higher-order failures likewise show varied recognition and recall patterns.For application failures, 29.1% fail both recognition and recall on measured associated facts, while 13.3% recall all of them; the corresponding pairs are 18.9% and 3.6% for understanding, and 36.6% and 5.3% for reasoning.Source: Figure 3 · p. 8, §5.2 · p. 8

5.3 Overall Knowledge Improvement

• Across both benchmarks, RKI averages 45.81, 18.41 points above the strongest-baseline average of 27.40, and achieves the best performance on each.Source: §5.3 · p. 8, Table 3 · p. 9

5.4 Recursive Knowledge-State Re-estimation

• With application weakness persisting after an intervention, 57.1% of units change recognition–recall pattern, while full recall of measured associated facts rises from 12.2% to 40.9%.Source: Figure 4 · p. 9, §5.4 · p. 9
• Overall score rises from 17.41 at Phase 0 to 31.50, 51.03, and 58.92 across updates, ending 8.42 points above the 50.50 full-data-training reference.Source: Figure 5 · p. 9, §5.4 · p. 9
• From Phase 2 to Phase 3, recall increases from 37.1 to 38.9, understanding from 62.1 to 69.6, and application from 56.4 to 69.2.Acquisition reaches 1 after the first intervention because all target facts are injected; this indicates exposure, not reliable recognition, recall, or use.Source: §5.4 · p. 9

5.5 Ablation and Analysis

• On SAFEINTERNALEVAL, Fixed Utilization scores 48.00 versus Fixed Access at 23.17, while Weakness Only scores 53.75 versus Full-Data Training at 50.50.Source: Table 4 · p. 10, §5.5 · p. 10
• RKI scores 58.92, 5.17 points above Weakness Only, while Frozen State scores 39.00, below both RKI and Weakness Only.The results support state-aware intervention selection beyond weakness localization and recursive re-estimation rather than reusing stale state estimates.Source: §5.5 · p. 10

• RKI aligns diagnosis, intervention, and re-evaluation through graph-structured knowledge units and selects interventions using the model’s current operational knowledge state.Source: §6 · p. 10
• 18.41 points: RKI improves the average score across HOTPOTEVAL and SAFEINTERNALEVAL over the strongest baseline.Source: §6 · p. 10
• Analyses show operational knowledge states provide intervention-selection information beyond weakness localization and can change after updates even when the observed weakness persists.Source: §6 · p. 10

• Operational knowledge state is estimated from diagnostic evidence and intervention history, not recovered as the model’s true latent state.Its effectiveness depends on the chosen diagnostic dimensions and estimation procedure.Source: §7 · p. 10
• RKI selects among a predefined set of candidate interventions and re-evaluates after each update, adding evaluation and training costs.The authors suggest exploring more flexible state representations and intervention spaces, and more efficient recursive improvement while preserving effectiveness.Source: §7 · p. 10

A Proof of Proposition 2

• No intervention rule conditioned only on an observed weakness can be optimal for two compatible states when those states require different optimal interventions.Source: A Proof of Proposition 2 · p. 13

B RKI Knowledge Units and Diagnostic Dimensions

• Detailed definitions of the knowledge units and diagnostic dimensions introduced in Section 4 are provided in the supplementary material.Source: B RKI Knowledge Units and Diagnostic… · p. 13

C Prompt Templates

• The prompt-template appendix shows knowledge-graph and relation-fact extraction prompts; the pipeline also uses prompts for iterative extraction, filtering, benchmark generation, and CPT- and SFT-data generation.Continuation prompts request entities and relations missing from previous rounds, while termination prompts determine whether extraction should continue.Source: C Prompt Templates · pp. 13–14, Figure 6 · p. 14, Figure 7 · p. 15

D Knowledge Graph Visualization

• The Hotpot graph visualization shows a two-hop Star Wars neighborhood with 130 entities and 162 edges, plus a selected subgraph of 15 entities and 25 edges.Colors and shapes distinguish entity types; all edges are undirected.Source: D Knowledge Graph Visualization · p. 16, Figure 8 · p. 16

E.1 Training Configurations

• CPT and SFT use separate training configurations, summarized in Table 7.Source: §E.1 · p. 16

E.2 Inference Configurations

• All models and benchmarks use greedy decoding, a 5,120-token maximum input, a 2,048-token maximum generation, and disabled thinking mode.Source: §E.2 · p. 16
• Table 8 presents representative source-corpus and evaluation examples from the public benchmarks.Source: Table 8 · p. 17
• Rubric dimensions are scored from 0 to 10, with scores of at least 6 passing; internal-benchmark understanding measures association coverage, while reasoning and application measure reasoning coverage.Source: Table 9 · p. 17

• Each benchmark pairs a source corpus for graph construction, diagnostic evaluation, and knowledge incorporation with a QA set for closed-book evaluation.Source: §E.3 · p. 17
• HOTPOTEVAL derives from HotpotQA, whose questions require combining evidence across multiple supporting passages, with aligned questions retained for evaluation.Source: §E.3 · p. 17
• SAFEINTERNALEVAL uses manually constructed, source-reviewed questions designed to avoid overlap with intervention-data knowledge units; its proprietary documents are not released or shown.Source: §E.3 · p. 17

E.4 Evaluation and Judging Protocol

• Open-ended evaluation uses DeepSeek-V4-Pro at temperature 0, scores each benchmark-specific dimension from 0–10, and counts a pass only if every dimension scores at least 6.Source: §E.4 · p. 17

E.5 Agreement Between LLM Judgments and Human Annotations

• Human annotators independently apply the same rubric and pass threshold to 100 stratified instances per benchmark without seeing LLM judgments.Source: §E.5 · p. 18
• 98% pass agreement on HOTPOTEVAL and 96% on SAFEINTERNALEVAL, with Cohen’s κ of 0.95 and 0.89, respectively, between LLM and human judgments.Source: §E.5 · p. 18, Table 10 · p. 18