Source-linked AI summary
Co-Evolving Structured Knowledge and Reasoning in Language Models
Ryan Thomas Noonan, Linxi Zhao, Menghan Xu, Akanksha Sarkar, Mihir Mishra, Dongyoung Go, Kilian Q. Weinberger, Yoav Artzi, Jennifer J. Sun
TL;DR
Existing retrieval and structured-knowledge approaches offer complementary benefits but remain limited by irrelevant text context, costly KB construction, or decoupled reasoning. KBEVO jointly learns KB construction and reasoning with QA outcome rewards, using reasoning signals to shape the structured KB. Across benchmarks, it improves over its SFT version and reaches performance comparable to Search-R1 while preserving structured knowledge control and editing.
Problem
Parametric knowledge is lossy and hard to inspect or update, while structured KBs are costly to build and existing approaches decouple knowledge construction from downstream reasoning.
Method
KBEVO jointly trains structured KB construction and question answering end-to-end with QA outcome rewards, then reuses an offline-indexed KB for targeted retrieval.
Results
Across model scales and benchmarks, KBEVO-GRPO improves average EM over KBEVO-SFT by 5.1 points at 1.7B and 9.8 points at 4B, with performance comparable to Search-R1.
Takeaways & Limitations
KBEVO combines competitive RL-trained retrieval and reasoning with structured knowledge that can be inspected, edited, controlled, and reused.
Takeaways & Limitations
QA rewards supervise only retrieved triplets, so they do not guarantee faithfulness across the full constructed KB; reward hacking may also remain possible.
Abstract
from arXiv · showhide
Retrieval-augmented methods improve factual accuracy by grounding language models in external knowledge, but retrieving over unstructured text often introduces irrelevant context and offers limited control over the retrieved information. Structured knowledge bases offer a more controllable alternative, yet they are expensive to construct and often brittle to reason over. To address these limitations, we propose KBevo: a co-evolving framework that jointly learns to construct a structured knowledge base and reason over it for knowledge-intensive question answering. By optimizing both components end-to-end with QA outcome rewards, our method enables reasoning success to directly improve the quality of the constructed knowledge base. This leads to larger, better-connected knowledge structures with higher answer reachability, while also improving compositional factual reasoning and controllability compared to standard retrieval baselines.
1 Introduction
Parametric knowledge is difficult to inspect, correct, and update, while unstructured retrieval offers limited support for reasoning and control. KBEVO addresses this gap by jointly learning structured knowledge-base construction and reasoning with QA outcome rewards.
- Parametric factual knowledge can be lossy, conflated, and expensive to update because changing it typically requires retraining model parameters.
- Retrieval-augmented systems ground generation in external text, but fixed-size chunks prioritize coverage over reasoning.
- Structured knowledge bases store explicit, queryable facts that support compositional reasoning, interpretability, editing, and direct control over available knowledge.
- Structured KBs remain costly and difficult to scale because schema choices constrain coverage and high-quality construction often requires human annotation or frontier-model inference.
- Existing approaches decouple knowledge representation from reasoning, so reasoning failures cannot directly improve the knowledge store.
- KBEVO jointly trains KB construction and question answering end-to-end with QA outcome rewards, using an offline-indexed KB for downstream retrieval and reasoning.
- KBEVO provides competitive knowledge-intensive QA performance, direct training-free knowledge editing, and evidence of more complete, effective KBs and grounded reasoning.
2 Related Work
Prior work retrieves external text or reasons over fixed knowledge graphs, whereas KBEVO co-evolves structured knowledge construction and question answering. Its pipeline extracts triplets, retrieves values for targeted queries, and jointly optimizes both phases with GRPO.
- Retrieval-Augmented LLMs and Search-Based Knowledge Agents: RAG retrieves passages from external corpora, while search agents interleave multi-turn tool use or search with step-by-step reasoning.
- Knowledge Storage, Compression, and Controllability in LLMs: Parametric knowledge is compact but lossy and entangled, making factual information difficult to inspect, edit, update, or remove.
- Structured Knowledge Base Construction and Reasoning: Prior KG construction methods treat extraction, schema induction, canonicalization, or filtering as standalone pipelines, whereas KBEVO is schema-free and uses downstream QA supervision.
- Co-Evolving Framework: Phase 1 extracts entity-relation-value triplets into a graph, and Phase 2 answers questions through targeted retrieval of values for queried entity-relation pairs.
- GRPO Training: For each question, KBEVO samples K candidate KBs and M answer rollouts per KB, rewarding answers with F1 and KBs by their average downstream QA reward.
- GRPO Training: GRPO computes separate group-relative advantages for the two phases and applies a joint clipped-surrogate update over their combined rollout batch.
- Inference: After construction, the indexed KB is fixed and reused across queries, separating document processing from query-time targeted retrieval.
3 Experimental Setup
The experiments compare KBEVO with direct generation and text-retrieval baselines on three multi-hop QA benchmarks and PopQA. All retrieval methods use the same source documents, while KBEVO retrieves extracted triplets instead of text chunks.
- Training Setting: The study trains QWEN3-1.7B and QWEN3-4B on 7k HotpotQA examples, using 6k Gemini-generated SFT trajectories before 500 GRPO steps with F1-based rewards.
- Baselines: Baselines include direct generation, RAG over text chunks, IRCoT with iterative BM25 retrieval, and a matched QWEN3 reimplementation of Search-R1.
- Benchmarks and Metrics: Evaluation covers HotpotQA, MuSiQue, and 2WikiMultiHopQA for multi-hop QA, plus PopQA for single-hop factual QA, with exact match as the primary metric.
- Retrieval and Knowledge Base Setup: Text-based baselines index document chunks, whereas KBEVO constructs and retrieves a structured KB from extracted triplets over the same benchmark-specific documents.
4 Results
Across knowledge-intensive QA and controlled analyses, KBEVO combines competitive reasoning with editable structured knowledge, while co-evolution improves knowledge coverage and grounded reasoning. Results also show that joint optimization and SFT warmup are important for effective structured reasoning.
- KBEVO-GRPO increases average EM by 5.1 points at 1.7B and 9.8 points at 4B over KBEVO-SFT across three multi-hop QA benchmarks and PopQA.
- 41.3 vs. 42.4 average EM at 1.7B and 46.6 vs. 49.0 at 4B shows KBEVO-GRPO achieves comparable overall performance to SEARCH-R1.The comparison uses matched Qwen3 training setups; the methods have complementary strengths across benchmarks.
- Structured knowledge enables direct fact updates that KBEVO uses for multi-hop reasoning without retraining, remaining substantially stronger than Search-R1 after counterfactual edits.The evaluation uses ConFiQA-MR settings with original and counterfactual knowledge stores.
- GRPO consistently improves answer reachability over SFT across hop depths, encouraging knowledge paths that better support multi-hop reasoning.
- KBEVO increases fully grounded answers from 48% to 55% and reasoning correctness from 45% to 55% over the constructed knowledge base.The analysis uses Gemini-2.5-Flash as a judge on QWEN3-4B with 1K HotpotQA examples.
- The learned KB reaches 35.6 average EM versus 26.2 for EDC and 24.6 for AutoSchemaKG, while a Gemini-constructed KB reaches 37.2.These inference-time KB swaps hold the KBEVO-1.7B GRPO reasoning policy fixed.
5 Limitation
The framework has several acknowledged limitations, including reliance on SFT warm-starting and incomplete faithfulness supervision across the constructed knowledge base.
- KBEVO relies on SFT for warm-starting, though larger models with stronger prompting may reduce this dependence.
- QA rewards supervise only retrieved triplets during training, so they do not guarantee faithfulness across the full constructed KB.Unsupported triplets primarily appear to introduce retrieval noise, while broader reward coverage and explicit verification remain future directions.
- The framework may remain vulnerable to reward hacking, although analysis suggests it is not the dominant behavior.
6 Conclusion
KBEVO jointly optimizes structured KB construction and reasoning with outcome-based rewards, using a two-phase objective and phase-specific reward normalization. Its GRPO variant achieves strong benchmark performance while retaining an inspectable, reusable, and editable knowledge base.
- 6 Conclusion: KBEVO-GRPO achieves the best results on MuSiQue and 2Wiki at both model scales while retaining a structured, inspectable, and reusable knowledge base.
- 6 Conclusion: KBEVO jointly optimizes KB construction and reasoning with outcome-based rewards instead of treating KB construction as fixed preprocessing.
- 6 Conclusion: The joint objective combines KB and QA losses, with λ balancing the two phases and λ = M matching their total trajectory weights.
- 6 Conclusion: Phase 1 constructs knowledge bases, while Phase 2 performs QA conditioned on each constructed KB.
- 6 Conclusion: Importance ratios compare current and previous policies for KB construction and QA trajectories.
- 6 Conclusion: Phase-1 baselines average QA rewards across K candidate KBs, whereas Phase-2 baselines average rewards across M rollouts conditioned on one KB.
A.2 Training and Inference Cost
KBEVO concentrates extra training expense in SFT warm-starting and GRPO rollouts, then reuses offline-built knowledge bases to reduce inference-time computation.
- A.2 Training and Inference Cost: The SFT warm-start uses 6k traces, with a 0.4-epoch checkpoint already sufficient.
- A.2 Training and Inference Cost: Each lookup injects roughly ten tokens, whereas document-based search appends a full passage every turn and grows context across turns.
- A.2 Training and Inference Cost: At inference, offline KB construction is performed once per document and reused across queries.
- A.2 Training and Inference Cost: Per-query cost is one Phase-2 generation plus embedding lookups, without K×M sampling, a teacher model, or schema induction.
- A.2 Training and Inference Cost: The additional training expense is confined to the SFT warm-start and the K + K×M GRPO rollouts.
B.1.1 SFT Data Generation
Synthetic two-phase supervision uses a teacher model to construct knowledge bases from supporting context and generate database-lookup reasoning traces for supervised fine-tuning.
- B.1.1 SFT Data Generation: Gemini 2.5 Flash generates synthetic two-phase supervision for each training example.
- B.1.1 SFT Data Generation: In Phase 1, the teacher constructs a knowledge base from the supporting context.
- B.1.1 SFT Data Generation: In Phase 2, the teacher generates a reasoning trace that answers the question through database lookups over the constructed knowledge base.
- B.1.1 SFT Data Generation: Knowledge is represented as triplets of the form (entity, relationship, value), using explicit entity names rather than pronouns.
- B.1.1 SFT Data Generation: Relationships should capture each entity characteristic separately and be as precise as possible without requiring external lookup information.
- B.1.1 SFT Data Generation: Triplet extraction avoids multiple values for an insufficiently precise entity–relationship pair and supplies the context as the extraction input.
B.1.2 Database and Retrieval Implementation
KBEVO stores knowledge as entity–relationship–value triplets and retrieves matching pairs using embedding similarity, with reverse-index entries added to improve access and learning.
- The database stores (entity, relationship, value) triplets.
- Retrieval embeds (entity, relationship) pairs and returns the top k=4 nearest matches above a 0.6 cosine-similarity threshold.
- Reverse-index entries swap each triplet’s entity and value, improving initial SFT performance, reinforcement-learning ability, and final performance.
- KBEVO uses a two-stage training configuration: supervised fine-tuning followed by joint optimization with GRPO.
B.1.3 Training hyperparameters
KBEVO uses supervised fine-tuning followed by GRPO-based joint optimization, while the Search-R1 comparison uses matched training and retrieval settings with prompt adaptations for QWEN3.
- KBEVO training: KBEVO first fine-tunes QWEN3-1.7B and QWEN3-4B for 3 epochs on Gemini-generated two-phase reasoning traces.The SFT learning rate is 5 × 10^-5, with per-device batch size 24 and maximum sequence length 2048.
- KBEVO training: GRPO continues from the SFT checkpoint for 500 steps using an outcome-based F1 reward and jointly optimizes KB construction and reasoning.Each question samples K=4 KB rollouts and N=32 QA rollouts, with M=8 QA rollouts per KB candidate.
- Search-R1 comparison: Search-R1 is reimplemented on QWEN3 from base models without SFT, following its original multi-turn tool-interaction training paradigm.
- Search-R1 comparison: Search-R1 retrieves the top three passages from an E5-base-v2 FAISS index, truncating retrieved responses to 1,024 characters.
- Search-R1 comparison: QWEN3 prompt adaptations use <thinking> tags and a single three-hop in-context demonstration to encourage targeted search and concise reasoning.The prompt requires one brief sentence inside each thinking block before tool calls or answers.
B.3 ConFiQA Evaluation Details
The ConFiQA evaluation uses shared original or conflict-free counterfactual conditions, contrasting structured-KB retrieval with Search-R1 retrieval over corresponding contexts and judging grounding and reasoning quality.
- Dataset and conditions: The evaluation retains the first 1,000 examples after one seed-42 shuffle and uses identical source examples and ordering across conditions.
- Dataset and conditions: MR-ORIG uses original contexts, paths, answers, and aliases, while MR-CF-100 and MR-CF-356 contain 100 and 356 conflict-free counterfactual examples.The latter is the maximum feasible counterfactual count under the forward-key definition.
- Retrieval setup: KBEVO aggregates labeled triplets into a shared database for each setting, without giving the model the current question’s gold path directly.
- Retrieval setup: Search-R1 instead indexes the corresponding 1,000 full ConFiQA contexts and retrieves the top three passages from that closed corpus.
- Metrics: Performance is measured by normalized exact match against condition-specific answers and aliases after lowercasing, punctuation removal, article removal, and whitespace normalization.
- Judging protocols: Knowledge-base evaluation judges triplet faithfulness and soundness, including ambiguity, triviality, nonspecificity, malformed spans, and reversed roles.
- Judging protocols: Answer evaluation judges database grounding and multi-hop reasoning correctness using labels such as fully grounded, partially grounded, ungrounded, no answer, correct, minor error, and major error.
C Detailed Analysis
Detailed analyses find that co-evolution improves KB structure, reachability, grounding, and reasoning, but increases some unfaithful triplets and requires checkpoint-aware interpretation.
- Evaluation caveats: Earlier-checkpoint diagnostics in Section 4.3 and Appendix C use retrieval top-k = 1, unlike the main k = 4 results, so their absolute QA values are not directly comparable to Table 2.
- KB structure and coverage: Co-evolution produces larger, better-connected KBs, fewer failed or redundant lookups, higher triplet quality, and better multi-hop answer reachability.
- Faithfulness trade-offs: GRPO increases hallucination and soundness-issue rates relative to SFT despite improving coverage and connectivity.
- Faithfulness analysis: Only about 5% of constructed triplets are retrieved during reasoning, yet unused triplets are nearly as faithful as used triplets.
- Faithfulness analysis: Removing unfaithful triplets slightly improves EM, while GRPO improves EM and grounding even when retrieved triplets are all faithful.These analyses indicate hallucinated triplets primarily add retrieval noise rather than drive the gains.