Source-linked AI summary
DeepRefine: Agent-Compiled Knowledge Refinement via Reinforcement Learning
Haoyu Huang, Jiaxin Bai, Shujie Liu, Yang Wei, Hong Ting Tsang, Yisen Gao, Zhongwei Xie, Yufei Li, Yangqiu Song
TL;DR
Agent-compiled knowledge bases suffer from incompleteness, incorrectness, and redundancy, while refinement lacks scalable defect localization and golden reference actions. DeepRefine uses multi-turn query-conditioned reasoning, abductive diagnosis, targeted graph actions, and a Gain-Beyond-Draft reinforcement-learning reward. Across five datasets and out-of-distribution evaluations, it consistently improves downstream performance, though its training data and systematic refinement framework remain limited.
Problem
Agent-compiled knowledge bases contain incompleteness, incorrectness, and redundancy, while refinement must localize defects at scale and learn without golden references.
Method
DeepRefine performs multi-turn query-conditioned interaction, abductive defect diagnosis, targeted refinement actions, and end-to-end reinforcement learning with Gain-Beyond-Draft reward.
Results
DeepRefine consistently improves constructed knowledge-base quality and downstream performance across five datasets and out-of-distribution evaluations.
Takeaways & Limitations
DeepRefine provides a general approach for incrementally refining pre-constructed knowledge bases using user queries rather than rebuilding them.
Takeaways & Limitations
Training data is constructed only from HotpotQA with easy filtering criteria, and integration with a robust systematic refinement framework remains underexplored.
Abstract
from arXiv · showhide
Agent-compiled knowledge bases provide persistent external knowledge for large language model (LLM) agents in open-ended, knowledge-intensive downstream tasks. Yet their quality is systematically limited by \emph{incompleteness}, \emph{incorrectness}, and \emph{redundancy}, manifested as missing evidence or cross-document links, low-confidence or imprecise claims, and ambiguous or coreference resolution issues. Such defects compound under iterative use, degrading retrieval fidelity and downstream task performance. We present \textbf{DeepRefine}, a general LLM-based reasoning model for \emph{agent-compiled knowledge refinement} that improves the quality of any pre-constructed knowledge bases with user queries to make it more suitable for the downstream tasks. DeepRefine performs multi-turn interactions with the knowledge base and conducts abductive diagnosis over interaction history, localizes likely defects, and executes targeted refinement actions for incremental knowledge base updates. To optimize refinement policies of DeepRefine without gold references, we introduce a Gain-Beyond-Draft (GBD) reward and train the reasoning process end-to-end via reinforcement learning. Extensive experiments demonstrate consistent downstream gains over strong baselines.
1 Introduction
Agent-compiled knowledge bases make external knowledge evolvable, but systematic defects and the cost of rebuilding motivate targeted refinement. DeepRefine addresses these challenges through query-conditioned reasoning, reinforcement learning, and downstream evaluation.
- Agent-compiled knowledge bases evolve through source ingestion, article synthesis, and maintained cross-references over time.
- Their main defects are incompleteness, incorrectness, and redundancy, appearing as missing links, imprecise assertions, and ambiguity issues.
- Refinement must localize defects in large relational knowledge bases and optimize policies without golden refinement actions or knowledge bases.
- DeepRefine uses answerability judgment, error abduction, and refinement-action generation, with query-conditioned interactions that localize defective neighborhoods.
- DeepRefine introduces Gain-Beyond-Draft reward and end-to-end reinforcement learning to optimize refinement using downstream task utility.
- Experiments across five datasets show that DeepRefine steadily improves state-of-the-art baseline methods.
2 Related Work
Related work frames DeepRefine against agent memory, reinforcement learning for sequential LLM decisions, and established knowledge-refinement methods.
- Agent-compiled knowledge bases organize relational and multi-hop dependencies, while agent memory systems emphasize persistent, evolving external memory.
- Reinforcement learning improves sequential LLM decision-making through environmental interaction and reward feedback, including retrieval and search optimization.
- Classical knowledge refinement methods address existing artifacts through graph completion, entity alignment, and disambiguation techniques.
3 Preliminaries
The paper represents pre-constructed knowledge as relational triples and formulates refinement as generating action sequences conditioned on user queries. These actions update the knowledge base for downstream retrieval-augmented generation tasks.
- Knowledge bases contain triples whose items may be entities, events, or document-grounded spans, with relations covering taxonomy, time, causality, or cross-document linkage.
- The refinement task updates a pre-constructed triple set Gf using user queries Q to improve suitability for downstream RAG tasks.
- Refinement is formulated as generating action sequences for each query, with actions subsequently applied to improve Gf.
- The framework defines possible action sequences through the policy model pθ over an output set S.
4 Methodology
DeepRefine refines pre-constructed knowledge bases through query-conditioned interactions, abductive defect diagnosis, targeted edits, and reinforcement-learning optimization. Its pipeline expands retrieval when answers are insufficient, identifies incompleteness, errors, and redundancy, and directly updates the full knowledge base.
- Answerability Judgement Loop: DeepRefine begins with dense retrieval and expands the query-specific subgraph across interactions when answerability is unsatisfied.The process stops when the query becomes answerable or the maximum expansion depth is reached, retaining a bounded interaction history for later reasoning.
- Error Abduction: It abductively diagnoses potential incompleteness, errors, and redundancy from the query-specific interaction history.For multi-hop queries, diagnosis targets missing or incorrect intermediate items and relations, as well as ambiguity that may affect answerability.
- Refinement Actions Generation: DeepRefine generates direct edits to the full knowledge base rather than reconstructing and reinserting retrieved subgraphs.The generated actions use edge insertion, edge deletion, and node replacement to address missing, incorrect, redundant, or ambiguous knowledge.
- Reward Design: The Gain Beyond Draft reward measures the change in task-specific generation accuracy between answers produced from refined and draft knowledge bases.Both answers use dense retrieval, while the reward compares their accuracy against the golden answer.
- Policy Optimization: DeepRefine’s refinement policy is optimized with GRPO using rewards computed from the refinement process.Training generates reasoning and refinement actions token by token, with group-relative advantages and clipped updates.
- Inference-Time Deployment: After training, a frozen DeepRefine can refine a pre-constructed knowledge base alongside an online query stream and an evolving knowledge-base stream.This supports incremental refinement during subsequent user interactions.
5 Experiments
DeepRefine is evaluated in OOD RAG and conversation settings across five datasets, constructors, and retrievers. It generally improves downstream performance while requiring less time than full knowledge-base reconstruction, though RL training is important for reliable gains.
- Experimental Setup: Experiments cover five OOD QA datasets spanning simple, multi-hop, and conversation question answering.The evaluation uses NQ, PopQA, 2WikiMultihopQA, Musique, and LOCOMO.
- Experimental Setup: DeepRefine is tested across Naive, AR1, and Graphify constructors with multiple graph and graph-based text retrievers.The retrievers include Subgraph Retriever, ToG, HippoRAG, and HippoRAG2.
- Main Results: DeepRefine-8B consistently improves downstream performance across most constructor–retriever combinations, with only marginal and slight degradations in some cases.The reported results are evaluated under out-of-distribution settings, and DeepRefine-8B generally performs better than DeepRefine-4B.
- Main Results: 53.02 vs. 52.61, HippoRAG2+DeepRefine with the naive constructor exceeds HippoRAG with AR1 on 2WikiQA.In Musique, HippoRAG+DeepRefine with the naive constructor also exceeds HippoRAG with AR1, 28.27 vs. 26.21.
- Efficiency: DeepRefine is more efficient than AutoGraph-R1 reconstruction because it targets potentially problematic regions rather than regenerating the full knowledge base.Code-formatted refinement actions and greedy maximum-coverage query selection further reduce processing and duplicated refinement.
- Ablation Study: Without RL training, DeepRefine yields marginal improvements or worse performance, indicating that RL optimization is important for using interaction history effectively.Table 3 compares the GBD-reward-finetuned model with DeepRefine-8B without RL under AR1 and Graphify constructors.
6 Conclusion
DeepRefine iteratively refines pre-constructed knowledge bases using multiple interactions and a GBD-reward-trained reinforcement-learning policy. Under OOD settings, it consistently improves knowledge-base quality and downstream performance over the original knowledge base.
- DeepRefine iteratively refines any pre-constructed knowledge base through multiple interactions and a GBD-reward-trained reinforcement-learning policy.
- Under OOD settings, DeepRefine consistently improves constructed knowledge-base quality and downstream performance compared with the original knowledge base.
A Details for Metrics
Benchmark evaluation uses token-level F1 as the evaluation metric.
- Token-level F1 is used as the benchmark evaluation metric.
A.2 Metrics for GBD Reward
The GBD reward uses Generation Accuracy, a combination of span matching and LLM-based correctness checking. Its transition-based reward matrix assigns the strongest incentive to successful draft-to-refined corrections.
- Generation Accuracy combines span matching with an LLM-based correctness check using DeepSeek-V3 as judge.
- The transition-based reward matrix compares draft and refined answer GenAcc across four transition states.
- A successful GenAcc correction from 0 →1 receives the highest reward, +1.0, encouraging evidence-based error correction.
B Coverage-Based Query Selection
The method selects refinement queries by greedy maximum coverage over query-related triples, then refines only the selected queries before evaluating all benchmark queries.
- Candidate queries are represented by top-k related triples plus up to m one-hop neighborhood triples ranked by embedding similarity.
- Table 4 reports hyperparameters for greedy maximum-coverage query selection.
- Greedy selection chooses queries maximizing newly covered triples until reaching budget B or coverage threshold ρ.
- Only selected queries pass through refinement, after which all benchmark queries are evaluated with the standard RAG pipeline.
C Additional Experiments
The benchmark evaluates refinement on controlled incompleteness, incorrectness, and redundancy defects using 501 HotpotQA-based samples. Results show the agentic framework improves downstream performance across issue types, with DeepRefine-8B outperforming Qwen3-8B, especially on redundancy.
- Benchmark Construction: The benchmark contains 501 HotpotQA-based samples, with 167 samples for each of three systematic error types.Each sample includes the original query, corrupted knowledge base, and corruption metadata.
- Benchmark Construction: Controlled corruption removes critical triples for incompleteness, alters correct claims for incorrectness, and inserts or modifies triples for redundancy.These strategies simulate missing evidence, imprecise information, and ambiguity in knowledge bases.
- Results: Both DeepRefine-8B and Qwen3-8B improve downstream performance across incompleteness, incorrectness, and redundancy defects.The evaluation uses a 2-hop interaction setting on the constructed benchmark.
- Results: Incompleteness causes the largest performance degradation and is easiest to improve, whereas redundancy causes moderate degradation and is hardest to resolve.Adding supporting evidence is more direct than correcting incorrect information or resolving coreference and disambiguation.
- Results: DeepRefine-8B outperforms Qwen3-8B on all three issue types, with the largest advantage on redundancy.The base model has almost no effectiveness on redundancy, while DeepRefine-8B improves performance to a certain extent.
- Case Studies: In a case study, DeepRefine inserts missing population edges linking a tourist city to its 2010 population value, shortening the reasoning path.The added edges make the answer directly retrievable.
- Case Studies: In a factual-contradiction case, DeepRefine deletes an incorrect edge and inserts the evidence-supported runner-up relation.This preserves alignment between the reasoning chain and supporting paragraphs.
D.3 Case 3: Redundancy
The redundancy case studies examine coreference resolution and entity disambiguation, showing how ambiguous knowledge items can mislead retrieval. DeepRefine locates the problematic subgraph and adds or rewrites information to make references more specific.
- Redundancy Cases: The redundancy cases focus on coreference resolution and disambiguation.Both issues can introduce ambiguity into query-related knowledge triples.
- Coreference Resolution: A coreference error represents “the girl’s phone number” without resolving which person the phrase refers to.The ambiguity remains even when contextual information appears in the query-related triples.
- Disambiguation: A disambiguation error arises when distinct people share the same name string in the knowledge base.This extraction strategy can introduce structural noise into related triples.
- Disambiguation: DeepRefine resolves the disambiguation case by locating the problematic subgraph and replacing “Ray Taylor” with a date-qualified identity.The added specificity prevents later same-name entities from being confused.
- Prompt Templates: DeepRefine uses separate prompt templates for answerability judgement, error abduction, and refinement action generation.The templates respectively assess KG-based answerability, analyze interaction-history error reasons, and generate KG refinement actions.
F Limitations
The paper identifies training-data quality and systematic refinement as unresolved limitations. Its training data comes from HotpotQA with easy filtering criteria, and robust systematic refinement remains underexplored.
- Limitations: The training data is constructed from HotpotQA with easy filtering criteria, limiting its quality.Higher-quality and more diverse training data is identified as a future direction.
- Limitations: Combining DeepRefine with a robust systematic refinement framework remains underexplored.The authors identify this combination as a way to improve performance and stability.