PaperlayerView exact revision on arXiv

Source-linked AI summary

KiRAG: Knowledge-Driven Iterative Retriever for Enhancing Retrieval-Augmented Generation

Jinyuan Fang, Zaiqiao Meng, Craig Macdonald

arXiv:2502.18397v1cs.CL

TL;DR

Multi-hop QA requires information gathered across reasoning steps, but iterative retrievers can be disrupted by noisy context and may not adapt to changing information needs. KiRAG iteratively retrieves knowledge triples to guide reasoning, and it improves retrieval and QA performance on multi-hop QA benchmarks.

  • Problem

    Existing iRAG can be disrupted by irrelevant documents or inaccurate thoughts, and its retrievers may not adapt to the changing information needs of multi-step reasoning.

    Source: IRDoc · pp. 1–2

  • Method

    KiRAG iteratively retrieves knowledge triples while building a reasoning chain that identifies missing knowledge needed to extend it.

    Source: §3.2 · p. 3

  • Results

    9.40% average R@3 and 5.14% average F1 improvements over existing iRAG models on multi-hop QA are reported for KiRAG.

    Source: §6 · p. 8

  • Takeaways & Limitations

    KiRAG achieves comparable retrieval performance to IRCoT, the best-performing baseline, on the single-hop QA dataset NQ.

    Source: §4.2 · p. 7

    The Aligner uses silver training data from only three multi-hop QA datasets, and KiRAG trains the Aligner while keeping the Constructor frozen.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Iterative retrieval-augmented generation (iRAG) models offer an effective approach for multi-hop question answering (QA). However, their retrieval process faces two key challenges: (1) it can be disrupted by irrelevant documents or factually inaccurate chain-of-thoughts; (2) their retrievers are not designed to dynamically adapt to the evolving information needs in multi-step reasoning, making it difficult to identify and retrieve the missing information required at each iterative step. Therefore, we propose KiRAG, which uses a knowledge-driven iterative retriever model to enhance the retrieval process of iRAG. Specifically, KiRAG decomposes documents into knowledge triples and performs iterative retrieval with these triples to enable a factually reliable retrieval process. Moreover, KiRAG integrates reasoning into the retrieval process to dynamically identify and retrieve knowledge that bridges information gaps, effectively adapting to the evolving information needs. Empirical results show that KiRAG significantly outperforms existing iRAG models, with an average improvement of 9.40% in R@3 and 5.14% in F1 on multi-hop QA.

1 Introduction

Standard RAG often misses information needed for multi-hop questions, while existing iterative approaches can be distracted by noisy context and fail to adapt retrieval to changing information needs. KiRAG uses document-grounded triples and iterative reasoning to address these challenges.

1 Introduction

  • Standard RAG often struggles with multi-hop questions because a single retrieval step may not gather all the information needed to answer them.iRAG models use multiple retrieval and reasoning steps to gather information for multi-hop questions.Source: §1 · p. 1, IRDoc · p. 1

IRDoc

  • The introduction illustrates iterative retrieval with competing document and thought-based approaches, including IRCoT and IRDoc.IRDoc replaces generated thoughts with top-ranked documents.Source: Figure 1 · p. 1
  • Existing iRAG retrieval can propagate irrelevant documents or inaccurate thoughts, while semantic-similarity retrievers may miss information needs that change across reasoning steps.For the example question, retrieval shifts from finding Kirton End’s location to finding Boston’s 2001 population.Source: IRDoc · pp. 1–2
  • KiRAG decomposes documents into compact, document-grounded knowledge triples and iteratively retrieves them for a more focused and factually reliable process.Its framework builds a triple-based reasoning chain and identifies missing triples at each iteration.Source: IRDoc · p. 2
  • KiRAG outperforms existing iRAG models on multi-hop QA, improving average R@3 by 9.40%, R@5 by 7.59%, and F1 by 5.14%.It also achieves comparable retrieval and QA performance to a state-of-the-art baseline on a single-hop dataset.Source: IRDoc · p. 2

2 Problem Formulation

The paper formalizes iRAG as iterative retrieval followed by answer generation. Its formulation focuses on improving the retriever for multi-hop QA.

2 Problem Formulation

  • At each iteration, the retriever selects documents from a corpus using the question and previously retrieved documents, and a reader produces the answer.The formulation denotes the retriever as pθ and the reader as pϕ.Source: §2 · p. 2
  • The paper focuses on enhancing the retriever to retrieve relevant documents from the corpus.The stated evaluation focus is multi-hop QA, described as a standard benchmark type for iRAG systems.Source: §2 · p. 2

3 KiRAG

KiRAG uses iterative retrieval over document-grounded knowledge triples, then ranks their source documents for answer generation. Its components progressively build a reasoning chain, identify candidate triples, and train an aligner to select useful next triples.

3.2 Knowledge-Driven Iterative Retrieval

  • At each iteration, KiRAG extends a chain of logically connected triples by identifying candidate triples and selecting one to continue the chain.The Reasoning Chain Aligner scores candidates using the question and current chain; an LLM-based Constructor selects a triple, and only the first triple of its generated chain is appended.Source: §3.2 · pp. 3–4
  • KiRAG extracts triples from retrieved documents independently, with query-independent extraction enabling offline precomputation across the corpus.An off-the-shelf retriever supplies an initial document pool, and extracted triples form a knowledge graph corpus.Source: §3.2 · p. 3

3.3 Document Ranking

  • KiRAG’s retriever uses retrieved triples to rank source documents, which an LLM-based reader then uses to generate an answer.Documents linked to multiple triples receive the highest associated triple score; documents are returned in descending score order.Source: §3.3 · p. 4

3.4 Training Strategy

  • The Reasoning Chain Aligner is trained on silver data adapted from multi-hop QA, using questions and supporting triple-based reasoning chains as labels.Training decomposes complete chains into incomplete chains and next triples, treating the correct next triple as positive and other candidates as negatives.Source: §3.4 · pp. 4–5

4 Experiments

KiRAG is evaluated on multi-hop and single-hop QA using retrieval and answer-quality metrics, against retrieval-augmented baselines and variants. Results report its overall performance, generalization, iterative retrieval behavior, and component effects.

4.2 Results and Analysis

  • KiRAG significantly improves on strongest baselines by 9.40% in R@3 and 7.59% in R@5, and by 3.12% in EM and 5.14% in F1.Retrieval results are reported in Table 1 and QA results in Table 2.Source: Table 1 · p. 5, Table 2 · p. 5, §4.2 · p. 6
  • Compared with document- and sentence-level variants, KiRAG retrieves better; compared with IRCoT, it improves R@3 by 10.42% on average.The paper attributes the variants’ weaker retrieval to noise in documents and sentences, and the IRCoT difference to hallucinated chain-of-thoughts.Source: §4.2 · p. 6
  • KiRAG’s recall and baselines’ recall decline at later reasoning steps, but KiRAG substantially exceeds RAG and IRCoT there despite comparable or slightly lower first-step recall.The analysis measures recall for the relevant document required at each step; Figure 3 reports retrieval performance across steps.Source: §4.2 · p. 6, Figure 3 · p. 6
  • On unseen Bamboogle and WebQA, KiRAG significantly outperforms all baselines; on single-hop NQ, its retrieval is comparable to IRCoT.These datasets were not included during training.Source: §4.2 · pp. 6–7, Table 3 · p. 6
  • Removing the Retriever significantly lowers HotPotQA performance but leaves 2Wiki and MuSiQue comparable; relying only on the knowledge graph may lose document context.The Constructor variant also performs significantly worse when it builds a chain from only the Aligner’s top-ranked triple.Source: §4.2 · p. 7
  • Removing the Aligner reduces R@3 by 8.67% and R@5 by 11.47%; removing training also significantly lowers retrieval and QA results.The paper links the Aligner’s absence to noisy triples and the training strategy to identifying relevant triples.Source: §4.2 · p. 7, Table 4 · p. 7
  • On 100 manually reviewed 2Wiki questions, KiRAG significantly outperforms E5 in retrieving relevant triples and in QA performance using those triples as context.The manually selected triples were treated as relevant for retrieval evaluation.Source: §4.2 · p. 7, Table 5 · p. 7
  • As iterative steps increase, retrieval and QA performance initially improve and then plateau, with optimal performance at a moderate step count.The comparison uses HotPotQA and 2Wiki development sets; Figure 4 examines the effect of L.Source: Figure 4 · p. 7, §4.2 · p. 8

5 Related Work

Prior RAG work improves retrieval, reading, or the connection between them, while iterative RAG targets multi-hop QA through repeated retrieval and reasoning. KiRAG differs from these approaches by using knowledge triples and a trained retriever for iterative retrieval that adapts to evolving information needs.

5 Related Work

  • RAG research enhances retrievers, readers, or modules connecting the two within the retriever-reader architecture.The cited directions include better retrieval [1a, 1b, 1c], answer comprehension and generation [2a, 2b, 2c], and bridging modules [3a, 3b, 3c].Source: §5 · p. 8
  • Iterative RAG models address multi-hop QA through multiple retrieval and reasoning steps, often relying on LLM-generated thoughts.IRCoT uses generated chain-of-thoughts for retrieval, while DRAGIN dynamically decides when and what to retrieve.Source: §5 · p. 8
  • Knowledge-graph RAG work uses existing graphs or generates graphs from documents, but primarily follows standard RAG with single-step retrieval.KiRAG instead focuses on iterative RAG and a trained retriever tailored to evolving information needs.Source: §5 · p. 8

6 Conclusion

KiRAG enhances iRAG retrieval by decomposing documents into knowledge triples, retrieving triples iteratively, and using them to rank documents for answer generation. It reports improved retrieval and QA performance on multi-hop QA.

6 Conclusion

  • KiRAG decomposes documents into knowledge triples, retrieves them iteratively, and uses them to rank documents for answer generation.The ranked documents are then inputs to answer generation.Source: §6 · p. 8
  • 9.40% average improvement in R@3 and 5.14% in F1 are reported for KiRAG on multi-hop QA.The conclusion characterizes these as significant retrieval and QA improvements.Source: §6 · p. 8

Limitations

The limitations concern the Aligner’s silver training data and the decision to keep the Constructor frozen. The appendix describes prompts for extracting triples, constructing reasoning chains, and generating answers.

Limitations

  • The Aligner’s silver training data is constructed from only three multi-hop QA datasets.The authors identify larger-scale and higher-quality training data as future work.Source: Limitations · p. 9
  • The Constructor is kept frozen to preserve adaptability to different LLMs, although further training could potentially improve performance.The authors contrast adaptability with relying on a specific fine-tuned LLM.Source: Limitations · p. 9

A.1 Prompt for Knowledge Triple Extraction

  • The extraction prompt asks for document-grounded triples in head-entity, relation, tail-entity form, with shared-relation tails aggregated.The prompt allows the document title or text phrases as entities.Source: §A.1 · p. 11, §A.3 · p. 12

A.2 Prompt for Reasoning Chain Construction

  • The Reasoning Chain Constructor is prompted to complete the current reasoning-chain step using candidate knowledge triples.Source: §A.2 · p. 11

A.3 Prompt for Answer Generation

B.1 Datasets

The experiments use five multi-hop QA datasets and one single-hop QA dataset, with retrieval corpora drawn from dataset-specific or Wikipedia sources. The baseline implementations are aligned for comparison, including use of the same document retriever and answer reader.

B.1 Datasets

  • The evaluation includes HotPotQA, 2Wiki, MuSiQue, Bamboogle, and WebQA for multi-hop QA, plus Natural Questions for single-hop QA.HotPotQA uses its authors’ corpus; 2Wiki and MuSiQue follow Trivedi et al. (2023), while other datasets use the Wikipedia corpus from Karpukhin et al. (2020).Source: §B.1 · p. 12
  • Public-test datasets use their full test sets, while HotPotQA, 2Wiki, and MuSiQue use full development sets for testing and 500 training questions apiece for development.The remaining training questions are used for training, and knowledge triples are precomputed for corpus documents.Source: §B.1 · p. 12

B.2 Baselines

  • KiRAG and its baselines use the same retriever for document retrieval and the same reader for answer generation.The experiments adapt or use released implementations for the compared models.Source: §B.2 · p. 12

B.3 Training and Hyperparameter Details

The Reasoning Chain Aligner is trained on reasoning chains built from multi-hop QA datasets, with implementation choices specified for retrieval, triple selection, construction, and evaluation.

B.3 Training and Hyperparameter Details

  • Training chains come from HotPotQA, 2Wiki, and MuSiQue; only the first chain producing the correct answer is retained, and questions whose chains all fail are filtered out.Development data is built with the same procedure for hyperparameter tuning.Source: §B.3 · p. 13
  • The Aligner uses Adam at 2e-5 learning rate, 0.01 weight decay, batch size 64, seven negatives per example, and τ=0.01 for 10 epochs; the best development R@5 checkpoint is selected.The supplied training-details passage ends after stating the 10-epoch duration.Source: §B.3 · p. 13
  • Experiments use at most five iterative steps, E5 or BGE retrieval with K0=10, Llama3 triple extraction, and top-20 candidate triples for chain alignment.Llama3, Mistral, and Gemma2 are tried as constructors; completed chains are filtered to triples in the candidate set.Source: §B.3 · p. 13
  • QA is evaluated with Llama3, Qwen2.5, Flan-T5, and TRACE readers; the supplied table captions identify BGE retrieval and QA performance but provide no cell values.The specific Hugging Face checkpoints are listed in Table 8.Source: §B.3 · pp. 13–14, Table 8 · p. 13, Table 9 · p. 14

C.1 Overall Performance of Using Different Retrievers and Readers

KiRAG’s reported results remain consistent with a BGE retriever and across different readers; using top-5 retrieved documents as context yields similar QA results to top-3.

C.1 Overall Performance of Using Different Retrievers and Readers

  • With BGE replacing E5, retrieval and QA results are consistent with those using E5, supporting KiRAG’s reported adaptability across retriever models.Tables 9 and 10 are identified as the corresponding retrieval and QA results; their cell values are not supplied here.Source: §C.1 · p. 14
  • Across different reader models, KiRAG consistently outperforms all baselines, and the reported interpretation is that its retrieval results support downstream QA.Table 11 reports QA performance by reader; the caption does not provide numerical values.Source: §C.1 · p. 14, Table 11 · p. 14

C.2 QA Performance based on Top-5 Documents

  • Using the top-5 retrieved documents as context produces QA performance similar to that with top-3 documents.Table 12 reports the top-5-context results, but its cell values are not supplied.Source: §C.2 · p. 14, Table 12 · p. 14

C.3 Retrieval Performance at Different Steps on MuSiQue Dataset

The supplementary analyses report retrieval trends on MuSiQue, performance on unseen datasets, a manual triple-labeling procedure, and sensitivity to retrieval and candidate counts.

C.3 Retrieval Performance at Different Steps on MuSiQue Dataset

  • KiRAG and baseline retrieval performance across steps on MuSiQue follows trends similar to those on HotPotQA and 2Wiki.Figure 9 presents the stepwise results; its numerical values are not supplied.Source: §C.3 · p. 14, Figure 9 · p. 15

C.4 QA Performance on Unseen Datasets

  • On unseen multi-hop and single-hop QA datasets, reported QA results align with Table 3 retrieval results and are described as maintaining high performance without overfitting to specific training data.Table 13 covers unseen-dataset QA performance; its cell values are not supplied.Source: §C.4 · p. 15, Table 13 · p. 15

C.5 Details and Examples of Manually Labeled Relevant Knowledge Triples

  • For 100 randomly sampled 2Wiki questions, Llama3 extracts triples from ground-truth relevant documents, and annotators select triples directly supporting each answer.Table 14 provides examples of the manually labeled triples.Source: §C.5 · p. 15, Table 14 · p. 16

C.6 Effect of the Number of Initially Retrieved Documents

  • Increasing the initial document count K0 beyond a certain point can degrade retrieval because larger pools raise the chance of noisy or irrelevant triples.K0 is varied from 10 to 100; Figure 10 reports results on three multi-hop QA development sets.Source: §C.6 · p. 15, Figure 10 · p. 15

C.7 Effect of the Number of Candidate Triples

  • Retrieval performance remains relatively stable as the candidate-triple count N varies from 10 to 40, which the authors attribute to the constructor’s ability to select useful triples from noisy candidates.Figure 11 reports results on three multi-hop QA development sets.Source: §C.7 · p. 15, Figure 11 · p. 15

C.8 Efficiency Analysis

Precomputed triple extraction reduces latency relative to online extraction without sacrificing retrieval performance; KiRAG also outperforms IRCoT across tested constructors.

C.8 Efficiency Analysis

  • KiRAG substantially reduces latency versus KiRAG (online) without compromising retrieval performance, and has latency comparable to IRCoT with significantly better retrieval performance.The authors attribute the online comparison’s efficiency benefit to precomputing query-independent knowledge-triple extraction.Source: §C.8 · p. 16

C.9 Performance of Using Different LLM-Based Constructor

  • With different frozen LLM-based Reasoning Chain Constructors, KiRAG consistently outperforms IRCoT in retrieval performance.The authors connect using a frozen LLM constructor with maintaining framework adaptability.Source: §C.9 · p. 16

Related papers

Loading 2502.18397v1…