DeFT-RAG: Incrementally Decomposing Multi-Hop Questions into Self-Contained Sub-Questions with Reusable Answers

1 Introduction

• Multi-hop questions require chaining facts across documents through bridge entities that the question never names, so retrieval from named entities often misses the full evidence chain.Most graph-based methods retrieve the full chain for only 6–10% of 3-hop MuSiQue questions.Source: §1 · p. 1
• Iterative RAG finds bridge entities step by step, but planning, dependency, rewriting, and tracking tasks each create distinct opportunities for error.Fixed top-k retrieval can further lower answer accuracy, and intermediate QA pairs are discarded even when later questions could reuse them.Source: §1 · p. 1
• DeFT-RAG generates one self-contained sub-question at a time from the original question and answered QA pairs, while rules update a QA table and a fine-tuned lightweight decision model makes yes/no decisions.It also caches QA pairs and reuses answers when later sub-questions have the same meaning.Source: §1 · p. 2
• Across three benchmarks and two backbones, DeFT-RAG achieves the best accuracy among thirteen baselines while giving the LLM fewer input tokens than any other iterative method.Its QA cache further reduces LLM calls without lowering accuracy.Source: §1 · p. 2

2.1 Retrieval-Augmented Generation

• Vanilla RAG retrieves passages relevant to a question from a corpus, commonly ranking them by dense or sparse similarity; fixed top-k retrieval sets the passage count in advance.The retrieved passages are supplied to the LLM to answer the question.Source: §2.1 · pp. 2–3

2.2 Multi-Hop Question Answering

• A multi-hop question requires an evidence chain of two or more facts, and later facts may depend on bridge entities that the question never names.In the example, answering where buses leave in the relevant city requires first identifying Kevin Drew and Toronto.Source: §2.2 · p. 3
• On MuSiQue, vanilla RAG retrieves the full evidence chain for only 7% of 3-hop questions because passages about unnamed bridge entities rank low against the question’s surface terms.The passage motivates graph-based and iterative RAG as two approaches to this problem.Source: §2.2 · p. 3

2.3 Graph-Based RAG

• Graph-based RAG follows links from question entities to related passages, but most such methods retrieve the full chain for only 6–10% of 3-hop MuSiQue questions.LinearRAG reaches 28% while giving the LLM about 4,100 words per question, most of which are outside the evidence chain.Source: §2.3 · p. 3

2.4 Iterative RAG

• Iterative RAG alternates sub-question retrieval and LLM reasoning, retrieving full evidence chains for 19–41% of 3-hop questions.Each answered sub-question can expose a bridge entity needed for a later retrieval.Source: §2.4 · p. 3
• Iterative methods can err by planning before bridge entities are known, linking dependencies in the wrong order, failing to rewrite dependent questions, or losing track of known facts.The section illustrates planning and dependency errors with LogicRAG and rewriting and tracking errors with other methods.Source: §2.4 · p. 3
• On one-retrieval-step MuSiQue questions, expanding from the top 5 to top 10 passages raises gold-passage coverage from 62% to 88% but accuracy only from 0.620 to 0.630.Giving the LLM only the gold passages from the top 10 raises accuracy to 0.793.Source: §2.4 · p. 3, Table 1 · p. 4
• Iterative methods make more retrieval and LLM calls than vanilla or graph-based RAG, then discard intermediate answers even though questions over the same corpus may share facts.Those intermediate answers could be reused to save retrieval and LLM calls on later questions.Source: §2.4 · p. 4

3 DeFT-RAG Design

• DeFT-RAG combines incremental question decomposition, rule-based QA-table updates, fine-grained yes/no decisions by a decision model, and a cache that reuses answers across questions.The framework also shows how these components work together in its query-answering process.Source: §3 · p. 4

3.1 Incremental Question Decomposition

• The LLM generates one self-contained sub-question at a time from the original question and known QA pairs, avoiding preplanned dependencies and sub-question rewriting.Each sub-question names the entities it asks about and targets the next missing piece of information.Source: §3.1 · p. 4

3.2 QA Table

• The QA table stores each answered sub-question and answer, records unanswered or repeated questions, and is the shared record used to continue and stop the process.Rule-based updates avoid an LLM or decision-model call, and the compact QA pairs keep inputs short.Source: §3.2 · p. 5

3.3 Decision Tasks

• A fine-tuned decision model makes relevance, direct-answer, and sufficiency yes/no decisions at task-specific thresholds and filters retrieved passages before they reach the LLM.The direct-answer threshold is 0.7, versus 0.5 for relevance and sufficiency; training labels come from benchmark training-split gold annotations, with no test questions used.Source: §3.3 · p. 5

3.4 Reusing Answers Across Questions

• Because self-contained sub-questions have the same meaning and answer across original questions, the QA cache can reuse a stored answer without retrieval or an LLM call.If exact normalized wording does not match, the decision model checks whether a similar stored sub-question has the same meaning, using a strict 0.8 threshold.Source: §3.4 · p. 5

3.5 Query Answering Process

• The process first tests whether retrieved passages support a direct answer, then iteratively answers sub-questions until the QA pairs suffice or the T-step limit triggers a direct-answer fallback.A direct answer can save LLM calls, while the final answer for iterative cases is built from the QA pairs.Source: §3.5 · pp. 5–6

4.1 Setup

• The evaluation uses 1,000 test questions per benchmark, two backbones, and 13 baselines, with systems sharing the backbone and embedding model.DeFT-RAG retrieves ten passages and keeps at most five relevant passages per step; the reported main results do not use answer reuse.Source: §4.1 · p. 6

4.2 Fine-Tuning the Decision Model

• The decision model is fine-tuned on training-split examples only, with no test questions or test-corpus passages in its training data.The training and development sets contain 48,036 and 7,806 examples, respectively.Source: §4.2 · p. 6

4.3 Main Results

• DeFT-RAG achieves the highest score on EM, F1, and LLM-judged accuracy in all six benchmark–backbone settings, with significant gains over most baselines.Its largest accuracy gains over the second-best system are on MuSiQue: 5.2 points with gpt-4o-mini and 5.3 with Qwen3.6.Source: §4.3 · p. 6, Table 2 · p. 7

4.4 Why DeFT-RAG Gives Better Answers

• DeFT-RAG supplies the full evidence chain for 46% of 3-hop and 18% of 4-hop MuSiQue questions using 390 passage words per question, the fewest of any system.Self-Ask uses 2,590 words while covering more 2-hop chains; LinearRAG uses 4,110 words while covering 19% of 4-hop chains.Source: §4.4 · pp. 6–7, Table 3 · p. 8
• Without relevant-passage selection, accuracy drops by 2.3 points on MuSiQue, 3.7 on 2Wiki, and 5.3 on HotpotQA; the HotpotQA drop is significant.On 2Wiki and HotpotQA, most of the loss comes from questions answerable directly, where evidence for the whole question must be found in one passage list.Source: §4.4 · p. 7, Table 4 · p. 8

4.5 Effect of Fine-Tuning the Decision Model

• Fine-tuning improves direct-answer and relevance decisions, while the enough-facts decision changes little; end-to-end MuSiQue accuracy rises from 0.587 to 0.623.The direct-answer decision improves from 0.850 to 0.933 and relevance from 0.931 to 0.979 on MuSiQue.Source: §4.5 · p. 7, Table 5 · p. 8

4.6 Reusing Answers

• With one cache shared across 1,000 MuSiQue questions, 23.1% of sub-questions are answered from cache and LLM calls fall by 12.9%, with no accuracy drop.Accuracy is 0.642 with reuse versus 0.629 without reuse (p=0.18).Source: §4.6 · p. 8

Graph-based RAG

• Graph-based methods retrieve once from entities named in the question, which rarely recovers the full evidence chain beyond two hops; DeFT-RAG finds bridge entities through successive self-contained sub-questions.Graph-based approaches build corpus graphs and follow links at answering time, whereas DeFT-RAG retrieves the full chain for more questions with three or more hops than those methods.Source: §5 · p. 8

Iterative RAG

• Iterative methods assign the LLM tasks such as planning, dependency decisions, rewriting, and tracking; DeFT-RAG generates one self-contained sub-question at a time and uses a QA table and fine-tuned decision model for the other tasks.DeFT-RAG uses 1,569 LLM input tokens per question versus 6,140 for Self-Ask and achieves the highest accuracy among these methods in every setting.Source: §5 · p. 8

Models for retrieval decisions

• Unlike methods that rerank a fixed number of passages or train the backbone to make retrieval decisions, DeFT-RAG uses a separate fine-tuned decision model while leaving the backbone unchanged.The paper states that fine-tuning this lightweight decision model is much cheaper than training the backbone LLM.Source: §5 · p. 8

• DeFT-RAG combines incremental self-contained sub-questions, QA-table updates and fine-tuned lightweight decision-model judgments, and a cache that reuses answers across questions.Source: §6 · p. 8
• Across three benchmarks and two backbones, DeFT-RAG achieves the best accuracy among thirteen baselines, with the largest gains on MuSiQue.It also gives the LLM fewer input tokens than any other iterative method.Source: §6 · p. 8
• Reusing answers reduces LLM calls without lowering accuracy.Source: §6 · p. 8

A Implementation Details

• The implementation retries once with changed input after an empty or repeated reply.The same rules are used on every benchmark and backbone.Source: A Implementation Details · p. 9
• A direct answer requires a decision-model sufficiency score of at least 0.7 and no keyword-detected composite family relation.Examples of such relations include grandparent and in-law questions.Source: A Implementation Details · p. 9
• Changing the direct-answer threshold from 0.7 to 0.6 or 0.8 changed at most 13 of 1,000 question paths and accuracy by at most 0.1 points per benchmark with gpt-4o-mini.The 0.7 threshold was chosen during development using stored answers from 2Wiki and the first 300 MuSiQue questions, then fixed across benchmarks and backbones.Source: A Implementation Details · p. 9

B Prompts

• Once entities are named, prompts seek directly stated facts, such as dates of birth for age comparisons or nationality for country-of-origin questions.The example question first asks who authored Y, then where that person was born.Source: B Prompts · p. 10
• Sub-question prompts ask for one self-contained next question, using resolved names and avoiding information already present in the QA pairs called “facts.”The initial prompt selects a description not built on another unresolved description; later prompts target the next missing piece.Source: B Prompts · p. 10
• After an unanswered or repeated sub-question, prompts prohibit asking it again and request different wording or another part of the main question.A repeated question is explicitly flagged as already asked.Source: B Prompts · p. 10
• Passage-answering and final-answer prompts both request short, direct answers, with the former using retrieved passages and the latter using the QA-table facts.The prompt examples use separate question-and-answer fields for each task.Source: B Prompts · p. 10
• For date comparisons, the prompt orders comparison by year, month, then day; for age, the earlier-born person is older.Source: B Prompts · p. 10
• Decision prompts separately assess passage relevance, whether passages suffice for a direct answer, whether accumulated facts answer the question, and whether two questions mean the same thing.Reuse requires the same explicitly named entity and information; differing scope or an unnamed reference makes the match “no.”Source: B Prompts · p. 10

C Decision Model Fine-Tuning Details

• Training samples 3,000 questions per benchmark, grouped into sets of 1,000 with question-specific corpora close in size to the corresponding test corpora.Each training request retrieves its top 10 passages from its group's corpus using DeFT-RAG's embedder and query instruction.Source: C Decision Model Fine-Tuning Details · p. 10
• Gold annotations define relevance, direct-answer sufficiency, and enough-facts labels; the same-meaning decision is not fine-tuned.Relevance labels include gold supporting passages and three randomly selected alternatives; enough-facts examples also include DeFT-RAG runs and self-contained sub-questions.Source: C Decision Model Fine-Tuning Details · p. 10
• The training data excludes test questions and skips training questions whose gold passage appears in a test corpus.Test questions come from each benchmark's official validation split, and no training question text is identical to a test question.Source: C Decision Model Fine-Tuning Details · p. 11
• The final data contains 48,036 training and 7,806 development examples, with a single yes/no probability temperature of 1.08 fitted on development examples.Source: C Decision Model Fine-Tuning Details · p. 11

D Experimental Details for Baselines

• All systems share the backbone, Qwen3-Embedding-4B, decoding settings, a 3,072-token output limit, and top-5 retrieval per step.Baselines retain method-defining settings from their released code or paper; answers use the same grading script and judge prompt, and missing answers count as wrong.Source: D Experimental Details for Baselines · p. 11

Index

• Graph-based methods that build a corpus graph use about 2,100–147,000 LLM calls, 11–114 million tokens, and up to 18 hours per corpus; DeFT-RAG only embeds passages, taking about five minutes without LLM calls.Graph-extraction costs recur when the corpus changes.Source: E Overhead Analysis · p. 11

Answering

• 1,569 input tokens per question, versus 1,926–13,928 for other iterative methods; DeFT-RAG also uses 33 output tokens and 4.8 LLM calls, fewer than Self-Ask and Hi-Q.Its median latency is 9.9 seconds, the highest among the methods.Source: E Overhead Analysis · p. 11

Where the latency comes from

• On MuSiQue, frequent local DM calls account for 12.5 of 20.4 seconds of mean answering time; DeFT-RAG makes 7.0, 6.3, and 3.2 calls per question on MuSiQue, 2Wiki, and HotpotQA, respectively.The calls take 1.6 to 1.8 s each, with the DM running on one GPU, one request at a time, over inputs of up to ten passages.Source: E Overhead Analysis · p. 11
• Replacing the local DM with Jev lowers median latency from 9.9 to 5.0 seconds, while accuracy stays within 1.2 points of the local DM on all three benchmarks.Mean answering times with Jev are 8.2 seconds on MuSiQue, 7.7 on 2Wiki, and 3.6 on HotpotQA; the authors attribute most added latency to how the DM is served.Source: E Overhead Analysis · p. 11

F Case Studies

• For a three-hop question, DeFT-RAG finds Sazerac Company, then New Orleans, then the 1970s mayor-election date; Self-Ask, Hi-Q, and LogicRAG fail to provide that answer.The DM judges the accumulated QA pairs sufficient after the third pair.Source: F Case Studies · p. 11
• On a naval-ships question, DeFT-RAG first asks a question that retrieves Caesar, then completes the chain through the North Sea and English Channel before stopping.The DM does not judge the initial retrieved answer sufficient; it stops after the chain is complete.Source: F Case Studies · p. 11
• 38% of nested sub-questions on 2Wiki bridge-comparison questions get no answer, versus 4% for single-relation sub-questions; 75 of 235 such questions reach the step limit or repeat themselves.The authors identify these cases as the main reason DeFT-RAG only matches Self-Ask on 2Wiki with gpt-4o-mini.Source: F Case Studies · p. 12
• In an age-comparison example, DeFT-RAG finds the right person but asks for his birth date instead of his signing date, producing the wrong final answer.The authors suggest the date-of-birth rule for age comparisons may have pulled the sub-question toward a birth date.Source: F Case Studies · p. 12
• Among 148 wrong answers on 2Wiki with gpt-4o-mini, about 19 were attributed to DM decisions; most others were attributed to the backbone, including nested sub-questions and misread passages.Source: F Case Studies · p. 12