Source-linked AI summary
Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answering
Yuling Shi, Maolin Sun, Zijun Liu, Mo Yang, Yixiong Fang, Tianran Sun, Xiaodong Gu
TL;DR
Complex multi-hop QA remains difficult because iterative RAG can suffer from inaccurate decomposition and error propagation across reasoning steps. RT-RAG addresses this with consensus-selected reasoning trees, entity-aware decomposition, and bottom-up retrieval with query refinement. Across three benchmarks, it reports average gains of 7.0% F1 and 6.0% EM over existing approaches.
Problem
Complex multi-hop QA remains challenging because inaccurate query decomposition and error propagation can disrupt coherent retrieval and reasoning.
Method
RT-RAG uses explicit entity-aware reasoning trees, consensus-based tree selection, bottom-up traversal, query rewriting, and hierarchical answer integration.
Results
7.0% F1 and 6.0% EM average gains over existing approaches are reported across three evaluated datasets.
Takeaways & Limitations
RT-RAG provides a structured approach for managing reasoning and retrieval in complex multi-hop QA.
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) has demonstrated significant effectiveness in enhancing large language models (LLMs) for complex multi-hop question answering (QA). For multi-hop QA tasks, current iterative approaches predominantly rely on LLMs to self-guide and plan multi-step exploration paths during retrieval, leading to substantial challenges in maintaining reasoning coherence across steps from inaccurate query decomposition and error propagation. To address these issues, we introduce Reasoning Tree Guided RAG (RT-RAG), a novel hierarchical framework for complex multi-hop QA. RT-RAG systematically decomposes multi-hop questions into explicit reasoning trees, minimizing inaccurate decomposition through structured entity analysis and consensus-based tree selection that clearly separates core queries, known entities, and unknown entities. Subsequently, a bottom-up traversal strategy employs iterative query rewriting and refinement to collect high-quality evidence, thereby mitigating error propagation. Comprehensive experiments show that RT-RAG substantially outperforms state-of-the-art methods by 7.0% F1 and 6.0% EM, demonstrating the effectiveness of RT-RAG in complex multi-hop QA.
1 Introduction
Multi-hop QA remains difficult because inaccurate query decomposition and error propagation undermine coherent retrieval and reasoning. RT-RAG addresses these problems with explicit reasoning-tree construction, consensus selection, and bottom-up evidence synthesis, achieving strong benchmark gains.
- Motivation: Multi-hop QA remains challenging for LLMs and retrieval-augmented systems, especially when answers require synthesizing information across Web documents.The introduction frames reliable retrieval and synthesis as critical for Web-based AI systems.
- Challenges: Inaccurate query decomposition produces poorly structured sub-questions, while error propagation carries early incorrect steps into later reasoning.These are identified as two critical failure modes in existing RAG-based multi-hop QA.
- Approach: RT-RAG constructs explicit reasoning trees through structured entity analysis and consensus-based selection to constrain decomposition and organize logical dependencies.The framework distinguishes core queries, known entities, and unknown entities, then selects a robust tree from multiple candidates.
- Contributions: RT-RAG combines hierarchical decomposition with bottom-up synthesis and adaptive mechanisms for reliable multi-hop reasoning.Its contributions include adaptive leaf-node determination and rejection sampling for inconsistent evidence.
- Results: 7.0% F1 and 6.0% EM average gains over existing approaches are reported across three evaluated datasets.The reported result is an average improvement over existing methods.
2 Related Work
RAG research has expanded from retrieve-then-generate systems toward iterative retrieval and structured decomposition, but complex multi-hop QA still challenges context maintenance and reasoning reliability. RT-RAG emphasizes explicit reasoning pathways to reduce decomposition errors and propagation across hops.
- RAG Evolution: RAG integrates retrieved external knowledge with generation and has expanded beyond question answering to dialogue, summarization, and long-form generation.The related work describes retrieval, generation, iterative retrieval, and reranking as successive architectural developments.
- Multi-hop QA: Multi-hop QA requires synthesizing multiple evidences, and benchmarks such as HotpotQA, 2WikiMQA, and MuSiQue evaluate this capability.These benchmarks are presented as established evaluations for multi-hop reasoning and evidence retrieval.
- Existing Directions: Existing methods mainly pursue iterative refinement or structured decomposition of complex questions.Iterative approaches sequence retrieval and reasoning, whereas decomposition methods break queries into simpler sub-problems or graph structures.
- Limitations: Iterative methods can accumulate errors from local decisions, while decomposition methods remain sensitive to initial structure quality and limited validation.These limitations motivate stronger control over reasoning structure in multi-hop retrieval.
- Motivation for RT-RAG: RT-RAG creates clear reasoning pathways designed to minimize error propagation and inaccurate query decomposition in complex multi-hop QA.Its motivation connects explicit reasoning structure to the two central failure modes identified in the introduction.
3 Methodology
RT-RAG decomposes questions into entity-aware reasoning trees, selects a consensus structure, and retrieves answers through adaptive post-order traversal. Query rewriting, rejection sampling, and hierarchical integration support evidence quality and coherence.
- Question Analysis: RT-RAG analyzes each question through a core query, known entities, and unknown entities before constructing smaller sub-questions.Known entities act as retrieval anchors, while unknown entities must be discovered during retrieval.
- Decomposition: The framework supports parallel, sequential, and direct decomposition patterns according to the structure of the query.Parallel sub-questions are independent, sequential ones pass answers forward, and direct questions require no decomposition.
- Tree Construction: Sub-questions form a dependency tree whose nodes represent retrieval questions, with recursive decomposition stopping at a depth limit or directly answerable leaves.The tree is represented as T = (V, E), where edges encode dependencies between questions.
- Tree Selection: RT-RAG generates multiple candidate trees and selects the most prevalent structure using consensus over tree depth and node-count patterns.If no satisfactory decomposition is found, the original question is reformulated and the process repeats.
- Retrieval and Aggregation: Post-order traversal retrieves leaf answers first, then combines child answers at non-leaf nodes while adaptively reopening unsupported nodes for direct retrieval.This fallback behavior is triggered when child answers are missing or cannot support their parent.
- Evidence Refinement: Query rewriting generates semantically equivalent alternatives when initial retrieval is insufficient, while rejection sampling selects frequent answers to reduce inconsistent evidence.The framework also permits rephrasing and repeating decomposition and retrieval when the initial result is unsatisfactory.
4 Experiments
Experiments evaluate RT-RAG across three multi-hop QA benchmarks, compare it with established retrieval and reasoning baselines, and analyze its components, tree depth, and model behavior. RT-RAG achieves consistent gains, with especially strong improvements on 2WikiMQA and MuSiQue.
- Experimental Setup: Experiments use MuSiQue, 2WikiMQA, and HotpotQA, comparing RT-RAG with direct prompting, CoT, retrieval-augmented, iterative, and structured methods.The evaluation uses standard Exact Match and F1 metrics across the three benchmarks.
- Dataset-Specific Analysis: RT-RAG improves most strongly on 2WikiMQA, gaining 12.5% F1 and 11.0% EM with GPT-4o-mini and 13.2% F1 and 14.0% EM with Qwen2.5-14B.The reported gains are associated with 2WikiMQA’s structured and unstructured evidence requiring precise decomposition and tracking.
- Dataset-Specific Analysis: RT-RAG also improves MuSiQue substantially, while HotpotQA shows more modest gains under its susceptibility to shortcuts and single-hop solutions.MuSiQue gains reach 13.0% F1 and 11.5% EM with Qwen2.5-14B; HotpotQA gains include 0.7% and 3.6% F1 and 1.5% EM for both models.
- Impact of Tree Depth: Increasing maximum tree depth from 1 to 2 substantially improves performance, while depth 3 offers a balance between decomposition granularity and computational efficiency.On 2WikiMQA, depth 1 to 2 raises F1 by 15.7% and EM by 16.0%; depth 2 to 3 yields smaller gains, including 1.9% F1 and 1.5% EM on MuSiQue.
5 Case Study
The case studies show RT-RAG combining focused reasoning through structured decomposition with resilience to missing information through adaptive restructuring. This combination supports reliable multi-hop answers when retrieval paths contain irrelevant or unavailable evidence.
- RT-RAG maintains relevant, dependency-linked reasoning where Self-Ask drifts into irrelevant retrieval and incorrect answers.The comparison illustrates structured decomposition as more reliable than self-guided exploration.
- When retrieval for a node returns no answer, RT-RAG converts its parent into a leaf node and restructures the reasoning path.This allows the system to seek information at a higher level rather than remain blocked by the missing sub-answer.
- RT-RAG derives the correct continent, North America, despite the missing birthplace information.
- The case studies identify precise, hallucination-resistant reasoning and resilience through adaptive restructuring as RT-RAG’s dual strengths.
6 Conclusion
The paper introduces RT-RAG, a hierarchical framework that combines consensus-based tree decomposition, adaptive leaf determination, rejection sampling, and hierarchical answer aggregation. Experiments report state-of-the-art performance on established multi-hop QA benchmarks, supporting its use for more reliable and accurate retrieval-augmented reasoning.
- RT-RAG is a hierarchical multi-hop QA framework that uses consensus-based tree selection to create robust reasoning pathways.
- Adaptive leaf determination prevents over-decomposition, while rejection sampling during retrieval minimizes hallucinations.
- RT-RAG integrates structured decomposition, dynamic retrieval, and hierarchical answer aggregation to model multi-hop reasoning.
- Experiments demonstrate state-of-the-art results on established multi-hop QA benchmarks.