Source-linked AI summary
How Much Reasoning Do Retrieval-Augmented Models Add beyond LLMs? A Benchmarking Framework for Multi-Hop Inference over Hybrid Knowledge
Junhong Lin, Bing Zhang, Song Wang, Ziyan Liu, Dan Gutfreund, Julian Shun, Yada Zhu
TL;DR
Knowledge-intensive multi-hop QA benchmarks can be contaminated by overlap with LLM pretraining, obscuring retrieval and reasoning ability. HybridRAG-Bench constructs recent hybrid text–graph environments and explicit-path questions to evaluate these capabilities, and experiments show low LLM-only accuracy with consistent gains from external retrieval. Its scope assumes that documents, graph facts, and answers postdate evaluated models’ pretraining cutoffs.
Problem
Pretraining overlap in existing benchmarks makes it difficult to distinguish genuine retrieval and reasoning from parametric recall.
Method
HybridRAG-Bench automatically couples recent, time-framed arXiv text with aligned knowledge graphs and generates diverse questions grounded in explicit reasoning paths.
Results
LLM-only prompting achieves 23–40% accuracy across three domains, while text-based RAG improves accuracy by 7–29 absolute points across most model–domain combinations.
Takeaways & Limitations
HybridRAG-Bench provides a controlled testbed for distinguishing retrieval and multi-hop reasoning from parametric memorization.
Takeaways & Limitations
The evaluation assumes all documents, graph facts, and answers are strictly newer than the evaluated models’ pretraining cutoff.
Abstract
from arXiv · showhide
Large language models (LLMs) continue to struggle with knowledge-intensive questions that require up-to-date information and multi-hop reasoning. Augmenting LLMs with hybrid external knowledge, such as unstructured text and structured knowledge graphs, offers a promising alternative to costly continual pretraining. As such, reliable evaluation of their retrieval and reasoning capabilities becomes critical. However, many existing benchmarks increasingly overlap with LLM pretraining data, which means answers or supporting knowledge may already be encoded in model parameters, making it difficult to distinguish genuine retrieval and reasoning from parametric recall. We introduce HybridRAG-Bench, a framework for constructing benchmarks to evaluate retrieval-intensive, multi-hop reasoning over hybrid knowledge. HybridRAG-Bench automatically couples unstructured text and structured knowledge graph representations derived from recent scientific literature on arXiv, and generates knowledge-intensive question-answer pairs grounded in explicit reasoning paths. The framework supports flexible domain and time-frame selection, enabling contamination-aware and customizable evaluation as models and knowledge evolve. Experiments across three domains (artificial intelligence, governance and policy, and bioinformatics) demonstrate that HybridRAG-Bench rewards genuine retrieval and reasoning rather than parametric recall, offering a principled testbed for evaluating hybrid knowledge-augmented reasoning systems. We release our code and data at github.com/junhongmit/HybridRAG-Bench.
1 Introduction
Existing multi-hop benchmarks can conflate retrieval and reasoning with parametric recall because their knowledge overlaps model pretraining data. HybridRAG-Bench addresses this by automatically generating recent, hybrid-knowledge questions grounded in explicit reasoning paths.
- Motivation: Pretraining contamination and static, manually curated knowledge sources make it difficult to isolate retrieval and reasoning contributions.Many established benchmarks were created before widespread LLM deployment and draw on knowledge likely present in pretraining corpora.
- Motivation: Later pretraining cutoffs increased CRAG QA accuracy by up to +102%, while general reasoning improvements were only +6.7%.Earlier models achieved near-zero accuracy on questions involving fast-changing facts, indicating benchmark performance can reflect pretraining exposure.
- Framework: HybridRAG-Bench collects recent, time-framed scientific corpora to reduce overlap with pretraining data and make correctness depend on retrieval.The framework supports user-specified topics and time frames.
- Framework: It aligns unstructured text chunks with extracted knowledge graphs and generates question-answer pairs grounded in explicit reasoning paths.The questions span single-hop lookup, conditional reasoning, multi-hop inference, hard multi-hop chains, counterfactual reasoning, and open-ended synthesis.
- Contributions: The reusable framework enables controlled evaluation of whether retrieval and multi-hop reasoning, rather than parametric memorization, drive performance.It also provides diagnostic instances for comparing RAG and KG-RAG strategies.
2 Related Works
Prior work includes knowledge-graph question answering, retrieval-augmented generation, and knowledge-grounded LLMs. However, these methods are often evaluated on static benchmarks whose overlap with model pretraining can overestimate performance, while dynamic alternatives do not fully support hybrid graph reasoning.
- KGQA Benchmarks: KGQA benchmarks evolved from Freebase-based datasets toward greater compositional complexity and question diversity.Examples include WebQuestions, WebQSP, CWQ, GrailQA, and KQA Pro.
- KGQA Benchmarks: LLMs’ increasing exposure to web-scale and scientific corpora creates growing risk that KGQA knowledge overlaps their internal memory.This motivates efforts such as adaptive or dynamic question generation.
- KGQA Benchmarks: Dynamic alternatives such as ArxivRoll and OKBench do not fully provide KGQA-oriented graph retrieval and multi-fact reasoning.Dynamic-KGQA does not definitively isolate parametric memory, while OKBench operates on single-document fragments.
- Retrieval-Augmented and Knowledge-Grounded LLMs: Text-based RAG retrieves unstructured documents, whereas KG-based approaches support relational and multi-hop reasoning over structured graphs.The literature explores these approaches to address limitations of parametric knowledge and avoid continual pretraining costs.
- Retrieval-Augmented and Knowledge-Grounded LLMs: Static benchmarks built from Freebase, DBpedia, and Wikidata can overestimate performance because their external knowledge may overlap models’ parametric knowledge.This problem affects widely used datasets including WebQSP, CWQ, GrailQA, HotpotQA, and MetaQA.
3 Problem Definition and Preliminaries
HybridRAG-Bench evaluates knowledge-intensive questions within domain-specific, time-indexed hybrid knowledge environments. Questions are answered using information available by their issue time, under an assumption that all required knowledge postdates model pretraining cutoffs.
- Domains, Corpora, and Knowledge Graphs: The framework supports multiple domains, each defined by document-selection criteria such as subject categories and keyword constraints.Evaluation is performed within one domain at a time.
- Domains, Corpora, and Knowledge Graphs: For each domain, documents are collected within a user-specified time window and used to form an evolving knowledge graph.The graph is induced from the domain corpus and changes with the available time range.
- Domains, Corpora, and Knowledge Graphs: Each domain has an independent knowledge graph whose entity types, relation schemas, topology, and temporal dynamics may differ.No entities or relations are shared across domains.
- Questions and Evaluation Scope: Each question is evaluated against the domain-specific knowledge-graph snapshot and documents available up to its issue time.The question includes an issue time and corresponding ground-truth answer.
- Questions and Evaluation Scope: The evaluation assumes all documents, graph facts, and answers are newer than the evaluated models’ pretraining cutoff.Under this assumption, correct answers cannot be recovered through parametric memorization alone.
4 Benchmark Construction
HybridRAG-Bench is an automated, configurable pipeline that builds hybrid knowledge environments and generates validated multi-hop benchmarks from time-framed arXiv corpora. Its questions combine graph-based reasoning paths with supporting text and span several reasoning types.
- Benchmark Construction: The framework collects time-framed arXiv corpora, constructs aligned knowledge graphs and text chunks, generates questions, and applies automated quality control.The pipeline supports user-specified topics and time ranges while filtering for answerability, document independence, and non-redundancy.
- Knowledge Graph Construction: HybridRAG-Bench retains structured relational scaffolding alongside unstructured evidence to support hybrid retrieval and reasoning.The knowledge graph represents entities and relations extracted from the document corpus, while textual evidence supplies contextual descriptions and qualifiers.
- Hybrid-Grounded Question–Answer Generation: Each generated question is conditioned on a sampled graph path, associated textual evidence, and in-context examples defining the desired format.The graph path supplies relational constraints, while textual context guides natural-language question construction.
- Hybrid-Grounded Question–Answer Generation: The benchmark covers single-hop, conditional single-hop, multi-hop, difficult multi-hop, counterfactual, and open-ended questions.These forms require relation lookup, alignment with textual conditions, intermediate-node inference, robustness to large candidate spaces, cautious counterfactual reasoning, or evidence synthesis.
- Quality Control: An LLM-as-a-Judge quality-control process removes questions requiring external knowledge or unsupported inference and retains only pairs that pass all checks.Additional filtering removes document-local references, ambiguity, and poor phrasing before final inclusion.
5 Experiment
Experiments show that HybridRAG-Bench remains difficult for LLMs without retrieval, rewards effective integration of text and graph evidence, and distinguishes retrieval and reasoning strategies across question types. Its KG construction recovers factual structure effectively while scaling predictably with document length.
- RQ1: Challenge Across LLM Scales: LLM-only accuracy remains 23–40% across domains despite scaling from LLaMA-3.1-8B to DeepSeek-V3.2-685B.The results indicate that parametric knowledge alone is insufficient for reliably answering the benchmark questions.
- RQ2: External Retrieval: Text-based RAG improves accuracy by 7–29 absolute points over LLM-only prompting across most model–domain combinations.The exception is DeepSeek-V3.2 on Arxiv-CY; naïve one-hop KG augmentation can instead add noise and reduce performance.
- RQ3: Structured Knowledge: Hybrid KG-RAG methods consistently outperform text-only RAG across all domains by combining relational scaffolding with unstructured evidence.Structured graphs particularly support relational reasoning, entity disambiguation, and multi-hop composition.
- RQ4: Retrieval and Reasoning Strategies: KG-RAG baselines exhibit substantial performance gaps across domains and question types, exceeding gains from scaling LLM size alone.These differences indicate that the benchmark distinguishes retrieval and reasoning strategies rather than measuring only model scale.
- Question-Type Breakdown: Graph-structured methods perform best on multi-hop and hard multi-hop questions, while hybrid methods help single-hop and conditional questions.Open-ended questions favor strong text retrieval, whereas counterfactual questions favor explicit reasoning mechanisms and naïve one-hop KG augmentation performs near zero.
- KG Construction Effectiveness: The KG construction pipeline recovers approximately 71% of verifiable facts, outperforming KGGen by about 5 absolute points.The evaluation uses 106 document corpora with 15 manually verified facts per corpus, and recovery is matched to at least one KG triplet.
- KG Construction Cost and Scalability: Extraction latency and token usage grow approximately linearly with document length, with no observed superlinear cost growth.The pipeline can be parallelized across documents, reducing end-to-end wall-clock time.
6 Conclusion
HybridRAG-Bench evaluates retrieval-intensive reasoning over hybrid knowledge while separating retrieval and reasoning from memorization. Its construction from recent literature and explicit reasoning paths supports diagnostic comparisons across LLM-only, RAG, and hybrid KG-RAG methods.
- HybridRAG-Bench constructs hybrid knowledge environments from recent scientific literature and generates questions grounded in explicit reasoning paths and supporting evidence.
- The framework enables diagnostic evaluation of LLM-only, RAG-based, and hybrid KG-RAG methods.
- HybridRAG-Bench poses substantial challenges to LLM-only baselines and distinguishes methods by their ability to retrieve and integrate graph and textual information.
- The framework relies on LLM-based knowledge extraction and question generation and is currently instantiated using scientific literature.
A.1 KG Construction Efficiency and Cost Analysis
KG evolution combines LLM extraction with sublinear entity and relation alignment, while empirical latency increases smoothly with document length and supports parallel extraction for lower end-to-end time.
- KG extraction uses one LLM call with cost O(n + m), followed by HNSW-based alignment with complexity O(n log N + m log M).Merging and insertion add linear cost O(n + m).
- Overall KG evolution has complexity O(n log N + m log M), and the implementation uses five LLM calls per corpus.The constant call count is described as practical for continuous, large-scale KG updates.
- Longer documents incur proportional increases in extraction time, with no evidence of superlinear blow-up.Extraction latency is dominated by the LLM forward pass and aligns with the reported theoretical complexity.
- Data-parallel extraction substantially lowers effective end-to-end KG construction time relative to summing individual document extraction times.
- Latency and token cost grow smoothly and predictably with corpus size, empirically validating EvoKG’s scalability.
A.2 Additional Per-Question-Type Results.
Additional results report accuracy by question type for evaluated language models, with standard deviations computed across five runs and best results highlighted in the reported tables.
- Per-question-type accuracy and standard deviation across five runs are reported for all evaluated LLMs and baselines.The reported models include DeepSeek-V3.2, Qwen2.5-72B, LLaMA-3.3-70B, and LLaMA-3.1-8B.
- Table 8 reports LLaMA 3.3-70B accuracy by question type and highlights the first- and second-best results.
- Table 9 reports LLaMA 3.1-8B accuracy by question type and highlights the first- and second-best results.