PaperlayerView exact revision on arXiv

Source-linked AI summary

ConRAG: Lightweight inference of multi-hop relations

Kilian Bänziger, Sonia Laguna, Markus Kreft, Robert Jakob, Kevin O'Sullivan, Lasse B. Strand, Julia E. Vogt

arXiv:2609.35193v1cs.LG

TL;DR

Recovering how two known entities are connected requires finding bridge entities and evidence across documents, a target existing multi-hop RAG systems do not explicitly address. CONRAG builds an entity-document graph and retrieves endpoint-connected paths; across two benchmarks, it improves bridge-entity and reasoning-chain recovery over strong RAG baselines while reducing indexing token cost.

  • Problem

    Existing multi-hop RAG systems typically retrieve toward an unknown answer rather than explicitly reconstructing an evidence-grounded chain between two known endpoints.

    Source: §1 · p. 2

  • Method

    CONRAG builds an entity-document graph using corpus co-occurrence and LLM-based entity filtering, then retrieves paths connecting the two endpoints.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Across MuSiQue and 2WikiMultiHopQA, CONRAG improves bridge-entity and reasoning-chain recovery over strong RAG baselines while reducing graph-indexing token cost by up to 1.5 orders of magnitude.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The derived benchmarks provide a testbed and recipe for future research on relation discovery.

    Source: §1 · p. 2

    Evaluation uses the gold decompositions as the only reference chains, so valid alternative routes are scored as errors and reported CR, CP, and BCM are conservative lower bounds.

    Source: §6 · p. 9

Abstract

from arXiv · show

Understanding how two entities are connected often requires tracing multi-hop relations across documents to identify intermediate entities and supporting evidence that explain a connection. This is a task that appears frequently in scientific research and other knowledge-intensive analyses. We formalise this setting as multi-hop relation inference: given two known endpoint entities, we aim to recover the bridge entities and evidence-grounded reasoning chains that connect them across a document corpus, and to generate an explanation grounded in the retrieved evidence. Existing multi-hop RAG systems typically seek an unknown answer entity rather than explicitly recovering the connection between two known endpoints and graph-based approaches often rely on costly LLM-extracted knowledge graphs that limit scalability to large document collections. We introduce ConRAG, which builds a lightweight entity-document graph from entity co-occurrence and LLM-based entity filtering. Its connective retrieval infers and semantically ranks paths between two endpoints. On MuSiQue and 2WikiMultiHopQA, ConRAG consistently improves bridge entity and reasoning chain recovery over strong RAG baselines, while reducing graph-indexing token cost by up to roughly 1.5 orders of magnitude. Our results show that endpoint-constrained path retrieval provides an effective and index-efficient approach to evidence-grounded relation discovery.

1 INTRODUCTION

The introduction defines multi-hop relation inference as recovering the bridge entities and evidence links between two known endpoints. It motivates endpoint-constrained retrieval and introduces CONRAG as a framework for this task.

1 INTRODUCTION

  • RAG supplies language models with retrieved external information for generation on knowledge-intensive tasks.The introduction presents external context as grounding generation in information beyond model parameters.Source: §1 · p. 1
  • Multi-hop relation inference targets the bridge entities and links connecting two known entities, rather than a single relation label or an unknown answer.The connection may require combining relations stated across documents.Source: §1 · pp. 1–2
  • Existing multi-hop RAG systems are not explicitly designed to reconstruct connected evidence chains between two given endpoints.Their retrieval is typically relevance- or answer-oriented, which can miss bridge documents with weak query similarity.Source: §1 · p. 2
  • CONRAG builds an entity-document graph and retrieves paths between endpoint-associated documents to surface bridge entities and supporting evidence.The overview describes LLM filtering of mined entity candidates and path scoring by question-document similarity.Source: §1 · p. 2, Figure 2 · p. 2
  • CONRAG improves bridge entity recovery and reasoning-chain recall and precision over RAG baselines, while reducing indexing token cost by up to 1.5 orders of magnitude.The evaluation reinterprets MuSiQue and 2WikiMultiHopQA by treating question and answer entities as endpoints and annotated decompositions as target chains.Source: §1 · p. 2

2 RELATED WORK

Related work spans explanatory paths over knowledge graphs, relation extraction, graph-based RAG, and multi-hop reasoning. CONRAG differs by making the evidence-grounded chain between known entities the output itself.

2 RELATED WORK

  • Prior relation-explanation methods rank paths over existing knowledge graphs or combine graph information with corpus sentences, while CONRAG induces its graph from raw text.CONRAG uses an LLM to explain recovered chains in prose.Source: §2 · p. 3
  • Cross-document relation extraction retrieves evidence to predict a predefined relation label, whereas CONRAG makes the bridge-entity and evidence chain the target.CONRAG builds on the evidence-path retrieval formulation of Lu et al. (2023) while removing reliance on gold mention annotations and a task-specific trained retriever.Source: §2 · p. 3
  • Graph-based RAG systems vary in graph construction and retrieval, while CONRAG uses a relation-free entity-document graph and ranks paths between two known endpoints.Its entity vocabulary comes from corpus co-occurrence statistics and LLM filtering.Source: §2 · p. 3
  • Multi-hop reasoning systems generally retrieve toward an unknown answer or produce a chain as a byproduct, rather than targeting an unlabeled chain between known entities.CONRAG makes that chain the output and assumes no predefined relation vocabulary.Source: §2 · p. 3

3 CONRAG: CONNECTIVE RETRIEVAL AUGMENTED GENERATION

ConRAG targets evidence-grounded recovery of the connection between two known entities. It builds a relation-free entity-document index and retrieves paths to identify bridge entities and supporting documents.

3 CONRAG: CONNECTIVE RETRIEVAL AUGMENTED GENERATION

  • ConRAG is designed to recover multi-hop connections between two entities.Source: §3 · p. 4

3.1 PROBLEM FORMULATION

  • Given anchor and target entities, the system returns an ordered chain and prose explanation, both entailed by retrieved documents.Source: §3.1 · p. 4

3.2 ENTITY QUALIFICATION

  • Entity candidates are mined as unigrams or short capitalized phrases occurring in at least two documents, then LLM-filtered in containment-aware batches.Source: §3.2 · p. 4

3.3 OFFLINE INDEXING

  • The index combines the corpus, a bipartite entity-document graph, and document embeddings; entities connect documents without relation triples, while documents remain the evidence source.Source: §3.3 · p. 4

3.4 ONLINE RETRIEVAL

  • Dense seeding is adjusted to cover both endpoints, injecting matching documents or using fallback searches when semantic retrieval misses an endpoint.Source: §3.4 · p. 5
  • Path search traverses distinct documents under a maximum length, removes high-degree entities, retains paths reaching target-side seeds, and ranks paths by mean query support.Source: §3.4 · p. 5
  • The generator receives endpoint seed evidence alongside path documents, while the generator determines which documents carry the relations.Source: §3.4 · p. 5

3.5 CONSTRUCTING MULTI-HOP RELATION INFERENCE BENCHMARKS

  • The benchmark converts linear MHQA decompositions into endpoint-only relation questions with ordered bridge entities and gold chains, excluding branching questions.Source: §3.5 · pp. 5–6

4 EXPERIMENTAL SETUP

The experiments adapt two multi-hop QA validation sets into relation-inference benchmarks and evaluate bridge recovery, chain recovery, and judged response quality. They compare ConRAG with four RAG baselines under shared generation settings.

Datasets

  • The benchmarks contain 2,066 MuSiQue examples across two-, three-, and four-hop questions and 6,785 two-hop 2WikiMultiHopQA examples.Source: §4 · p. 6

Evaluation

  • Evaluation measures bridge containment match, chain recall, and chain precision, plus LLM-judged response quality and average win rate across three 1,000-question samples.Source: §4 · p. 6

Baselines and implementation

  • ConRAG is compared with Naïve RAG, LightRAG, PathRAG, and HippoRAG 2; methods share embeddings and generation prompts, with most baselines capped at ten retrieved documents.Source: §4 · p. 6

5 RESULTS

ConRAG outperforms the baselines on relation inference across both datasets, while qualitative comparisons, token costs, and retrieval results assess complementary aspects of performance. Its retrieval also shows a tradeoff between recall and retrieval time as hub suppression changes.

5.1 RELATION INFERENCE

  • By over 6.2 points on MuSiQue and 9.2 points on 2WikiMultiHopQA in chain recall versus HippoRAG 2, ConRAG outperforms baselines on both datasets.Table 1 reports Bridge Contains Match Accuracy, Chain Recall, and Chain Precision on both datasets.Source: §5.1 · p. 7, Table 1 · p. 7
  • Baseline rankings vary across metrics and datasets, while all models show lower chain recall as reasoning-hop count increases.Only HippoRAG 2 tends to be the strongest baseline in chain recall; ConRAG consistently performs above baseline level.Source: §5.1 · pp. 7–8, Figure 3 · p. 7

5.2 QUALITATIVE PERFORMANCE

  • ConRAG attains the higher overall qualitative win rate, though Naïve RAG is second and exceeds ConRAG on 2WikiMultiHopQA diversity.ConRAG wins more than twice as much against LightRAG on MuSiQue; its tighter, higher-precision evidence trades breadth of stated content for relevance.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 INDEXING AND INFERENCE EFFICIENCY

  • ConRAG uses a comparable generation-token footprint to HippoRAG 2, while its indexing process removes most candidate entities and preserves bridge entities.Table 3 reports average prompt tokens and index construction tokens.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 RETRIEVAL

  • ConRAG achieves the highest document recall on both benchmarks at a comparable document budget, while higher hub suppression raises retrieval time.Retrieval time grows strongly beyond phub = 0.8 without significantly improving overall document recall.Source: §5.4 · p. 9

6 CONCLUSION

CONRAG uses a relation-free entity-document graph and endpoint-constrained retrieval to recover connections between known entities. The conclusion reports stronger relation recovery and document recall than strong RAG baselines, alongside lower indexing-token requirements.

6 CONCLUSION

  • Across both benchmarks, CONRAG improves bridge-entity and reasoning-chain recovery over strong RAG baselines while achieving higher document recall and lower graph-indexing token requirements.The conclusion presents endpoint-constrained path retrieval as an effective, index-efficient approach to evidence-grounded relation discovery.Source: §6 · p. 9
  • CONRAG builds a relation-free entity-document graph from corpus co-occurrence and LLM-based entity filtering, then retrieves paths linking endpoints through bridge entities and evidence.The conclusion argues that this approach does not require richly extracted knowledge graphs.Source: §6 · p. 9
  • The benchmark reference chains may omit valid routes, so alternative paths are scored as errors and reported CR, CP, and BCM are conservative lower bounds.Path-enumeration cost also rises with admitted high-degree entities, making hub suppression a trade-off between those connections and tractability.Source: §6 · p. 9

AI USE STATEMENT

  • The authors report using generative AI for dataset reformatting, iterative methodology feedback, code assistance, manuscript editing, and literature-related tasks, with outputs reviewed by the authors.They state that the authors take responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The reproducibility materials include the pipeline, fixed hyperparameters, benchmark construction details, baseline configurations, prompts, computational token costs, and an anonymous repository.The passage identifies Section 3, Algorithm 1, Table 5, Appendix B, Appendix E, and Table 3 as sources of experiment details.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXTENDED RELATED WORK

  • Related graph-based RAG approaches still rely on LLM corpus passes for entity or relation extraction or summarisation, while CONRAG uses co-occurrence-mined mentions qualified by an LLM and retrieves between known endpoints.The passage distinguishes CONRAG by requiring neither external structure such as citation links nor a relation extractor.Source: A EXTENDED RELATED WORK · p. 16

B.1 ADDITIONAL CONRAG IMPLEMENTATION DETAILS

This appendix records CONRAG’s fixed experiment parameters, baseline configurations, benchmark processing, and pooled-corpus setup. The derived relation-inference examples use known anchors and targets with annotated intermediate bridges and links.

B.1 ADDITIONAL CONRAG IMPLEMENTATION DETAILS

  • CONRAG’s experiment parameters are fixed across both benchmarks, and entities above the 90th degree-distribution percentile are excluded from path enumeration when phub = 0.9.The parameters cover entity qualification, indexing, traversal, hub suppression, path length, and path selection.Source: §B.1 · p. 16, Table 5 · p. 16

B.2.1 NAÏVE RAG

  • Naïve RAG retrieves the top 10 documents by embedding similarity from an exact FAISS index and uses GPT-4o for generation, with no LLM indexing call.Documents are indexed individually, and the query and documents use text-embedding-3-small.Source: §B.2.1 · pp. 16–17

B.2.2 LIGHTRAG

  • LightRAG uses GPT-4o-mini for entity and relation extraction, then hybrid retrieval capped at 10 documents before GPT-4o generation.Its embeddings and storage use text-embedding-3-small, NanoVectorDB, and NetworkX.Source: §B.2.2 · p. 17

B.2.3 PATHRAG

  • PathRAG retrieves 10 entity nodes and relationship edges to form a candidate pool, then prunes it to 15 highest-scoring relational paths.The setup uses a per-hop decay rate of 0.8 and retention threshold of 0.3.Source: §B.2.3 · p. 17

B.2.4 HIPPORAG 2

  • HippoRAG 2 uses GPT-4o-mini for graph extraction, adds synonymy edges above cosine similarity 0.8, and retrieves 10 documents via PPR.GPT-4o serves as recognition memory and generator.Source: §B.2.4 · p. 17

B.3.1 ORIGINAL DATASETS

  • MuSiQue and 2WikiMultiHopQA provide decompositions or evidence annotations with intermediate answers, enabling their reinterpretation as relation-inference benchmarks.MuSiQue includes linear and branching question structures; 2WikiMultiHopQA includes compositional and inference questions based on chained triples.Source: §B.3.1 · pp. 17–18

B.3.2 PROCESSING FOR MULTI-HOP RELATION INFERENCE

  • The processing defines the answer as target, the starting entity as anchor, and intermediate answers or triple objects as bridges, with the ordered chain forming gold links.For some MuSiQue decompositions, GPT-4o-mini extracts the anchor; manual review found 92 of 100 random extractions named the gold chain’s starting entity.Source: §B.3.2 · p. 18

B.3.3 QUESTION STATISTICS

  • The evaluation samples three sets of 1,000 questions per dataset and retrieves over pooled, deduplicated validation corpora of approximately 26.3k MuSiQue and 56.7k 2WikiMultiHopQA documents.MuSiQue questions have two to four hops, while all selected 2WikiMultiHopQA questions are two-hop; retrieval uses the full pooled corpus rather than each question’s context bundle.Source: §B.3.3 · p. 19, §B.3.4 · p. 19

B.3.5 LIMITATIONS OF THE DERIVED BENCHMARK

The derived benchmark’s reference chains and document support constrain how relation-retrieval metrics should be interpreted.

B.3.5 LIMITATIONS OF THE DERIVED BENCHMARK

  • Alternative valid routes count as false links, and 312 MuSiQue questions plus 95 2WikiMultiHopQA questions contain at least one gold link unsupported by any document.Restricting MuSiQue evaluation to 1,754 grounded questions raises every CONRAG metric, so reported scores on the full set are conservative.Source: §B.3.5 · p. 19, Table 6 · p. 19

C.1 DETAILED RESULTS OVERVIEW

Hop-stratified results show CONRAG’s strongest pattern in bridge recovery, with chain-recovery performance weakening as hop count increases.

C.1 DETAILED RESULTS OVERVIEW

  • CONRAG leads BCM by 11.9, 13.8, and 4.9 points on two-, three-, and four-hop MuSiQue questions, respectively.Table 7 stratifies results by dataset and hop count.Source: §C.1 · p. 19, Table 7 · p. 20
  • On two-hop questions, CONRAG gains chain precision and recall on 2WikiMultiHopQA and beats HippoRAG 2 in MuSiQue chain recall.The nominal two-hop LightRAG precision lead is within one standard deviation.Source: §C.1 · p. 19, Table 7 · p. 20
  • At three and four hops, every system recovers fewer than one in ten and one in twenty gold links, respectively; CONRAG’s four-hop chain precision falls below baselines despite leading chain recall.CONRAG still names most gold bridges on four-hop questions.Source: §C.1 · p. 19, Table 7 · p. 20

C.2 WIN CASE ANALYSIS

In a two-hop MuSiQue example, CONRAG retrieves the bridge document connecting Marzuki Alie to 15 February 1942, while HippoRAG 2 retrieves no chain. The contrast illustrates how reaching the connecting evidence can determine whether a chain is produced.

C.2 WIN CASE ANALYSIS

  • On the cited two-hop MuSiQue example, CONRAG retrieves the Battle of Palembang bridge document and predicts Marzuki Alie → Palembang → 15 February 1942; HippoRAG 2 returns no chain from its top-ten documents.Source: §C.2 · p. 20
  • Palembang links Marzuki Alie’s birthplace to the Battle of Palembang, which occurred during the Pacific theatre of World War II from 13–15 February 1942.The bridge document is the only document tying the anchor’s birthplace to the target date.Source: §C.2 · p. 20
  • HippoRAG 2 retrieves topically adjacent wartime documents, but its context does not establish a connection between Marzuki Alie and 15 February 1942.The retrieved documents describe Marzuki Alie but do not mention the date or a linking event.Source: §C.2 · pp. 20–21

C.3 FALSE NEGATIVE ANALYSIS

The false-negative analysis finds that abstentions dominate systems’ missed gold chains, while many CONRAG alternative paths are supported by its retrieved context. Because evaluation accepts only the gold decomposition, valid alternative routes can still count as errors.

C.3 FALSE NEGATIVE ANALYSIS

  • The gold-only reference scores every valid route that deviates from the gold decomposition as an error.Table 8 partitions the false-negative pool; Table 9 classifies alternative-path candidates and describes blind validation against each system’s retrieved documents.Source: §C.3 · p. 21, Table 8 · p. 21, Table 9 · p. 21
  • 70 of 80 distinct MuSiQue candidates and 58 of 61 2WikiMultiHopQA candidates were supported by CONRAG’s own context.Abstentions dominate every system’s false negatives; baselines gain similar absolute numbers of valid alternatives, while LightRAG and PathRAG have markedly lower validity rates.Source: §C.3 · p. 22
  • For Australia and Lyons’ death in 1939, the alternative route through Joseph Lyons is valid but receives zero because neither link string matches the gold chain.The gold route instead passes through Robert Menzies, who succeeded Lyons.Source: §C.3 · p. 22
  • For Mickey Mouse and Walt Disney, the gold route goes through Minnie Mouse, while the alternative route goes through Ub Iwerks.The passage describes Iwerks as drawing Mickey Mouse’s original sketches and co-creating the character with Disney.Source: §C.3 · p. 22

D.1 ENTITY FILTERING

Entity filtering sharply reduces the vocabulary while largely preserving bridge coverage, which path enumeration depends on. Endpoint-constrained seeding can recover anchors and targets that are absent from the graph.

D.1 ENTITY FILTERING

  • About 4% of the initial MuSiQue vocabulary and 6% of the initial 2WikiMultiHopQA vocabulary remain after document-frequency cutting and LLM filtering.The document-frequency cut removes about 85% of mined candidates, and LLM filtering removes roughly two thirds of the remainder.Source: §D.1 · p. 22
  • Bridge coverage is largely unaffected by document-frequency cutting and LLM filtering, preserving entities needed for path enumeration.Anchors and targets lose more coverage but are recovered through endpoint-constrained seeding.Source: §D.1 · p. 22
  • Figure 5 tracks mined candidates and gold anchors, targets, and bridges retaining entity nodes after each indexing stage on both benchmarks.The hubs stage applies only to the traversal adjacency.Source: Figure 5 · p. 23

E OVERVIEW OF LLM PROMPTS

The appendix prompts extract benchmark anchors, filter mined entity candidates, generate evidence-grounded connection chains and explanations, and evaluate explanations through pairwise judgments. The anchor-extraction prompt is used during benchmark construction, not as part of CONRAG itself.

E.1 DATASET ANCHOR ENTITY EXTRACTION

  • The anchor prompt distinguishes a chain’s starting name from the surrounding nested description and instructs the model to return the name exactly.Examples include returning “Mickey Mouse” rather than “Mickey Mouse’s spouse”; outputs are JSON objects with an index and anchor.Source: §E.1 · p. 23
  • For MuSiQue questions with natural-language first steps, GPT-4o-mini extracts the single proper name where the reasoning chain starts.It is used once during benchmark construction; structured <entity> » <relation> questions allow direct parsing instead.Source: §E.1 · p. 23

E.2 ENTITY FILTERING

  • The entity-filtering prompt labels items true when they are specific named entities and false when they are generic, descriptive, grammatical, or not specific referents.Examples include people, places, organisations, works, dates, and languages as true entities.Source: §E.2 · p. 24
  • GPT-4o labels batches of mined candidates using their surface strings and corpus capitalization rates; only candidates labelled true enter the entity vocabulary.The prompt uses containment-aware batches and accounts for CONRAG’s indexing LLM token cost.Source: §E.2 · p. 24

E.3 GENERATION

  • The generation prompt asks for evidence-grounded chains and a prose explanation of how two named entities relate, using only retrieved passages.It directs the model to identify intermediates, cover distinct supported connections, and return no chains when only incidental overlap or no connection is supported.Source: §E.3 · pp. 24–25
  • All systems share the same GPT-4o generation prompt, so the retrieved passage block is the difference between their generation inputs.The output fields mirror the chain and explanation; the model may return an empty chain list when the passages support no connection.Source: §E.3 · p. 25

E.4 EVALUATION

  • The evaluation prompt compares two answers on comprehensiveness, diversity, and empowerment, then selects an overall winner.For Table 2, GPT-4o-mini compares CONRAG’s explanation with one baseline per question, and win rates count selections of CONRAG.Source: §E.4 · p. 26
Loading 2609.35193v1…