Read the Route: Attention-Steered Hypergraph Retrieval for Multi-hop RAG

• Diffusion and similarity methods can miss evidence about implicitly referenced bridge entities, while exploration methods require repeated LLM generation and incur growing computational costs.Fixed propagation can spread relevance to irrelevant entities across hops; generation-based methods also underuse graph structure.Source: §1 · pp. 1–2
• HyperAddress separates semantic guidance from structural exploration, using LLM signals to guide graph computation instead of generating reasoning at every hop.Source: §1 · p. 2
• Addressing heads identify the entities described by question phrases before generation, covering several hops in one forward pass and requiring no labels for identification.Ablating these heads halves the LLM’s accuracy on two-hop questions whose bridge is described.Source: §1 · p. 2
• HyperAddress runs Personalized PageRank on a description-labelled hypergraph, using addressing-head scores for its restart distribution and transition weights.Source: §1 · p. 2
• 3.5 points of Recall@5 and 7.4 points of All@5 on average over PropRAG across three multi-hop benchmarks.Source: §1 · p. 2

• Graph-based RAG indexes represent corpus structure using summaries, entity relations, propositions, related facts, questions, or entity co-occurrence.Examples include RAPTOR’s summary tree, knowledge graphs, and relation-free passage or sentence links.Source: §2 · p. 3
• Multi-hop retrieval either propagates relevance or searches by similarity, or uses an LLM to choose successive nodes, paths, or sub-questions.Source: §2 · p. 3
• Attention-based retrieval work scores prompt passages or retrieves contextual information, while interpretability studies examine reference resolution and bridge-entity resolution.Source: §2 · p. 3

Motivation

• The section investigates whether an LLM can identify described entities before generation, unlike exploration-based retrieval that generates at every hop.The analysis examines attention because Transformers move information between positions through attention.Source: §3 · p. 3

Analysis Setup

• The analysis uses Qwen3-4B to read retrieved passages followed by a question in one prompt without generating tokens, and selects eight heads from 1,152 using 600 unlabeled probes.For each head, the pointer score measures the contribution from an entity’s mentions to the head’s output at question tokens.Source: §3 · p. 3

Observation 1: Sparse Addressing Heads

• The eight addressing heads rank the target first on 59.0% of held-out probes, and mean-ablation lowers reader accuracy from 76.5% to 48.5% on probes and from 40.1% to 19.7% on two-hop MuSiQue.The ablation has a larger effect when the bridge is described than when it is named, while random heads cause smaller drops.Source: §3 · pp. 3–4, Table 1 · p. 3

Observation 2: Word-Level Multi-hop Addressing

• 79.8% of three-hop and 75.6% of four-hop MuSiQue questions have all bridges among the five highest-scoring candidates when scores are read across question words.For later hops, the described bridge is highest-scoring at its own description about twice as often as chance; last-token scoring substantially lowers later-bridge top-five rates.Source: §3 · p. 4

Observation 3: Context-Dependent Addressing

• In-context addressing heads outperform embeddings on target ranking, while encoding the question or passages separately reduces the heads’ mean reciprocal rank.The paired in-context gains are .174 on probes and .198 on MuSiQue; removing rotary position encoding changes mean reciprocal rank by less than .01.Source: §3 · p. 4, Table 2 · p. 4

4.1 Offline: Description-Labelled Hypergraph

• The offline index stores passages as vertices, entities as hyperedges, and passage-specific descriptions of entities on their incidences.An LLM writes one to three descriptions per mentioned entity; incidence weights are |e|^-1/2, reducing probability flow through entities mentioned by many passages.Source: §4.1 · pp. 4–5
• The index supports non-parametric continual learning: adding a passage requires its own address book and joining the hyperedges of named entities, without reprocessing other passages.Incidence labels let the initial restart match a question’s description and bring the passage stating the relation into the reader’s context.Source: §4.1 · p. 5

4 Attention-Steered Hypergraph Retrieval

• HyperAddress runs Personalized PageRank on a description-labelled hypergraph, with addressing heads setting the restart distribution and transition weights over retrieval rounds.The reader processes top-ranked passages and the question in one forward pass per round, without generating tokens.Source: §4 · p. 4, §4.2 · p. 5

4.2 Online: Attention-Steered PageRank

• At query time, the walk alternates between passages and entity hyperedges, and the reader’s scores reweight read incidences in both directions.Between rounds, HyperAddress replaces one question description with the entity name it most likely refers to; the rewritten question adds a restart for the next round.Source: §4.2 · pp. 5–6

5.1 Setup

• Evaluation uses 1,000 questions each from MuSiQue, 2Wiki, and HotpotQA; design choices were fixed on MuSiQue development questions and kept across benchmarks.Results use the remaining 800 questions per benchmark; retrieval metrics include Recall@k and All@5.Source: §5.1 · p. 6

5.2 RQ1: Retrieval Effectiveness

• The margin over PropRAG grows with question depth: on MuSiQue it is 5.3 points of Recall@5 and 8.7 of All@5, versus 2.1 and 4.2 on two-hop HotpotQA.On every benchmark, the gain in All@5 exceeds the gain in Recall@5; on 2Wiki, All@5 is 92.8% versus 86.1% with IRCoT using the 27B model.Source: §5.2 · p. 6
• Averaged over three benchmarks, HyperAddress improves Recall@5 by 3.5 points and All@5 by 7.4 points over PropRAG, ranking first on every benchmark and metric.Every improvement over the strongest baseline of its column is significant at the 95% level.Source: §5.2 · p. 6, Table 3 · p. 7

5.3 RQ2: Where the Gain Comes From

• On MuSiQue test, steering with addressing heads adds 8.4 Recall@5 points over the walk without reading; the gain rises from 7.6 points at two hops to 10.1 at four.One reader forward pass without generation or rewriting raises Recall@5 from 74.7 to 80.1, accounting for 5.4 of the 8.4 points; a second steering round reaches 81.7.Source: §5.3 · p. 7, Figure 4 · p. 7
• Scoring every question word with addressing heads matters: removing labels lowers Recall@5 by 3.2 points, and the address book increases one-round steering gain from 1.7 to 5.4 points.At four hops, the alternative head sets reach 50.0 and 47.8 Recall@5, against 47.8 for the unsteered walk and 57.9 for HyperAddress; removing steered transitions costs 4.9 points.Source: §5.3 · pp. 7–8

5.4 RQ3: Indexing and Query Cost

• HyperAddress answers in 0.51 seconds without generating query-time tokens; its index takes less than half the tokens and about half the GPU time of HippoRAG 2 and PropRAG.PropRAG takes more than twice as long per query, HippoRAG 2 five times as long, and 27B IRCoT 23 times as long.Source: §5.4 · p. 8, Figure 5 · p. 8

5.5 RQ4: Question Answering

• Using HyperAddress passages, Qwen3.5-27B gives the best answers on all benchmarks, averaging 2.6 EM points above PropRAG and 2.6 F1 points above 27B IRCoT.On MuSiQue, it is 3.6 EM and 4.0 F1 points above 27B IRCoT.Source: §5.5 · p. 8, Table 4 · p. 8

5.6 RQ5: Other Readers

• Steering improves retrieval with every tested reader; with a 1.7B reader, HyperAddress reaches 81.5 Recall@5, 2.6 points above the strongest baseline, 27B IRCoT.The gain over the unsteered walk grows from 6.8 points with the 1.7B reader to 8.8 with either tested 8B reader.Source: §5.6 · p. 8, Table 5 · p. 8

6 Conclusion

• HyperAddress combines a one-pass LLM reading of top passages and the question with addressing-head control of both restart and transition weights in Personalized PageRank.The walk runs on a description-labelled hypergraph and generates no tokens at query time.Source: §6 · p. 8
• Across three multi-hop benchmarks, HyperAddress outperforms nine baselines, with gains growing as the number of hops increases.The conclusion states that attention-steered graph retrieval retrieves more evidence than generating every hop, in a fraction of the time.Source: §6 · p. 9

A Implementation Details

• HyperAddress constants are selected on MuSiQue development questions and kept across benchmarks and readers.NV-Embed-v2 encodes questions with separate retrieval and addressing instructions.Source: A Implementation Details · p. 10, Table 6 · p. 10
• Rewriting replaces a question span with the name of the entity most likely described, when span-label cosine similarity is at least 0.7.The rewritten question's restart, weighted 2, is added to restarts from earlier question versions.Source: A Implementation Details · p. 10
• In generated-name variants, Qwen3-4B names a bridge from the passages and question, then maps the name to a candidate entity by embedding similarity.The selected entity replaces the rewriting choice and receives the full restart share for steering.Source: A Implementation Details · p. 10

Binding

• On 300 probes, a made-up name became the top candidate in 25% of cases when the passage bound the description to it, versus 3% when it replaced an unrelated entity.The heads ranked the entity bound by the text first, regardless of its name.Source: §A.1 · p. 10

Positional rotation

• Removing rotary position encoding changed probe mean reciprocal rank from .734 to .728.The reported change was less than .01.Source: §A.1 · p. 10
• Removing rotary position encoding changed MuSiQue mean reciprocal rank from .705 to .702, supporting a content-based rather than position-based match.The reported change was less than .01.Source: §A.1 · p. 10

Decomposition

• Table 7 reports MuSiQue Recall@5 for variants from Figure 4 and Section 5.3, overall and by number of hops.The passage does not provide the table's result values.Source: §A.2 · p. 10

Cost

• Table 8 reports index-building costs, including generated tokens and GPU hours for 11,656 passages.HyperAddress's head selection is included in the index cost.Source: §A.2 · p. 10
• For Figure 5, Table 8 lists costs, including the tokens generated to build each index and the tokens generated to answer each question.Source: §A.2 · p. 10

All-question evaluation

• The comparison runs methods on all 1,000 questions of each benchmark, including development questions, alongside published numbers under the same protocol.Published numbers use NV-Embed-v2 and 70B-parameter LLMs; the runs use Qwen3.5-27B.Source: §A.3 · p. 10

Reproduction comparison

• NV-Embed-v2 reproductions were within 0.4 points of published Recall@5.The passage does not state specific per-benchmark Recall@5 values.Source: §A.3 · p. 10
• HippoRAG 2 reproductions were within 0.6 points on MuSiQue and HotpotQA and 3.0 points above published results on 2Wiki.These are differences between the runs and the published numbers.Source: §A.3 · p. 10

Single-hop results

• HyperAddress ranks first on NQ and PopQA, scoring 2.7 Recall@5 points above HippoRAG 2 on NQ and 12.5 above PropRAG on PopQA.The comparisons cover all 1,000 questions in each single-hop dataset.Source: §A.4 · p. 10
• On PopQA, complete evidence appears in the top five for 38.1% of questions, compared with 3.0% with dense retrieval.Source: §A.4 · p. 10

MuSiQue tables

• Table 7 reports MuSiQue test Recall@5 overall and for groups of 415, 258, and 127 questions by hop count.The caption says all comparisons are significant at paired-bootstrap 95% except appended labels; it does not provide result values.Source: Table 7 · p. 11
• Table 8 reports MuSiQue index costs in generated tokens and GPU hours for 11,656 passages, plus mean per-question cost over the first 100 test questions.The caption notes that IRCoT retrieves from the dense index.Source: Table 8 · p. 11

A.5 Baselines

• Baselines use HippoRAG 2 corpora and questions, with methods run using released configurations or code where available and HGRAG reimplemented from its paper.LLMs use greedy decoding without thinking mode; HyperAddress's reader runs in Transformers, and its encoder and several baselines use NV-Embed-v2.Source: §A.5 · p. 11
• Table 9 reports Recall@5 on all questions under a shared protocol, comparing the authors' runs with published numbers.The caption specifies 1,000 questions.Source: Table 9 · p. 11
• Table 10 reports Recall@5 and All@5 on 1,000 single-hop questions each from NQ and PopQA.Published numbers are attributed to HippoRAG 2 and PropRAG with Llama-3.3-70B; unreported values are marked with a dash.Source: Table 10 · p. 11
• ICR and QRHead score the top twenty NV-Embed-v2 passages with Qwen3-4B; QRHead uses sixteen heads selected on NQ data.Both rebuild question-token attention from query states and cached keys; ICR uses default calibrated scoring and places the most relevant passage last.Source: §A.5 · p. 11