PaperlayerView exact revision on arXiv

Source-linked AI summary

REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing

Jiawen Tao, Xiaokun Yuan, Yaoming Li, Chenxu Liu, Mengzhou Wu, Tong Yang, Maxm Pan

arXiv:2609.36984v1cs.AI

TL;DR

Multi-hop benchmarks treat correct answers on predefined reasoning chains as evidence of composition, but do not establish dependence on every intended step. REALHOP audits and rebuilds benchmarks around evidence necessity. Across its evaluations, BNR rises sharply and long-context model spread persists under repeated open-ended evaluation.

  • Problem

    Predefined reasoning chains and correct answers do not establish whether success depends on evidence for every intended step.

    Source: §1 · p. 1

  • Method

    REALHOP diagnoses bypasses, rebinds entities, factorizes selected relations, adds complete competing paths, and verifies construction before behavioral deletion tests.

    Source: §1 · pp. 2–3

  • Results

    27.4% to 94.4% BNR on 790 paired MuSiQue questions; BNR reaches about 89% on REALHOP-FRAMES and above 96% on REALHOP-LONGBENCH, with a wider long-context model spread.

    Source: §1 · p. 2, §9 · p. 11

  • Takeaways & Limitations

    The results support evidence necessity as a benchmark design criterion alongside answer accuracy.

    Source: §9 · p. 11

    BNR depends on the chosen evidence unit, judge, and trial budget, and only indirectly reflects processing; constructed synthetic entities and planned chains do not cover all natural information needs.

    Source: §8 · p. 10

Abstract

from arXiv · show

Complex questions often require multi-hop reasoning that connects facts distributed across sources or distant regions of a long context through intermediate steps. Benchmarks commonly evaluate this ability with questions built around predefined reasoning chains, treating a correct answer as evidence that the intended composition was used. Yet answer correctness alone leaves open whether success depends on the evidence associated with each intended step: models may instead rely on memorized associations, shorter paths, or partial evidence. We examine this dependence using the Behavioral Necessity Rate (BNR), which measures how often targeted evidence removal prevents answer recovery on initially correct instances. Across five existing benchmarks, panel-mean BNR ranges from 16.6% to 48.9%, exposing a substantial gap between annotated structure and observed dependence. Guided by this diagnosis, we introduce REALHOP, a diagnose-construct-verify framework that rebinds entities, factorizes selected relations, adds complete competing paths, and places evidence at traceable locations. Structural and semantic checks precede freezing; behavioral interventions follow. On 790 paired MuSiQue questions, REALHOP raises panel-mean BNR from 27.4% to 94.4% while retaining high Full accuracy. It also yields high BNR on REALHOP-FRAMES and REALHOP-LONGBENCH. On 216 long-context questions, the matched multiple-choice spread across 16 models grows from 13.9 to 59.2 points and persists under repeated open-ended evaluation. Together, these results show that a conceptually coherent chain and a correct final answer do not by themselves establish multi-hop reasoning. Verifying that success depends on every intended hop is therefore as fundamental to multi-hop evaluation as measuring answer accuracy itself.

1. Introduction

The introduction questions whether correct answers on predefined multi-hop chains depend on evidence for every intended step. It presents a behavioral audit and REALHOP, which rebuilds and tests benchmarks for evidence necessity.

1. Introduction

  • Multi-hop benchmarks use predefined chains to connect distributed facts through intermediate steps, but answer accuracy alone leaves the actual evidence dependence unresolved.The introduction asks whether success requires evidence for each intended step and whether benchmark success establishes multi-hop reasoning ability.Source: §1 · p. 1
  • Answer recall, unannotated shorter paths, and surface cues can preserve answers after gold-hop evidence is removed.Thus, coherent annotations or lower accuracy under added noise alone do not establish dependence on the intended chain.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • 16.6–27.4% BNR on three standard benchmarks and 43.5–48.9% on two adversarial constructions show that correct answers often survive gold-evidence deletion.The audit uses targeted deletion among initially correct instances, with matched-background Control to distinguish evidence-specific failure from generic deletion sensitivity.Source: §1 · p. 2
  • REALHOP rebinds entities, factorizes selected relations, adds complete competing paths, and records evidence locations before structural, semantic, and behavioral checks.Construction is frozen after structural and semantic checks; subsequent evidence-deletion tests measure whether answers depend on intended support.Source: §1 · pp. 2–3
  • 27.4% to 94.4% BNR on 790 paired MuSiQue questions, 88.9–89.9% on REALHOP-FRAMES, and 96.5–98.1% on REALHOP-LONGBENCH demonstrate higher evidence dependence across settings.MuSiQue Full accuracy remains 68.5–83.5%; the LongBench result covers three audited models.Source: §1 · pp. 2–3
  • The 16-model accuracy spread on REALHOP-LONGBENCH grows from 13.9 to 59.2 points and persists under repeated open-ended evaluation.The matched multiple-choice comparison uses a protocol shared with the source questions.Source: §1 · p. 3

2. Related Work

Prior work documents shortcut routes in multi-hop benchmarks and develops construction methods to make them harder. The paper distinguishes these efforts from directly testing whether each intended support is necessary for a correct answer.

Standard multi-hop QA and shortcut-resistant evaluation

  • HotpotQA analyses identify single-hop solutions, sentence cues, and word-matching routes [4] [11] [18], while models may know individual hops yet fail to compose them [19].Standard multi-hop datasets annotate supporting chains so correctness can be read as evidence of connected reasoning [7] [22] [25] [29] [31].Source: §2 · p. 3

Construction work

  • MuSiQue composes connected single-hop questions [25], while plausible distractors and counterfactual chains target shortcut artifacts [33].Other constructions add implicit or generative hops [6] [20].Source: §2 · p. 3

Long-context multi-hop reasoning

  • LongBench v2 [2], Loong [27], RULER, and BABILong [14] test locating and combining facts in extended contexts, but do not establish dependence on every intended hop.FRAMES provides source-linked questions without annotated hop chains [13].Source: §2 · p. 3

Evidence necessity through intervention

  • Sufficiency tests whether selected evidence is enough, whereas necessity tests whether a correct prediction survives its removal [16] [5] [32].Attention weights and chain-of-thought traces are unreliable proxies for dependence [10] [15] [26] [30]; DiRe measures disconnected reasoning using complementary support subsets [24].Source: §2 · pp. 3–4
  • REALHOP intervenes on individual hops and uses matched background deletion to separate evidence-specific failure from generic deletion sensitivity.Unlike prior audits, its diagnosis feeds into construction through complete competing paths and structural, semantic, and behavioral checks.Source: §2 · p. 4

3. Problem Formulation and Benchmark Audit

The section separates annotated structural coherence from a model's behavioral dependence on each support. It defines a deletion-based audit and reports that correct answers often survive removal of annotated evidence.

3.1. The Conceptual–Behavioral Gap

  • An annotated evidence graph claims a coherent, sufficient structure for an answer but does not establish that a model must use every step.Conceptual necessity concerns whether removing a target leaves any sufficient answer path; behavioral necessity concerns whether the model loses correctness after that removal.Source: §3.1 · p. 4
  • A model can answer correctly through shortcuts, answer cues, unannotated evidence, or parametric knowledge rather than the annotated chain.Behavioral necessity is evaluated by comparing correctness on the original context with correctness after removing a target's evidence.Source: §3.1 · p. 4

3.2. Audit Protocol and Metrics

  • BNR compares Full with DROP-ONE among Full-correct items, sampling items and their declared evidence targets for deletion.A matched-background Control provides a separate specificity check and is not part of BNR.Source: §3.2 · p. 4
  • BNR is the item-macro proportion of testable targets whose removal prevents gold-answer recovery, so longer chains receive no greater weight.It is model- and protocol-specific and is reported alongside Full accuracy.Source: §3.2 · p. 4

3.3. Audit of Existing Multi-Hop Benchmarks

  • The audit covers 790 MuSiQue questions and stratified 300-question samples from HotpotQA and 2WikiMultiHopQA, plus 300-item samples from each adversarial split.The audit uses different deletion units across datasets: a gold paragraph for MuSiQue and support sentences associated with one title for HotpotQA and 2Wiki.Source: §3.3 · p. 4
  • 16.6–27.4% BNR on standard benchmarks and 43.5% and 48.9% on adversarial constructions show that correct answers still often survive gold-hop deletion.Control remains near Full and well above Drop, supporting evidence-specific sensitivity rather than an effect of deletion volume alone.Source: §3.3 · pp. 4–5, Table 1 · p. 5
  • FalseCoTQA and Plausible Distractors reach 43.5% and 48.9% BNR, respectively, but remain far below REALHOP-MUSIQUE's 93.9–95.0%.The comparisons are descriptive and unpaired.Source: §3.3 · p. 5
  • The audit motivates constructions that make every intended binding consequential in standard and long-context multi-hop QA.Source: §3.3 · p. 5

4. Method

REALHOP constructs evidence paths for every reasoning step, realizes them at traceable context locations, and checks the resulting items before behavioral evaluation. Its adapters vary how reasoning graphs are obtained and evidence is placed across MuSiQue, FRAMES, and LongBench settings.

4. Method

  • The construct phase first builds evidence from a question, context, answer, and dependency graph, then realizes it in context, with separate checks determining sample acceptance.The two stages are summarized in Figure 2.Source: §4 · p. 5, Figure 2 · p. 6

4.1. Construct Evidence

  • Entities in questions, planned facts, and answers are rebound while preserving relations, roles, answer type, and reasoning structure; selected relations may be split into premises that work together.Original passages remain as background, and intermediate answers are not revealed in the question.Source: §4.1 · p. 5
  • For each reasoning step, REALHOP adds a complete alternative path with a similar but insufficient relation, using Flexible or Strict rules to control where the mismatch may occur.At least one complete Strict path remains for each step after problematic paths are removed.Source: §4.1 · p. 5

4.2. Realize in Context

  • Generated gold facts are inserted into original supporting passages and competing facts into compatible passages; the realized text is checked against planned facts, locations, and semantic requirements.The exact text expressing each fact and evidence unit is recorded; problematic paths are removed before retained evidence and paths are checked against construction requirements.Source: §4.2 · p. 6

4.3. Adapt Across Settings

  • After the dataset is fixed, Full and targeted Drop evaluations test answering and evidence dependence, while matched-background Control helps distinguish evidence loss from generic text-removal sensitivity.MuSiQue uses annotated dependency graphs; FRAMES reconstructs linked articles and infers chains, while LongBench starts from a fictional graph and places evidence across long-document chunks.Source: §4.2 · p. 6, §4.3 · pp. 6–7

5. Experimental Setup

The experiments assess whether REALHOP increases behavioral necessity, isolates the components behind the challenge, and separates long-context models consistently. Evaluations use paired source and constructed questions, with protocols adapted to each setting.

Experimental aims

  • The evaluation tests whether REALHOP increases behavioral necessity, which construction components create the challenge, and whether long-context construction separates models consistently across trials.Source: §5 · p. 7

MuSiQue and FRAMES

  • MuSiQue contains 790 paired questions spanning 2–4 hops, while the FRAMES audit uses 81 matched source/constructed pairs with inferred hop spans; both compare Full and targeted Drop for three models.MuSiQue confidence intervals resample paired questions; FRAMES reports Avg@3 over recorded trials.Source: §5 · p. 7

LongBench evaluation

  • REALHOP-LONGBENCH contains 216 open-ended questions with prompts of about 11k–575k tokens, evaluated across 16 models and three trials, with a separate matched multiple-choice comparison.BNR uses open-ended Full/Drop runs for three models, while the format-matched comparison runs source and constructed questions once under the official multiple-choice template.Source: §5 · p. 7

6. Results

REALHOP raises behavioral dependence on intended evidence across MuSiQue, FRAMES, and long-context questions while retaining substantial Full accuracy. Its long-context construction also produces a wider model-performance spread than the source questions.

6.1. Behavioral Effects on REALHOP-MUSIQUE

  • 93.9–95.0% BNR on REALHOP-MUSIQUE, versus 25.2–28.9% on Original, accompanies 68.5–83.5% Full accuracy; Figure 3 summarizes the source and constructed comparison.The paired BNR increases are 65.5 points for DeepSeek, 68.7 for Gemini, and 66.8 for Qwen; Full accuracy differs by only 2.6 points across models on source MuSiQue, while the constructed between-model spread is 15.0 points.Source: §6.1 · pp. 7–8, Figure 3 · p. 8
  • Six earlier models show paired accuracy drops of 25.3–47.9 points on model-specific subsets of 305–353 questions, but these runs do not measure BNR.The subsets differ from the current-model panel, so these declines establish increased difficulty but not greater reliance on source shortcuts.Source: §6.1 · p. 8

6.2. REALHOP-FRAMES Hop-Level Audit

  • On 81 matched FRAMES pairs, BNR rises from 6.6–8.1% on source questions to 88.9–89.9% after construction, while Full accuracy remains 68.4–86.0%.The reported increase is not obtained by counting unsolved Full questions as evidence-dependent successes.Source: §6.2 · p. 8

6.3. REALHOP-LONGBENCH Across Repeated Trials

  • REALHOP-LONGBENCH spans 26.1–82.6% Avg@3, and its matched multiple-choice spread is 59.2 points versus 13.9 on source questions; open-ended and multiple-choice rankings correlate at Spearman ρ = 0.86, p < 10−4.Across three runs, 15 of 16 models vary by no more than about six percentage points.Source: §6.3 · p. 9, Table 2 · p. 9
  • Targeted deletion on REALHOP-LONGBENCH yields 97.6% BNR for DeepSeek V4 Flash, 98.1% for Gemini 3.6 Flash, and 96.5% for Qwen 3.7 Max.Source: §6.3 · p. 9

7. Analysis

The component analysis separates difficulty from behavioral necessity by comparing competitor-path variants and targeted all-hop interventions. Rewriting the gold path and adding competitors affect these outcomes differently.

Ablation setup

  • The analysis uses a reviewed 73-item cohort from a stratified 80-question sample, with Gold-only, parallel Flexible and Strict competitor conditions, and a Full condition combining both families.Each condition’s BNR is conditioned on its own Full-correct subset and interpreted alongside its Full accuracy.Source: §7 · pp. 9–10

Difficulty

  • Competitor insertion accounts for 85.5% of the earlier-panel mean Original-to-Full accuracy reduction, while Strict paths cost more accuracy than Flexible competitors overall.Both effects are much larger on the earlier model panel than on the current panel.Source: §7 · p. 10, Table 3 · p. 10

Behavioral necessity

  • Rewriting the gold path more than doubles panel-mean BNR, and adding either competitor family raises it by roughly 30 further points, with a smaller additional gain when both are combined.The ablation reports BNR only for current models and weights models equally in its mean.Source: §7 · p. 10, Table 3 · p. 10

8. Limitations

BNR tests whether targeted evidence removal prevents recovery among initially correct answers, but does not establish that a model internally used the removed hop.

8. Limitations

  • BNR is an indirect behavioral measure whose estimate depends on the chosen evidence unit, judge, and trial budget.Most reported BNR uses one Full/Drop pair per condition.Source: §8 · p. 10
  • REALHOP uses synthetic entities and planned chains inside source carriers, leaving the full range of natural information needs outside its scope.For REALHOP-LONGBENCH, the matched multiple-choice check uses one trial and is not score-equivalent to open-ended Avg@3.Source: §8 · p. 10
  • FRAMES controls use approximate matching with unverified trial counts, while common-Full results are conditional on a shared subset.Drop indirectly reflects processing rather than directly measuring it.Source: §8 · p. 10

9. Conclusion

REALHOP’s conclusion is that annotated structure and correct answers do not establish behavioral necessity; its paired MuSiQue analysis shows the result persists on jointly answerable questions. The appendices document the external audit and matched-background specificity checks.

9. Conclusion

  • Panel-mean BNR rises from 27.4% to 94.4% on REALHOP-MUSIQUE, reaches about 89% on REALHOP-FRAMES and above 96% on REALHOP-LONGBENCH, while long-context model separation persists across repeated trials.The conclusion treats evidence necessity as a benchmark design criterion rather than an assumption inferred from annotations or answer accuracy.Source: §9 · p. 11

A. External Multi-Hop Audit Details

  • The external audit covers 790 MuSiQue questions and stratified 300-question samples from HotpotQA and 2WikiMultiHopQA, plus 300-item samples from each of two adversarial constructions.The audit removes one gold paragraph for MuSiQue or support sentences associated with one title for HotpotQA and 2WikiMultiHopQA.Source: A. External Multi-Hop Audit Details · p. 16

Appendix Contents

  • Table 7 reports percentages of planned competitor-endpoint matches among Full answers on the 790 REALHOP-MUSIQUE questions, describing how often Full answers match a planned competitor endpoint.Source: Table 7 · p. 17

B. Matched-Background Specificity

  • Control survival remains 89.7–93.0% on REALHOP-MUSIQUE, while targeted Drop survival is 5.7–6.2%, separating targeted deletion effects from generic background deletion.The Control matches deletion volume using background spans outside gold supports and newly inserted competitor text.Source: B. Matched-Background Specificity · pp. 16–17, Table 5 · p. 16

C. MuSiQue Common-Full Analysis

  • 66.6–69.7-point BNR gains persist on the jointly Full-correct MuSiQue subset, compared with 65.5–68.7 points using each side’s own Full-correct population.The common-set result supports persistence on jointly answerable questions, not equivalence of populations.Source: C. MuSiQue Common-Full Analysis · p. 17, Table 6 · p. 17

D. Competitor-Endpoint Error Analysis

The competitor-endpoint analysis finds that many incorrect answers match planned alternatives, while a historical six-model comparison shows a broad difficulty increase on REALHOP-MUSIQUE. These analyses address distinct outcomes: endpoint errors and accuracy change, respectively.

D. Competitor-Endpoint Error Analysis

  • The endpoint analysis uses incorrect Full responses on 790 REALHOP-MUSIQUE questions and compares them with the terminal answers of planned competitors.Full correctness uses the same responses and denominator as Figure 3.Source: D. Competitor-Endpoint Error Analysis · p. 17
  • 64.3–69.5% of incorrect Full answers exactly match planned competitor endpoints across models, with wrong-hit rates of 11.1–20.3%.Exact matching excludes paraphrases and other branch-induced errors, making this a conservative signature of selecting introduced alternatives.Source: D. Competitor-Endpoint Error Analysis · p. 17

E. Historical Model Difficulty

  • Mean paired accuracy falls from 57.9% on source MuSiQue to 18.2% on REALHOP-MUSIQUE across six earlier models, a 39.7-point reduction.All six models decline; DeepSeek R1 remains strongest in both conditions but has the largest decrease, and these runs measure difficulty rather than BNR.Source: E. Historical Model Difficulty · p. 18, Table 8 · p. 18

F. Component Ablation Details

The appendix details REALHOP’s competitor ablations, long-context interventions, and construction checks. It distinguishes targeted hop deletion from random-window sensitivity and records the limits of routing, realization, and validation.

F. Component Ablation Details

  • Flexible competitors significantly reduce paired accuracy for five of six models, while DeepSeek R1’s reduction is borderline; positive interaction terms indicate overlapping rather than additive failure mechanisms.The sampled ablation cohort contains 73 reviewed items spanning two to four hops after seven exclusions.Source: F. Component Ablation Details · p. 18

G. REALHOP-LONGBENCH Targeted Interventions

  • LongBench v2 targeted-intervention BNR is computed over Full-correct questions with complete targeted Drop evaluations, averaging targeted hops within each question and then questions.The supplied table identifies the intervention results but does not provide its cell values.Source: G. REALHOP-LONGBENCH Targeted… · p. 19, Table 10 · p. 19

H. Random-Window Diagnostics

  • 11.4–23.8 points of accuracy are lost under random windows on REALHOP-LONGBENCH, versus 1.4–2.3 points on source LongBench v2; other long-context audits fall only 1.9–2.7 points.Loong falls by 17.0–22.0 points; random masks probe spatial sensitivity, not hop necessity, and all-mask survival on LongBench v2 exceeds the 51.2% single-indispensable-location prediction.Source: H. Random-Window Diagnostics · pp. 20–21, Table 11 · p. 20, Figure 4 · p. 20

I. Evidence Construction Details

  • Selected hops are factorized into complementary premises that jointly establish a relation, while preserving the original question graph’s annotated hop count.Plans link evidence units and atomic facts to stable identifiers and corresponding text locations.Source: §I.1 · p. 21, §I · p. 21

I.1. Plans and Traceability

  • Construction records link graph facts to text spans and track competitor plans, routes, and realizations; MuSiQue deletion removes a supporting paragraph rather than each premise.The linked records allow traceability but do not make premise-level necessity the intervention unit.Source: §I.1 · p. 22

I.2. Flexible and Strict Competitor Contracts

  • Flexible and Strict competitors impose different mismatch contracts, and the Full condition combines both families; planned coverage is not itself a guarantee of behavioral necessity.Strict requires the sole mismatch at the protected hop, whereas Flexible can also permit a downstream dependency break when safe.Source: §I.2 · p. 22

I.3. Routing, Realization, and Validation Boundaries

  • Routing and validation enforce fact coverage and reject ambiguous bindings or unintended answer paths, but structural checks do not certify complementary premises or behavioral necessity.MuSiQue deletion can remove co-located gold and competitor text, and competitor pruning is limited to two iterations without using Full/Drop outcomes.Source: §I.3 · pp. 22–23

I.4. FRAMES Adapter and Hop-Level Audit

  • FRAMES questions lack native hop labels, so the adapter assembles linked articles and proposes chains with evidence spans before applying the MuSiQue construction procedure.The adapted audit retains 81 questions and 208 hops per side; each Drop removes a hop’s recorded evidence spans jointly, not each split premise independently.Source: §I.4 · pp. 23–24

I.5. FRAMES Matched Controls and Sensitivity

The FRAMES matched-control analyses test whether BNR increases persist on jointly answerable questions and under tighter deletion-length constraints. Human spot checks assess constructed items, while the evaluation prompts specify how model responses are scored.

I.5. FRAMES Matched Controls and Sensitivity

  • 81.8–83.5 points: BNR increases persist on the jointly Full-correct subset across models.The intersection retains 72, 71, and 61 questions; relative to each side’s own Full-correct population, the increase changes by +0.81, +0.83, and +0.03 points, respectively.Source: §I.5 · p. 25
  • Control and Drop are aligned by sample ID and hop, with comparisons using the same question–hop conditions and equal weighting of questions.These are mean judge scores on a 0/0.5/1 scale, not binary accuracy or BNR; control trial counts cannot be verified from the stored average scores.Source: §I.5 · p. 25
  • +0.59, −2.19, and −0.58 points: constructed-side Control–Full mean-score differences show that background deletion is not assumed to have exactly zero effect.The reported intervals reflect question-sampling variability conditional on saved scores, not independent uncertainty in model trials, judges, or generated contexts; no multiplicity correction or equivalence test is claimed.Source: §I.5 · pp. 25–26
  • 60.6–69.8 points: the constructed-side Control–Drop gap remains in the 5% length band, which retains 38 conditions from 30 questions per model.All constructed-side Control–Drop intervals remain above zero in every band; tighter subsets do not establish exact matching of passage position, deletion geometry, or semantic relevance, nor universal necessity of every evidence unit.Source: §I.5 · p. 25

J.1. Human Spot Check of REALHOP-MUSIQUE

  • 2/100 sampled REALHOP-MUSIQUE items retained gold-chain errors after adjudication, both involving a pronoun with the wrong antecedent.The competitor sentences were not judged to give the gold answer.Source: §J.1 · p. 26

J.2. Human Spot Check of REALHOP-LONGBENCH

  • 50/50 sampled REALHOP-LONGBENCH items were accepted after adjudication.The text attributes the higher acceptance rate as consistent with local-neighborhood rewriting that gives gold and competitor relations explicit antecedents.Source: §J.2 · p. 27

K. Evaluation Prompts

  • The short-answer judge sees the question, references, and response—not the document—and only a fully correct verdict counts as correct for most reported scores.FRAMES maps correct, partial, and incorrect verdicts to 1, 0.5, and 0, then counts only 1 as correct for Full accuracy and BNR.Source: K. Evaluation Prompts · p. 27
  • The long-context multiple-choice evaluation uses a frozen A–D option order with the gold terminal and three planned competitor terminals as choices.The source and constructed evaluations use the same official LongBench v2 0-shot multiple-choice template.Source: K. Evaluation Prompts · p. 28

Related papers

Loading 2609.36984v1…