PaperlayerView exact revision on arXiv

Source-linked AI summary

Locally Sound, Globally Insufficient: The Local-Global Gap in Multi-Hop Reasoning

Bohao Chu, Hendrik Damm, Qianli Wang, Hui Wang, Shuning Zhang, Christoph M. Friedrich, Norbert Fuhr

arXiv:2609.32496v1cs.CL

TL;DR

Locally supported reasoning traces can still fail to align with the question or establish the submitted answer, and evidence-support verifiers largely miss these failures. The paper introduces E-CLOSURE, which trains evidence support, question alignment, and answer closure through supported generation and bidirectional counterfactual constraints. Across three benchmarks and three backbones, it achieves the strongest reported average accuracy and trace reliability among fine-tuned methods, with the lowest average LGG rate.

  • Problem

    A multi-hop trace can be locally sound yet globally insufficient, while conventional evidence-support verifiers largely miss these failures.

    Source: §1 · pp. 1–2

  • Method

    E-CLOSURE trains evidence-to-step support, question-to-trace alignment, and trace-to-answer closure using supported generation anchors and bidirectional counterfactual constraints.

    Source: §1 · p. 2

  • Results

    Across three benchmarks and three backbones, E-CLOSURE achieves the highest average answer accuracy (92.8%) and trace reliability (89.0%), with the lowest average LGG rate (6.2%) among evaluated fine-tuned methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The diagnostic identifies a substantial class of globally insufficient responses that remain locally sound, supporting separate checks of question alignment and answer closure.

    Source: §1 · p. 2

    The setup assumes the contextual evidence uniquely warrants the reference answer.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Reliable multi-hop reasoning requires more than locally supported steps: a trace can be sound at every reasoning step yet still fail to answer the question as a whole. We call this failure regime the local-global gap (LGG), in which the trace is locally sound yet globally insufficient. Local soundness requires each step to be supported by the available evidence and preceding steps, whereas global sufficiency requires the reasoning trace to align with the question and establish the submitted answer. In a human-adjudicated diagnostic of 2,598 responses across three multi-hop QA benchmarks and three models, we find that the LGG occurs in every benchmark-model combination and accounts for nearly half of globally insufficient responses overall. However, conventional faithfulness verifiers that check claims against the evidence largely miss these failures: at thresholds retaining at least 95% of reliable traces, recall for LGG cases is substantially lower than that for locally unsound traces. To address these failures, we formalize three dependencies for reliable reasoning: evidence-to-step support, question-to-trace alignment, and trace-to-answer closure. Instead of post-hoc diagnosis, we introduce E-Closure to supervise these dependencies during training, combining generation supervision on supported original and counterfactual responses with bidirectional switching constraints. Averaged over three benchmarks and three backbones, existing fine-tuning baselines improve accuracy and local soundness over the base models, but at the cost of global sufficiency. E-Closure improves both: among all fine-tuned methods, it achieves the highest average accuracy (92.8%) and trace reliability (89.0%) while yielding the lowest LGG rate (6.2%).

1 INTRODUCTION

A reasoning trace can be supported step by step yet fail to answer the question, either by pursuing the wrong target or by stating an answer inconsistent with what it establishes. The paper diagnoses this local–global gap and introduces E-CLOSURE to supervise the dependencies needed for reliable reasoning.

1 INTRODUCTION

  • A trace may support every substantive step yet pursue the wrong question target or establish one answer while submitting another.The film example has a trace establishing “Moonstruck” but submits “Please Give.”Source: §1 · p. 1
  • In 2,598 responses, 145 LGG cases appeared in every benchmark-model combination, accounting for 46.3% of globally insufficient responses.The cases comprised 35.2% path deviation, 47.6% answer decoupling, and 17.2% incomplete reasoning.Source: §1 · p. 2
  • At thresholds retaining at least 95% of reliable traces, conventional evidence-support verifiers reached at most 11.0% recall for LGG, versus 40.8% for locally unsound responses.Alignment and closure judges detected 66.7% of path deviations and 56.5% of answer-decoupling cases within the LGG under the same retention criterion.Source: §1 · p. 2
  • Across three benchmarks and three backbones, E-CLOSURE achieves the highest average answer accuracy (92.8%) and trace reliability (89.0%), and the lowest average LGG rate (6.2%) among evaluated fine-tuned methods.It combines supported original and counterfactual generation with bidirectional supervision of evidence-to-step support, question-to-trace alignment, and trace-to-answer closure.Source: §1 · p. 2
  • Reliable reasoning requires evidence-to-step support, question-to-trace alignment, and trace-to-answer closure, which E-CLOSURE supervises through paired interventions during training.The intervention must switch the dependent step, trace, or answer when its conditioning evidence, question target, or trace changes, with preferences reversing in both directions.Source: §1 · p. 2

2 RELATED WORK

Prior work supports multi-hop reasoning through retrieval, chain-of-thought, decomposition, and interleaved retrieval and generation, while reasoning evaluation covers several complementary properties. This paper separates question alignment and answer closure from step-level evidence support and uses dependency-specific bidirectional counterfactual supervision.

Multi-hop reasoning

  • Sufficient evidence alone does not ensure that a trace addresses the question or establishes the submitted answer, motivating the paper’s evidence-present setting.This setting focuses on reasoning failures rather than failures caused by insufficient evidence.Source: §2 · p. 2

Reasoning quality and evaluation

  • Existing evaluation approaches assess complementary reasoning qualities, but do not explicitly separate question alignment and answer closure as global requirements alongside step-level evidence support.The paper formalizes this distinction to identify cases where all substantive steps are supported yet alignment or closure fails.Source: §2 · pp. 2–3

Counterfactual and preference learning

  • E-CLOSURE combines counterfactual augmentation and preference optimization through dependency-specific switching constraints and generation anchors.Evidence interventions supervise evidence-to-step and trace-to-answer switching; question interventions supervise question-to-trace switching, with separate signals for the first affected step, full trace, and answer.Source: §2 · p. 3

3 PROBLEM FORMULATION AND DIAGNOSIS

The section distinguishes locally supported reasoning from globally sufficient responses, which must both pursue the question’s target and establish the submitted answer. A human-adjudicated study finds that locally sound but globally insufficient traces are common, while conventional evidence-support verifiers detect them poorly.

3.1 PROBLEM SETUP AND DEFINITIONS

  • The formalization treats each instance as a question, contextual evidence set, and reference answer, and separately assesses soundness even when alignment and closure hold.The study pools results over all 2,598 responses, while Figure 1 also presents dataset–model percentages and pooled rows with conditional denominators for LggShare and LggRisk.Source: §3.1 · p. 3, Figure 1 · p. 4
  • Global sufficiency requires an aligned trace that establishes the submitted answer, whereas local soundness requires every substantive step to be supported by evidence and preceding steps.The LGG is a locally sound response that is not globally sufficient; path deviation and closure failure are its two first-broken-link categories.Source: §3.1 · p. 3

3.2 DIAGNOSTIC STUDY

  • The diagnostic sampled 1,300 held-out questions across three benchmarks, retained 866 after evidence–gold screening, and generated 2,598 responses from three models.Each retained response received two independent annotations; disagreements and consistency-check violations triggered adjudication, affecting 823 responses (31.7%).Source: §3.2 · pp. 3–4
  • Trace reliability is the joint rate of responses that are both locally sound and globally sufficient, not necessarily the product of their separate rates.The reported measures also distinguish answer accuracy, global sufficiency, LggShare among globally insufficient responses, and LggRisk among answer errors.Source: §3.2 · p. 4

3.3 DIAGNOSTIC RESULTS

  • 46.3% of globally insufficient responses were locally sound, and the LGG accounted for 27.3% of incorrect responses in 2,598 adjudicated responses.The diagnostic also found 371 locally unsound but globally sufficient responses, showing that local soundness and global sufficiency can diverge in either direction.Source: §3.3 · p. 4
  • Among LGG cases, 47.6% were answer decoupling, 35.2% were path deviation, and 17.2% involved incomplete reasoning.Answer decoupling made up 70.7% of closure failures; path deviation and answer decoupling together accounted for 82.8% of LGG cases.Source: §3.3 · pp. 4–5
  • Global sufficiency and answer correctness are distinct: 28 responses were correct but globally insufficient, while 199 were incorrect but globally sufficient.The latter can still address the question and establish the submitted answer while relying on unsupported premises, which local soundness assesses separately.Source: §3.1 · p. 3, §3.3 · p. 5
  • At thresholds retaining at least 95% of reliable traces, alignment and closure judges detected 42.1% and 42.8% of LGG cases, while conventional faithfulness verifiers detected at most 11.0%.Alignment detected 66.7% of path deviations, while closure detected 56.5% of answer-decoupling cases.Source: §3.3 · p. 5, Table 1 · p. 5

4 E-CLOSURE: EDGE CLOSURE LEARNING

E-CLOSURE trains evidence-to-step support, question-to-trace alignment, and trace-to-answer closure using supported original and counterfactual responses. Its bidirectional constraints require preferences to switch when the conditioning input changes.

4 E-CLOSURE: EDGE CLOSURE LEARNING

  • E-CLOSURE combines generation supervision on supported original and counterfactual responses with bidirectional constraints on evidence-to-step support, question-to-trace alignment, and trace-to-answer closure.The constraints target the affected step, trace, or answer as the conditioning input changes.Source: §4 · p. 5, Figure 2 · p. 6

4.1 INTERVENTION VIEWS

  • The evidence intervention replaces a supporting item and changes the trace from its first dependent step, while the question intervention changes the requested target with evidence held fixed.The evidence intervention supplies the step and closure views; the question intervention supplies the trace-alignment view.Source: §4.1 · p. 6

4.2 BIDIRECTIONAL CLOSURE OBJECTIVE

  • The bidirectional margins prefer the original output under its original input and reverse that preference under the counterfactual input, preventing a fixed preference from satisfying the objective.The objective combines generation loss over three supported anchors with six dependency-direction margins; λ weights the margins relative to generation, with default λ = 1, γx = 0.5, and βx = 0.1.Source: §4.2 · p. 6

4.3 TRAINING AND INFERENCE

  • Training uses teacher forcing, with supported anchors as generation targets and swapped outputs used only in margins; inference retains the standard mapping without extra verifier calls or decoding changes.The three anchors are trained only under their supported inputs.Source: §4.3 · p. 6

5 EXPERIMENTS AND RESULTS

Across three benchmarks and three backbones, the experiments compare training methods on answer and reasoning quality, test objective ablations, and assess dependency switching under interventions.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses three multi-hop QA benchmarks, three backbones, adapted-method results averaged over three seeds, and paired hierarchical bootstrap intervals.Methods are averaged equally across nine dataset–backbone settings.Source: §5.1 · p. 7, Figure 3 · p. 8

5.2 MAIN RESULTS

  • SFT and CF-SFT improve answer accuracy and LSR over the backbone but reduce closure and GSR, while LGG rises from 5.4% to 10.4%–10.7%.Figure 3 aggregates results across nine dataset–backbone settings.Source: §5.2 · p. 7, Figure 3 · p. 8
  • E-CLOSURE leads the evaluated methods in average accuracy (92.8%), LSR (95.2%), GSR (91.5%), and TRR (89.0%), with a 6.2% LGG rate.Its LGG rate remains above the backbone’s 5.4%.Source: §5.2 · p. 7, Figure 3 · p. 8
  • Against RM, E-CLOSURE raises average accuracy from 90.5% to 92.8% and TRR from 81.2% to 89.0%, while lowering LGG from 11.6% to 6.2%.Table 2 reports accuracy and TRR gains over CF-SFT in all nine settings and over RM in seven.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 OBJECTIVE ABLATIONS

  • Removing any dependency objective lowers TRR by 4.27–6.12 percentage points; the largest drop follows removal of question-to-trace supervision.The full objective has the best point estimates across all seven metrics; paired LGG reduction intervals lie above zero only for removing T.Source: §5.3 · pp. 8–9, Table 3 · p. 9
  • Bidirectional supervision improves TRR by 6.89 percentage points and reduces LGG by 2.35 points versus one-sided supervision, with both confidence intervals above zero.The comparisons support a contribution from reverse preferences.Source: Figure 4 · p. 8, §5.3 · p. 9

5.4 DEPENDENCY INTERVENTION RESULTS

  • E-CLOSURE improves switching on both global dependencies over four comparators, with gains over RM for question-to-trace and trace-to-answer switching in all nine settings.On trace-to-answer switching, it beats RM by 69.59 percentage points; question-guided trace selection remains weaker, particularly on 2WikiMultiHopQA at 59.90%.Source: §5.4 · p. 9, Table 4 · p. 9

6 CONCLUSION

The conclusion presents the local–global gap as a distinct reasoning failure and motivates training that supervises all three dependencies.

6 CONCLUSION

  • Locally sound traces can still fail to align with the question or establish the submitted answer, and conventional faithfulness verifiers miss most such failures.The diagnostic study was human-adjudicated.Source: §6 · p. 9
  • E-CLOSURE combines supported-generation supervision with bidirectional counterfactual switching constraints for evidence support, question alignment, and answer closure.The method does not change inference.Source: §6 · p. 9
  • Across three benchmarks and three backbones, E-CLOSURE improves all four reported quality dimensions and lowers LGG relative to evaluated fine-tuning baselines, while gains over untuned backbones are modest.Its average LGG rate remains higher than the untuned backbones’.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies documentation and controls for the diagnostic study, training, and evaluation.

REPRODUCIBILITY STATEMENT

  • Appendix A documents diagnostic sampling, annotation guidelines, and adjudication procedures.The statement also points to diagnostic results for reference-free detectors.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendices B and C detail counterfactual data construction and training objectives, while Appendix D specifies training hyperparameters, seeds, checkpoint selection, and hardware.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Generation and training reproducibility measures include fixed random seeds, CUDA determinism settings, and generation temperature 0.Evaluation protocols, human validation, and uncertainty estimation are described in Appendices D.1, D.2, and D.5.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The statement records AI assistance, while the appendix documents the diagnostic sample, generation and annotation procedures, adjudication, and failure composition.

AI USE STATEMENT

  • ChatGPT polished author-written text; Cursor with Grok-family models assisted coding, and GPT-5.6 Sol supported counterfactual construction and semantic evaluation.The authors state they reviewed and verified every manuscript sentence and take responsibility for the work.Source: AI USE STATEMENT · p. 10

A DIAGNOSTIC STUDY: ADDITIONAL DETAILS

  • The diagnostic appendix describes question sampling, response generation, eligibility screening, annotation, and final notation: q is the question, Dq its evidence, and y = (r, a) the response.Here r is the reasoning trace and a the submitted answer.Source: A DIAGNOSTIC STUDY: ADDITIONAL DETAILS · p. 16

A.1 DIAGNOSTIC SAMPLE

  • The diagnostic sampled 1,300 questions across three benchmarks and retained 866 after screening; prevalence estimates therefore concern the retained evidence-eligible subset.Retained counts were 278 for 2WikiMultiHopQA, 255 for MuSiQue, and 333 for HotpotQA.Source: §A.1 · p. 16, §A.3.1 · pp. 18–19

A.2 GENERATION PROTOCOL

  • The study generated one completion per question–model pair, yielding 3,900 initial generations; 3,873 responses from 1,291 matched questions remained after incomplete outputs were excluded.The shared structured prompt requested a numbered reasoning trace followed by a submitted answer.Source: §A.2 · pp. 16–17

A.3.1 QUESTION-LEVEL ELIGIBILITY SCREENING

  • Two screeners independently assessed eligibility using only each question, its evidence, and reference answer; 866 questions were retained and 434 excluded after adjudication.Retained questions had to be well posed, uniquely answerable from evidence, and matched to the reference answer.Source: §A.3.1 · pp. 17–18

A.3.2 RESPONSE-LEVEL ANNOTATION RULES

  • Response annotation assessed substantive-step soundness, question alignment, answer closure, and then correctness and completeness, with information revealed in stages.Local soundness requires evidence-warranted claims and inferences; closure asks whether the trace supports the submitted answer.Source: §A.3.2 · pp. 19–20

A.3.3 AGREEMENT AND ADJUDICATION

  • A seventh annotator adjudicated 823 responses (31.7%); disagreements affected 22.9% of responses for step soundness and 5.0% for completeness.Directly annotated binary variables showed 95.0%–99.9% raw agreement.Source: §A.3.3 · pp. 20–21

A.4 FAILURE COMPOSITION

  • Among 2,598 responses, 145 were locally sound but globally insufficient: 51 failed alignment and 94 remained aligned but failed closure.Of the 94 closure failures, 25 were incomplete reasoning and 69 answer decoupling; path deviation and answer decoupling comprised 82.8% of LGG cases.Source: §A.4 · p. 21

A.5.1 EVALUATION SETUP

The appendix evaluates whether detectors identify locally unsound and globally insufficient responses at a low false-positive rate. Evidence-support detectors are less sensitive to LGG than alignment and closure detectors, whose strengths differ across LGG subtypes.

A.5.1 EVALUATION SETUP

  • The 2,598 responses are partitioned into reliable, LGG, and locally unsound groups; the LGG includes 51 path deviations and 94 closure failures.The closure failures comprise 25 incomplete-reasoning and 69 answer-decoupling cases.Source: §A.5.1 · p. 22
  • R@95 is the failure percentage detected at the highest empirical threshold that flags at most 5% of reliable responses; AUROC measures ranking separation.The appendix reports 95% percentile confidence intervals from 2,000 question-level bootstrap resamples.Source: §A.5.1 · p. 22
  • RAGAS and VeriScore assess the complete response against evidence, AlignScore assesses substantive steps, and global detectors withhold evidence to isolate alignment or closure.Detector backbones differ from the diagnostic generators’ model families to reduce potential evaluator-family bias.Source: §A.5.1 · p. 22, Table 15 · p. 23

A.5.2 DEPENDENCY-SPECIFIC DETECTOR SENSITIVITY

  • At R@95, evidence-support detectors detect 6.2%–11.0% of LGG cases, while alignment and closure detect 42.1% and 42.8%, respectively.AlignScore’s AUROC for distinguishing LGG from reliable responses is 0.469.Source: §A.5.2 · p. 23, Table 16 · p. 23
  • Within the LGG, alignment detects 66.7% of path deviations, while closure detects 56.5% of answer-decoupling cases and 64.0% of incomplete reasoning cases.For answer decoupling, alignment detects 17.4%, and evidence-support detectors detect 8.7%–10.1%.Source: §A.5.2 · p. 23

A.5.3 OPERATING-POINT ROBUSTNESS

The appendix tests whether LGG detection differences persist when thresholds are calibrated on reliable responses and transferred to held-out questions. Alignment and closure retain higher LGG recall than evidence-support detectors, but still miss more than half of LGG cases.

A.5.3 OPERATING-POINT ROBUSTNESS

  • Figure 5 plots detector-score distributions by response group, while Figure 6 plots LGG recall against reliable-response false-positive rate across thresholds.Figure 6 marks R@95 at the most permissive threshold with FPR at most 5%; its dashed diagonals indicate chance performance.Source: Figure 5 · p. 24, Figure 6 · p. 24
  • Five dataset-stratified folds calibrate each R@95 threshold on four folds’ reliable responses and apply it unchanged to the held-out fold.Questions are evaluated once using a threshold calibrated on other questions; bootstrap resamples repeat calibration and evaluation with fixed folds.Source: §A.5.3 · p. 24
  • With cross-fitted thresholds, alignment and closure each retain 42.8% LGG recall, versus 6.2%–10.3% for evidence-support detectors; held-out FPRs are 4.5%–5.1%.Each global detector still misses more than half of LGG cases at this operating point.Source: §A.5.3 · p. 24, Table 17 · p. 24

B TRAINING DATA CONSTRUCTION: ADDITIONAL DETAILS

The appendix describes how the authors screen sources, construct and verify original and counterfactual responses, and pack validated families for E-CLOSURE and baseline training. The resulting corpus contains 2,852 families, and a stratified human audit found that 290 of 300 sampled families passed across all three inspected views.

B.1 SAMPLING

  • The source pool samples 5,000 2WikiMultiHopQA, 15,000 MuSiQue, and 10,000 HotpotQA questions; 23,000 unique questions enter construction after screening and processing.More MuSiQue and HotpotQA questions are processed to offset higher screening exclusion rates and improve retained-data balance.Source: §B.1 · p. 25

B.2 SOURCE SCREENING

  • Source screening retains 7,276/23,000 questions (31.6%) and requires at least two support passages, a uniquely warranted reference answer, and no answer support from the remaining context.Malformed or underdetermined questions, broken entity links, unsupported answers, and impractically long answers are also rejected.Source: §B.2 · pp. 25–26

B.3 TARGET RESPONSE CONSTRUCTION

  • Every screened source yields an accepted target response whose steps are locally supported, whose trace pursues the question target, and whose submitted answer matches the reference.Reference-blind verification separately checks local soundness, alignment, closure, and evidence grounding; all 7,276 screened sources yield an accepted target.Source: §B.3 · p. 26

B.4 EVIDENCE INTERVENTION

  • An evidence intervention replaces one supporting passage and preserves the trace prefix until its first affected step; 5,169/7,276 accepted targets yield a verified intervention.The edited passage changes one semantic claim while preserving unrelated content, and the edited response must establish the new answer.Source: §B.4 · pp. 26–27

B.5 QUESTION INTERVENTION

  • A question intervention changes one target-determining component while keeping evidence fixed; 3,757/5,169 accepted evidence-intervention sources also yield a verified question intervention.The edited question must be natural, unambiguous, multi-step, and not reveal its answer; matched question–response pairs are generation anchors.Source: §B.5 · pp. 27–28

C EDGE CLOSURE LEARNING: ADDITIONAL DETAILS

  • Packed families require an accepted target, evidence intervention, and exactly one question intervention; matched views are generation targets, while crossed pairs score dependency-specific margins.The appendix frames these validated interventions as training signals for evidence-to-step, question-to-trace, and trace-to-answer dependencies.Source: §B.6 · p. 28, C EDGE CLOSURE LEARNING: ADDITIONAL… · p. 29

B.6 PACKING, SPLITTING, AND AUDIT

  • The final corpus contains 2,852 families—1,276 from 2WikiMultiHopQA, 559 from MuSiQue, and 1,017 from HotpotQA—and splits into 2,566 training and 286 development families.Splitting at the family level keeps each source and its derived views in one partition.Source: §B.6 · p. 28
  • After adjudication, 290/300 audited families pass (96.7% unweighted; 97.3% corpus-weighted); a family passes only if all three audited views pass.Source: §B.6 · p. 28, Table 20 · p. 29

C.1 TRAINING VIEWS AND VALIDITY

The appendix describes E-CLOSURE’s original and counterfactual training views, edge-specific scores, and bidirectional margin objective, alongside validation of the evaluator used for experimental outcomes. The high-reasoning-effort evaluator shows high agreement with human judgments, including on derived global sufficiency and LGG labels.

C.1 TRAINING VIEWS AND VALIDITY

  • The three training views pair the original response with the original input, an evidence-edited response with edited evidence, and a target-edited response with the edited question.Evidence interventions supply evidence-to-step and trace-to-answer pairs; question interventions supply question-to-trace pairs.Source: §C.1 · p. 29

C.2 EDGE-SPECIFIC SCORES

  • Edge-specific scores use the first affected step for evidence-to-step, the complete trace for question-to-trace, and the submitted answer for trace-to-answer.Scores average token log probabilities with teacher forcing; the evidence score holds the validated prefix fixed.Source: §C.2 · pp. 29–30

C.3 COMPLETE OBJECTIVE AND DESIGN RATIONALE

  • The objective combines generation anchors with bidirectional margins, requiring preferences to reverse under each intervention; separate trace and answer margins give those spans dedicated supervision.At inference, E-CLOSURE uses the standard question-and-evidence-to-trace-and-answer mapping without auxiliary verifiers, extra model calls, or decoding changes.Source: §C.3 · p. 30

D EXPERIMENTS: ADDITIONAL DETAILS

  • Experimental outcomes use GPT-5.6 Sol with high reasoning effort and frozen guidelines, validated against human adjudications on the same 2,598 diagnostic responses.For closure and completeness judgments, trace claims are treated as premises; evidence facts cannot fill gaps in the trace.Source: §D.1 · p. 31

D.1 EVALUATOR AND VALIDATION

  • Evaluator raw agreement ranges from 97.31% to 99.73% across five directly evaluated criteria, with 99.08% agreement and 0.962 failure F1 for derived global sufficiency.The largest discrepancies are concentrated in local soundness.Source: §D.1 · p. 31, Table 21 · p. 32
  • On human-labeled LGG cases, the high configuration identifies 126 of 145 and reaches 91.97% precision, 86.90% recall, and 0.894 F1.Across configurations, high has the highest mean agreement (99.07%) and mean failure F1 (0.962) across Acc, L, A, C, and K.Source: §D.1 · p. 32, Table 22 · p. 32

D.2 EXPERIMENTAL EVALUATION PROTOCOL

The appendix specifies a shared evaluation and training protocol, defines comparison objectives and ablations, and describes how checkpoints and aggregate results are produced.

D.2 EXPERIMENTAL EVALUATION PROTOCOL

  • Rates are computed within dataset–backbone–seed settings before equal-weight aggregation, and aggregate main results use paired hierarchical bootstrap confidence intervals.The uncertainty estimates use 5,000 resamples; backbone results use fixed generations without training-seed resampling.Source: §D.2 · p. 33

D.3 COMPARISON OBJECTIVES

  • SFT uses only the original validated response, while CF-SFT adds matched evidence- and question-counterfactual responses as generation anchors.RM retains the three anchors and adds bidirectional preferences over complete responses, providing a comparison with E-CLOSURE’s dependency-specific preferences.Source: §D.3 · p. 34
  • E-CLOSURE adds bidirectional margins on the first evidence-affected step, the question-conditioned trace, and the trace-conditioned answer, alongside generation anchors.The six comparisons allocate one third of total margin weight to each dependency, split equally between its two directions.Source: §D.3 · pp. 34–35
  • Ablations remove one dependency’s margins or retain only original-condition margins, while preserving generation anchors and the total margin coefficient.Removed-edge variants retain the intervention data and anchors associated with that edge.Source: §D.3 · p. 35

D.4 TRAINING IMPLEMENTATION AND COMPUTE

  • All adapted methods share source partitions, sampling, context serialization, and optimization settings; 48 independent LoRA adapters are trained across methods, backbones, and seeds.Checkpoint selection uses weighted development answer accuracy under original and edited inputs.Source: §D.4 · pp. 35–36

D.5.1 EVALUATION SETUP

  • 866 diagnostic questions—278 from 2WikiMultiHopQA, 255 from MuSiQue, and 333 from HotpotQA—are evaluated using one response per checkpoint and question.The main comparison equally weights nine dataset–backbone settings; adapted-method metrics are averaged over three training seeds within each setting.Source: §D.2 · p. 32, §D.5.1 · p. 37

D.5.2 HUMAN AUDIT OF EXPERIMENTAL RESULTS

The appendix audits automated evaluation, checks accuracy under an alternative scoring rule, and reports ablation and dependency-switch analyses. The full objective improves trace reliability over every ablation, while some individual failure comparisons remain less certain.

D.5.2 HUMAN AUDIT OF EXPERIMENTAL RESULTS

  • Automated and human judgments agree on answer correctness for 359/360 audited responses, while the evaluator misses 14 of 35 human local-soundness failures.For closure failures, it identifies 31 of 32 human-labeled cases and flags one additional response.Source: Table 26 · p. 37, §D.5.2 · p. 37
  • Under human labels, E-CLOSURE has the highest TRR point estimate at 84.44% and the lowest LGG estimate at 5.56% (5 responses) in the seed-42 audit.Its TRR gain over CF-SFT is 6.67 points with a 95% CI of [1.11, 12.22]; the comparison with RM remains uncertain.Source: Table 27 · p. 38, §D.5.2 · p. 38

D.5.3 DETERMINISTIC MATCHING ACCURACY

  • E-CLOSURE retains the highest aggregate deterministic matching accuracy at 88.44%, versus 87.16% for RM and 84.96% for CF-SFT.Deterministic matching can reject semantic equivalents or accept incomplete answers, so this scoring rule differs from semantic correctness.Source: Table 29 · p. 39, §D.5.3 · p. 39

D.6.1 ABLATION SETUP

  • Ablations on Qwen3-8B retain the same generation anchors and training protocol while varying which dependency edges or directions receive margin supervision.The total margin coefficient is preserved by scaling retained terms after edge or direction removal.Source: §D.6.1 · p. 40

D.6.2 DETAILED RESULTS

  • The full objective reaches 87.34% pooled TRR and 88.07% pooled deterministic accuracy, exceeding all four ablations on deterministic accuracy.Pooled TRR weights questions equally, unlike the main comparison’s equal dataset weighting.Source: §D.6.2 · p. 40, Table 34 · p. 44

D.7.1 EVALUATION PROTOCOL

  • The evaluation tests switching by changing evidence, question, or supplied trace while holding the other specified inputs fixed; successful switching requires positive margins in both directions.The scores use fixed candidates and do not require response generation or LLM judging.Source: §D.7.1 · pp. 44–45

D.6.3 FAILURE DECOMPOSITION

  • Removing question-to-trace margins raises path deviation from 2.27% to 4.08%, incomplete reasoning from 3.66% to 4.70%, and answer decoupling from 0.50% to 1.62%.For one-sided supervision, the largest increase is incomplete reasoning, from 3.66% to 5.62%.Source: §D.6.3 · p. 40, Table 36 · p. 45

D.6.4 PAIRED COMPARISONS

  • The full objective improves TRR over every ablation, with all four confidence intervals above zero; LGG reduction intervals exclude zero only for w/o T and one-sided supervision.The closure gain over w/o C is 1.15 points, with an interval of [−0.12, 2.35].Source: §D.6.4 · p. 40, Table 37 · p. 45

D.7.3 PER-SETTING RESULTS

E-CLOSURE’s dependency-switching results vary by dataset for question-guided trace changes, while paired comparisons show gains over CF-SFT for trace and answer switching.

D.7.3 PER-SETTING RESULTS

  • E-CLOSURE achieves 100% C-switching across 2,574 source–checkpoint evaluations, while mean T-switching ranges from 59.90% on 2WikiMultiHopQA to 98.69% on HotpotQA.T-switching is 94.64% on MuSiQue; S-switching is near ceiling across fine-tuned methods.Source: §D.7.3 · pp. 45–46, Table 39 · p. 46

D.7.4 PAIRED COMPARISONS

  • E-CLOSURE improves T-switching by 4.54 points and C-switching by 10.06 points over CF-SFT, with both 95% intervals above zero; the S-switch difference is −0.03 points.Perfect C-switching on these development sources does not imply perfect closure in freely generated responses.Source: Table 40 · p. 46, §D.7.4 · p. 46
Loading 2609.32496v1…