Source-linked AI summary
EVAR: Evidence-Validated Hypothesis Admission for Budget-Aware Narrative Reasoning
Peilin Liu, Zhiquan Ji, Jinglong Ping
TL;DR
Non-interactive long-form narratives challenge LLMs because decisive evidence is dispersed and unsupported intermediate hypotheses can contaminate later reasoning. EVAR compiles source-linked evidence, budgets refinement, and validates hypotheses before admission; experiments report improved task performance and evidence grounding with controllable inference cost.
Problem
Non-interactive long-form narratives disperse decisive premises across events, creating risks of premature commitment and unsupported intermediate reasoning.
Method
EVAR builds an immutable, source-linked atomic evidence store, routes instances using inferred difficulty, and gates hypothesis admission through validation challenges.
Results
Experiments on NarraCrime and three public benchmarks show improved task performance and evidence grounding with controllable inference cost.
Takeaways & Limitations
EVAR frames evidence-grounded narrative reasoning as admitting only hypotheses explicitly supported by source evidence before answer synthesis.
Takeaways & Limitations
EVAR requires multiple LLM calls, depends on extraction and verification quality, and is evaluated mainly on English benchmarks using fixed-judge automatic diagnostics.
Abstract
from arXiv · showhide
Large language models (LLMs) often produce fluent but weakly grounded conclusions when reasoning over non-interactive, long-form narratives. A central failure mode is that unsupported intermediate hypotheses can enter the reasoning trajectory and contaminate subsequent inference, especially when evidence is scattered across distant parts of the story. To address this problem, we propose EVAR, an evidence-validated hypothesis admission framework for budget-aware narrative reasoning. EVAR first compiles the narrative into an immutable evidence store of source-linked atomic claims and assigns an instance-specific inference budget from unresolved gaps and uncertainty signals. During refinement, EVAR directly proposes candidate hypotheses for unresolved gaps, constructs hypothesis-conditioned validation challenges, and verifies each candidate against the locked store before admission: supported hypotheses enter the answer-supporting state, unverifiable ones are quarantined, and contradictory ones are discarded. A sufficiency-based stopping mechanism further avoids unnecessary refinement. Experiments on NarraCrime and multiple public reasoning benchmarks show that EVAR improves both task performance and evidence faithfulness while maintaining controllable inference cost.
1 Introduction
EVAR addresses weakly grounded reasoning in non-interactive long-form narratives by preserving source-linked evidence, allocating instance-specific refinement budgets, and validating hypotheses before admission. It evaluates task quality and evidence faithfulness across NarraCrime and public benchmarks.
- Motivation: Non-interactive narratives scatter decisive premises across disjointed events, exposing premature commitment and unsupported reasoning that conventional benchmarks may obscure.Models can form early judgments from incomplete premises and rationalize them into coherent but unfaithful explanations.
- Motivation: Uniform self-revision adds inference cost and may fail to improve or degrade accuracy when reliable feedback is unavailable.EVAR responds by routing easy cases directly and sending hard cases into verifier-gated refinement.
- Method: EVAR compiles narratives into an immutable store of source-linked atomic claims, then assigns instance-specific inference budgets from unresolved gaps and uncertainty signals.The framework treats evidence storage and budget-aware routing as prerequisites for refinement.
- Method: During refinement, EVAR proposes hypotheses for unresolved gaps and admits them only after hypothesis-conditioned validation against the evidence store.Supported hypotheses enter the answer-supporting state; unverifiable ones are quarantined and contradictions discarded, with stopping once evidence suffices.
- Evaluation: NarraCrime contains 300 cases across Easy, Medium, and Complex splits, while evaluation covers task content recovery and atomic-claim faithfulness beyond final-verdict quality.The benchmark varies story length, evidence density, and suspect complexity.
2 Related Work
Prior approaches expand or revise reasoning through search, critique, decomposition, or external retrieval, but these strategies do not ensure faithful intermediate inference. EVAR instead validates candidate hypotheses against locked narrative evidence under instance-specific budgets.
- Reasoning expansion: Chain-of-Thought, Self-Consistency, and Tree-of-Thoughts expand reasoning space, but larger reasoning spaces do not guarantee faithful inference.CoT may rationalize biased answers, while self-correction without reliable feedback may fail or degrade performance.
- Revision and attribution: Self-Refine and Reflexion use iterative revision, yet revision without reliable external feedback does not consistently correct factual errors and requires additional inference.RARR improves attribution by retrieving external sources, unlike EVAR’s closed-world setting.
- Structured reasoning: Structured frameworks use critique, decomposition, strategy discovery, or graph-based state expansion, while unsupported intermediate conclusions can propagate through long-form reasoning.Long narratives still require integrating dispersed information from long contexts.
- Positioning: SABA assesses premise sufficiency before synthesis, whereas EVAR centers evidence-validated admission under instance-specific budgets.EVAR retains source-linked atomic evidence and validates hypotheses before each state update.
- Positioning: Prior abductive, narrative, and social-deduction studies mainly provide testing grounds, while EVAR targets reliable, budget-aware reasoning over non-interactive narratives.The related work distinguishes EVAR’s reasoning objective from the use of these tasks as evaluation settings.
3 Method
EVAR frames closed-world long-form narrative reasoning as evidence-first hypothesis admission: it compiles source-grounded claims, routes instances by difficulty, and permits refinement only when hypotheses pass validation. Final synthesis uses the locked evidence store and admitted hypotheses, with unsupported content excluded and refinement stopping when evidence is sufficient.
- Task Definition: EVAR treats fixed-narrative reasoning as inferring a conclusion from dispersed information without follow-up questions or external retrieval.The task is closed-world and requires integrating information across long contexts.
- Evidence Store Construction: EVAR atomizes the narrative into source-grounded claims, preserves provenance, attaches localized consistency metadata, and stores the resulting evidence immutably.Claims must be explicitly expressed in the narrative and retain valid source identifiers; inferred motives and causal bridges are excluded.
- Budget-aware Routing: EVAR estimates instance difficulty from unresolved gaps, uncertain or conflicting evidence, and issue severity, then converts that score into a capped refinement budget.The assigned budget determines whether an instance bypasses refinement or enters budgeted refinement.
- Hypothesis Admission: During refinement, EVAR re-identifies remaining gaps, proposes candidate hypotheses, and creates validation challenges without treating candidates or challenges as answer evidence.Challenges request supporting evidence, counterevidence, and indispensable unsupported premises, while candidates remain outside the answer-supporting state until verified.
- Hypothesis Admission: Only supported hypotheses with evidence links are admitted; unverifiable hypotheses are quarantined, contradictions are discarded, and the answer-supporting state excludes blocked material.Final synthesis is restricted to the locked evidence store and admitted hypotheses, and refinement stops when no blocking gap remains or the state is sufficient.
4 Experiments
Experiments evaluate EVAR across NarraCrime and public reasoning benchmarks using task-quality, evidence-faithfulness, and inference-cost measures. EVAR improves performance across benchmarks while keeping inference cost controllable.
- Datasets: Evaluation covers NarraCrime, HotpotQA, StrategyQA, and BBH, with NarraCrime results reported across Easy, Medium, and Complex splits.NarraCrime contains difficulty-controlled cases spanning increasing story length, evidence density, and cross-event reasoning complexity.
- Baselines and protocol: EVAR is compared with direct answering, chain-of-thought, self-feedback, sampling, structured reflection, and trajectory-search baselines under a shared backbone and protocol.The main experiments use DeepSeek-V3.2, with all compared methods receiving the same input and following a unified protocol.
- Metrics: NarraCrime evaluation combines verdict and content-recovery metrics with atomic-claim faithfulness diagnostics.The task metrics are RVS, IR, ASR, and EC; the faithfulness diagnostics are UCR and CR.
- Main results: 78.6 NC-Complex RVS and 83.3 EC are achieved by EVAR, versus 69.6 and 77.6 for GoT, respectively.On NarraCrime-Complex, the reported comparison also gives EVAR 78.6 RVS versus 68.0 for SELF-DISC., and 83.3 EC versus 76.3.
- Main results: 78.2 HQA Ans and 74.1 SF are achieved by EVAR versus 76.6 and 72.7 for GoT, while NarraCrime-Complex cost falls from T = 35.5 to T = 15.8.The reported cost reduction is 55.5%.
5 Further Analysis
Further analyses show that EVAR’s performance and faithfulness depend on combining budget routing, evidence storage, and verifier-gated admission. Budget sweeps and early-stopping comparisons indicate controllable cost with diminishing returns from additional refinement.
- Ablations: Removing budget routing causes the largest ablation performance drop, while removing admission increases unsupported claims and removing the evidence store weakens grounding.The results support the combined role of locked evidence, verifier-gated admission, and budget-aware control.
- Evidence faithfulness: EVAR substantially reduces unsupported claim rate and contradiction rate versus strong reasoning baselines on NarraCrime-Complex.The pattern is consistent with filtering unverifiable and contradictory hypotheses before they enter the active reasoning state.
- Cost–utility trade-off: +7.8 NarraCrime-Complex RVS, +9.6 EC, and +12.3 StrategyQA accuracy result when Bmax increases from 0 to 1.Increasing Bmax from 3 to 4 yields only +1.3, +0.6, and +0.5, showing diminishing returns at higher budgets.
- Cost–utility trade-off: EVAR’s low-to-medium budget regime provides the main gains, while larger budgets offer limited additional benefit; Bmax = 4 is described as a reasonable performance–cost trade-off.Figure 2 reports NC-Complex RVS, NC-Complex EC, StrategyQA accuracy, and cost T across budget caps.
- Early-stopping effectiveness: EVAR with early stopping achieves stronger performance than fixed-K and no-early-stopping variants while avoiding unnecessary refinement cost.The no-early-stopping variant has higher normalized iteration count and cost but slightly worse performance; Fixed-K is more efficient but still underperforms full EVAR.
- Robustness: Routing assigns iterative refinement more frequently to harder instances, and EVAR remains effective across additional backbone models.These robustness analyses are reported in Appendix B.
6 Conclusion
EVAR grounds long-form narrative reasoning in source-linked atomic evidence and validates hypotheses before they affect answer synthesis. Experiments report improved task performance and evidence grounding with controllable inference cost.
- EVAR is a budget-aware test-time framework that grounds inference in a source-linked atomic evidence store and gates hypothesis admission through validation challenges.
- Experiments on NarraCrime and three public benchmarks show improved task performance and evidence grounding with controllable inference cost.
Limitations
EVAR has higher inference cost than single-pass prompting and depends on automated evaluation and evidence-processing quality. Its experiments focus mainly on English benchmarks.
- EVAR requires multiple LLM calls and is costlier than single-pass prompting.
- Performance depends on evidence extraction and verification quality.
- Automatic metrics rely on a GPT-5.5 judge, so scores should be interpreted as fixed-judge diagnostics rather than human-verified assessments.
- Experiments focus mainly on English benchmarks, leaving multilingual, open-ended, and interactive settings for future work.
A Full EVAR Procedure
EVAR assigns refinement selectively from unresolved gaps and uncertainty, then admits only evidence-supported hypotheses into the answer-supporting state. Verification, sufficiency, and budget limits control the refinement process.
- A Full EVAR Procedure: Candidate hypotheses are generated for unresolved gaps but cannot influence answer synthesis until explicitly verified.The procedure separates candidate generation from state update to prevent unsupported statements from being reused as narrative evidence.
- A Full EVAR Procedure: EVAR computes an instance-specific refinement budget from unresolved gaps, uncertainty, and conflict signals in the immutable evidence store.K = 0 routes an instance directly to answer synthesis; K > 0 permits up to K refinement iterations.
- A Full EVAR Procedure: Three validation challenges test each candidate for direct support, conflicting evidence, and indispensable unsupported prerequisites.
- A Full EVAR Procedure: Supported hypotheses enter the state with evidence links, while unknown hypotheses are quarantined and contradictory hypotheses are discarded.
- A Full EVAR Procedure: Refinement stops when no blocking gap remains, state sufficiency reaches τSUF, or the assigned budget is exhausted.Verification determines whether an individual hypothesis enters the state; sufficiency determines whether the admitted information is collectively adequate.
- A Full EVAR Procedure: The final answer can use original evidence and admitted hypotheses with evidence links, but not quarantined or contradictory hypotheses or validation challenges.
- A Full EVAR Procedure: The locked evidence store remains unchanged while later iterations may build on admitted hypotheses, creating a one-way evidential flow into the answer-supporting state.
B Additional Analysis
The additional analyses examine whether EVAR's complexity-aware routing matches observable difficulty and whether its gains persist across backbone families. Together, they distinguish the proposed mechanism from uniform extra computation or backbone-specific effects.
- B Additional Analysis: Routing effectiveness tests whether complexity-aware control activates refinement more often for instances with stronger observable difficulty signals.
- B Additional Analysis: Backbone robustness tests whether evidence-validated hypothesis admission gains persist across model families.
- B Additional Analysis: These analyses distinguish EVAR's control mechanism from uniformly adding computation or benefiting only from a particular model.
B.1 Routing Effectiveness
EVAR routes instances selectively, using diagnostic signals to reserve iterative refinement for cases likely to require stronger evidence integration. This preserves a fast path for simpler cases while making routing responsive to inferred difficulty.
- Routing motivation: EVAR assigns refinement budgets from unresolved evidence gaps and uncertainty signals rather than applying uniform reasoning depth.Uniform self-revision can add cost without improving accuracy, while uniformly small budgets may under-refine difficult cases.
- Metric interpretation: Iterative-routing rates measure whether at least one refinement iteration was warranted, not realized iteration counts or inference cost.Actual computational cost is measured by average call count T, because routed instances can receive different budgets and stopping times.
- Routing behavior: 33% versus 68% of HotpotQA examples were iteratively routed for exactly two versus more than two supporting facts, a 35-percentage-point difference.The groups use supporting-fact counts as a diagnostic proxy for evidence-chain breadth.
- Routing behavior: 26%, 45%, and 71% of StrategyQA examples were iteratively routed in the shortest, middle, and longest question-length groups, respectively.Question length is treated as a lightweight, reproducible proxy for information and compositional structure.
- Metric interpretation: The routing analysis is a sanity check using diagnostic proxies, not evidence that those proxies causally explain every individual routing decision.The groups are not ground-truth difficulty labels, and routing rate is not itself a performance score.
B.2 Backbone Robustness
Across four backbone models, EVAR maintains higher verdict quality and evidence coverage, lower unsupported-claim rates, and fewer calls than GoT. It is not the absolute lowest-cost method, since CoT and Self-Refine use fewer calls while achieving weaker quality and faithfulness.
- Cross-backbone results: 76.9–82.4 RVS, 82.1–86.0 EC, and 7.1–9.4 UCR were achieved by EVAR across four backbones, with the best value on each metric.The comparison includes Qwen3-235B, GPT-5.5, Gemini-2.5, and Claude-4.5 on NarraCrime-Complex, alongside the default backbone evaluation.
- Cross-backbone results: EVAR improves RVS by 8.1–9.8 points and EC by 5.2–6.5 points relative to GoT across the tested backbones.These gains are accompanied by lower UCR rather than increased unsupported content.
- Faithfulness and efficiency: EVAR uses 14.9–16.4 calls per instance versus GoT’s 33.9–36.1, corresponding to approximately 55% fewer calls.Selective verifier-gated refinement is therefore more efficient than expanding GoT’s larger reasoning structure in this comparison.
- Faithfulness and efficiency: EVAR reduces UCR by 6.7–7.0 points relative to GoT while retaining its RVS and EC advantages.The persistent UCR reduction indicates that the quality gains are not obtained by trading faithfulness for more aggressive generation.
- Cost–quality trade-off: CoT and Self-Refine require fewer calls than EVAR but have substantially lower RVS and EC and higher UCR.EVAR occupies a higher-cost, higher-quality operating point rather than minimizing absolute inference cost.
- Cost–quality trade-off: EVAR’s gains persist across model families, although evidence extraction and verification still rely on backbone capability.The cross-backbone consistency reduces the likelihood of a model-specific prompt artifact but does not establish independence from backbone quality.
C Details of Semantic Matching and Faithfulness Metrics
The evaluation separates recovery of required content from evidence faithfulness, using semantic proposition matching for coverage and atomic-claim labels for support status. EVAR’s inference procedure combines evidence-store construction, budgeted routing, and verifier-gated refinement before final answer synthesis.
- Evaluation dimensions: Coverage measures whether outputs recover annotated intents, action schemas, and supporting evidence, while faithfulness measures unsupported or contradictory claims.These properties are evaluated separately because an answer can cover reference evidence while adding speculation, or avoid speculation while omitting required content.
- Semantic matching: Predicted and reference propositions are encoded with all-mpnet-base-v2, and cosine similarity with δ = 0.8 identifies candidate semantic matches.Semantic matching accommodates equivalent propositions expressed with different wording rather than requiring exact strings.
- Semantic matching: Matching is performed within each content type and uses the same encoder, threshold, and one-to-one rule across methods; recall is micro-averaged over instances.IR, ASR, and EC respectively evaluate intent, action-schema, and supporting-evidence proposition recovery.
- Faithfulness metrics: Outputs are decomposed into atomic claims, each labeled Support, Unknown, or Contradict against gold evidence annotations.Atomic decomposition separates multiple factual commitments within a single sentence so their evidential status can be judged independently.
- Faithfulness metrics: UCR counts Unknown or Contradict claims, whereas CR isolates Contradict claims and necessarily satisfies CR ≤ UCR.UCR − CR represents claims that are unverifiable but not contradicted.
- Inference procedure: EVAR compiles a stable evidence store, computes complexity and refinement budgets, routes instances, and validates hypotheses before state updates.The inference procedure returns a textual answer and, for NarraCrime, a normalized verdict distribution; supported hypotheses may enter the answer-supporting state while unverified ones do not.