Source-linked AI summary
Learning Evidence Sufficiency Boundaries for Selective Answering in Grounded Multi-Hop QA
Haruto Sato, Yuki Tanaka, Ren Nakamura, Aoi Kobayashi, Mei Ito
TL;DR
Grounded multi-hop QA needs systems to abstain when partial evidence is insufficient, answer at the minimal sufficient context, and remain stable with redundant evidence. The paper introduces Evidence Sufficiency Boundary Training, which trains these transitions directly and achieves the strongest boundary localization and lowest overall unsupported-answer rate among tested systems while retaining competitive raw QA utility.
Problem
Grounded multi-hop QA can make unsupported answers appear plausible because partial evidence may identify relevant entities or relations without determining the answer.
Method
Evidence Sufficiency Boundary Training constructs ordered four-level evidence chains and supervises abstention before sufficiency, answering at the minimal sufficient context, stability afterward, and answer recall.
Results
0.807 flip accuracy was highest on the evidence chain, while overall unsupported-answer rate was lowest at 0.095 on external non-answerable evaluation among tested systems.
Takeaways & Limitations
Evidence-boundary supervision provides a concrete way to train and measure when grounded multi-hop QA systems should transition from abstention to answering.
Takeaways & Limitations
The conclusions are strongest for short-answer grounded QA with explicit support facts, using one backbone scale, one formal training seed, and automatic validation procedures.
Abstract
from arXiv · showhide
Grounded question answering systems should answer only when the supplied evidence supports the answer. In multi-hop QA, this requirement is difficult because partial evidence can make an unsupported answer appear plausible. We study selective answering through evidence sufficiency boundaries: for the same question, a model should abstain under unsupported or partially supported context, answer when the context first becomes sufficient, and keep the answer stable when redundant evidence is added. We introduce Evidence Sufficiency Boundary Training, a generation-native training framework that constructs ordered evidence chains and supervises the abstain-to-answer transition directly. The method combines level supervision, a boundary flip margin, post-boundary stability, and answer recall protection. We build evidence chains from HotpotQA, 2WikiMultiHopQA, and MuSiQue, then evaluate models with chain metrics, raw QA utility, and unsupported-answer rates on external non-answerable sets. With Qwen2.5-3B-Instruct and LoRA adaptation, Evidence Sufficiency Boundary Training gives the strongest boundary localization among the tested systems, with flip accuracy of 0.807 compared with 0.781 for a token-level abstention baseline. It also achieves the lowest overall unsupported-answer rate on external non-answerable evaluation, 0.095 compared with 0.101 for the same baseline, while retaining competitive raw QA F1. The results show that grounded selective answering improves when training marks the evidence level where refusal should give way to answering.
1 Introduction
Grounded multi-hop QA requires models to distinguish insufficient evidence from the first context that supports an answer. The paper introduces Evidence Sufficiency Boundary Training to supervise this transition and evaluate boundary behavior alongside answer utility and unsupported answering.
- Partial evidence can identify an entity or relation while leaving a multi-hop answer underdetermined.
- Evidence sufficiency boundaries order contexts from unsupported and partial evidence to minimal sufficiency and redundancy.
- Evidence Sufficiency Boundary Training converts each question family into four-level evidence chains and trains abstention before, answering at, and stability after the boundary.
- The generation-native objective combines pre-boundary abstention, boundary activation, post-boundary stability, and recall protection.
- The evaluation reports boundary-sensitive metrics alongside raw QA and unsupported-answer metrics that standard EM/F1 do not capture.
2 Evidence Sufficiency Boundaries
The paper defines selective answering around the point where an ordered context chain first becomes sufficient for the gold answer. This boundary separates abstention on insufficient contexts from answering on sufficient contexts while testing stability under redundancy.
- A grounded QA system should answer only when the supplied context contains enough evidence; otherwise it should emit a designated abstention response.
- For multi-hop questions, one required fact may warrant abstention, whereas the complete supporting facts should produce the answer.
- The ordered chain uses c0 and c1 for insufficient contexts, c2 for the first sufficient context, and c3 for sufficient context with redundancy.
- Boundary-aware evaluation distinguishes models that answer every level from models that abstain at every level, separating hallucination control from answer utility.
- Metrics: Flip accuracy records whether the first positive answer score occurs at k⋆, while pre-boundary abstention measures abstention on C0 and C1.
- Metrics: Unsupported-answer rates are reported for both answerable raw QA examples and filtered external non-answerable examples.
3 Evidence Chain Construction
The chain-construction pipeline converts grounded multi-hop QA examples into validated four-level evidence chains. It preserves a common family set across training views while varying how evidence levels supervise answering or abstention.
- HotpotQA, 2WikiMultiHopQA, and MuSiQue provide multi-hop questions whose answers depend on support facts embedded in larger contexts.
- The formal training split contains 2,400 question families, with 800 from each benchmark, producing 9,600 chain rows.
- Each family is converted into C0 unsupported or weakly related context, C1 partial support, C2 minimal sufficient context, and C3 sufficient context with redundancy.
- Programmatic filtering rejects chains when pre-boundary contexts leak the answer, C2 fails to support it, or redundancy changes it.
- The same chain families support answer-only, token-level abstention, and question-level refusal training views with different level labels.
4 Evidence Sufficiency Boundary Training
Evidence Sufficiency Boundary Training uses one generation path to learn when to abstain or answer across ordered evidence contexts. Its objective combines direct level targets with losses that localize the threshold at minimal sufficiency and preserve answer availability afterward.
- Model Interface: The model receives an instruction, question, and context, then generates either a short answer or <ABSTAIN> through one generation path.This avoids a separate inference-time classifier.
- Model Interface: Prefix scoring compares answer-token likelihood with the abstention token at the generation boundary, optionally extending beyond the first answer token.The simplest setting uses p = 1.
- Level Supervision: Level supervision assigns direct generation targets to insufficient and sufficient contexts, but cross-entropy alone may leave weak margins or unstable adjacent-level scores.The objective trains the output string for each context.
- Boundary Losses: Boundary losses keep insufficient contexts below the answer threshold, make the C1-to-C2 transition explicit, and prevent answer-score collapse after redundant evidence.The full objective combines level, pre-boundary, and flip losses, with additional stability and recall-protection terms described in the training framework.
- Boundary Losses: The objective is local to each evidence chain: place the answer threshold at minimal sufficiency and keep the same answer available afterward.Training views derive from matched source families, differing in supervision format.
- Training Design: Evidence Sufficiency Boundary Training uses the evidence chain as its supervision unit, unlike answer-only, question-level, and token-level alternatives.Its score compares answer activation with abstention within the chain.
5 Experimental Setup
The experiments compare four supervision approaches on matched multi-hop evidence-chain families. Evaluation includes chain behavior, ordinary answer utility, and grounding reliability on held-out non-answerable contexts.
- Implementation: All experiments use Qwen2.5-3B-Instruct with LoRA adaptation as the backbone configuration.Training uses the Hugging Face Transformers and PEFT implementations.
- Baselines: Answer-only training receives sufficient contexts and tests whether ordinary QA supervision can learn selective behavior without negative evidence levels.Its training data contains only C2 and C3 contexts.
- Baselines: Token-level abstention learning maps insufficient rows to a rejection token and sufficient rows to the gold answer, using rejection probability during decoding.The baseline follows SEAL’s core design and uses the same source families.
- Baselines: Question-level refusal tuning trains uncertainty and refusal for unknown questions while pairing the refusal branch with same-family C2/C3 QA examples.The positive answer budget is matched across the comparison.
- Metrics: The evaluation reports chain metrics for evidence-boundary learning, raw QA metrics for answer utility, and unsupported-answer rates for grounding reliability.External evaluation uses filtered non-answerable examples held out from training chains.
6 Main Results
Evidence Sufficiency Boundary Training achieves the strongest reported flip accuracy and lowest overall unsupported-answer rate, while the token-level baseline remains stronger on some activation, stability, and raw-QA measures.
- Boundary Behavior: 0.807 flip accuracy is the highest reported result for Evidence Sufficiency Boundary Training, which also reaches 0.934 pre-boundary abstention rate.The token-level baseline exceeds it on boundary activation and post-boundary stability.
- Boundary Behavior: Evidence Sufficiency Boundary Training improves family-level transition measurement by tying negative and positive contexts within the same evidence chain.Answer-only training answers almost everywhere, while question-level refusal fails to activate reliably at C2.
- Raw QA Utility: 0.584 F1 with unsupported-answer rate 0.083 shows competitive raw QA utility for Evidence Sufficiency Boundary Training, although the token-level baseline reaches 0.590 F1 and 0.081 answerable unsupported rate.Answer-only training has the highest EM and F1 but an unsupported-answer rate of 0.248.
- Raw QA Utility: The raw-QA advantage does not belong to Evidence Sufficiency Boundary Training: its strength appears instead in boundary-sensitive metrics and external non-answerable suppression.The raw-QA metric serves as a guardrail against winning by refusing answerable questions.
- External Non-Answerable Evaluation: 0.095 overall unsupported-answer rate is the lowest on external non-answerable evaluation, versus 0.101 for the token-level abstention baseline.Evidence Sufficiency Boundary Training is better on HotpotQA and MuSiQue, while the token-level baseline is better on 2WikiMultiHopQA.
- External Non-Answerable Evaluation: Contexts with entity overlap and missing bridge evidence remain difficult, especially in the 2WikiMultiHopQA result.The external result links stronger flip behavior with lower unsupported answering on held-out negative contexts.
7 Analysis
The analysis shows that boundary-sensitive evaluation reveals failure modes hidden by ordinary answerable QA results. Evidence Sufficiency Boundary Training localizes the abstain-to-answer transition well, but remains weaker on post-boundary stability and raw QA F1.
- What the Boundary Metrics Add: Answer-only training ranks first on raw QA but answers before evidence becomes sufficient, a failure exposed by chain metrics.External reliability results would rule out answer-only training, showing why ordinary answerable validation is insufficient.
- What the Boundary Metrics Add: 0.570 boundary activation and 0.456 post-boundary stability mark the question-level refusal baseline’s weak transition into answering.Its sample-level refusal supervision is too coarse for a local evidence transition in this setting.
- What the Boundary Metrics Add: The token-level abstention baseline is the strongest comparison, with raw QA slightly above Evidence Sufficiency Boundary Training and stronger activation and stability.Its rejection token participates directly in token prediction, and decoding uses rejection probability during search.
- Where the Method Still Loses: Post-boundary stability is the largest gap because redundant evidence can reduce answer confidence after minimal sufficient context.The current penalty for score drops from C2 to C3 is weaker than direct answer-generation pressure at every sufficient level.
- Where the Method Still Loses: The method also trails answer-only training and the token-level abstention baseline on raw QA F1 despite recall protection recovering much of answer utility.Evidence Sufficiency Boundary Training spends training signal on insufficient evidence, creating a deliberate utility trade-off.
- What the Boundary Metrics Add: The boundary task is relevant to deployed retrieval settings because systems receive partial, redundant, or distractor-heavy contexts.Evidence Sufficiency Boundary Training provides a controllable way to train and test behavior before, at, and after the sufficiency boundary.
8 Related Work
Related work spans grounded multi-hop QA, retrieval-augmented generation, faithfulness evaluation, selective prediction, and evidence allocation. This paper’s distinction is to make evidence level part of the evaluation instance.
- Grounded QA and multi-hop reasoning: Multi-hop QA benchmarks require compositional evidence across multiple facts or documents, covering bridge, comparison, and inference-style questions.The cited benchmarks include HotpotQA, 2WikiMultiHopQA, and MuSiQue.
- Retrieval-augmented generation: Retrieval-augmented systems bring non-parametric evidence into generation, but retrieved context can remain insufficient or distracting.Prior systems improve retrieval timing, fusion, and attribution without eliminating evidence-control problems.
- Hallucination and faithfulness evaluation: Faithfulness research measures factual consistency and unsupported claims through human annotation, automated judging, or model self-consistency.This paper follows that direction while making evidence level part of the test instance.
- Abstention, refusal, and calibration: Selective prediction and calibration provide established tools for rejecting uncertain inputs and estimating prediction reliability.QA work also studies unanswerable questions and selective answering under distribution shift.
- Evidence allocation in neighboring work: Neighboring work studies how models allocate supervision, memory, and evidence across contexts.Examples include effective context lengths, associative memory graphs, and retrieval-conditioned fusion weights.
9 Limitations
The study’s conclusions are scoped by a single backbone scale and seed, three multi-hop QA benchmarks, automatic evaluation, and weaker post-boundary stability and raw QA F1 than the token-level baseline.
- Scope and evaluation: Experiments use one backbone scale, Qwen2.5-3B-Instruct, and one formal training seed.Larger models may place the sufficiency boundary differently.
- Scope and evaluation: Evidence chains come from three multi-hop QA benchmarks, so conclusions are strongest for short-answer grounded QA with explicit support facts.The validation and unsupported-answer evaluation also use automatic procedures; human verification would strengthen reliability claims.
- Remaining performance gaps: The method lags the token-level abstention baseline on post-boundary stability and raw QA F1.The proposed next step is more direct supervision of answer identity across sufficient contexts while retaining pre-boundary abstention signals.
10 Ethics Statement
The ethics statement notes that abstention can suppress useful answers when evidence is sufficient and should be presented as evidence-based refusal rather than proof that no answer exists.
- Ethical considerations: Abstention can suppress useful answers when evidence is sufficient, so evaluations should report false-abstention rates alongside unsupported-answer rates.The experiments use public QA benchmarks and derived contexts.
- Ethical considerations: Generated abstention should be presented to users as an evidence-based refusal, not as proof that no answer exists in the world.This distinction limits how users should interpret the system’s refusal behavior.
AI Assistance Statement
The draft states that AI-assisted editing was used, while assigning responsibility for the research content and final wording to the authors.
- AI-assisted editing was used in preparing the draft.
- The authors remain responsible for the research claims and experimental results.
- The authors also remain responsible for the citations and final wording.