Source-linked AI summary
Guaranteeing Faithful Evidence Extraction in Speculative Retrieval-Augmented Generation
Quentin Signé, Mohand Boughanem, Jose Moreno, Thiziri Belkacem
TL;DR
Safety-critical QA requires answers that faithfully reproduce certified documentation, but existing retrieval and hybrid methods do not guarantee that quoted spans are verbatim. CHyD constrains extraction-mode decoding to continuous spans from retrieved context, achieving near-perfect extraction correctness across diverse benchmarks with a trade-off in fluency-oriented metrics.
Problem
Existing RAG and hybrid QA approaches do not guarantee that quoted or extracted spans match retrieved evidence verbatim, a requirement in safety-critical documentation use.
Source: §1 · p. 1Method
Results
CHyD achieves near-perfect extraction correctness across diverse QA benchmarks, with modest reductions in fluency-oriented metrics.
Source: §1 · p. 2, §5.3 · p. 9Takeaways & Limitations
The authors conclude that prioritizing evidence correctness over fluency and latency makes hybrid generation suitable for safety-critical domains such as aircraft maintenance.
Source: §7 · p. 11CHyD guarantees extractive faithfulness within brackets, but the base LLM still decides which span to extract, leaving the query-to-span reasoning link unresolved.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Large Language Models (LLMs) are increasingly used as interfaces for information retrieval, but they remain prone to hallucinations and faithfulness errors, in which the generated answers diverge from the retrieved evidence. While Retrieval-Augmented Generation (RAG) and recent hybrid or semi-extractive approaches mitigate this issue, they do not guarantee that quoted or extracted spans are verbatim from the retrieved context. This limitation can have severe consequences in safety-critical domains, where answers must exactly match certified documentation. We introduce Constrained Hybrid Decoding (CHyD), a novel faithfulness-first paradigm for speculative RAG. While traditional speculative decoding is optimized for inference speed, CHyD repurposes this architecture to ensure faithful verbatim evidence extraction when the extraction mode is correctly triggered. Our approach enforces hard decoding constraints that restrict generation to continuous spans present in the retrieved documents. This design provides a robust but straightforward guarantee: any explicitly quoted span in the output appears verbatim in the provided context. We evaluate our method across state-of-the-art LLMs on diverse abstractive, extractive, and semi-extractive QA benchmarks, including technical datasets motivated by aircraft maintenance. Results show that existing hybrid methods frequently hallucinate quoted spans, with exact extraction accuracy dropping below 40% in technical domains. In contrast, our approach achieves near-perfect extraction faithfulness regardless of the model used. Although enforcing hard constraints introduces a trade-off with fluency-oriented metrics, our method improves exact answer correctness and remains competitive overall, highlighting its suitability for safety-critical information retrieval applications.
1 Introduction
In safety-critical QA, answers must be correct and verbatim to support compliance with certified documentation, yet existing RAG and hybrid approaches do not guarantee exact extraction. CHyD constrains extraction decoding to continuous verbatim spans and reports near-perfect extraction correctness across diverse benchmarks.
1 Introduction
- In safety-critical settings such as aircraft maintenance, answers must reproduce certified procedures, thresholds, or conditions because paraphrasing can invalidate compliance.The paper also describes exactness and faithfulness to retrieved context as requirements for decision-support QA.Source: §1 · p. 1
- Existing RAG, hybrid, and semi-extractive approaches can improve factuality but do not guarantee verbatim extraction from retrieved context.The paper identifies speculative RAG and semi-extractive methods as approaches that may still produce hallucinated quoted spans.Source: §1 · pp. 1–2
- CHyD restricts extraction-mode decoding to continuous verbatim spans from retrieved documents, guaranteeing that quoted spans appear exactly in context.The guarantee applies when the model enters extraction mode.Source: §1 · p. 2
- The paper contributes CHyD, an evaluation against prior speculative and semi-extractive methods, and SEMAeroSQuAD, a dataset tailored to safety-critical semi-extractive QA.The reported evaluation finds near-perfect extraction correctness across diverse benchmarks, with modest reductions in fluency-oriented metrics.Source: §1 · p. 2
2 Related Work
Prior approaches address faithfulness through soft probabilistic preferences, pure extraction, or hybrid copying, but each leaves a guarantee or capability gap. CHyD uses constrained decoding to make verbatim evidence extraction the objective of hybrid generation.
2 Related Work
- RAG and logit-based methods can reduce context inconsistencies, but probabilistic generation or weighting does not guarantee verbatim evidence.Pointer-generator copying is also described as a soft preference rather than a hard guarantee.Source: §2 · p. 2
- Purely extractive methods provide strong copying guarantees but lack the fluency and synthesis needed for complex tasks such as multi-source QA.The cited examples include encoder-only QA, SEBRAG, and span prediction by Mallick et al. [20].Source: §2 · p. 2
- Hybrid and semi-extractive approaches combine generated text with quoted spans, but freely generated quotations can be absent from retrieved context, as in SEMQA [25].Cheng et al. [2] combine extractive and abstractive readers; SEMQA [25] introduces a semi-extractive format and QuoteSum.Source: §2 · p. 2
- Speculative RAG systems such as CoG [14], REST [8], CopySpec, and NEST [17] use copying primarily to accelerate inference, not to guarantee correctness.CHyD instead treats constrained copying as a correctness mechanism.Source: §2 · p. 3
- CHyD addresses the hybrid-framework gap by switching from free generation to extraction mode, where hard constraints allow only valid continuous context spans and the closing trigger.The figure describes “[” as the extraction trigger; the method assigns disallowed tokens −∞ and permits “]” to resume free generation.Source: §2 · p. 3, Figure 1 · p. 3
3 Approach
CHyD alternates free generation with an extraction mode that permits only token continuations matching a continuous span in retrieved context. A decoding mask enforces that constraint while allowing extraction to end and free generation to resume.
3 Approach
- CHyD combines free-form generation with extraction mode, enforcing extraction correctness at decoding time rather than relying on post-hoc validation or soft preferences.The approach is intended for settings where faithfulness guarantees are necessary.Source: §3 · p. 3
- The model switches between Standard and EEC modes; “[” enters extraction, while “]” returns generation to Standard mode.The transition function updates the mode based on the previous mode and generated token.Source: §3 · p. 4
- The token-selection function f returns next tokens that extend the generated prefix as an exact continuous match in at least one retrieved passage.This defines the valid-token set used during extraction.Source: §3 · p. 4
- During EEC, a constraint mask preserves logits for valid continuations and the closing token while assigning disallowed tokens zero sampling probability.The resulting distribution allows either continued extraction or an exit to free generation.Source: §3 · p. 4
- CHyD can perform multiple extractions across retrieved passages; if no valid next token remains, it exits extraction mode with “]”.EOS generation is disallowed during EEC so quoted spans close before termination.Source: §3 · p. 4
4 Experimental Setup
The experiments compare CHyD across abstractive, extractive, and semi-extractive QA settings, using separate measures for answer quality, correctness, and verbatim evidence faithfulness. The setup also introduces a technical semi-extractive test set and evaluates decoding under controlled contexts and retrieval noise.
4.1 Datasets
- The evaluation spans abstractive MESAQA, extractive MedMCQA, and semi-extractive QuoteSum, alongside the newly introduced SEMAeroSQuAD technical test set.SEMAeroSQuAD targets whether evidence is reproduced verbatim while fluent surrounding content is generated.Source: §4.1 · pp. 4–5
- SEMAeroSQuAD preserves SQuAD 2.0 gold spans inside LLM-generated surrounding text, creating a controlled closed-domain extraction test with a possible surrounding-text bias.The dataset uses aeronautical and space-related questions; its quoted spans remain unchanged from SQuAD 2.0.Source: §4.1 · p. 5
4.2 Baselines
- The baselines are SEMQA for semi-extractive QA, NEST for speculative RAG, and DistilBERT-SQuAD as a purely extractive reference.NEST pastes retrieved spans for inference speed; DistilBERT predicts answer token indices rather than generating text probabilistically.Source: §4.2 · pp. 5–6
4.3 Evaluation Metrics
- EFA measures whether quoted spans occur continuously in retrieved contexts, complementing correctness metrics and answer-quality scores that do not test verbatimness.EM, F1, and ALR assess extraction correctness; ROUGE-L, BERTScore, MAUVE, and SEMQA assess answer quality.Source: §4.3 · p. 6
- Binary EM and ALR results are tested with McNemar, continuous metrics with Wilcoxon signed-rank, and raw p-values receive Holm-Bonferroni correction.Generation parameters are held consistent across methods within each dataset, and generative-model results average two independent runs.Source: §4.3 · p. 6, §4.4 · p. 7
4.4 Implementation Details
- Three few-shot LLMs are evaluated, with prompts specifying context-based answers, abstention when context is insufficient, and bracketed evidence for CHyD and SEMQA.NEST receives context-only answer instructions without explicit citation formatting.Source: §4.4 · p. 6
- Baseline datasets use gold contexts to isolate decoding, while SEMAeroSQuAD also tests a gold context accompanied by four distractor passages.The technical dataset includes both single-context and multi-context settings.Source: §4.4 · p. 6
- CHyD enters EEC mode on “[”, follows valid suffixes from a tree of tokenized retrieved contexts, and can exit to resume free generation.Tokens such as “]”, “].”, and “],” are accepted as exit tokens.Source: §4.4 · pp. 6–7
5 Results
CHyD’s hard constraints deliver near-perfect verbatim extraction while preserving competitive answer quality and answerability; the results also expose trade-offs in fluency and latency.
5 Results
- Overall averages cover MedMCQA, MESAQA, QuoteSum, and single-context SEMAeroSQuAD, excluding multi-context SEMAeroSQuAD to avoid double-weighting the dataset.The evaluation compares CHyD with NEST and SEMQA across extraction faithfulness, answer quality and fluency, and fluency-faithfulness trade-offs.Source: §5 · p. 7
5.1 Extraction Correctness and Faithfulness
- CHyD reaches 0.981–1.000 EFA across models and datasets, while SEMQA falls below 0.371 on technical MESAQA.On MedMCQA with Mistral-N, CHyD achieves EM 0.415 versus SEMQA’s 0.309 and NEST’s 0.136; DistilBERT averages 0.994 EFA but has EM 0.022 on MedMCQA and 0.012 on MESAQA.Source: §5.1 · pp. 7–8, §5.3 · p. 9
5.2 Answer Quality and Fluency
- CHyD stays near SEMQA in overall Llama-3.1 ROUGE-L (difference 0.002), but MESAQA latency is 6.014–6.583 seconds versus SEMQA’s 1.592–2.470 seconds.CHyD also slightly exceeds SEMQA on single-context SEMAeroSQuAD ROUGE-L, 0.663 versus 0.662.Source: §5.2 · pp. 8–9, Table 5 · p. 8
5.3 Fluency-Faithfulness Trade-offs
- SEMQA can score highly on generation metrics despite unfaithful quotes, while NEST’s opportunistic copying targets speed; CHyD instead prioritizes extraction faithfulness.The paper reports CHyD EFA of 0.981–1.000 across LLMs and datasets, compared with SEMQA’s model-dependent average EFA of 0.639–0.871.Source: §5.3 · p. 9
5.4 Qualitative Analysis
- In Qwen3 examples, NEST truncates a numerical answer and SEMQA generates an absent quote, while CHyD answers correctly with exact source spans.The examples illustrate that fluent answers or quoted generation alone do not ensure context faithfulness.Source: §5.4 · p. 9, Table 6 · p. 9
5.5 Answerability Detection
- CHyD preserves abstention performance: with Llama-3.1 in the multi-context setting, it ties SEMQA at 0.832 accuracy and 0.896 F1.Across nearly all configurations, CHyD’s answerability metrics are nearly identical to SEMQA’s.Source: §5.5 · p. 10, Table 7 · p. 10
6 Discussion and Limitations
CHyD prioritizes faithfulness over fluency or latency for safety-critical QA, while its guarantees and practical scope depend on when extraction is triggered and the task setting. Its constraints also leave query-to-span reasoning with the base LLM.
6 Discussion and Limitations
- Strict extraction constraints may reduce fluency or abstraction, limiting paraphrasing flexibility even as they preserve competitive answer quality.The paper prioritizes correctness over stylistic considerations in safety-critical domains.Source: §6 · p. 10
- Faithful extraction is guaranteed only after the model enters extraction mode, so CHyD still depends on its mode-switching conditions.The model must learn when to enter EEC mode; rare missed or delayed triggers can leave spans unquoted or unextracted.Source: §6 · p. 10
- Hard extraction constraints also reduce the model’s ability to paraphrase or summarize evidence, potentially making CHyD less suited to open-domain or creative applications.The paper identifies this as a separate limitation from mode switching.Source: §6 · p. 10
- CHyD has higher latency than SEMQA; the increase comes from evidence-extraction phases and scales with the number of quoted spans, not answer length.The approach is not optimized for inference time.Source: §6 · p. 10
- SEMAeroSQuAD enables controlled closed-domain faithfulness evaluation, but automatic generation may introduce bias, making complementary human evaluation valuable.The suggested human evaluation would assess fluency, comprehensiveness, and correctness.Source: §6 · p. 10
- CHyD guarantees extractive faithfulness, but the base LLM still decides which span to extract, leaving the query-to-span logical link unresolved.The paper identifies reasoning faithfulness as a subject for future research.Source: §6 · p. 10
- CHyD shifts speculative RAG toward strict evidence extraction, with future work proposing to combine constrained extraction and fast speculative decoding.The paper presents this as a promising direction for improving efficiency while preserving correctness guarantees.Source: §6 · p. 10
7 Conclusion
Existing hybrid and semi-extractive QA can alter quoted spans, a problem for safety-critical uses requiring exact matches to certified documentation. CHyD enforces verbatim extraction at decoding time, trading some fluency for strong evidence-faithfulness guarantees.
7 Conclusion
- Existing hybrid and semi-extractive QA can produce fluent answers while altering quoted spans, which is unacceptable when aircraft-maintenance answers must match certified documentation.The conclusion notes that SEMQA may cite spans absent from source documents.Source: §7 · p. 10
- CHyD enforces verbatim extraction at decoding time, shifting hybrid generation toward correctness guarantees rather than fluency or efficiency.The framework is presented as a principled solution for safety-critical QA.Source: §7 · pp. 10–11
- CHyD trades slight fluency losses and modest reductions in ROUGE-L and BERTScore for strong evidence-faithfulness guarantees.Source: §7 · p. 11
- The authors conclude that prioritizing evidence correctness over fluency and latency can make hybrid generation suitable for safety-critical domains.They aim to combine speculative decoding with faithfulness constraints to improve efficiency in future work.Source: §7 · p. 11