Source-linked AI summary
Retrieving Supporting Evidence for Generative Question Answering
Siqing Huo, Negar Arabzadeh, Charles L. A. Clarke
TL;DR
LLMs can generate convincing hallucinations, and retrieval-based generation does not guarantee that claims are supported by evidence. This paper tests post-generation retrieval checks, including verification of whole answers and individual factual claims. The authors report over 80% accuracy, while noting that the process misses some incorrect answers and cannot eliminate hallucinations.
Problem
LLMs can confidently generate hallucinations, and retrieval-augmented generation cannot guarantee that generated claims are supported by retrieved material.
Source: §1 · p. 1Method
The paper tests post-generation retrieval, using an LLM to verify answers against retrieved evidence and to decompose answers into factual claims for separate checking.
Source: §1 · p. 1Results
Takeaways & Limitations
Verification can reduce hallucinations but cannot eliminate them, so caution remains warranted when relying on LLM-generated answers, especially in critical circumstances.
Source: §1 · p. 1Evaluation uses MS MARCO, which may have been included in the GPT models’ training data, and the authors plan to explore other benchmark collections.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Current large language models (LLMs) can exhibit near-human levels of performance on many natural language-based tasks, including open-domain question answering. Unfortunately, at this time, they also convincingly hallucinate incorrect answers, so that responses to questions must be verified against external sources before they can be accepted at face value. In this paper, we report two simple experiments to automatically validate generated answers against a corpus. We base our experiments on questions and passages from the MS MARCO (V1) test collection, and a retrieval pipeline consisting of sparse retrieval, dense retrieval and neural rerankers. In the first experiment, we validate the generated answer in its entirety. After presenting a question to an LLM and receiving a generated answer, we query the corpus with the combination of the question + generated answer. We then present the LLM with the combination of the question + generated answer + retrieved answer, prompting it to indicate if the generated answer can be supported by the retrieved answer. In the second experiment, we consider the generated answer at a more granular level, prompting the LLM to extract a list of factual statements from the answer and verifying each statement separately. We query the corpus with each factual statement and then present the LLM with the statement and the corresponding retrieved evidence. The LLM is prompted to indicate if the statement can be supported and make necessary edits using the retrieved material. With an accuracy of over 80%, we find that an LLM is capable of verifying its generated answer when a corpus of supporting material is provided. However, manual assessment of a random sample of questions reveals that incorrect generated answers are missed by this verification process. While this verification process can reduce hallucinations, it can not entirely eliminate them.
1 Introduction
LLMs can produce convincing hallucinations despite gains in text generation, and retrieval-augmented generation still cannot guarantee that claims are supported. This paper tests post-generation evidence checks, finding over 80% accuracy while acknowledging that verification does not eliminate hallucinations.
1 Introduction
- LLMs can convincingly hallucinate despite improvements in question answering, summarization, and machine translation.Such errors matter particularly when generated text is shown directly to users in critical circumstances such as health and medicine.Source: §1 · p. 1
- Current LLMs lack access to external knowledge for self-detection, while retrieval-augmented generation can still produce claims absent from or contradictory to retrieved material.The paper positions evidence attribution as connecting generated text to supporting sources.Source: §1 · p. 1
- Post-generation retrieval checks the generated response against external evidence, unlike retrieval-augmented generation, which retrieves before generation.The approach uses the LLM and a retrieval pipeline without few-shot prompting or an external query-document relevance model.Source: §1 · p. 1
- The approach decomposes generated text into factual claims using the LLM alone, without training or human intervention and with minimal prompt engineering.The authors draw inspiration from prior work on decomposing text into atomic factual claims.Source: §1 · p. 1
- Over 80% accuracy: with relevant retrieved material, the LLM correctly detects its own hallucinations in a majority of cases, but verification does not eliminate them.The authors recommend continued caution, especially in critical circumstances.Source: §1 · p. 1
2 Experimental Setup
The experiments use GPT-3.5-turbo on MS MARCO (V1) and compare BM25 with a multi-stage neural retrieval pipeline. The neural stack combines sparse and dense retrieval with two reranking stages.
2 Experimental Setup
- The experiments use GPT-3.5-turbo at temperature 0 and evaluate on 6,980 questions from the MS MARCO (V1) small development set.The passage collection contains over 8 million passages.Source: §2 · p. 2
- BM25 serves as an exact-matching baseline with k1 = 0.82 and b = 0.68, standard values tuned for MS MARCO passage retrieval.The authors speculate that exact matching may help retrieve support for terms in generated answers.Source: §2 · p. 2
- The neural stack pools the top 100 SPLADE and ANCE results, reranks them with MonoT5, then reranks MonoT5’s top 10 with DuoT5.The pipeline achieves MRR@10 of 0.40 on the MS MARCO (V1) small development set.Source: §2 · p. 2
3 Experiment 1
Experiment 1 uses the LLM to answer a question, retrieves evidence using the question together with that answer, and asks the LLM whether the evidence supports its response.
3 Experiment 1
- The LLM first generates an answer to the question.This is the initial response that the later evidence check evaluates.Source: §3 · p. 2
- The question and generated answer are combined to query a corpus expected to contain supporting passages.The combined query is intended to retrieve evidence relevant to the question and potentially supportive of the answer.Source: §3 · p. 2
- The LLM receives the question, its generated answer, and retrieved evidence, then judges whether the evidence supports the answer.This constitutes the experiment’s self-verification step.Source: §3 · p. 2
3.1 Methodology
The first experiment generates an answer, retrieves evidence using the question combined with that answer, and asks the LLM to assess whether the evidence supports it. Retrieved answers may be a single BM25 or neural passage or a concise reader summary of three neural passages; validation first checks whether both answers address the question.
3.1 Methodology
- The first experiment queries a passage corpus with the question and generated answer, then asks the LLM whether retrieved evidence supports that answer.The LLM first generates an answer to the question.Source: §3.1 · pp. 2–3
3.1.1 Retrieved Answer
- Retrieved answers are either the top BM25 or neural passage, or a reader-generated concise summary of the top three neural passages.The reader produces a question-based summary to avoid abrupt shifts, repetition, or excessive length from simply concatenating passages.Source: §3.1.1 · p. 3
- Validation first checks whether the generated and retrieved answers directly address the question; if either does not, the pair is classified as “Not Related.”Only pairs where both answers directly address the question proceed to the support assessment.Source: §3.1.2 · p. 3
3.1.2 Validating Generated Answer
- The outcomes are “Yes” for supported, “No” for likely hallucination, and “Not Related” for clarification, unknown answers, or irrelevant retrieval; qrel passages provide a human-curated comparison.Qrel passages are assumed to address the question and are included for insight into query construction, not as part of the automatic validation task.Source: §3.1.2 · pp. 3–4
3.2 Results
Manual checks found that the LLM’s support judgments were often correct, but also exposed recurring errors in classifying answers and evidence. These included missed synonyms and implied information, false negatives for detailed passages, and mistaken inferences or evidence attribution.
3.2 Results
- Generated answers addressed the question about 93% of the time, while reader-extracted answers ranked ahead of neural and BM25 answers for relevance.The reported relevance order was reader-extracted, neural-retrieved, then BM25-retrieved answers.Source: §3.2.1 · p. 4
3.2.1 Step 1: Classifying Answers
- For non-direct answers, classifications had many false negatives; accuracy was high only for generated answers and near 50% for retrieved-answer types.The authors suggest a different prompt may be needed for classifying retrieved answers.Source: §3.2.1 · p. 4
3.2.1.1 Classify Generated Answer
- The LLM sometimes misclassified an answer that acknowledged limitations and offered an alternative, despite prompt instructions against that classification.The authors suggest better prompt engineering or fine-tuning may be needed.Source: §3.2.1.1 · p. 4
3.2.1.2 Classify Reader Extracted Answer
- The LLM sometimes failed to recognize synonymous expressions in reader-extracted answers, or demanded exact wording.For example, it missed that “someone sangs a song” and “a song is recorded by someone” express the same answer.Source: §3.2.1.2 · p. 4
3.2.1.3 Classify Neural or BM25 Retrieved Answer
- Detailed retrieved passages often triggered false negatives, and the LLM also missed implied information such as laptop sizes listed as models.The authors note that MS MARCO passages are not tailored to specific questions and suggest extracting more concise answers before classification.Source: §3.2.1.3 · p. 5
- About 93% of questions were classified as supported after excluding “Not Related” cases.This is the LLM’s classification, not the manual accuracy estimate.Source: §3.2.2 · p. 5, Table 3 · p. 5
3.2.2 Step 2: Classifying Question-answers Pair
- 80% accuracy for samples classified “Yes” and 90% for samples classified “No”; false positives were slightly more common than false negatives.The manual review sampled 100 cases from each classification cell for each answer type.Source: §3.2.2 · pp. 5–6, Table 4 · p. 5
- Pair-level judgments could accept partially supported answers, misread evidence across dates, or confuse answer text with evidence; qrel passages were less reliable for validation.For example, the LLM treated a population figure from 2015 as evidence of a 2019 increase without a stated rate.Source: §3.2.2 · pp. 5–6
4 Experiment 2
Experiment 2 decomposes generated answers into factual statements, retrieves evidence for each, and asks the LLM to validate and correct them. The authors report that about 80% of claims are supported on average, but manual checks expose classification errors, especially for contradictory cases.
4.1 Methodology
- The method extracts factual statements from the answer, retrieves evidence for each statement using a query combined with the original question, then validates and corrects each statement against that evidence.Supported statements and post-edited contradictory statements can be recomposed into an attributed answer.Source: §4.1 · pp. 7–8
4.2 Results
- The extracted statements can usefully isolate list items for separate checking, but may be excessively detailed or repetitive, leaving granularity suboptimal.For the “Rise Up” question, the existence of the song and of singer Andra Day were unnecessary statements because they were already implied.Source: §4.2 · p. 8, Figure 20 · p. 8
- On average, about 80% of claims per query were supported, and LLM classifications suggested neural-retrieved evidence was slightly better than BM25 evidence; no hallucination was found in about 70% of responses.Excluding “Neither” cases, the LLM judged about 85% of claims supported.Source: §4.2 · p. 8
- Using neural-method retrieved evidence, the LLM’s classifications achieved an average accuracy of about 80%, but performed noticeably worse on samples it predicted to be in the “Contradictory” class.For each retrieval method (BM25 and Neural), the authors randomly selected 100 statement-evidence pairs from each classification cell.Source: §4.2 · p. 9
- The LLM sometimes mislabeled irrelevant evidence as “Contradictory,” yet post-editing often returned the original statement; edits were usually reasonable when classification was correct.The authors also observed that the model could explain its incorrect classification using the prompt’s definition of “Neither.”Source: §4.2 · p. 9, Figure 21 · p. 9
5 Limitations
The experiments are limited by their reliance on one model and a single benchmark setting. Possible overlap between MS MARCO and model training data makes the results potentially best-case rather than definitive.
5 Limitations
- The experiments used only gpt-3.5-turbo and simple prompts with minimal prompt engineering, limiting evidence about other models or prompting strategies.The model was chosen for its convenient and inexpensive API; the authors say excessive prompt engineering could harm reproducibility.Source: §5 · p. 9
- The authors do not know whether MS MARCO questions and passages appeared in the model’s training data, so the possibility of overlap remains unresolved.They note that if the questions and corpus were included in training and all questions were answered by the corpus, the experiment may represent a “best case” scenario.Source: §5 · p. 9
- The study uses MS MARCO, where all questions have corpus answers, although those answers need not agree with the LLM’s generated answers.The authors plan to explore other benchmark collections.Source: §5 · p. 9
6 Conclusion
The paper tests whether retrieval can help an LLM detect hallucinations in its generated text. The authors report verification in over 80% of cases, but caution that errors mean the approach cannot be relied on alone.
6 Conclusion
- The second experiment addresses cases where retrieved evidence only partially supports a whole answer by checking factual statements separately and aiming to produce an attributed answer.The paper reports that the approach verifies generated texts in over 80% of cases, but manual examination found unreasonable decisions, fabricated supporting evidence, and missed implied connections.Source: §6 · p. 9
- The authors caution against relying solely on the approach because they observed more false positives than false negatives when checking whether an answer was hallucination-free.They propose testing more prompts, task-specific models, and different LLMs in future work.Source: §6 · p. 9