Source-linked AI summary
Chain-of-Verification Reduces Hallucination in Large Language Models
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, Jason Weston
TL;DR
Large language models can generate plausible but factually incorrect information, motivating methods for correcting their own responses. Chain-of-Verification drafts an answer, independently checks planned verification questions, and revises the response; it reduces hallucinations across list-based questions, closed-book QA, and longform generation.
Problem
Large language models can produce plausible factual errors, particularly on lesser-known facts, despite increased model size and training data.
Method
CoVe generates a baseline response, plans verification questions, answers them independently, and incorporates the results into a final verified response.
Results
CoVe improves performance over baseline responses across list-based tasks, closed-book QA, and longform generation.
Takeaways & Limitations
A language model can improve its own response by deliberating on and verifying the answer.
Takeaways & Limitations
CoVe does not remove hallucinations completely, covers only directly stated factual inaccuracies, and requires additional computational expense.
Abstract
from arXiv · showhide
Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models. We study the ability of language models to deliberate on the responses they give in order to correct their mistakes. We develop the Chain-of-Verification (CoVe) method whereby the model first (i) drafts an initial response; then (ii) plans verification questions to fact-check its draft; (iii) answers those questions independently so the answers are not biased by other responses; and (iv) generates its final verified response. In experiments, we show CoVe decreases hallucinations across a variety of tasks, from list-based questions from Wikidata, closed book MultiSpanQA and longform text generation.
1 INTRODUCTION
Large language models can produce plausible but factually incorrect statements, especially for less frequent facts. Chain-of-Verification (CoVe) addresses this by planning and answering verification questions before revising the response.
- Motivation: LLMs may generate plausible hallucinations when they fail on lesser-known facts.These facts occur relatively rarely in training corpora, and incorrect alternatives can appear credible.
- Approach: CoVe drafts a response, plans verification questions, answers them, and incorporates the checks into a revised response.The factored version answers verification questions without conditioning on the original response.
- Approach: Verification questions are typically answered more accurately than the corresponding facts in the original longform generation.This provides the basis for correcting inaccuracies in the final response.
2 RELATED WORK
Prior work addresses hallucination through training-time correction, generation-time correction, and external augmentation, while related reasoning methods improve logical or mathematical tasks without explicitly targeting hallucination.
- Hallucination mitigation: Hallucination remains a general problem across tasks such as summarization and open-domain dialogue despite scaling training data or model size.Existing mitigation methods are grouped into training-time correction, generation-time correction, and augmentation.
- Training-time correction: Training-time methods adjust model weights to reduce the probability of hallucinated generations.Examples include reinforcement learning and contrastive learning.
- Generation-time correction: Generation-time methods add reasoning or consistency checks on top of the base LLM to improve reliability.Approaches include token probabilities, multiple samples, confidence scores, validation, and cross-examination.
- Augmentation: Tool-based approaches mitigate hallucinations by grounding generation in factual documents or external fact-checking resources.Retrieval augmentation and attribution to external documents are examples.
- Related reasoning work: Chain-of-thought, deductive verification, and self-verification improve reasoning on logical or mathematical tasks without explicitly addressing hallucination.Self-verification predicts a masked question from an answer as evidence for correctness.
3 CHAIN-OF-VERIFICATION
CoVe uses one language model to generate a baseline, plan factual checks, execute them, and produce a revised response. Its factored variant independently answers verification questions to reduce copying and interference, at additional computational cost.
- Assumption: CoVe assumes the base LLM can generate and execute a verification plan and incorporate its analysis into an improved response.The assumption applies when the model can be prompted in few-shot or zero-shot fashion.
- Core pipeline: CoVe generates a baseline response, plans verification questions, executes them, and generates a final verified response.Each step uses the same LLM prompted in different ways.
- Baseline response: The baseline is an ordinary left-to-right generation that serves as the response CoVe seeks to improve.Experiments directly compare this baseline with the final verified response.
- Plan and execute verifications: Verification questions test factual claims in the baseline response, after which their answers are checked for inconsistencies or mistakes.The questions can target specific claims, such as dates in a historical statement.
- Execute verifications: Factored verification answers each question in a separate prompt without the original baseline response or other answer contexts.This reduces copying and interference, can support more questions, and may be batched despite requiring more prompts.
- Revise: Factor+Revise adds an explicit cross-check prompt for each question before the final response incorporates the verification results.The final prompt receives the baseline, question-answer pairs, and, when available, inconsistency detections.
4 EXPERIMENTS
CoVe is evaluated across list-based entity questions, closed-book MultiSpanQA, and longform biography generation using factuality and answer-quality metrics. Across these tasks, CoVe improves precision or F1, with factored and factor+revise variants producing especially strong gains.
- Benchmark setup: The experiments cover list-based questions, closed-book MultiSpanQA, and longform biography generation, spanning entity-set answers to multiple freeform sentences.The study compares CoVe with several baselines across these benchmark settings.
- List-based tasks: 0.17 → 0.36 Wikidata precision: CoVe more than doubles precision while reducing average hallucinated entities from 2.95 to 0.68.Average non-hallucinated entities decrease from 0.59 to 0.38, a relatively smaller reduction.
- Closed-book QA: 23% improvement in MultiSpanQA F1 over the few-shot baseline, from 0.39 → 0.48, with gains in both precision and recall.MultiSpanQA is evaluated in a closed-book setting on factoid-based questions with multiple independent answers.
- Longform generation: 28% FACTSCORE increase in longform generation, from 55.9 → 71.4, while the average number of provided facts decreases from 16.6 → 12.3.CoVe improves results for both rare and more frequent facts in the FACTSCORE breakdown.
- CoVe variants: Factored CoVe consistently outperforms joint CoVe, while 2-step CoVe also outperforms joint CoVe on the Wikidata and Wiki-Category list tasks.The results support separating verification from the original response to reduce repetition or copying during verification.
- CoVe variants: 71.4 FACTSCORE for factor+revise versus 63.7 for factored CoVe and 60.8 for joint CoVe in longform generation.The factor+revise variant explicitly cross-checks verification answers for inconsistency before revision.
5 CONCLUSION
Chain-of-Verification (CoVe) reduces hallucinations by having a language model deliberate on and self-correct its own responses. It improves verification accuracy and performance over the original response without changing the underlying model.
- CoVe reduces hallucinations by deliberating on and self-correcting a language model’s responses.
- Breaking verification into simpler questions lets models answer those questions more accurately than the original query.
- Factored CoVe prevents verification answers from attending to previous answers, helping alleviate repetition of the same hallucinations.
- CoVe provides substantial gains over the original language-model response by asking the same model to verify its answer.
- Tool use, such as retrieval augmentation during verification, is identified as an extension that could bring further gains.
6 LIMITATIONS
CoVe improves factual accuracy but does not eliminate hallucinations and has a scope limited to directly stated factual inaccuracies. Its gains are bounded by the base model’s capabilities, while additional verification tokens increase computational expense.
- CoVe can still generate incorrect or misleading information even when it improves over the baseline.
- The experiments address directly stated factual inaccuracies, not hallucinations arising from incorrect reasoning steps or opinions.
- Verification outputs can improve interpretability but increase computational expense by generating more tokens.
- CoVe’s improvement is limited by the base model’s ability to identify mistakes and know what it knows.
- Combining CoVe with external tools is proposed as a potentially fruitful direction but is not explored in this work.
7 COVE - FURTHER DETAILS
The longform CoVe workflow verifies factual claims in generated biographies through targeted questions and independent cross-checking. Figure 3 identifies Factor + Revise as the most effective longform variant.
- A biography claim about the Mexican–American War is checked with targeted questions about its dates, Texas annexation, and Texas’s secession.
- CoVe Factor + Revise independently cross-checks verification results, discards inconsistent facts, and regenerates the response from consistent facts.
8 PROMPT TEMPLATES
The appendix specifies few-shot prompt templates for generating, verifying, and revising longform biographies. The templates progressively transform a baseline response into a fact-checked final answer, including an explicit consistency step.
- 8.1 GENERATE BASELINE RESPONSE: The biography task uses three few-shot examples to prompt baseline generation, with other tasks using three examples from their respective tasks.
- 8.2 PLAN VERIFICATIONS: Biography generations are split into passages because of context-length constraints, whereas other tasks do not require this split.
- 8.2 PLAN VERIFICATIONS: The planning template generates a verification question for each fact observed in each biography passage.
- 8.3 EXECUTE VERIFICATIONS: The execution template generates an answer for every planned verification question using three few-shot examples.
- 8.4 GENERATE FINAL VERIFIED RESPONSE: Factored CoVe presents the original generation and verification results as separate sources, then synthesizes an answer from mutually consistent facts.
- 8.5 FACTOR+REVISE: IDENTIFY WHICH FACTS ARE CONSISTENT: Factor + Revise explicitly identifies which facts agree between the two sources before splicing the consistent facts together.
9 CHATGPT EXAMPLE SCREENSHOTS
The examples show hallucinations in longform answers across multiple queries, while shortform answers appear more factual. They also illustrate that yes/no verification can be misled by an incorrectly phrased question.
- ChatGPT generates multiple hallucinations in longform responses, including incorrect claims about Hillary Clinton and Michael Bloomberg.
- Rewriting the query and requesting a longform answer does not prevent several hallucinations, despite producing a slightly different response.
- Shortform questions, including potential verification questions, appear to receive more factual answers than the longform responses.
- A separate example shows a hallucinated claim that John F. Kennedy Jr. was born in Washington D.C.
- The model can answer a factual question correctly yet answer an incorrectly phrased yes/no verification question incorrectly, tending to agree with the question’s wording.