Chain-of-Verification Reduces Hallucination in Large Language Models

1 INTRODUCTION

• Hallucinations are plausible-looking incorrect facts, especially for rarely occurring facts, and can be exacerbated in longform generation by exposure bias.The passage notes that larger models can generate more correct factual statements, but even the largest can fail on lesser-known facts.Source: §1 · p. 1
• Factored CoVe variants separate verification steps to avoid attending to the model’s own hallucinations, further improving performance across list-based questions, closed-book QA, and longform generation.The authors report that models can repeat hallucinations when they attend to them in their own generated context.Source: §1 · p. 1, Figure 1 · p. 2
• CoVe plans verification questions, answers them to check an initial draft, and uses the results to produce a revised response.The paper reports that independently answered verification questions tend to provide more accurate facts than the original longform answer.Source: §1 · p. 1, Figure 1 · p. 2

• Training-time correction adjusts model weights to reduce hallucinations, including through reinforcement learning, contrastive learning, and other methods (Roit et al., 2023; Wu et al., 2023; Chern et al., 2023b; Sun et al., 2023b; Li et al., 2023).Source: §2 · p. 2
• Generation-time correction builds reliability checks on base-model reasoning, using token probabilities (Kadavath et al., 2022), multiple samples (Manakul et al., 2023), or low-confidence detection and validation (Varshney et al., 2023).Source: §2 · p. 2
• Inconsistency checks include direct and indirect queries for references (Agrawal et al., 2023) and LM-versus-LM cross-examination (Cohen et al., 2023); CoVe uses self-consistency without multi-LLM debate.For QA, Cohen et al. (2023) report that inconsistency-based detection can outperform confidence scores.Source: §2 · p. 3
• Tool-based approaches ground or check generations using retrieval-augmented documents (Shuster et al., 2021; Jiang et al., 2023b; Yu et al., 2023), chain-of-thought verification (Zhao et al., 2023), fact-checking tools, or document attribution.Fact-checking tools include Chern et al. (2023a), Galitsky (2023), and Peng et al. (2023); attribution approaches include Menick et al. (2022), Rashkin et al. (2023), and Gao et al. (2023).Source: §2 · p. 3
• Reasoning methods for logical and mathematical tasks include chain-of-thought (Wei et al., 2022), deductive verification (Ling et al., 2023), and self-verification (Miao et al., 2023; Jiang et al., 2023a; Weng et al., 2022).For math problems, self-verification predicts the masked question from the answer as evidence of correctness.Source: §2 · p. 3

3 CHAIN-OF-VERIFICATION

• CoVe has four steps: draft a baseline response, plan verification questions, answer them to identify inconsistencies, and generate a revised response.The method assumes a base LLM can be prompted to plan and execute its own verification and incorporate the analysis into an improved response.Source: §3 · p. 3

3.1 BASELINE RESPONSE

• The baseline is a standard left-to-right LLM generation, serving as the response CoVe aims to improve and compare against its final output.The paper notes baseline generations are typically prone to hallucination.Source: §3.1 · p. 3

3.2 PLAN VERIFICATIONS

• Verification planning prompts the model, given the query and baseline response, to generate questions that test factual claims without requiring templated phrasing.Questions need not closely match the original wording.Source: §3.2 · p. 4

3.3 EXECUTE VERIFICATIONS

• Execution uses the LLM alone, without tool-use, and ranges from joint planning and answering to separate or independent verification and cross-check prompts.In 2-Step, answers see questions but not the baseline; factored answers also avoid interference between answer contexts and can be batched, while factor+revise explicitly checks each question-answer pair against the baseline.Source: §3.3 · pp. 4–5

3.4 FINAL VERIFIED RESPONSE

• A final few-shot prompt generates the improved response using the prior reasoning, baseline response, and verification question-answer pairs.When using Factor+Revise, the prompt also receives the cross-check inconsistency output.Source: §3.4 · p. 5

4.1.1 WIKIDATA

• The Wikidata benchmark uses 56 generated profession-and-city questions, typically with about 600 gold entities each, and evaluates micro-averaged precision plus positive and negative answer counts.Gold entity lists come from the Wikidata knowledge base; model-generated lists are typically much shorter.Source: §4.1.1 · p. 5

4.1.2 WIKI-CATEGORY LIST

• Wiki-Category tests 55 “Name some” questions drawn from Wikipedia categories, typically with about eight answers each, using micro-averaged precision and positive and negative counts.The selected examples exclude questions requiring logical operations.Source: §4.1.2 · p. 5

4.1.3 MULTISPANQA

• MultiSpanQA uses 418 factoid-based closed-book questions with multiple independent answers and spans of up to three tokens per item.The questions originate from Natural Questions, and no supporting documents are provided.Source: §4.1.3 · p. 5

4.1.4 LONGFORM GENERATION OF BIOGRAPHIES

• The biography task prompts models to generate biographies of selected entities and evaluates them with FACTSCORE, using retrieval-augmented Instruct-Llama to fact-check responses.FACTSCORE was developed by Min et al. (2023) and reported to correlate well with human judgments.Source: §4.1.4 · p. 6

4.2 BASELINES

• Experiments use greedy decoding with few-shot Llama 65B as the main baseline; CoVe uses the same model and examples, while Llama 2 instruction-tuned baselines are tested zero-shot with and without chain-of-thought.For list tasks, prompts constrain Llama 2 outputs to comma-separated answers and NER post-processing extracts answers; biography comparisons also include existing model results reported by Min et al. (2023).Source: §4.2 · p. 6

4.3 RESULTS

• CoVe raises Wikidata precision from 0.17 to 0.36, MultiSpanQA F1 from 0.39 to 0.48, and biography FACTSCORE from 55.9 to 71.4.On Wikidata, hallucinated negatives fall from 2.95 to 0.68, while correct positives fall from 0.59 to 0.38; biography FACTSCORE gains come with average facts decreasing from 16.6 to 12.3.Source: Table 1 · p. 6, §4.3 · pp. 7–8, Table 2 · p. 7
• Few-shot Llama 65B beats Llama 2 Chat across tasks, while CoVe surpasses InstructGPT, ChatGPT, and PerplexityAI overall but trails PerplexityAI on rare facts.The authors report FACTSCORE as normalized and comparable across models; clipping Llama 2 70B Chat to 10 sentences changes its score from 41.3 to 42.7.Source: Table 3 · p. 7, Figure 2 · p. 7, §4.3 · p. 8
• Factored and 2-step CoVe outperform joint CoVe; factor+revise raises longform FACTSCORE further, while LLM-generated and open verification questions beat their tested alternatives.Factored reaches 63.7 versus joint's 60.8 on longform, and factor+revise reaches 71.4; 2-step is best on Wikidata and factored is best on Wiki-Category. Rule-based questions reduce Wiki-Category precision, and yes/no questions perform worse for factored CoVe; on Wikidata, about 70% of entities are correctly answered by individual verification questions versus about 17% in baseline lists.Source: §4.3 · pp. 8–9, Table 4 · p. 8

5 CONCLUSION

• CoVe provides substantial performance gains over the original language-model response by having the same model deliberate on and verify its answer.The conclusion describes CoVe as an approach to reduce hallucinations through self-correction.Source: §5 · p. 9
• Verification questions receive more accurate answers than the original query when verification is broken into simpler questions.Source: §5 · p. 9
• Factored CoVe, which prevents the model from attending to its previous answers during verification, helps alleviate copying the same hallucinations.Source: §5 · p. 9
• The authors propose adding tool use, such as retrieval augmentation during verification, as an extension likely to bring further gains.Source: §5 · p. 9

6 LIMITATIONS

• CoVe can still produce incorrect or misleading information, and the experiments address directly stated factual inaccuracies rather than hallucinations in reasoning or opinions.The verifications can make its decisions more interpretable to users, but generating additional output tokens increases computational expense, as with other reasoning methods such as Chain-of-Thought.Source: §6 · p. 9
• CoVe’s potential improvement is limited by the model’s capabilities, including its ability to identify and know what it knows.The authors do not explore external tools, but suggest they could be fruitful to combine with CoVe to provide information beyond model weights.Source: §6 · p. 9

• In longform generation, CoVe Factor + Revise is most effective, independently cross-checking facts to discard inconsistencies and regenerate the response from consistent facts.The method aims to retain facts consistent with its executed verifications and disregard inconsistent ones.Source: Figure 3 · p. 13

8 PROMPT TEMPLATES

• The provided templates are for longform biography generation; templates for other tasks are similar but use few-shot examples from those tasks.Source: §8 · p. 14

8.1 GENERATE BASELINE RESPONSE

• The longform biography baseline uses three few-shot examples, and other tasks use three examples from their respective tasks.The appendix says other tasks use the same standard few-shot setup with examples from that task.Source: Table 5 · p. 14, §8 · p. 14

8.2 PLAN VERIFICATIONS

• For biography verification planning, the prompt splits the generation into passages and requests a verification question for each fact in each passage.A passage may contain multiple facts; splitting is used because of excessive context length and is unnecessary for the other tasks.Source: Table 6 · p. 14

8.3 EXECUTE VERIFICATIONS

• In verification execution, the model answers each planned question using three few-shot examples.Source: Table 7 · p. 15

8.4 GENERATE FINAL VERIFIED RESPONSE

• For the factored final-response step, the model receives the original generation and verification results as another source, then synthesizes an answer from facts consistent between them.In the biography case, the original generation is split into passages because of context length.Source: Table 8 · p. 15

8.5 FACTOR+REVISE: IDENTIFY WHICH FACTS ARE CONSISTENT

• In Factor + Revise, the model explicitly identifies facts consistent between the two sources so they can be spliced together.Source: Table 9 · p. 16

• ChatGPT generates several hallucinations in response to the Figure 4 question, including Hillary Clinton and Michael Bloomberg.Source: Figure 4 · p. 17
• For a rewritten query, ChatGPT gives a slightly different longform answer but still generates several hallucinations, including Hillary Clinton and Michael Bloomberg.Source: Figure 5 · p. 17
• Shortform questions, which could serve as verification questions, appear to be answered more factually than the longform answers in Figures 4 and 5.Source: Figure 6 · p. 18
• A different query produces another hallucination: the answer says John F. Kennedy Jr. was born in Washington D.C.Source: Figure 7 · p. 18
• Questions asking for a fact can be answered correctly while yes/no verification is incorrect, as the model tends to agree with incorrectly phrased questions.Source: Figure 8 · p. 19