Source-linked AI summary
Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework
Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, Lidong Bing
TL;DR
LLMs and CoT prompting still face factuality problems in knowledge-intensive reasoning. Verify-and-Edit post-edits uncertain CoT chains using retrieved external knowledge, and the framework improves accuracy across open-domain QA tasks, though its benefits are strongest for challenging reasoning settings.
Problem
LLMs can generate factually incorrect content, and CoT reasoning remains vulnerable when accurate supporting knowledge is unavailable.
Method
Verify-and-Edit identifies uncertain predictions, generates verifying questions, retrieves external knowledge, edits rationales, and regenerates predictions.
Results
The framework improves accuracy across multiple open-domain QA tasks, including 3.8x improvement over similar retrieval-augmented models on Adversarial HotpotQA.
Takeaways & Limitations
Knowledge retrieval combined with reasoning can produce more factually grounded CoTs and improve end-task prediction performance.
Takeaways & Limitations
Verify-and-Edit works best for complex open-domain QA and depends on consistency scores to separate correct from incorrect predictions.
Abstract
from arXiv · showhide
As large language models (LLMs) have become the norm in NLP, demonstrating good performance in generation and reasoning tasks, one of its most fatal disadvantages is the lack of factual correctness. Generating unfactual texts not only leads to lower performances but also degrades the trust and validity of their applications. Chain-of-Thought (CoT) prompting improves trust and model performance on complex reasoning tasks by generating interpretable reasoning chains, but still suffers from factuality concerns in knowledge-intensive tasks. In this paper, we propose the Verify-and-Edit framework for CoT prompting, which seeks to increase prediction factuality by post-editing reasoning chains according to external knowledge. Building on top of GPT-3, our framework lead to accuracy improvements in multiple open-domain question-answering tasks.
1 Introduction
The paper targets factuality problems in LLM generation and proposes Verify-and-Edit to post-edit uncertain CoT reasoning with external knowledge before producing predictions. Across open-domain QA tasks, the framework improves factual alignment and accuracy.
- Motivation: Existing CoT methods generally use generated reasoning chains as-is, while only a few approaches improve the chains themselves.The paper frames reasoning-chain quality as relevant to both interpretability and end-task performance.
- Motivation: Factual correctness is a major weakness of LLMs, undermining the validity and trust of knowledge-access applications.Instruction tuning does not fully resolve the problem when source-of-truth knowledge is unavailable during fine-tuning.
- Approach: Verify-and-Edit selects uncertain predictions, generates verifying questions, retrieves external knowledge, edits rationales, and generates new predictions.Highly consistent predictions are left unchanged, while lower-consistency cases proceed through the editing stages.
- Contribution: The framework post-edits CoT-style reasoning chains to produce more factually aligned predictions.The paper presents this as its first attempt to post-edit CoT-style chains for improved prediction performance.
- Evaluation: Experiments cover two reasoning-focused open-domain QA tasks and the Fever fact-verification task.The evaluated QA datasets are Adversarial HotpotQA and 2WikiMultihop.
- Results: 3.8x accuracy improvement is reported over similar retrieval-augmented models on Adversarial HotpotQA.On 2WikiMultihop, Verify-and-Edit reaches 33.6% accuracy versus 27.7% for CoT Self-Consistency.
2 Related Work
Related work improves factuality through prompt engineering, result calibration, or retrieval augmentation. Verify-and-Edit is positioned as a natural conversational approach that edits reasoning chains rather than merely selecting or calibrating outputs.
- Chain-of-Thought: CoT decomposes complex problems into intermediate steps while providing interpretable reasoning chains.It has been shown more capable than standard prompting for complex problems.
- Factuality: Hallucination and unsupported facts can undermine CoT validity because autoregressive decoding may generate nonfactual content.Final-answer accuracy is reported to correlate with the factuality and consistency of reasoning explanations.
- Prompt Engineering: Prompt-engineering methods guide reasoning or retrieve information, including ReAct, self-ask, and least-to-most prompting.ReAct combines reasoning and actions, while self-ask explicitly generates follow-up questions.
- Result Calibration: Result-calibration methods adjust answer probabilities using explanation quality or sample consistency.Examples include calibrators based on factuality and consistency and self-consistency decoding over diverse reasoning paths.
- Retrieval Augmentation: Retrieval-augmented approaches provide external documents or search results to support question answering and factuality.The paper describes retrieval-augmented reasoning paths as a natural way to increase factual alignment.
3 Verify-and-Edit Framework
Verify-and-Edit uses consistency to identify uncertain predictions, verifies rationale details through generated questions and external retrieval, then regenerates answers from edited CoTs. The design aims to preserve LLM generation and reasoning while improving factual grounding.
- Design Goals: The framework targets knowledge-intensive tasks requiring multi-step reasoning and external knowledge.Its goal is to improve factual reasoning chains and final-answer accuracy.
- Framework Stages: VE separates into finding uncertain predictions, editing rationales with supporting facts, and generating final answers from edited rationales.This organization is intended to follow a human-like process of checking uncertain answers against external information.
- Uncertainty Detection: Self-consistency samples diverse reasoning paths and provides an uncertainty estimate for selecting instances to edit.The framework uses consistency because it is reported to correlate highly with accuracy.
- Rationale Editing: VE improves both factual content and reasoning by treating rationales as combinations of facts and inference.External sources supply supporting facts, while generative prompting produces relevant and logical rationales.
- Rationale Editing: A verifying question is generated from the original question and rationale before retrieval, reducing noise from potentially incorrect entities in the rationale.The question is designed to target the information needed for the original query rather than blindly searching the generated claim.
- Answer Generation: The edited CoT is fed back into the LLM to generate a new answer with more factually grounded reasoning.The framework uses retrieved knowledge as a memory refresh for information the model may not have recalled correctly.
- Design Properties: The framework is described as simpler and more conversational than ReAct, potentially making its reasoning easier for humans to understand and revise.The paper reports that the setup mitigates factuality concerns and boosts end-task performance in experiments.
4 Experiment Setup
The evaluation tests Verify-and-Edit with GPT-3 in-context learning on three datasets spanning multi-hop open-domain QA and fact verification, using several retrieval systems and comparison baselines. Editing is applied to low-consistency instances.
- Task Selection: The experiments target tasks that are open-domain and rely on multi-hop reasoning to reach later predictions.These properties motivate interaction with external knowledge and reliance on generated rationales.
- Datasets: Evaluation uses Adversarial HotpotQA, 2WikiMultihop, and Fever.The first two are multi-hop question-answering datasets, while Fever is a fact-verification dataset.
- Model and Prompts: GPT-3 instruct-series models are used with in-context learning, including text-davinci-003 as the experimental backbone.Adversarial HotpotQA and 2WikiMultihop use six-shot prompts, while Fever uses three-shot prompts.
- Baselines: The study compares Standard Prediction, Original CoT, CoT Self-Consistency, and a calibrator against Verify-and-Edit.These baselines cover direct prediction, explanation generation, consistency decoding, and probability calibration.
- Verify-and-Edit Configuration: VE edits instances whose self-consistency score is below the majority threshold, while high-consistency answers remain unchanged.The implementation uses the same consistency baseline to estimate uncertainty and generates verifying questions with a two-shot setup.
- Retrieval Systems: Four retrieval settings are evaluated: Wikipedia-API, DrQA, Google, and dataset-provided contexts.Dataset contexts serve as an oracle-like upper-bound setting using supporting and distractor paragraphs.
- Retrieval Systems: For most retrieval systems, the top three sentences ranked by Sentence-BERT similarity are selected as context.This selection is applied after retrieval for Wikipedia-API, DrQA, and dataset contexts.
5 Results and Analysis
Across open-domain QA and fact verification, Verify-and-Edit improves performance by editing uncertain reasoning chains with retrieved knowledge, with gains depending on retrieval quality and task characteristics.
- Using Self-Consistency: Low-consistency predictions are more often incorrect, supporting consistency as an uncertainty signal for selecting instances to edit.Incorrect samples show a left-skewed consistency distribution, with few incorrect samples at higher consistencies.
- Results on HotpotQA: 4.5% and 4.8% EM improvements over baseline are achieved with Wikipedia and DrQA retrieval, while Google retrieval reaches a 6.5% improvement.The Google result is 3.8x the 1.7% highest EM improvement reported for ReAct.
- Results on HotpotQA: 56.8% EM and 60.94 AUC are obtained using adversarially augmented dataset paragraphs.This setup uses highly compressed contexts and a nearly ideal retrieval setting.
- Results on 2WikiMultiHop: 2WikiMultiHop accuracy improvements range from 3.4% to 5.9% with open-domain retrieval and reach 9.5% when selecting from dataset paragraphs.The calibrator trails all Verify-and-Edit variations in this comparison.
- Results on fact verification: Verify-and-Edit consistently improves over baseline on Fever, with Google retrieval producing a further 1.9% improvement.The authors note that Fever requires less multi-hop reasoning than the other evaluated tasks.
- Evaluating reasoning chains: Humans selected Verify-and-Edit chains as more factually consistent 53% of the time versus 17% for CoT-SC, while annotators used Google search 100% of the time.Annotators required 1.5 minutes on average to validate one data point.
6 Conclusions
The paper introduces Verify-and-Edit for open-domain question answering, post-editing CoT reasoning chains with retrieved knowledge to improve factuality and end-task performance.
- Verify-and-Edit post-edits CoT-style reasoning chains for better end-task performance in open-domain question answering.It combines knowledge retrieval with natural, conversational reasoning-chain edits.
Limitations
The framework is best suited to open-domain question-answering tasks requiring complex reasoning and depends on consistency-based selection of uncertain examples.
- Verify-and-Edit works best for open-domain question-answering tasks requiring complex reasoning.Less complex or commonsense datasets that do not require knowledge retrieval may yield smaller improvements.
- The method relies on consistency estimates to separate correct from incorrect predictions and select mostly incorrect samples for editing.It often produces larger improvements on more challenging examples.
- The authors plan to reduce noise introduced during rationale editing and incorporate additional knowledge resources, including knowledge bases.
Ethics Statement
The framework may mitigate hallucinations and unfactual details, but its retrieval and GPT-3 components retain ethical risks.
- Verify-and-Edit may mitigate ethical concerns surrounding hallucinations and unfactual details in LLM generation.
- Google retrieval may return potentially toxic information present in search results.
- Using GPT-3 leaves the framework exposed to concerns such as toxic responses and biased behavior.
A Dataset Processing
The appendix describes dataset-specific sampling and preprocessing procedures, with licensing information for HotpotQA, 2WikiMultihop, and Fever materials.
- The Adversarial HotpotQA subset reduces context length and retains two supporting and two adversarial paragraphs with relevant sentences.
- HotpotQA is licensed under CC BY-SA 4.0, 2WikiMultihop under Apache License 2.0, and Fever annotations incorporate Wikipedia material.
- Experiments use a random sample of 1,000 from HotpotQA’s 12,576-example development set, with indices released for reproducibility.
B Experiment Costs
The experiments use text-davinci-003 and present question-answering examples with rationales, verification questions, retrieved facts, and edited answers.
- $0.02/1K tokens was the reported inference cost for text-davinci-003, with total spending of 273$.The paper states that experiments used the text-davinci-003 API.
- The QA examples pair original questions with answers and CoT rationales that state intermediate facts before giving the final answer.Examples cover racing, music, geography, wrestling, royal honors, and albums.
- Verification prompts ask targeted questions about rationale details or the reason supporting an answer.Examples include questions about teams, composers, lifespans, and film release dates.
- Retrieved or supplied facts are used to support corrected answers in the worked examples.The examples show fact-based reasoning for film releases and other question-answering items.
D Human Study
The human study compared CoT-SC and Verify-and-Edit reasoning chains using two English-proficient NLP Ph.D. student volunteers.
- Two NLP Ph.D. student volunteers, proficient in English, conducted the human evaluation after receiving study instructions.The volunteers understood the data-collection use and were in consensus.
- The study presented CoT-SC and Verify-and-Edit chains in random order for comparison.Each volunteer evaluated 50 samples and took 1.25 hours on average.
E Qualitative Examples
Qualitative AdvHotpotQA examples show retrieved facts correcting fabricated or inaccurate details, while a second reasoning pass can resolve relevance errors.
- AdvHotpotQA examples: Table 5 presents three detailed examples from AdvHotpotQA with facts retrieved using Google.
- AdvHotpotQA examples: Retrieved Google facts corrected an incorrect Spider-Man attribution to Ghost Rider and resolved a brand-versus-hero answer mistake.The example concerns a song associated with Marvel Comics.
- AdvHotpotQA examples: Google retrieval corrected a plausible but unsupported claim that Tony Robinson had written seven children’s books to the fact that he published 16.The retrieved fact supplied a concrete correction for a claim without an indicated confidence level.
- AdvHotpotQA examples: A second reasoning pass corrected an initial claim while disregarding a verified fact that had become irrelevant to the original question.The example presents this as a reason for feeding both rationale sentences back into the CoT setup.