Source-linked AI summary
LM vs LM: Detecting Factual Errors via Cross Examination
Roi Cohen, May Hamri, Mor Geva, Amir Globerson
TL;DR
Modern language models often generate factually incorrect text, raising the question of whether such errors can be detected automatically. LMVLM addresses this problem with a zero-shot, multi-turn cross-examination between an examinee LM and an examiner LM that probes claims for inconsistencies. Across factual question-answering benchmarks, it detects over 70% of incorrect claims at above 80% precision and substantially improves factual-error detection, though its multiple-LM queries can be costly to scale.
Problem
Modern LMs generate factually incorrect text, motivating automatic detection of whether a model's claims are correct or incorrect.
Method
LMVLM uses prompting to create a zero-shot, multi-turn cross-examination between an examinee LM and an examiner LM that probes claims for inconsistencies.
Results
Across multiple datasets and examination settings, LMVLM detects over 70% of incorrect claims while maintaining precision above 80% and outperforms baselines.
Takeaways & Limitations
The results demonstrate the potential of interacting LMs for capturing factual errors in LM-generated claims.
Takeaways & Limitations
The method requires multiple queries to the examinee and examiner LMs, which can be costly when using external APIs and may constrain scaling.
Abstract
from arXiv · showhide
A prominent weakness of modern language models (LMs) is their tendency to generate factually incorrect text, which hinders their usability. A natural question is whether such factual errors can be detected automatically. Inspired by truth-seeking mechanisms in law, we propose a factuality evaluation framework for LMs that is based on cross-examination. Our key idea is that an incorrect claim is likely to result in inconsistency with other claims that the model generates. To discover such inconsistencies, we facilitate a multi-turn interaction between the LM that generated the claim and another LM (acting as an examiner) which introduces questions to discover inconsistencies. We empirically evaluate our method on factual claims made by multiple recent LMs on four benchmarks, finding that it outperforms existing methods and baselines, often by a large gap. Our results demonstrate the potential of using interacting LMs for capturing factual errors.
1 Introduction
Modern language models frequently produce inconsistent, non-attributable, or factually incorrect text, motivating automatic factuality detection. LMVLM uses law-inspired cross-examination between an examinee and examiner LM to expose inconsistencies and improve detection.
- Modern LMs generate inconsistent, non-attributable, and factually incorrect text, reducing their reliability.
- LMVLM frames factuality testing as a multi-turn interaction in which an examiner LM questions the LM that generated the claim.The examiner decides whether the original claim was correct after examining the examinee's answers.
- The method treats contradictions among the examinee's responses as evidence relevant to whether its original statement is correct.This approach views calibration as reasoning about the support a fact receives from other model-generated statements.
- Over 70% of incorrect claims were detected while maintaining precision above 80% across datasets and examination settings.The experiments used CHATGPT, GPT-3, and LLAMA in factual question-answering.
- Examiner LMs ask multiple questions and use paraphrasing, argument validation, claim decomposition, and evidence requests to reveal inconsistencies.
- The contributions are a two-LM factuality-testing formulation, a zero-shot prompted implementation, and improved detection accuracy across benchmarks.EXAMINER and EXAMINEE may also be the same LM prompted into different roles.
2 LM Cross-Examination
LMVLM conducts claim verification through a three-stage, multi-turn exchange: setup, follow-up questioning, and a final factuality decision. The examiner uses prior answers and iterative questioning to seek inconsistencies before classifying the claim.
- LMVLM maintains a multi-turn interaction in which each prompt incorporates outputs from previous rounds.The examination uses three stages, each guided by a specific prompt.
- Stage 1: Setup: Stage 1 assigns the examiner role, provides the examinee's claim, and asks the examiner to generate questions.Those questions are then presented individually to the examinee for answers about its claim.
- Stage 1: Setup: The examinee answers examiner-generated questions, producing responses that the examiner can inspect for inconsistencies with the initial claim.
- Stage 2: Follow-up Questions: The examiner reviews the answers and iteratively requests follow-up questions until it has none or reaches the turn threshold.The follow-up process conditions examiner outputs on previous examinee answers and can steer questions toward the claim.
- Stage 3: Factuality Decision: Stage 3 prompts the examiner to output “correct” or “incorrect,” treating any other final response as rejection of the claim.
3 Related Work
LMVLM differs from prior factuality and reasoning approaches by probing claims through multi-turn interaction without reference text or external knowledge bases. It focuses on zero-shot binary factual-error detection and generates an elaborate explanation through prompting rather than training.
- Attribution and Fact Checking: Unlike attribution and fact-verification methods, LMVLM does not assume reference text or an external knowledge base.It directly probes an LM's claim for inconsistencies.
- Consistency Across Generations: LMVLM differs from consistency methods by using multi-turn dialogue as a central mechanism for checking claims.
- Model Calibration: LMVLM treats factual-error detection as a binary variation of calibration, replacing a continuous error probability with a correct-or-incorrect prediction.
- Model Calibration: The work performs zero-shot factual-error detection with two output categories: whether a claim is correct or incorrect.The authors envision extending the binary output to a probabilistic correctness estimate.
- Multi-Agent LMs: The method belongs to interactive multi-agent LM approaches that use one model to assist or evaluate another model's behavior.
- Chain of Thought Reasoning: The approach relates to chain-of-thought reasoning but generates its explanation through different prompts rather than in-context training or fine-tuning.
4 Experiments
The experiments evaluate cross-examination for zero-shot factual-error detection across multiple question-answering datasets, models, and examination settings. The method consistently outperforms baselines, with follow-up questions contributing substantially to performance.
- Experimental setup: The evaluation covers factual question answering and fact completion across LAMA, TriviaQA, Natural Questions, and PopQA in a closed-book setting.Claims are generated by recent LMs and compared against dataset gold answers or aliases.
- Experimental setup: The method converts model answers into claims, conducts staged examiner interactions, and ends with a correct-or-incorrect factuality decision.The examination uses prompts for setup, follow-up questions, and final classification.
- Results: Across all settings, LMVLM outperforms baselines, including 85.4 F1 versus ≤65.2 for CHATGPT on PopQA and 77.2 versus ≤60.1 for GPT-3 on TriviaQA.The largest gains are reported in recall while baselines retain reasonable precision.
- Results: On the Falsehoods dataset, LMVLM rejects 87%-98% of false claims with CHATGPT and 75%-99% with GPT-3 across datasets.The reported figures are accuracies for examiner decisions on deliberately generated false claims.
- Ablations: Removing follow-up iterations reduces GPT-3 performance, with F1 falling from 78 to 68.3 on NQ and from 77.2 to 71.1 on TriviaQA.Recall decreases by 6%-10%, supporting the importance of follow-up questioning.
5 Analysis of Cross-Examinations
The analysis shows that examiners typically ask multiple questions and use several strategies to expose inconsistencies. Failures often arise when the examinee supplies mutually consistent false information, while examiner behavior differs across models.
- Examination statistics: Examiners ask 6-7 questions on average, typically including 1-2 follow-up steps that influence the final decision.In 9%-15% of cases, the examiner does not produce a concrete correct-or-incorrect decision.
- Observed strategies: Examiners commonly rephrase claims, repeat questions with different wording, validate logical implications, decompose claims, and request attribution.These strategies are observed across manual analyses of examinations by GPT-3 and CHATGPT.
- Observed strategies: Logical question decomposition appears in about 75% of CHATGPT examinations but only 10% of GPT-3 examinations.The strategy appears in 70% of CHATGPT’s correct detections and 33% of GPT-3’s correct detections.
- Failure patterns: Wrong intermediate answers occur mainly when the original claim is false and appear in about half of false claims detected by the examiner.They occur in about 80% of accepted false claims and 45% of rejected correct claims.
- Failure patterns: Many failures occur when the examinee encodes mutually consistent false facts, making inconsistencies difficult for the examiner to uncover.The analysis also suggests that CHATGPT’s more frequent validation through logical questions may contribute to its stronger performance than GPT-3’s.
6 Conclusion
LMVLM detects factuality errors by cross-examining claims through multi-turn interactions that test consistency with other model beliefs. The authors report substantial improvements and identify extensions involving interpretability, multiple examiners, and improved question generation.
- 6 Conclusion: LMVLM performs zero-shot factuality-error detection through prompting between examiner and examinee LMs.The interaction is multi-turn and designed to reveal inconsistencies implying incorrect claims.
- 6 Conclusion: The authors report substantially improved detection of factual errors made by LMs in factual question answering.
- 6 Conclusion: The method tests whether an LM-generated claim is faithful and consistent with several other beliefs generated by the same LM.This connects self-consistency with factual consistency by treating the LM as the information source.
- 6 Conclusion: Future extensions include analyzing related beliefs, using multiple examiner instances, and training examiners to generate better questions.
Limitations
The method has cost, model-capability, and examiner-reliability limitations. It requires multiple LM queries, may work less well with smaller models, and can be affected by logical flaws in examiner operation.
- Limitations: Multiple queries to examinee and examiner LMs can make LMVLM costly when using external APIs.The authors identify this cost as important when scaling the method to many claims.
- Limitations: LMVLM may perform less well with smaller models because success depends mainly on examiners’ instruction-following and long-context reasoning abilities.These abilities are described as mostly demonstrated by models larger than 10B parameters.
- Limitations: Logical flaws in examiner operation may affect the examination and lead to inaccurate decisions.The authors nevertheless report that the method remains useful on average despite such flaws.
A Additional Evaluation
The additional evaluation tests claim acceptance rather than rejection and adds an AYS-plus-LMVLM ensemble. LMVLM generally achieves higher precision and the best F1 scores across the reported settings.
- A Additional Evaluation: The evaluation changes the performance criterion from claim rejection to claim acceptance and introduces an AYS + LMVLM ensemble.Rejected claims from AYS are subsequently evaluated with LMVLM Majority for a final decision.
- A Additional Evaluation: 81.6 precision for CHATGPT on PopQA exceeds baseline values of ≤60, while 68.7 for GPT-3 exceeds baseline values of ≤52.4.
- A Additional Evaluation: LMVLM surpasses the baselines across all settings, obtaining the best F1 scores of 83.3−95.8 for CHATGPT and 74−94.9 for GPT-3.
- A Additional Evaluation: LMVLM maintains good recall relative to the baselines, although AYS obtains the best recall scores.
B Falsehoods Data
The falsehoods data are generated by prompting models to answer questions or complete sentences incorrectly, while also phrasing the output as an argument.
- B Falsehoods Data: For question-answering prompts, models are explicitly asked to provide a wrong answer and phrase it as an argument.
- B Falsehoods Data: For sentence-completion prompts, models are asked to complete the sentence with a wrong answer, which is concatenated with the model answer.
C Example Cross-Examinations
The examples show cross-examination evaluating claims by asking follow-up questions and using the examinee’s answers to reach a correctness decision. Across factual claims, the examiner reaches correct or incorrect judgments for both true and false statements.
- Incorrect claims: The examiner correctly identifies the song claim as incorrect because Ben Findon was one of several co-writers, not the sole writer.The cited examples name Mike Myers and Robert Puzey as additional writers.
- Cross-examination procedure: Cross-examination uses an examiner’s follow-up questions to investigate an examinee’s original claim before issuing a correctness decision.The examples include questions about associated facts, sources, related people, and historical evidence.
- Correct claims: The examiner correctly accepts that Lee Daniels was a producer of Precious after the examinee lists multiple producers and related supporting information.The listed producers include Lee Daniels, Gary Magness, Sarah Siegel-Magness, Oprah Winfrey, Tom Heller, and Tyler Perry.
- Incorrect claims: The examiner rejects the claims that an octahedron has eight vertices and that Augustus was the first Roman Emperor to sport a beard.The Augustus examination cites the absence of historical depictions showing him with a beard and his association with a clean-shaven appearance.
- Correct claims: The examiner accepts that Asmara is Eritrea’s capital after the examinee states that the government recognizes no other official capital.The example explicitly concludes that the original claim is correct.
- Incorrect claims: The examiner rejects the claim that Igor Stravinsky was born in Zagreb after the examinee answers that he was born in Oranienbaum, Russia.The cross-examination also asks about Stravinsky’s parents’ nationality.