Source-linked AI summary
Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large Language Models
Sina J. Semnani, Jirayu Burapacheep, Arpandeep Khatua, Thanawan Atchariyachanvanit, Zheng Wang, Monica S. Lam
TL;DR
Wikipedia’s scale makes comprehensive verification difficult, despite the importance of correcting internal contradictions. The paper introduces CLID and CLAIRE, which combines retrieval with LLM reasoning to surface potential contradictions for human review. It estimates that at least 3.3% of Wikipedia facts contradict another fact, with contradictions also present in AmbigQA and FEVEROUS.
Problem
Wikipedia’s scale makes comprehensive verification challenging, while internal contradictions indicate errors requiring correction through consultation of original sources.
Source: §1 · p. 1Method
Results
At least 3.3% of Wikipedia facts contradict another; 4.0% of AmbigQA questions and 7.3% of FEVEROUS Supports claims also contain corpus contradictions.
Source: §1 · p. 2Takeaways & Limitations
Contradictions are measurable in Wikipedia and its dependent NLP benchmarks, challenging assumptions of unambiguous answers and corpus consistency in fact verification.
Source: §1 · p. 2The study focuses exclusively on Wikipedia, leaving other document types, structured data sources, and crosslingual inconsistencies for future work.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Wikipedia is the largest open knowledge corpus, widely used worldwide and serving as a key resource for training large language models (LLMs) and retrieval-augmented generation (RAG) systems. Ensuring its accuracy is therefore critical. But how accurate is Wikipedia, and how can we improve it? We focus on inconsistencies, a specific type of factual inaccuracy, and introduce the task of corpus-level inconsistency detection. We present CLAIRE, an agentic system that combines LLM reasoning with retrieval to surface potentially inconsistent claims along with contextual evidence for human review. In a user study with experienced Wikipedia editors, 87.5% reported higher confidence when using CLAIRE, and participants identified 64.7% more inconsistencies in the same amount of time. Combining CLAIRE with human annotation, we contribute WIKICOLLIDE, the first benchmark of real Wikipedia inconsistencies. Using random sampling with CLAIRE-assisted analysis, we find that at least 3.3% of English Wikipedia facts contradict another fact, with inconsistencies propagating into 7.3% of FEVEROUS and 4.0% of AmbigQA examples. Benchmarking strong baselines on this dataset reveals substantial headroom: the best fully automated system achieves an AUROC of only 75.1%. Our results show that contradictions are a measurable component of Wikipedia and that LLM-based systems like CLAIRE can provide a practical tool to help editors improve knowledge consistency at scale.
1 Introduction
Wikipedia’s scale makes comprehensive accuracy checks difficult, even as internal contradictions can signal errors requiring correction. The paper introduces corpus-level inconsistency detection, CLAIRE, and WIKICOLLIDE, and reports both human-study gains and measurable inconsistencies in Wikipedia and related benchmarks.
1 Introduction
- Corpus-level inconsistency detection seeks another fact in the same corpus that contradicts a given fact, addressing a largely unexplored task.Wikipedia contradictions can arise from outdated information, editors’ limited awareness of related content, or human error; the corpus’s scale makes comprehensive verification challenging.Source: §1 · p. 1
- 64.7% more inconsistencies: eight experienced editors identified more within the same time using CLAIRE than with search engines.CLAIRE presents candidate contradictions with disambiguating context and explanations of specialized terminology.Source: §1 · p. 2
- Approximately 3.3% of English Wikipedia facts contradict other corpus statements; 4.0% of AmbigQA questions and 7.3% of FEVEROUS Supports claims also have contradictions.The paper presents these as lower-bound estimates of inconsistency.Source: §1 · p. 2
- WIKICOLLIDE contains 955 facts, 34.7% of which are inconsistent, and targets Level 5 Vital Articles prioritized for improvement.Unlike synthetic datasets, it captures genuine ambiguities and complex factual relationships in real-world content.Source: §1 · p. 2
- 75.1% AUROC: CLAIRE outperforms baselines on the WIKICOLLIDE test set while leaving substantial headroom.The study also establishes strong baselines on WIKICOLLIDE.Source: §1 · p. 2
2 Related Work
Prior work covers fact verification and inconsistency detection at narrower scopes, while WIKICOLLIDE targets naturally occurring contradictions across Wikipedia’s corpus. This setting requires searching for and aggregating evidence across multiple articles.
2 Related Work
- Fact-verification systems commonly retrieve relevant passages and use language models to assess whether claims align with them.Work on claim extraction and decomposition has also addressed breaking claims into smaller facts.Source: §2 · pp. 2–3
- Many Wikipedia fact-verification datasets create Refutes examples by synthetically modifying true statements, whereas WIKICOLLIDE captures naturally occurring corpus contradictions.WikiContradict (Hou et al., 2024) uses editor-added inconsistency tags, but many tagged cases have since been resolved.Source: §2 · p. 2
- WikiContradiction (Hsu et al., 2021) focuses on contradictions within one article, while WIKICOLLIDE extends the scope across the corpus.The corpus-level setting adds the challenge of searching for and aggregating evidence across multiple articles.Source: §2 · p. 2
3 Corpus-Level Inconsistency Detection (CLID)
CLID classifies atomic facts according to whether another item in the corpus contradicts them. Unlike typical fact verification, it requires finding contradictory evidence or exhaustively checking that none exists across the corpus.
3 Corpus-Level Inconsistency Detection (CLID)
- CLID is a binary classification task over atomic facts, which are short, self-contained statements independently verifiable as single pieces of information.An atomic fact is corpus-level inconsistent if at least one other item in the corpus contradicts it; otherwise, it is consistent.Source: §3 · p. 3
- CLID asks whether documents in the corpus contain evidence contradicting an atomic fact.Its formal objective is to determine whether a subset of corpus documents contains such evidence.Source: §3 · p. 3
- Unlike fact verification, which typically assumes corpus consistency, CLID requires finding refuting evidence or exhaustively checking the corpus for contradictions.Fact verification typically treats any supporting or refuting evidence as sufficient.Source: §3 · p. 3
4 CLAIRE: A Human-in-the-Loop Assistant for Corpus-Level Inconsistency Detection
CLAIRE interleaves retrieval and LLM verification to find potential contradictions, while clarifying ambiguous entities and specialized terms for users. In a study, editors found more inconsistencies per hour and reported greater confidence with the tool.
4 CLAIRE: A Human-in-the-Loop Assistant for Corpus-Level Inconsistency Detection
- CLAIRE interleaves evidence retrieval and verification, allowing verification insights to guide further retrieval.It uses a ReAct architecture; an inconsistency score from 0 to 1 helps users prioritize candidates because LLMs alone do not reliably verify inconsistencies at high accuracy.Source: §4 · p. 3
- Clarification disambiguates entities using additional context, while explanation makes specialized terminology more accessible to users.The tool also presents candidate contradictions with supporting evidence and explanations through a browser extension.Source: §4 · p. 3, §4.1 · p. 4
- Unlike fact verification, inconsistency detection continues searching for contradictory evidence within the corpus after finding supporting evidence.Figure 1 illustrates the distinction with a FEVEROUS example.Source: Figure 1 · p. 4
4.1 User Study
- Editors identified 64.7% more inconsistencies per hour using CLAIRE.The study involved eight experienced Wikipedia editors completing randomized 30-minute tasks with and without the extension.Source: §4.1 · p. 4
- 87.5% of participants reported increased confidence when using CLAIRE.Editors particularly valued contradictions surfaced across article boundaries, which typically require extensive manual cross-referencing.Source: §4.1 · p. 4
5 Inconsistency Rates in the English Wikipedia and NLP Datasets
The analysis estimates a lower-bound share of inconsistent Wikipedia facts and finds inconsistencies in both article categories and question-answering datasets. These findings challenge the assumption that Wikipedia provides a consistent source of truth for fact verification.
5 Inconsistency Rates in the English Wikipedia and NLP Datasets
- An estimated 3.3% ± 1.7% [1.6%, 5.0%] of Wikipedia facts contradict other information in the corpus, with 99% confidence.This is a lower bound based on 700 uniformly sampled atomic facts from the November 1, 2024 dump; 23 of 44 potentially inconsistent facts were manually confirmed.Source: §5 · p. 5
Inconsistency rates vary across article cat-
- History articles had the highest inconsistency rate (17.7%), followed by Everyday Life (16.9%) and Society & Social Sciences (14.3%).Mathematics (5.6%) and Technology (9.4%) had markedly lower rates; numerical discrepancies were the most common error type in history articles.Source: Inconsistency rates vary across… · p. 5
- 4.0% ± 1.1% of AmbigQA examples contradicted information elsewhere in the corresponding Wikipedia dump.The question-answer pairs were converted into declarative facts for the analysis.Source: Inconsistency rates vary across… · p. 5
- 7.3%±0.5% of FEVEROUS claims labeled Supports were involved in corpus-level inconsistencies.Their verification outcome depends on which Wikipedia article is chosen as evidence, challenging fact verification's assumption of a consistent corpus.Source: Inconsistency rates vary across… · p. 5
6 WIKICOLLIDE: A Dataset for Corpus-Level Inconsistency Detection
WIKICOLLIDE is a human-annotated benchmark of real Wikipedia inconsistencies, built with CLAIRE-assisted analysis. It documents evidence and reasoning, while its analysis shows numerical discrepancies are the largest inconsistency category.
6 WIKICOLLIDE: A Dataset for Corpus-Level Inconsistency Detection
- The benchmark contains real, previously unknown Wikipedia inconsistencies, rather than relying on synthetic contradictions.Inconsistent facts include verified evidence documents, reasoning, and inconsistency categories; consistent facts include up to 40 reviewed Wikipedia evidence passages.Source: §6 · p. 5
- The dataset covers diverse topics and includes implicit contradictions, temporal conflicts, and divergent interpretations.Representative examples involve multi-hop reasoning, calculations, contextual interpretation, entity disambiguation, and domain expertise.Source: §6 · p. 5
6.1 Dataset Construction
- Construction sampled popular pages, extracted atomic facts, then filtered for facts more likely to be inconsistent.From 10,000 sampled text blocks, GPT-4o extracted 89,300 facts; retrieval and LLM filtering reduced candidates to 1,880.Source: §6.1 · p. 6
- Annotators reviewed CLAIRE's gathered evidence and reasoning; consistent labels represent strong verification, not proof that no contradiction exists.For each fact labeled consistent, annotators reviewed an average of 21 potential evidence passages.Source: §6 · p. 5, §6.1 · p. 6
- WIKICOLLIDE contains 955 manually labeled atomic facts, of which 34.7% are inconsistent.The dataset divides facts evenly between validation and test sets, with consistent and inconsistent labels randomly distributed.Source: Table 1 · p. 5, §6.1 · p. 6
6.2 Analysis of WIKICOLLIDE
- Numerical discrepancies made up 54.7% of inconsistencies, and 42% of those discrepancies were off-by-one errors.Logical contradictions accounted for 17.5%; the remaining 27.8% involved differing definitions, temporal or spatial conflicts, entity disambiguation errors, and divergent categorizations.Source: §6.2 · p. 6
7 Evaluating Automatic Corpus-Level Inconsistency Detectors using WIKICOLLIDE
On WIKICOLLIDE, CLAIRE outperforms other systems on test Accuracy and AUROC, while error analysis and ablations expose remaining challenges and performance differences across system configurations.
7.1 Evaluated Systems
- Systems include CLAIRE and retrieve-and-verify/NLI pipelines; experiments compare LLM backbones using shared retrieval resources and comparable evidence budgets.Retrieve-and-verify scores all retrieved evidence in one LLM call; the NLI pipeline assesses each passage individually and flags a fact if any passage contradicts it.Source: §7.1 · pp. 6–7, §7.2 · p. 7
7.3 Evaluation Metrics
- AUROC is primary because false positives waste review effort and false negatives miss inconsistencies; accuracy and F1 are also reported.ROC curves vary inconsistency-score thresholds for retrieve-and-verify and CLAIRE, and the required number of contradictory passages for NLI.Source: §7.3 · p. 7
7.4 Results
- On the test set, CLAIRE outperforms other systems by at least 0.3 Accuracy points and 2.1 AUROC points.Table 3 reports GPT-4o performance on the WIKICOLLIDE validation and test sets; test thresholds are selected to maximize validation F1.Source: §7.4 · p. 7, Table 3 · p. 7
7.5 Error Analysis
- Systems can confuse same-named entities or flag acceptable discrepancies involving rounding, language variants, different time periods, or perspective and belief.These cases illustrate context-dependent false positives: apparent differences do not always indicate genuine inconsistencies.Source: §7.5 · pp. 7–8
Legitimate variation in scholarly interpre-
- Apparent contradictions about historical events or scientific classifications may reflect evolving scholarly consensus rather than genuine inconsistencies.This is a separate form of legitimate variation identified in the error analysis.Source: Legitimate variation in scholarly… · p. 8
7.6 Ablation Studies
- The highest F1 occurs when CLAIRE has both explain and clarify tools; more retrieved documents generally improve performance, but by less than 3%.The ablations vary available agent tools, thought-action-observation steps, and documents returned per query.Source: §7.6 · p. 8
- RankGPT reranking consistently improves all systems and metrics on the validation set.Table 4 compares performance with and without reranking.Source: §7.6 · p. 8, Table 4 · p. 8
- On validation, GPT-4o achieves the highest scores, o3-mini is competitive with generally higher precision, and Llama-3.1-70B underperforms.With the same prompt, o3-mini rarely produces intermediate inconsistency scores, instead clustering at the extremes.Source: §7.6 · p. 8, Table 5 · p. 8
8 Conclusion
The paper introduces corpus-level inconsistency detection and CLAIRE, an agent system that combines retrieval with LLM reasoning to surface and contextualize potential contradictions. It reports that such contradictions are measurable at Wikipedia scale and that automated detection can support human review.
8 Conclusion
- CLAIRE combines retrieval with LLM reasoning to detect and contextualize potential contradictions for human review.The paper introduces corpus-level inconsistency detection as the task of identifying contradictory information within large knowledge repositories.Source: §8 · p. 8
- WIKICOLLIDE captures real inconsistencies that synthetic datasets often miss.Experiments on the dataset show that retrieval and verification remain challenging.Source: §8 · p. 8
- Approximately 3.3% of Wikipedia facts conflict with other information in the corpus, amounting to millions of contradictory statements.The authors characterize corpus-level inconsistencies as a measurable phenomenon in large-scale knowledge corpora.Source: §8 · p. 8
- Automated systems exhibit systematic errors, but the authors conclude they can aid in maintaining knowledge consistency at scale.The paper suggests a broader cycle in which LLMs help curate cleaner corpora, which can improve human knowledge access and AI systems.Source: §8 · pp. 8–9
Limitations
The paper evaluates corpus-level inconsistency detection only on Wikipedia, leaving other corpora and cross-language inconsistencies for future work.
Limitations
- The study is limited to Wikipedia; it does not examine technical texts, structured data sources, or inconsistencies across language versions.The authors identify academic, medical, and legal documents, databases, and knowledge graphs as unexamined applications.Source: Limitations · p. 9
Ethical Considerations
The authors report their ethical and resource practices and explain the sampling calculation used to estimate Wikipedia inconsistency rates.
Ethical Considerations
- The study reports no anticipated risks or ethical concerns from publishing WIKICOLLIDE; annotators were compensated at least $16 per hour and study participants $20 per hour.The study received institutional review board approval, obtained informed consent, collected no personally identifiable information, and released the dataset under a license compatible with Wikipedia’s.Source: Ethical Considerations · p. 9
- The experimental cost did not exceed $4,000, while direct carbon-footprint estimation was difficult because the work used commercial LLM APIs.A CPU-based machine served the Wikipedia index.Source: Ethical Considerations · p. 9
A Establishing Lower Bounds for Inconsistency Rates
- At least 664 claims were required for a 99% confidence level and 5% margin of error under the Cochran sample-size calculation.The calculation assumes maximum variance with p = 0.5; the same statistical approach was used for AmbigQA and FEVEROUS sample sizes.Source: A Establishing Lower Bounds for… · p. 12
A.1 Inconsistencies Per Wikipedia Article Category
The category analysis reports uneven inconsistency rates across Wikipedia topics. The accompanying passages describe dataset annotation, study feedback, and the tools and prompts used in the system.
A.1 Inconsistencies Per Wikipedia Article Category
- 17.7% in History, 16.9% in Everyday Life, and 14.3% in Society & Social Sciences were the highest category rates; Mathematics was 5.6% and Technology 9.4%.The authors suggest that narrative-heavy subjects are particularly susceptible to inconsistencies, while quantifiable domains show lower rates.Source: §A.1 · pp. 12–13, Figure 5 · p. 13
B.1 Annotation Tool
- A weak GPT-4o mini baseline filters obviously consistent claims, while a report-generation stage gives annotators two-sided reasoning for final judgments.The binary baseline is a simplified retrieve-and-verify system; report generation uses two GPT-4o calls, one arguing each side, and includes the agent trace.Source: §B.1 · p. 13
B.2 Annotators
- Seventeen qualified annotators were recruited; they averaged 6.5 minutes evaluating each fact.Requirements included native English fluency in the US or UK, a graduate degree, and a Prolific approval rate of at least 95%.Source: §B.2 · p. 13
B.3 Annotation Guidelines
- Inconsistency judgments can depend on whether rounding is acceptable or an imprecise phrase can be matched to a specific value.The authors note that thresholds are not always clear and established explicit guidelines to address such ambiguity.Source: §B.3 · pp. 13–14
C.1 Distribution of the Topics
- The dataset passages point readers to a topic-distribution figure and examples pairing claims with evidence.The figures are described as showing covered topics and detailed claim-and-evidence examples.Source: §C.1 · p. 14, §C.2 · p. 14
D.1 Browser Extension Implementation
- The browser extension highlights potentially inconsistent claims on Wikipedia pages and provides explanations in a side panel.It uses a JavaScript frontend and Python backend; GPT-4o extracts atomic claims from page content.Source: §D.1 · p. 14, Figure 13 · p. 19
D.3 Perceived Usefulness and Qualitative Feecback
- Editors said manual cross-checking is time-consuming and cognitively cumbersome; they valued accessibility and AI-content verification but raised speed, false-positive, and interface concerns.The authors describe a tradeoff between reducing editorial burden and balancing detection sensitivity with efficiency and usability.Source: §D.3 · pp. 15–16
E.1 Implementation of Explain and Clarify Tools
- Clarify retrieves information to disambiguate entities, while explain prompts the model to provide topic background; the prompt suite includes fact extraction and agent tools.The prompts mark boundaries for few-shot examples where applicable.Source: §E.1 · p. 16, §E.2 · p. 19