Source-linked AI summary
CoVer: Conflict-Aware Claim Verification
Shuning Zhang, Dai Shi, Bohao Chu, Hui Wang, Yuwei Chuai, Yifan Wang, Jingruo Chen, Simin Li, Xin Yi, Hewu Li
TL;DR
Social media fact-checking must handle adversarial conflicts among evidence sources and contradictory subclaim signals. The paper introduces ContraNote and CoVer, a structured adjudication framework that prioritizes evidence over noise. CoVer reports strong performance across conflict, prioritization, and CONFACT evaluations, while its current scope excludes multimodal evidence and remains limited in some languages and domains.
Problem
Automated social media fact-checking faces evidence-level conflicts between sources and aggregation-level conflicts among contradictory subclaim signals.
Method
CoVer normalizes evidence schemas, establishes factual consensus, and verifies support in a structured pipeline, alongside the ContraNote dataset from X’s Community Notes.
Results
CoVer achieves strong performance across benchmarks, including 86.0% accuracy on ContraNote Conflict, 88.5% on Prioritization, and 88.4% and 89.4% on CONFACT-HumC and CONFACT-ModC.
Takeaways & Limitations
ContraNote supplies real-world conflict and prioritization testbeds, while CoVer prioritizes evidence over noise during conflicting-evidence adjudication.
Takeaways & Limitations
The framework and dataset focus predominantly on English textual claims and metadata, excluding multimodal conflicts and requiring validation in low-resource languages and specialized domains.
Abstract
from arXiv · showhide
Social media fact-checking has long been challenged by evidence-level and aggregation-level conflicts, where erroneous evidence mimics authoritative news sources. To capture this challenge and support conflict verification tasks, we present ContraNote, a large-scale real-world dataset curated from X's Community Notes system. It includes 33,686 posts for evaluating evidence-level conflict resolution, and 54,474 instances for evaluating aggregation-level prioritization. Additionally, we propose CoVer, a factual adjudication framework with three-stage pipelines: evidence schema normalization, factual consensus and support verification. This prioritizes evidence over noise to prevent it from compromising the final verdict. Technical evaluations show that CoVer achieves strong performance compared with state-of-the-art baselines across ContraNote (86.0% Acc., 68.0% mac. F1, 64.5 bal. Acc. on Conflict; and 88.5% Acc., 88.5 mac. F1 and 89.2 bal. Acc. on Prioritization), CONFACT-HumC (88.4% Acc.) and CONFACT-ModC (89.4% Acc.).
1 Introduction
Automated social media fact-checking must resolve both conflicting evidence sources and contradictory subclaim signals. CoVer addresses these challenges with structured evidence adjudication, while ContraNote provides real-world evaluation tasks derived from X’s Community Notes.
- Evidence-level conflicts arise when retrieved sources take opposing stances on one fact, while aggregation-level conflicts arise from divergent subclaim verdicts.
- Social media conflicts are often adversarial and popularity-biased, limiting frequency-based aggregation methods.
- 33,686 posts form ContraNote’s conflict task, evaluating support or refutation from opposed Community Notes.
- CoVer uses evidence schema normalization, factual consensus, and support verification to prioritize evidence over noise.
- 86.0% accuracy and 64.5% balanced accuracy were achieved on ContraNote Conflict, while Prioritization reached 88.5% accuracy and 89.2% balanced accuracy.
2 Background and Related Work
Prior work spans community-based and automated fact-checking, truth discovery, knowledge-conflict analysis, and conflict-resolution frameworks. CoVer differs by focusing specifically on adjudicating conflicting evidence rather than merely retrieving, verifying, or aggregating information.
- Community-based and automated fact-checking emerged partly in response to expert-based methods’ scalability, selection-bias, and public-trust challenges.
- Automated fact-checking research addresses suspicious claims, verification, explanation generation, and community-note synthesis.
- Truth discovery: Truth-discovery methods estimate source reliability, whereas this work models consensus among conflicting evidence items.
- Taxonomy and biases in knowledge conflicts: LLM knowledge conflicts include intra-context, inter-context, and parametric discrepancies, alongside confirmation, self-generation, and selection biases.
- Detection and resolution frameworks: Resolution research uses factual-consistency models, multi-agent debates, contrastive argument synthesis, calibration, and uncertainty estimation.
3 Problem Definition
The problem formalizes fact-checking when evidence documents or decomposed subclaims yield contradictory signals. The objective is to learn a verification function that maps a claim and its conflicting evidence to an overall verdict.
- Evidence-level conflict occurs when documents for one subclaim simultaneously support and refute it.
- Aggregation-level conflict occurs when local subclaim verdicts have opposing logical implications for the overall claim.
- A correct attribution alongside false causality exemplifies mixed signals requiring synthesis into a coherent conclusion.
4 ContraNote Dataset
ContraNote is built from conflicting viewpoints in X’s Community Notes and supports separate conflict and prioritization benchmarks. Its analyses show heterogeneous evidence sources, diverse disagreement patterns, and high label reliability.
- ContraNote uses X’s open Community Notes repository, where multiple notes on one post can express divergent viewpoints.
- 2,276,724 notes covering 1,502,486 posts were filtered to identify 443,148 posts with multiple notes and stance contrast.
- ContraNote Conflict contains 33,686 posts: 27,445 Refuted and 6,241 Supported instances defined by note stance and helpfulness criteria.
- The dataset analysis identifies missing context with 22,615 instances and factual errors with 20,918 as primary misleading reasons.
- Refuting notes average 1.77 per post and 289.08 characters, exceeding supporting notes in detail and length.
- News accounts for 49.1% and social media platforms 24.6% of cited sources, indicating heterogeneous evidence requiring provenance assessment.
- Note-necessity disagreement is the primary conflict pattern at 33%, while English is the majority language at 62.98%.
- Dataset labels achieved Fleiss’ κ =0.82 and aligned with majority-vote annotations in 94.2% of sampled cases.
5 CoVer
CoVer predicts claim support or refutation through evidence schema normalization, factual consensus, and support verification. Its structured adjudication preserves evidence cues, filters noise, and applies deterministic aggregation before final verification.
- 5 CoVer: CoVer predicts Supported or Refuted labels using evidence schema normalization, factual consensus, and support verification.The framework operates on a claim and an evidence set.
- 5 CoVer: Evidence schema normalization preserves textual content and schema-level cues across QA, source-pointer, candidate-note, and fact-checking evidence.The retained fields include answer, page and line identifiers, note status and helpfulness, and article text or snippets.
- 5 CoVer: Factual consensus jointly predicts each evidence item’s stance and four quality components under structured output constraints.The quality dimensions are directness, attribute alignment, schema reliability, and informativeness, with scores normalized to [0, 1].
- 5 CoVer: CoVer computes quality-weighted supporting and refuting strengths, selects a dominant stance, and filters consensus evidence using a quality threshold.Refute breaks ties to avoid unsupported positive predictions.
- 5 CoVer: An additional adjudication call maps consensus relations to Supported or Refuted, while support verification runs only after a Supported consensus.If consensus is Refuted, the algorithm terminates; otherwise, selected supporting evidence must independently validate the claim.
6 Experiments
Experiments evaluate CoVer across conflict-resolution, prioritization, and fact-checking datasets using controlled evidence settings, baseline comparisons, ablations, and efficiency analysis. CoVer generally outperforms baselines, while removing key modules substantially reduces performance.
- Main Results: CoVer surpasses baselines on CONFACT, ContraNote Conflict, ContraNote Prioritization, FEVER, and ECON.It achieves 88.4% accuracy on CONFACT-HumC, 89.4% on CONFACT-ModC, 86.0% on ContraNote Conflict, 88.5% on ContraNote Prioritization, 93.4% on FEVER, and 77.5% on ECON.
- Main Results: 69.6 mac. F1 is achieved by CoVer with retrieval enabled on ContraNote Conflict, versus 59.2 mac. F1 for the strongest baseline.This evaluation pairs each method with an upstream retriever and evaluates end-to-end claim-level predictions.
- Main Results: 81.5% accuracy for the metadata shortcut baseline on ContraNote Conflict is accompanied by 44.9 mac. F1 and 50.0 bal. Acc.Its high accuracy is attributed to class imbalance, while mac. F1 and bal. Acc. are by chance.
- Ablation Study: Removing factual consensus drops ContraNote Prioritization accuracy from 88.5% to 42.5% and ContraNote Conflict accuracy from 86.0% to 69.0%.The ablation targets tasks requiring nuanced arbitration.
- Ablation Study: Removing evidence schema normalization reduces ECON accuracy from 77.5% to 45.5%, with further declines on FEVER and ConflictBank.FEVER falls from 93.4% to 82.4%, while ConflictBank falls from 73.5% to 68.8%.
- Ablation Study: Removing support verification lowers ContraNote Prioritization accuracy from 88.5% to 68.5%, while removing all configuration lowers it to 49.7%.The all-configuration ablation also reduces CONFACT-HumC accuracy to 80.6%.
7 Conclusion
The paper addresses evidence-level and aggregation-level conflicts in automated social media fact-checking by introducing CoVer and ContraNote. CoVer combines evidence structuring, factual consensus, and support verification, and performs strongly on the two ContraNote tasks.
- The paper introduces CoVer to resolve evidence-level and aggregation-level conflicts through evidence schema normalization, factual consensus, and support verification.
- ContraNote contains 33,686 items for conflict resolution and 54,474 items for evidence prioritization.
- 86.0% and 88.5% accuracies are achieved on the ContraNote Conflict and Prioritization tasks, respectively.The corresponding balanced accuracies are 64.5% and 89.2%.
8 Limitations
The framework and dataset have limited language and modality coverage, rely partly on crowdsourced consensus, and are primarily evaluated with gold evidence. Statistical power also limits some claims on ContraNote Conflict.
- Scope and coverage: CoVer and ContraNote focus on textual claims and metadata, predominantly in English, leaving low-resource languages and specialized domains insufficiently validated.The setting excludes manipulated images, deepfakes, and out-of-context videos.
- Label validity: ContraNote ground truth reflects crowdsourced consensus and algorithmic helpfulness rather than absolute factual truth, with possible rating manipulation and shared annotator assumptions.These factors may introduce cultural or ideological bias into labels.
- Evaluation setting: Primary experiments use gold evidence to isolate adjudication from retrieval, while the end-to-end retrieval-enabled experiment is limited in scale.The authors therefore position CoVer as complementary to retrieval systems.
- Statistical power: Some pairwise improvements on ContraNote Conflict do not reach statistical significance, limiting the strength of the claims.The authors estimate that larger samples would be needed for key comparisons.
9 Ethical Considerations
The paper frames automated fact-checking as an information-integrity tool with risks of misclassification and obligations concerning data privacy and platform terms.
- Information integrity: Automated fact-checking errors can suppress accurate information or inadvertently spread misinformation through incorrect verdicts.No automated system is infallible.
- Information integrity: CoVer should currently be treated as a decision-support tool rather than an authority.
- Data privacy: ContraNote was derived from public X Community Notes data through the official X API, and reuse must comply with platform terms and creator privacy.
B.2 Dataset Analysis
ContraNote analyzes real-world evidence conflicts across sources, languages, note relations, and dataset labels, while extending prior fact-checking benchmarks with distinct conflict tasks.
- Flagging reasons: 22,615 instances cite missing context and 20,918 cite factual errors as the leading reasons for misleading flags.Other categories include unverified claims, outdated information, satire, manipulated media, and other reasons.
- Note characteristics: Refuting notes average 1.77 per post and 289.08 characters, versus 1.61 notes and 163.64 characters for supporting notes.
- Evidence sources: 49.1% of cited links come from web and news pages, while social-media and video platforms contribute 24.6%.The remaining links span Wikipedia, official sources, fact-checking sites, and other sources.
- Conflict relations: 33% of manually coded cases show note-necessity disagreement, where competing notes contest moderation necessity rather than establish factual truth.
- Language distribution: English accounts for 62.98% of posts with successfully retrieved text, followed by Spanish at 8.04% and Portuguese at 6.25%.Other posts use French, Japanese, German, Chinese, and other languages.
- Benchmark design: ContraNote separates evidence-level and aggregation-level conflicts, with Conflict labels for claims and Prioritization labels for competing evidence.This extends earlier fact-checking datasets with naturally occurring conflicts within the same post.
- Label validation: Human labels agreed with ContraNote labels on 94.2% of 500 sampled instances, with substantial interannotator agreement of Fleiss’ κ = 0.82.The sampled judgments used posts, notes, and cited evidence.
- Dataset bias: ContraNote may exhibit population and ideological skew because Community Notes contributors may not represent all demographics.The paper reports note proportions of 18.6% left, 47.0% center, and 34.4% right.
C.2 Additional Results on Baselines
Additional experiments compare CoVer with diverse baselines, test controls and extensions, and evaluate rationale factuality, weighting sensitivity, and statistical inference choices.
- Baseline comparisons: ClaimDecomp reaches macro-F1 scores of 67.28 on CONFACT-HumC, 76.21 on CONFACT-ModC, 54.29 on ContraNote Conflict, and 65.58 on Prioritization.MADAM-RAG and an AVeriTeC-style baseline provide additional ContraNote comparisons.
- Additional datasets: CoVer obtains macro-F1 scores of 92.60 on AVeriTeC, 84.00 on WikiContradict, and 92.20 on AmbiFC.The authors note that these datasets differ in task formulation and evidence structure.
- Shortcut controls: 81.5% shortcut accuracy on ContraNote Conflict falls to 44.9 macro-F1 and 50.0 balanced accuracy, indicating class imbalance rather than reliable verification.The same rule reaches 50.0% accuracy on Prioritization.
- Three-way verification: 80.6% accuracy and 68.0 macro-F1 are achieved in the three-way setting, compared with 66.5 macro-F1 for the strongest baseline.The labels are Supported, Partially Supported, and Refuted.
- Rationale factuality: CoVer rationale support rates are 85.2% on ContraNote Conflict and 82.3% on Prioritization, versus 76.9% and 72.6% for strongest baselines.The automatic procedure agreed with human annotations on 97% of 100 manually annotated instances for GPT-5.5.
- Decision controls: Changing the tie default decreases performance by 2.15 percentage points on Prioritization but improves it by 4.19 points on CONFACT-HumC.The effect of tie-breaking is dataset-dependent.
- Ablation controls: Removing structured-output constraints reduces accuracy by 1.57 percentage points relative to the constrained configuration.
E.2 Error Analysis
CoVer’s qualitative error analysis identifies three principal failure categories: temporal ambiguity, domain-expertise requirements, and unsupported implications attached to real events. The authors propose corresponding improvements through temporal normalization, expert routing, and finer-grained decomposition.
- Failure categories: Three principal failure categories emerge from CoVer’s qualitative analysis.The analysis does not assign category percentages because the available coding record lacks a frequency table.
- Temporal ambiguity: Temporal ambiguity causes CoVer to select outdated interpretations or fail to restrict verdicts to the claim’s intended period.Claims and notes may concern different stages of an evolving event or omit the relevant timeframe.
- Domain expertise: Specialized legal, medical, scientific, or policy cases may cause CoVer to overweight fluent explanations when expert interpretation is required.The model can identify competing stances, but the technical distinction may not be explicit in the supplied evidence.
- Unsupported implications: CoVer sometimes treats evidence for a real event as support for an attached causal, intentional, or generalized implication.This motivates conservative support verification, although tightly coupled factual and implied components remain difficult.
- Improvement directions: The proposed improvement directions are explicit temporal normalization, domain-aware or expert routing, and finer-grained separation of event facts from implications.The authors present these as directions for improvement rather than quantified findings.