Source-linked AI summary
Integrating Stance Detection and Fact Checking in a Unified Corpus
Ramy Baly, Mitra Mohtarami, James Glass, Lluis Marquez, Alessandro Moschitti, Preslav Nakov
TL;DR
Existing fact-checking datasets do not integrate document retrieval, stance detection, rationale extraction, and fact checking, despite their interdependence. This paper introduces the first corpus unifying these annotations for Arabic and reports benefits for stance detection and fact checking, while leaving source reliability and other factors for future work.
Problem
Existing datasets do not integrate document retrieval, stance detection, rationale extraction, and fact checking, despite the interdependence of these tasks and the need for explanatory evidence.
Method
The paper builds an Arabic corpus by retrieving documents for claims and annotating document stance, supporting rationales, and related fact-checking information.
Results
The unified annotations, particularly gold rationales, benefit both stance detection and fact checking.
Takeaways & Limitations
Integrating these annotations in one corpus supports joint study of stance detection, rationale extraction, document retrieval, and fact checking.
Takeaways & Limitations
Future work must address source reliability, language style, and temporal information.
Abstract
from arXiv · showhide
A reasonable approach for fact checking a claim involves retrieving potentially relevant documents from different sources (e.g., news websites, social media, etc.), determining the stance of each document with respect to the claim, and finally making a prediction about the claim's factuality by aggregating the strength of the stances, while taking the reliability of the source into account. Moreover, a fact checking system should be able to explain its decision by providing relevant extracts (rationales) from the documents. Yet, this setup is not directly supported by existing datasets, which treat fact checking, document retrieval, source credibility, stance detection and rationale extraction as independent tasks. In this paper, we support the interdependencies between these tasks as annotations in the same corpus. We implement this setup on an Arabic fact checking corpus, the first of its kind.
1 Introduction
Fact checking requires retrieving documents, assessing their stance and source reliability, and aggregating evidence into a factuality prediction. The paper targets the gap between these interdependent tasks by creating an Arabic corpus with stance and rationale annotations.
- Automation is needed because false claims, clickbait, hoaxes, and satire are produced faster than fact-checking organizations can address them.
- The paper targets Arabic, whereas most previous fact-checking research has focused on English.
- Fact-checking systems retrieve relevant documents, determine document stance toward claims, and aggregate stance strength while considering source reliability.
- Existing datasets do not provide an integrated corpus supporting the interdependence between fact checking and stance detection.
- The corpus identifies stance-marking sentences so systems can provide document extracts that explain their decisions.
2 Related Work
Prior work largely treats fact checking, stance detection, evidence extraction, and credibility as separate or limited resources. This paper distinguishes its corpus through Arabic, naturally retrieved claims and documents, and annotated supporting evidence.
- Existing datasets commonly study fact checking, stance detection, and evidence extraction separately rather than as one integrated resource.
- Non-English fact-checking datasets are rare and often use translated English material focused on US politics.
- The corpus uses claims relevant to the Arab world that were originally made in Arabic.
- Retrieving documents for each claim produces a stance-label distribution argued to be more natural than FNC's article-claim pairs.
- Annotated evidence for agree and disagree labels supports supervised stance or fact-checking models and user-facing explanations.
- FEVER differs through artificially altered Wikipedia claims, a Wikipedia-only knowledge base, and identical stance and factuality labels.
3 The Corpus
The corpus combines factuality-labeled Arabic claims with retrieved documents, stance labels, and supporting evidence. It was assembled from VERIFY and REUTERS, filtered and annotated through multiple stages, and shows a stance distribution with challenges for factuality prediction.
- The corpus labels claims as true or false and claim-document pairs as agree, disagree, discuss, or unrelated.
- Claims came from VERIFY for false claims and REUTERS headlines for true claims, with topical matching across sources.
- The corpus excludes multimedia claims and filters out rhetorical statements, multiple facts, accusations, and denials.
- 1,381 initial claims were reduced to 422 worth-checking claims: 219 false from VERIFY and 203 true from REUTERS.
- Documents were retrieved with Google custom search using claim sub-queries built from high-TF.DF entities, adjectives, nouns, and verbs.
- Arabic speakers annotated claim-document stance with four labels, using 3–5 annotators per pair and majority voting with performance-based tie breaking.
- Very few documents disagree with true claims, while documents agreeing with false claims remain numerous, complicating factuality prediction from stance.
4 Experiments and Evaluation
The experiments evaluate FNC stance-detection systems on the Arabic corpus and examine how document representations and gold rationales affect performance. The corpus is harder than FNC, while rationale-based representations improve stance and fact-checking results.
- The FNC baseline uses a gradient boosting classifier with hand-crafted polarity, refute, similarity, and overlap features.
- The evaluation applies FNC stance-detection models to the Arabic corpus using 5-fold cross-validation and reports accuracy, macro-average F1-score, and weighted accuracy.
- The corpus is harder than FNC: weighted accuracy for the baseline is 55.6 on this corpus versus 75.2 on FNC.The authors attribute this difference to realistic retrieval in their corpus and many easily identifiable unrelated pairs in FNC.
- Gold rationales improve stance detection by 5–12 absolute points over the best-sentence representation and 3–9 points over the full-document representation.Using the full document plus rationale performs best, reaching 80.2% accuracy with ATHENE.
- Gold-rationale results are an upper bound because rationales were available at test time, although the authors expect training-only use could still yield sizable improvements.
- The fact-checker reaches 56.2 accuracy with full documents and 59.7 with the best sentence plus rationale, compared with a 50.5 majority baseline.It aggregates predicted document stances to determine claim factuality.
5 Conclusion and Future Work
The paper presents a unified corpus combining stance detection, stance rationales, relevant-document retrieval, and fact checking. It reports benefits from these annotations and identifies source reliability, language style, and temporal information as future extensions.
- The corpus unifies stance detection, stance rationale, relevant document retrieval, and fact checking, making it the first corpus with this combination for Arabic and generally.
- Unified annotations, especially gold rationales, benefit both stance detection and fact checking in the experiments.
- Future work will cover source reliability, language style, and temporal information.