Source-linked AI summary
ClaimRank: Detecting Check-Worthy Claims in Arabic and English
Israa Jaradat, Pepa Gencheva, Alberto Barron-Cedeno, Lluis Marquez, Preslav Nakov
TL;DR
Manual fact-checking is time-consuming, and identifying claims worth checking is its first automated step. ClaimRank detects check-worthy claims using annotations from nine organizations and supports English and Arabic. Though trained on political debates, it can also process interviews and news articles.
Problem
Manual fact-checking is effort- and time-consuming, and automated fact-checking first needs to identify claims worth checking.
Source: §1 · p. 1Method
ClaimRank detects check-worthy claims using annotations from nine fact-checking organizations, supporting their individual selection strategies or their union in English and Arabic.
Source: §1 · p. 1Results
Originally trained on political debates, ClaimRank can also work on interviews and news articles and was adapted to Arabic using translated training data and cross-language embeddings.
Source: §5 · p. 4Takeaways & Limitations
ClaimRank can help fact-checkers and journalists prioritize which claims to consider first.
Source: §5 · p. 4The reported results are not comparable to Gencheva et al. (2017) because the evaluation setup differs in data split, debates, and disabled features.
Source: §3.5 · p. 3
Abstract
from arXiv · showhide
We present ClaimRank, an online system for detecting check-worthy claims. While originally trained on political debates, the system can work for any kind of text, e.g., interviews or regular news articles. Its aim is to facilitate manual fact-checking efforts by prioritizing the claims that fact-checkers should consider first. ClaimRank supports both Arabic and English, it is trained on actual annotations from nine reputable fact-checking organizations (PolitiFact, FactCheck, ABC, CNN, NPR, NYT, Chicago Tribune, The Guardian, and Washington Post), and thus it can mimic the claim selection strategies for each and any of them, as well as for the union of them all.
1 Introduction
Automated fact-checking begins by identifying claims worth checking, since manual fact-checking is effort- and time-consuming. ClaimRank detects such claims and learns selection strategies from nine fact-checking organizations, supporting English and Arabic.
- Manual fact-checking is effort- and time-consuming, motivating automated systems.Source: §1 · p. 1
- Identifying claims worth fact-checking is the first step toward an automated fact-checking system.Source: §1 · p. 1
- ClaimRank supports English and Arabic and can predict claim selections from each of nine fact-checking organizations or their union.The authors describe it as the only such system they are aware of for Arabic and for mimicking individual organizations’ strategies.Source: §1 · p. 1
2 Related Work
Related work includes systems that target claim check-worthiness and prior efforts using fact-checker annotations. The authors distinguish ClaimRank by its features, language support, organization-specific modeling, and deployed system.
- ClaimBuster was the first work to target check-worthiness.It was trained on annotations by students, professors, and journalists and used sentiment, TF.IDF-weighted words, part-of-speech tags, and named entities.Source: §2 · p. 1
- Compared with ClaimBuster, ClaimRank uses richer features, supports English and Arabic, learns from nine reputable organizations, and mimics each organization’s selection strategy.Source: §2 · p. 1
- The authors’ previous debate-focused work used four English debates; ClaimRank adds debates, Arabic translations, Arabic support, and a deployed system.The current work uses seven English debates and Arabic translations of two debates.Source: §2 · p. 1
- Patwari et al. (2017) also studied the 2016 US Election campaign using similarly obtained data, but used fewer features and did not mimic website-specific strategies or deploy a system.Source: §2 · p. 1
3 System Overview
ClaimRank processes submitted text into sentence-level check-worthiness scores and provides English and Arabic handling; the final system uses MUSE embeddings, reported as best overall.
3.1 General Architecture
- Submitted text is language-detected, split into sentences, scored individually, and optionally reordered to mimic a source’s selection strategy.Scores appear beside sentences with color codes, and the sentence list and scores are stored together in the session.Source: §3.1 · p. 2
3.2 Features
- Features combine TF.IDF-weighted words, POS tags, named entities, sentiment, sentence length, lexicons, topics, embeddings, and discourse relations.Structural features include sentence location within a debate or intervention.Source: §3.2 · p. 2
3.3 Model
- The English model reuses a two-hidden-layer neural network with contextual features, ReLU layers, and a sigmoid check-worthiness classifier.The model’s positive-class probability supplies the ranking score; it is trained for 100 iterations using stochastic gradient descent.Source: §3.3 · p. 2
3.4 Adaptation to Arabic
- Arabic support uses custom sentence splitting, Farasa tokenization and segmentation, and mapping Arabic and English POS tags to the Universal tagset.Segmentation addresses Arabic conjunctions and clitics attached to words, which otherwise increase vocabulary size and data sparseness.Source: §3.4 · pp. 2–3
3.5 Evaluation
- Evaluation trains on five English debates and tests on two debates or their Arabic translations, comparing VecMap, MUSE, and Attract-Repel embeddings.VecMap and MUSE project embeddings into a joint space; Attract-Repel uses pretrained English-Arabic embeddings.Source: §3.5 · p. 3
- All results exceed a random baseline; MUSE is best for Arabic, whose MAP is competitive with English, while Attract-Repel performs better for English.The authors associate Attract-Repel’s weaker Arabic results with embeddings trained on unsegmented Arabic, and its stronger English results probably with monolingual synonymy/antonymy constraints.Source: §3.5 · p. 3
- These results are not comparable to Gencheva et al. (2017) because the evaluation split, debates, and enabled features differ.This evaluation uses a train/test split rather than cross-validation, includes debates beyond those involving Hillary Clinton and Donald Trump, and disables metadata and discourse-parse features.Source: §3.5 · p. 3
- MUSE vectors are the final system choice, performing best overall on MAP, P@20, and P@50.The reported measures reflect a workload where fact-checkers may consider only 20 or 50 claims.Source: Table 1 · p. 4, §3.5 · p. 4
4 The System in Action
The interface presents sentence-level scores and color cues, with options to sort claims or mimic media selection.
4 The System in Action
- Users can enter text, view sentence-level scores and color intensity, and sort results so the most check-worthy sentences appear first.The interface also lets users choose to mimic different media.Source: §4 · p. 4
5 Conclusion and Future Work
ClaimRank prioritizes check-worthy claims to help fact-checkers and journalists decide where to focus their efforts first. Though originally trained on political debates, it supports English and Arabic and can work on other text genres.
5 Conclusion and Future Work
- ClaimRank prioritizes check-worthy claims and can mimic the sentence-selection strategies of each of nine fact-checking organizations or their union.The system learns from selections by those organizations.Source: §5 · p. 4
- Although originally trained on political debates, ClaimRank can also work on interviews and regular news articles, and supports Arabic as well as English.Arabic support uses manual training-data translation and cross-language embeddings.Source: §5 · p. 4
- Future work plans include training on more debates, speeches, and other genres, and adding support for more languages.Source: §5 · p. 4