Source-linked AI summary
Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness
Pepa Atanasova, Alberto Barron-Cedeno, Tamer Elsayed, Reem Suwaileh, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, Preslav Nakov
TL;DR
The paper addresses how to prioritize political claims for fact-checking when manual review can cover only a limited number of claims. It overviews CLEF-2018 CheckThat! Task 1, which ranks debate and speech sentences in English and Arabic, and reports the participating systems and benchmark results. The best systems reached mean average precision of 0.18 for English and 0.15 for Arabic, while the authors identify substantial room for improvement and release datasets and scoring scripts.
Problem
Manual fact-checking can cover only a relatively small number of claims, motivating systems that prioritize claims for review.
Method
The lab evaluates systems that rank sentences from political debates and speeches by check-worthiness, using a MAP-based information-retrieval evaluation.
Results
The best systems achieved mean average precision of 0.18 on English and 0.15 on Arabic test datasets.
Takeaways & Limitations
The released datasets and scoring scripts support further research in check-worthiness estimation, while system performance leaves substantial room for improvement.
Takeaways & Limitations
Full automation is not yet a viable alternative because of existing technology limitations and low trust among human users.
Abstract
from arXiv · showhide
We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims, with focus on Task 1: Check-Worthiness. The task asks to predict which claims in a political debate should be prioritized for fact-checking. In particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact checking. We offered the task in both English and Arabic, based on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign. A total of 30 teams registered to participate in the Lab and seven teams actually submitted systems for Task~1. The most successful approaches used by the participants relied on recurrent and multi-layer neural networks, as well as on combinations of distributional representations, on matchings claims' vocabulary against lexicons, and on measures of syntactic dependency. The best systems achieved mean average precision of 0.18 and 0.15 on the English and on the Arabic test datasets, respectively. This leaves large room for further improvement, and thus we release all datasets and the scoring scripts, which should enable further research in check-worthiness estimation.
1 Introduction
The lab targets the manual bottleneck of fact-checking by ranking political-debate or speech sentences according to their check-worthiness. Task 1 forms the first stage of a broader pipeline, while full automation remains constrained by technology and user trust.
- Fact-checking remains a demanding manual effort, so only a relatively small number of claims can be checked and prioritization is needed.
- Task 1 asks systems to predict which claims in a political debate or speech should be prioritized for fact-checking.
- The task produces a ranked list of sentences ordered by their worthiness for fact-checking.It is the first stage before claims are extracted, normalized, and fact-checked.
- Full automation is not yet viable because of limitations in existing technology and low trust among human users.
- The lab offered the task in English and Arabic using political debates and speeches, with English debate fragments illustrating marked check-worthy sentences.
2 Related Work
Prior work framed check-worthiness as part of a multi-step automatic fact-checking process and modeled selection using annotated political debates. This lab follows related setups while manually verifying claims, including repeated instances, and adding Arabic data for one fact-checking organization.
- Automatic fact-checking can include identifying check-worthy statements, generating questions, retrieving information, and inferring statement veracity.
- ClaimBuster used manually annotated historical US election debates and an SVM with sentiment, TF.IDF, and related features to classify sentence types.
- A closely related study used 5,415 sentences from four 2016 US campaign debates annotated by nine fact-checking organizations and modeled sentence context.
- ClaimRank extended claim-selection modeling across nine fact-checking organizations and supported Arabic through cross-language English-Arabic embeddings.
- This lab manually verified selected sentences, adjusted claim boundaries, included all instances of selected claims, added Arabic data, and focused on one fact-checking organization.
3 Evaluation Framework
The evaluation used the CT-CWC-18 corpus to rank check-worthy sentences from English and Arabic political debates and speeches, using mean average precision as the official measure. The framework also reported several complementary ranking metrics and controlled access to external fact-checking data.
- Data: The dataset annotations identify check-worthy claims at the sentence level using publicly available FactCheck.org analyses.The dataset also manually verifies selected sentences, adjusts claim boundaries, and includes all instances of selected claims.
- Data: CT-CWC-18 contains five English debates and five speeches, with Arabic translations for five debates and Donald Trump’s acceptance speech.The first three debates were released for training, while the remaining debates and speeches were used for testing.
- Data: External fact-checking annotation datasets were forbidden, although participants could retrieve Web or Twitter information subject to URL-sanity checks.The provided script aimed to prevent use of information from fact-checking websites.
- Evaluation Measures: Mean average precision was the official measure because the task ranks check-worthy instances toward the top of each list.For each debate or speech, average precision uses precision at rank k and an indicator for whether the claim at that position is check-worthy.
- Evaluation Measures: The evaluation additionally reports mean reciprocal rank, mean R-Precision, and mean precision@k, macro-averaged over testing debates and speeches.These measures complement the official MAP ranking metric.
4 Overview of Participants’ Approaches
Participants applied diverse representations and classifiers, including neural networks, multilayer perceptrons, support vector machines, random forests, nearest neighbors, and learning-to-rank models. English results were ranked by MAP, with the strongest primary submission outperforming the random baseline across measures and exceeding the n-gram baseline in only two runs.
- Models and Representations: Participants used word embeddings, linguistic features, lexicons, syntactic dependencies, sentiment, topics, named entities, and contextual sentences across multiple classifiers.The approaches included recurrent neural networks, multilayer perceptrons, SVMs, random forests, k-nearest neighbors, and MART learning-to-rank.
- Neural Approaches: The Copenhagen system combined word2vec embeddings, part-of-speech tags, and syntactic dependencies in an attention-based GRU neural network.Its combination with another approach helped cross-validation, but the neural network alone performed better on the test dataset.
- Neural Approaches: The blue team used an attention-based LSTM with 100 hidden dimensions and the five preceding sentences as context.This approach explicitly incorporated local discourse context around the target sentence.
- English Results: All systems beat the random baseline on every measure, but only Prise de Fer’s primary run and Copenhagen’s contrastive 1 run beat the n-gram baseline.The English results table was ranked by MAP, the official evaluation measure.
5 Evaluation Results
Seven teams submitted English runs and two submitted Arabic runs. Prise de Fer led the English primary results, while Arabic systems showed a reversal in relative team performance.
- Seven teams submitted English runs, while two teams also submitted Arabic runs.
- English: Prise de Fer achieved the best English primary performance overall, leading on MAP and six of nine evaluation measures.
- English: The Copenhagen team’s unofficial recurrent-neural-network contrastive run outperformed its primary submission on the English test set.
- English: All English systems beat the random baseline, but only Prise de Fer’s primary run and Copenhagen’s contrastive run surpassed the n-gram baseline.
- Arabic: For Arabic, UPV–INAOE–Autoritas outperformed bigIR in English, whereas bigIR outperformed UPV–INAOE–Autoritas in Arabic.The authors suggest translation direction and the presence or absence of context as possible reasons.
- Arabic: All Arabic runs beat the random baseline, while the n-gram baseline remained strong but could be surpassed.
6 Discussion
Speeches were easier than debates for the participating systems, and score-level ensembles produced small MAP gains in both languages. Ablations indicate further gains may come from combining systems or their features and models.
- Speeches were easier than debates, possibly because the test speeches contained about twice as many check-worthy claims.
- Summing min-max-normalized system scores yielded small MAP improvements over the best individual system in both English and Arabic.
- Removing most individual systems lowered ensemble MAP, although excluding blue improved MAP and some other evaluation measures.
- Ablation patterns suggest potential for improving results by combining participating approaches at the feature or model level.
- The reciprocal-rank ensemble performed much worse than summing normalized prediction scores.
7 Conclusion and Future Work
The lab focused on ranking political claims for fact-checking in English and Arabic, using shared evaluation resources and participant systems. Future iterations aim to expand the data and annotations to support semi-supervised and multi-task learning.
- The lab’s Task 1 asked systems to predict which claims in political debates should be prioritized for fact-checking in English and Arabic.
- 30 teams registered and 7 submitted systems; the strongest approaches used neural networks, distributional representations, lexicons, and syntactic dependency measures.
- Mean average precision reached 0.18 on English and 0.15 on Arabic test datasets.
- The authors released all datasets and scoring scripts because the results leave substantial room for further improvement.
- Future lab iterations plan to add annotated and unannotated debates and speeches, enabling semi-supervised learning.
- They also plan annotations from multiple fact-checking organizations to support multi-task learning.