Overview of the CLEF-2019 CheckThat!: Automatic Identification and Verification of Claims

1 Introduction

• Rapid online claim spread, difficult human judgments, and few large-scale benchmarks make automatic fake-news detection challenging.The introduction notes that fact-checking systems must act before false claims reach many people.Source: §1 · p. 2
• The 2019 lab organized political-claim verification into check-worthiness estimation and factuality verification.Task 1 prioritizes claims for fact-checking; Task 2 covers useful Web pages, passages, and factuality judgments.Source: §1 · pp. 2–3, Figure 1 · p. 2
• The lab offered tasks independently to ease participation, focus evaluation, and enable more meaningful comparisons between systems.The tasks correspond to the main blocks of the verification pipeline.Source: §1 · pp. 2–3
• Task 1 extended an English dataset with new annotations, while Task 2 built an Arabic dataset from scratch with judgments for four subtasks.The datasets were released as CT19-T1 and CT19-T2.Source: §1 · p. 3
• Fourteen teams participated, and the most successful systems relied on supervised machine learning; the released corpora and scripts are intended to support further research.The introduction identifies check-worthiness estimation and automatic claim verification as target areas for that research.Source: §1 · p. 3

2 Overview of Task 1: Check-Worthiness

• Task 1 ranks sentences from political debates, interviews, or speeches by fact-checking priority, assigning each sentence a score used to order the list.The task aims to help fact-checkers prioritize their efforts.Source: §2 · p. 4

2.1 Dataset

• The new English test set extends CT-CWC-18 with labeled claims from three press conferences, six speeches, six debates, and one post.Annotations were derived from factcheck.org and manually refined to include claim occurrences and sentences containing fact-checked claims.Source: §2.1 · p. 5

2.2 Overview of the Approaches

• Most successful Task 1 approaches used supervised classification to assign check-worthiness scores, with successful systems generally treating sentences independently.Common representations included bag-of-words, part-of-speech tags, named entities, and sentiment analysis; classifiers included SVMs and neural networks.Source: §2.2 · pp. 5–6
• Copenhagen used domain-specific and syntactic embeddings in an LSTM, pre-trained on earlier Trump and Clinton debates and weakly supervised with ClaimBuster.Its primary submission used contrastive ranking loss; its contrastive2 run added the previous sentence representation.Source: §2.2 · p. 6

2.3 Evaluation

• Task 1 used MAP as its official measure, with MRR, mean R-Precision, and mean precision@k also reported across test debates and speeches.AveP uses precision at rank k for positions containing check-worthy claims.Source: §2.3 · p. 8
• Eleven teams submitted 21 runs, but only primary runs counted toward official rankings; oversampling and added data points did not improve the cited teams’ results.Copenhagen and ´e proibido cochilar used external datasets; nlpir01 tried oversampling, while ´e proibido cochilar tried oversampling and additional data points.Source: §2.3 · p. 8, results · p. 8
• TheEarthIsFlat’s second contrastive run achieved the best MAP score overall, while Copenhagen’s system was reported as best-performing and its performance decreased without ranking loss.The Table 3 caption identifies primary and contrastive submissions and reports ranks by evaluation measure.Source: results · p. 8, Table 3 · p. 9

3 Overview of Task 2: Evidence and Factuality

• Task 2 uses retrieved Web pages to rank and classify usefulness, identify evidence passages, and classify claims as true or false.The subtasks were proposed independently to support focused evaluation and more meaningful comparisons.Source: §3 · pp. 9–10

3.1 Dataset

• The CT19-T2 corpus contains 69 Arabic claims from multiple sources; useful-page and passage labels were manually produced, with test passage labels for 33 of 59 claims.The passage-label coverage restriction affects only Subtask C.Source: §3.1 · pp. 11–12
• Google’s original search ranking surprisingly outperformed all seven Subtask A runs, suggesting systems must distinguish usefulness from relevance while benefiting from relevance-based ordering.The authors attribute the baseline’s result to useful pages also needing to be relevant.Source: §3.2 · p. 13, Table 5 · p. 13

3.3 Subtask B

• In Subtask B, most systems struggled to retrieve useful pages, while runs ranked poorly in two-way classification could rank among the best in four-way classification.The authors note that useful pages comprise 8% of training examples and 22% of test examples, and that distinguishing useful from very useful was difficult.Source: Evaluation Measures · p. 14, Table 6 · p. 14

3.4 Subtask C

• Subtask C’s TheEarthIsFlat run 1 achieved recall of 0.94; the positional baseline performed almost as well as bigIR’s four runs.The authors suggest passage position may be useful for predicting passage usefulness.Source: §3.4 · p. 15, Table 7 · p. 15

3.5 Subtask D

• In Subtask D, all official runs beat the simple majority-class baseline in both cycles.Cycle 1 used pages without usefulness labels; cycle 2 released those labels, but low cycle-1 participation made their effect difficult to assess.Source: §3.5 · p. 16, Table 8 · p. 16

• Task 1 prioritized claims in political debates, while Task 2 ranked and classified useful Web pages, identified useful passages, and predicted claim factuality.Source: §4 · p. 17
• CheckThat! released English datasets derived from fact-checking sources and Arabic datasets to support further work on check-worthiness estimation and claim verification.Source: §4 · p. 17
• Fourteen teams submitted 57 runs; Task 1’s strongest approaches used neural networks and logistic regression, while Task 2’s top subtask A runs used learning-to-rank and other subtasks used classifiers.Source: §4 · p. 17
• The authors propose expanding check-worthiness annotations and adding annotation sources because agreement across media sources was low, suggesting selection bias.Source: §4 · p. 17
• Low participation in Task 2 subtask D’s first cycle prevented the planned study of how useful pages affect claim verification; the authors plan to require participation in both cycles.They also plan to extend Task 2 beyond Arabic.Source: §4 · p. 17