Source-linked AI summary
Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection
Lev Konstantinovskiy, Oliver Price, Mevan Babakar, Arkaitz Zubiaga
TL;DR
Claim detection supports factchecking, but limited organisations and fast information flows make manual verification difficult. This paper develops an expert-informed annotation schema, crowdsourced benchmark, and universal-sentence-representation classifier, achieving F1 0.83 and over 5% relative improvement over ClaimBuster. The resulting benchmark and system support more consistent automated claim detection.
Problem
Limited factchecking capacity and rapidly spreading information create a need to automate claim detection before assessing claim veracity.
Method
The paper develops an expert-informed seven-label annotation schema, crowdsourced dataset and benchmark, then classifies claims using universal sentence representations.
Results
F1 0.83, with over 5% relative improvement over ClaimBuster, was achieved by the universal-sentence-representation classifier for binary claim classification.
Takeaways & Limitations
The annotation schema and benchmark provide a more formal basis for replicating and extending claim detection, while the classifier offers competitive binary claim classification.
Takeaways & Limitations
Prior claim-detection work’s lack of a formal claim definition limited replication and extension, motivating this paper’s crowdsourced annotations based on an explicit task definition.
Abstract
from arXiv · showhide
In an effort to assist factcheckers in the process of factchecking, we tackle the claim detection task, one of the necessary stages prior to determining the veracity of a claim. It consists of identifying the set of sentences, out of a long text, deemed capable of being factchecked. This paper is a collaborative work between Full Fact, an independent factchecking charity, and academic partners. Leveraging the expertise of professional factcheckers, we develop an annotation schema and a benchmark for automated claim detection that is more consistent across time, topics and annotators than previous approaches. Our annotation schema has been used to crowdsource the annotation of a dataset with sentences from UK political TV shows. We introduce an approach based on universal sentence representations to perform the classification, achieving an F1 score of 0.83, with over 5% relative improvement over the state-of-the-art methods ClaimBuster and ClaimRank. The system was deployed in production and received positive user feedback.
1 INTRODUCTION
The paper targets claim detection, the under-addressed step of identifying checkable claims before verification. It develops an expert-informed annotation schema, crowdsourced dataset, and automated classifier for this stage of factchecking.
- Factchecking comprises monitoring media, detecting claims, checking claims, and publishing verification results.
- Most prior factchecking research focused on determining claim veracity while assuming an available list of claims.
- Automated claim detection could expand media monitoring and reduce the time factcheckers spend on semi-manual spotting.
- The annotation schema avoids subjective importance and check-worthiness criteria to define claims more objectively.
- The study introduces an expert-developed seven-label schema, a crowdsourced dataset of 5,571 labelled sentences, and a benchmark methodology.
- The proposed system uses universal sentence representations rather than only word-level representations and outperforms ClaimBuster and ClaimRank.
2 RELATED WORK
Prior claim-detection studies differ in their definitions, annotation sources, and use of contextual or editorial judgments. The paper positions its approach against these methods by formalising claim annotation and crowdsourcing labels under an explicit task definition.
- Much automated factchecking research addresses claim veracity, leaving the preceding claim-detection stage less studied.
- Previous claim-detection approaches: ClaimBuster ranks claims by importance using TF-IDF, POS, and NER features with an SVM classifier, requiring a cutoff for factchecking.
- Previous claim-detection approaches: ClaimRank predicts whether claims would be highlighted by factchecking organisations using sentence and wider-context features.
- Previous claim-detection approaches: Other approaches use topic-specific context, publicly available factchecking articles, or veracity-oriented datasets as claim-detection resources.
- Defining claims: Some prior work lacks a formal claim definition and relies on external organisations’ selections, limiting replication and producing inconsistent annotations.
- Defining claims: The paper distinguishes its crowdsourced annotations from prior datasets assembled from external factchecking organisations and online articles.
- Defining claims: ClaimBuster defines claims through public interest in whether a sentence is true or false, incorporating an importance judgment that this paper avoids.
3 DATASET
The dataset was built through an iterative, factchecker-informed annotation process designed to define checkable claims more consistently while separating claim content from subjective importance and topic. It contains crowdsourced labels for UK political TV subtitles, with seven categories later mapped to binary claims and non-claims.
- 3.1 Our Claim Definition and Process: The annotation guidance evolved through trials with factcheckers and volunteers to reduce subjective judgments about whether sentences were claims.The process replaced earlier accept/reject, implicit/explicit, personal-experience, and opinion categories with a more comprehensive typology.
- 3.1 Our Claim Definition and Process: The schema separates claim definition from importance and topic, treating both as context-dependent or politically variable.Importance was considered subjective and changing, while topic classification could vary across populations.
- 3.2 Annotation Guidance: Seven mutually exclusive categories were defined, including personal experience, quantity, correlation or causation, current laws or rules, prediction, other claim, and not a claim.Examples include numerical comparisons, rankings, and claims about publicly available evidence.
- 3.2 Annotation Guidance: Not a claim represented about 55% of annotations, while Other contained 952 sentences, or 23% of the dataset.The Other category grouped overlapping or less-defined types, and a sample analysis supported retaining this aggregation.
- 3.4 Agreement: Binary annotation agreement reached Krippendorff’s alpha values of 0.70 and 0.53 under two mappings, compared with 0.46 across all seven categories.The second mapping was selected for experimentation because factcheckers prioritized recall and retaining the initial categories.
- 3.4 Agreement: Majority voting selected 4,777 binary-labelled sentences—3,973 non-claims and 804 claims—whereas retaining all seven categories yielded 4,080 sentences.The binary filter required at least three annotations and agreement from more than half of the annotators.
4 METHODS
The method uses universal sentence representations to capture the diversity of sentences in political TV shows. InferSent produces sentence embeddings that account for word order through a recurrent neural network.
- 4 METHODS: The system uses InferSent universal sentence representations to create sentence embeddings for political TV-show sentences.The approach is intended to capture sentence diversity beyond word-level representations.
- 4 METHODS: InferSent accounts for word order with a recurrent neural network rather than averaging word embeddings.Words are first converted to common-crawl GloVe representations.
5 METHODS
The study compares sentence representations and classifiers for claim detection, including universal sentence representations, lexical baselines, and adaptations of prior systems. Models are evaluated on 5,571 sentences using stratified 5-fold cross-validation, precision, recall, and F1-score.
- Sentence representations: Universal sentence representations are produced with a bidirectional long-short-term memory network using embeddings pre-trained on Natural Language Inference tasks.The study also tests concatenated POS and NER feature vectors with the sentence embeddings.
- Classifiers: The experiments compare supervised classifiers trained on sentence representations implemented with scikit-learn.
- Baselines: ClaimBuster baselines combine TF-IDF, POS, and NER features in different configurations.
- Baselines: Additional baselines use averaged Word2vec or GloVe embeddings, optional PCA, and TF-IDF with logistic regression.
- Comparison systems: ClaimRank is reimplemented without speaker, applause, laughter, and speaker-crossover features unavailable or unsuitable for the dataset.The authors train FNN, SVM, and logistic regression classifiers on the remaining features.
- Evaluation: 5,571 sentences are evaluated with stratified 5-fold cross-validation using precision, recall, and F1-score.The dataset contains 1,570 claims and 4,001 non-claims, producing a 30/70 class imbalance.
- Evaluation: Confidence intervals are reported for precision and recall, with wider recall intervals attributed to the small number of positive examples.
5 CLAIM DETECTION
The binary claim-detection experiments prioritize recall while balancing precision and recall through F1. The CNC model achieves the strongest overall F1 among the compared systems.
- Evaluation priorities: Recall is prioritized because factcheckers do not want to miss important claims, while F1 measures the balance between precision and recall.
- Results: 0.83 F1 is achieved by CNC with logistic regression, the highest overall F1 among the evaluated techniques.CNC stands for “Claim/No Claim.”
- Results: 0.79 F1 is achieved by ClaimBuster, below CNC’s reported F1 score.
- Results: TF-IDF achieves high precision but low recall, whereas GloVe has F1 scores 2% lower than CNC and substantially lower recall.
6 MULTI-CLASS CLASSIFICATION
The multi-class experiment extends CNC to seven categories using 4,080 sentences with sufficient agreement. Performance is stronger for frequent or readily identifiable categories and weaker for sparse categories.
- Setup: 4,080 sentences with sufficient agreement are used to train a multinomial logistic regression on CNC features for seven-category classification.
- Overall performance: 0.70 microaveraged F1 is achieved overall, while macroaveraged F1 reaches 0.48.The authors describe the macroaveraged result as leaving significant room for improvement.
- Category results: “Quantity” and “Not a claim” yield the best F1-scores among the larger classes.“Quantity” is relatively easy to identify using numbers and quantitative words.
- Category results: “Current laws” and “Correlation or causation” obtain low F1 scores among categories with the fewest instances.The passage notes that small class sizes may significantly affect these results.
- Error patterns: The classifier often confuses “Not a claim” and “Other type of claim” with other actual labels.The paper expects more annotated sentences to improve these categories, while the binary classifier is currently more accurate and usable.
7 DEPLOYMENT AND IMPACT
The claim-detection model was integrated into Full Fact’s live-transcript workflow, where it highlights detected claims for factcheckers. Deployment saved review time and supported skimming beyond users’ immediate domains.
- Deployment: Model deployment impacted Full Fact, while the crowdsourcing exercise educated its volunteer community.
- Deployment: The live-factchecking interface highlights model-detected claims in bold and lets factcheckers manually highlight additional claims in yellow.
- Impact: Factcheckers began skimming claims in transcripts outside their immediate domains, where reviewing the entire transcript was previously not viable.
- Evaluation in use: All claims manually highlighted in four live factchecking sessions were also detected by the model.Precision was harder to measure because factcheckers selectively highlighted claims using domain expertise and current-affairs awareness.
- User feedback: A factchecker reported reviewing only highlighted transcript sections instead of reading the whole transcript after live factchecking.
8 DEPLOYMENT AND IMPACT
The deployed “Live” factchecking tool highlighted claims during Prime Minister’s Questions, while the annotation exercise also had educational effects on volunteers. Volunteers reportedly became more scrupulous media consumers.
- The annotation exercise was educational for volunteers, who became more scrupulous media consumers.
- The “Live” factchecking tool highlighted claims during Prime Minister’s Questions on 12 September 2018.
- The figure notes that transcript errors came from the broadcast’s closed captions.
8 CONCLUSION
The paper presents an expert-informed annotation schema, an annotated political-TV dataset, and a universal-sentence-representation classifier for claim detection. The classifier achieves F1 0.83 and over 5% relative improvement over baseline methods, while finer seven-category classification remains open for improvement and future work includes broader languages and sources.
- F1 0.83 with over 5% relative improvement over baseline classifiers, including ClaimBuster, is reported for binary claim classification.
- Future work includes expanding the dataset to other languages, organizations, and news sources.