Source-linked AI summary
A simple but tough-to-beat baseline for the Fake News Challenge stance detection task
Benjamin Riedel, Isabelle Augenstein, Georgios P. Spithourakis, Sebastian Riedel
TL;DR
The paper addresses automatic evaluation of news-body stance toward headlines as support for fact checking. It presents UCLMR’s simple end-to-end system and reports third place in FNC-1 with competitive performance, while identifying weak ‘agree’ and ‘disagree’ classification.
Problem
Automatic stance evaluation is needed to help fact checkers assess how news sources relate to particular claims.
Method
UCLMR combines lexical and similarity features in a single end-to-end MLP stance detection system.
Results
UCLMR achieved an FNC-1 score of 81.72% and placed third, performing almost on par with the top two teams.
Takeaways & Limitations
The authors propose UCLMR as a simple but tough-to-beat baseline for FNC-1 stance detection.
Takeaways & Limitations
Performance was mediocre or poor on the ‘agree’ and ‘disagree’ labels, alongside a limited dataset and few instances of those labels.
Abstract
from arXiv · showhide
Identifying public misinformation is a complicated and challenging task. An important part of checking the veracity of a specific claim is to evaluate the stance different news sources take towards the assertion. Automatic stance evaluation, i.e. stance detection, would arguably facilitate the process of fact checking. In this paper, we present our stance detection system which claimed third place in Stage 1 of the Fake News Challenge. Despite our straightforward approach, our system performs at a competitive level with the complex ensembles of the top two winning teams. We therefore propose our system as the 'simple but tough-to-beat baseline' for the Fake News Challenge stance detection task.
1 Introduction
The paper frames automatic stance detection as a way to assist fact checkers by evaluating how news bodies relate to headlines. It presents UCLMR’s system and its FNC-1 competition performance.
- Automatic stance evaluation is proposed as a first step toward assisting human fact checkers in detecting inaccurate claims.
- FNC-1 asked systems to estimate a news body’s stance toward a headline.The possible labels were ‘agree’, ‘disagree’, ‘discuss’, and ‘unrelated’.
- The paper describes UCLMR’s FNC-1 system, summarizes its performance, reviews the competition, and outlines future work.
2 System description
UCLMR uses a single end-to-end system that combines simple lexical and similarity features with a one-hidden-layer MLP classifier. The paper also documents its feature construction, training, and tuning procedures.
- System overview: The system passes lexical and similarity features through a one-hidden-layer MLP and is designed as a single end-to-end stance detector.
- Representations and features: The inputs use headline and body TF vectors plus cosine similarity between their ℓ2-normalised TF-IDF vectors.
- Representations and features: The TF and TF-IDF vocabularies contain the 5,000 most frequent words, with stop words excluded; TF-IDF vocabulary is defined on training and test sets.
- Representations and features: The concatenated feature vector has total size 10,001 and is fed into the classifier.
- Classifier: The MLP has one hidden layer of 100 units, ReLU activation, a softmax output, and predicts the highest-scoring stance label.
- Training and tuning: Training minimizes cross entropy with ℓ2 weight regularization and dropout applied to both perceptron layers.
- Training and tuning: Hyperparameters were tuned during development using random search over a grid and cross-validation on various data splits.
3 Results
UCLMR achieved a strong overall FNC-1 score, driven mainly by distinguishing related from unrelated headline-body pairs. However, performance was weak on the ‘agree’ and ‘disagree’ labels.
- 81.72% was UCLMR’s FNC-1 score on the competition evaluation.
- 96.55% accuracy was achieved on classifying headline-body pairs as ‘related’ or ‘unrelated’.The authors attribute satisfactory general performance mainly to this near-perfect classification and default-like ‘discuss’ predictions for related instances.
- The system’s ‘agree’ performance was average at best, while its ‘disagree’ accuracy was clearly poor.The paper identifies these labels as especially relevant to automating stance evaluation.
4 Competition
UCLMR placed third among 50 FNC-1 teams, nearly matching the two leading systems despite using a simple single system. Its score also exceeded the official baseline by a substantial margin.
- UCLMR placed third among 50 participating FNC-1 teams.
- The two leading teams scored 82.02% and 81.97%, respectively.
- UCLMR performed almost on par with the top two teams and substantially ahead of the remaining teams and official baseline.
- UCLMR achieved competitive results with a simple, single, end-to-end system rather than the more elaborate approaches used by other submissions.
- Discussions with other teams indicated that their test performance on the key ‘agree’ and ‘disagree’ labels was not much better, if at all.
5 Future work
Future work will analyze how the straightforward system works, what drives its performance, and what limits it. The authors especially target mediocre agree/disagree performance while proposing the system as a simple but tough-to-beat FNC-1 baseline.
- 5 Future work: The authors plan in-depth analyses of the straightforward system’s operation, performance contributors, and limitations.
- 5 Future work: These analyses will investigate mediocre agree and disagree performance alongside the limited dataset and few instances of specific labels.
- 5 Future work: The system is proposed as a simple but tough-to-beat baseline for FNC-1 because of its competitive performance and basic implementation.