Source-linked AI summary
SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours
Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, Arkaitz Zubiaga
TL;DR
RumourEval addresses the need to determine the veracity of social-media rumours and understand the discourse surrounding them. It introduces a shared task with stance and veracity challenges using annotated Twitter data and contextual resources. Participants generally handled discourse-oriented classification well, while direct true/false verification remained difficult.
Problem
Determining the veracity of web claims and interpreting community reactions matters because false rumours can affect perceptions, behaviour, journalism, and disaster response.
Method
The shared task evaluates SDQC classification of replies in rumour conversations and true/false prediction for source tweets, using annotated data and time-controlled contextual resources.
Results
Participants achieved good results in identifying support, denial, questions, and comments, but systems did not reach the most-common-class baseline for true/false veracity prediction.
Takeaways & Limitations
RumourEval provides a benchmark dataset and competitive approaches for future research on rumour and fake-news detection.
Takeaways & Limitations
Accurately determining whether a story is true or false remained difficult, including assigning the correct confidence to veracity judgments.
Abstract
from arXiv · showhide
Media is full of false claims. Even Oxford Dictionaries named "post-truth" as the word of 2016. This makes it more important than ever to build systems that can identify the veracity of a story, and the kind of discourse there is around it. RumourEval is a SemEval shared task that aims to identify and handle rumours and reactions to them, in text. We present an annotation scheme, a large dataset covering multiple topics - each having their own families of claims and replies - and use these to pose two concrete challenges as well as the results achieved by participants on these challenges.
1 Introduction and Motivation
RumourEval addresses the difficult, socially consequential problem of determining both rumour veracity and how participants orient toward rumours in conversation. It introduces two complementary subtasks: SDQC stance classification and true/false veracity prediction.
- Rumours can distort perceptions and behaviour, while growing reliance on social media increases their potential disruptive impact.
- RumourEval frames rumour analysis as assessing claims and community responses to determine whether circulating stories are true or false.The paper defines a rumour as a circulating story of questionable veracity that appears credible but is difficult to verify.
- Subtask A - SDQC Support/ Rumour stance classification: Unlike conventional stance detection, SDQC evaluates support toward an entire rumour and includes questions and comments that characterize conversational context.The task also requires interpreting tree-structured discussions because individual tweets may depend on preceding context.
- Subtask A - SDQC Support/ Rumour stance classification: Subtask A classifies responses as supporting, denying, querying, or commenting on a rumour.It targets the discourse surrounding the original statement, including direct and nested replies in a conversation tree.
- Subtask B - Veracity prediction: Subtask B predicts whether an unsubstantiated, news-related source tweet is true or false using the tweet and available resources.Ground truth was manually established from official statements or other trustworthy evidence.
2 Data & Resources
The task uses annotated Twitter conversation trees covering multiple breaking-news events, with source tweets, replies, veracity labels, SDQC interactions, and time-controlled external context. Training and test resources were assembled through expert and crowdsourced annotation procedures.
- Twitter data are organized as tree-structured conversations in which each reply has a parent and threads begin with a source rumour tweet.The data were annotated for both veracity and SDQC labels under a published scheme.
- The training set contains 297 threads from eight events, comprising 297 source tweets and 4,222 replies, or 4,519 tweets total.
- The test set contains 28 additional threads and 1,080 tweets, including 28 source tweets and 1,052 replies.Twenty test threads came from training events, while eight came from two newly collected events.
- 2.3 Context Data: Participants received event-related Wikipedia articles and archived linked-URL content restricted to revisions available before the source tweet.The temporal restriction controlled access to information revealed after rumour resolution.
- 2.4 Data Annotation: Rumours were identified through sampling and expert journalist annotation, while tweet-level SDQC labels for Subtask A were assigned through crowdsourcing.The overall inter-annotator agreement was 63.7%, with higher agreement for source tweets than replies.
3 Evaluation
The evaluation measures SDQC classification accuracy and veracity prediction using macroaveraged accuracy combined with confidence calibration. Baselines include the most common class and an SDQ-only accuracy baseline for Task A.
- SDQC stance classification: SDQC stance classification is evaluated with classification accuracy despite a label distribution skewed toward comments.
- Veracity prediction: Veracity prediction uses macroaveraged accuracy and RMSE ρ for confidence differences on correct predictions, assigning incorrect examples an RMSE of 1.
- Veracity prediction: The final veracity score combines normalized confidence error with macroaveraged accuracy.The paper gives the example formula acc = (1 −ρ)acc.
- Baselines: The primary baseline is the most common class, with an additional Task A SDQ baseline excluding the low-impact comment class.The SDQ baseline measures accuracy over support, deny, and query labels only.
4 Participant Systems and Results
RumourEval evaluated systems on stance classification toward rumours and on source-tweet veracity classification. Participants used varied formulations and features, with sequential modelling and reply-stance information highlighted among the strongest approaches.
- Eight submissions addressed Subtask A, classifying tweets as supporting, denying, querying, or commenting on a rumour.
- Comments were the most frequent SDQC class, and recognising them helps reduce noise in the other information-bearing classes.Comments can also express implicit support, so absence of dispute provides a soft agreement signal.
- The best Subtask A system treated SDQC prediction sequentially, using previous tweets’ features and labels rather than classifying each tweet independently.Other systems addressed class imbalance or used source–reply tweet pairs.
- Feature engineering and class-imbalance strategies were associated with strong performance, while the best Subtask A result combined LSTM sequence modelling with word embeddings.
- Subtask B classified source-tweet veracity under open and closed resource settings, using either two-way or three-way labels.The best system added features based on the distribution of support, denial, and questioning labels among replies.
5 Conclusion
The shared task established a benchmark for analysing rumour discussions and veracity in social media. Its results show useful progress in interpreting conversational reactions, while direct true-or-false verification remains difficult.
- RumourEval brought together approaches for analysing claims and community interactions, with many systems identifying support, denial, questioning, and commentary.
- The open veracity setting permitted use of the 20160901 English Wikipedia dump as an external resource.
- Commentary around a story can help determine its veracity, but systems did not reach the most-common-class baseline for direct true-or-false judgments.
- The benchmark includes thousands of documents and novel news stories, alongside competitive results and diverse approaches for future research.