Source-linked AI summary
RumourEval 2019: Determining Rumour Veracity and Support for Rumours
Genevieve Gorrell, Kalina Bontcheva, Leon Derczynski, Elena Kochkina, Maria Liakata, Arkaitz Zubiaga
TL;DR
Automated rumour checking needs broader datasets and a better understanding of how discussion stance relates to veracity. RumourEval 2019 extends the shared task with Reddit, Russian and Danish data, and optional additional materials, while continuing stance and veracity prediction. The proposal builds on RumourEval 2017 results and aims to encourage more information-rich approaches, with a stated limitation that participants should exploit task A outputs more innovatively in task B.
Problem
Automated rumour verification remains in its infancy, with broader evidence on stance–veracity relations and more extensive datasets required.
Method
The shared task combines stance classification and rumour-veracity prediction across expanded multilingual data, allowing optional temporally appropriate additional materials.
Results
RumourEval 2017 subtask A teams achieved accuracies from 0.635 to 0.784, while the proposal uses those results to motivate richer task B information use.
Takeaways & Limitations
The 2019 task broadens rumour research with Reddit, Russian, and Danish data and a new baseline system for stance analysis.
Takeaways & Limitations
The proposal identifies more innovative use of task A outputs in task B as an unresolved need.
Abstract
from arXiv · showhide
This is the proposal for RumourEval-2019, which will run in early 2019 as part of that year's SemEval event. Since the first RumourEval shared task in 2017, interest in automated claim validation has greatly increased, as the dangers of "fake news" have become a mainstream concern. Yet automated support for rumour checking remains in its infancy. For this reason, it is important that a shared task in this area continues to provide a focus for effort, which is likely to increase. We therefore propose a continuation in which the veracity of further rumours is determined, and as previously, supportive of this goal, tweets discussing them are classified according to the stance they take regarding the rumour. Scope is extended compared with the first RumourEval, in that the dataset is substantially expanded to include Reddit as well as Twitter data, and additional languages are also included.
Overview
RumourEval 2019 continues automated rumour verification through stance classification and veracity prediction, addressing the need for broader evidence and datasets. It expands the task with Reddit, new languages, and a more flexible information setting intended to encourage richer approaches.
- Overview: Automated rumour verification remains limited, while broader understanding of stance–veracity relations and more extensive datasets are needed.Research has progressed beyond baselines, but the proposal identifies these as remaining requirements.
- RumourEval 2017: RumourEval 2017 classified discussion posts by support, deny, query, or comment, and classified source rumours as true, false, or unverified.The veracity task used open or closed information settings.
- RumourEval 2017: 0.635 to 0.784 was the accuracy range for eight teams in RumourEval 2017 subtask A.Open subtask B had one participating team with 0.393 accuracy, while five closed-variant teams scored between 0.286 and 0.536.
- How RumourEval 2019 will be different: The 2019 competition adds Reddit data and Russian and Danish languages to the existing data.Reddit provides more focused discussions and longer texts, while the new languages broaden the multilingual scope.
- How RumourEval 2019 will be different: Participants may use temporally appropriate additional materials without being required to do so, combining the former subtask B variants.This design prioritises information-rich pragmatic approaches while retaining a comparison forum for closed-world entries.
- Subtasks: Task A tracks reply stances toward a rumour’s accuracy, providing an intermediate challenge and information that can support task B.Task B predicts rumour veracity from the claim, task A stance data, and other selected information.
Data and Resources
The proposed corpus retains source rumours with tree-shaped discussions while expanding RumourEval 2017 with new English Twitter, English Reddit, Russian Twitter, and Danish Twitter data.
- Each source text asserts a potentially true or false rumour and is linked to a tree-shaped discussion of support, denial, comment, or query.
- 297 source tweets and 7100 discussion tweets from RumourEval 2017 will become training data for 2019.
- The proposed augmentation includes new English Twitter test data and English Reddit data.
- The proposed augmentation also includes Twitter data in Russian and Danish.
- Topics are identified through debunking projects, candidate source posts through attention, and final source texts are manually selected and expert-labeled for veracity.For tweets, retweet counts have been used as an indicator of suitable source posts.
- Discussion stances will be crowdsourced with multiple annotators and quality testing, building on a methodology used successfully in RumourEval 2017.Previous annotation experience reported high interannotator agreement.
Evaluation
RumourEval-2019 evaluates stance classification with a macroaveraged F1 focused on support, denial, and query, and evaluates rumour veracity with accuracy and confidence calibration.
- Task A: Stance classification: Task A uses macroaveraged F1 over support, denial, and query, disregarding the dominant comment class.The design aims to reward performance on the stance types considered more helpful for establishing rumour veracity.
- Task B: Veracity classification: Task B uses microaveraged accuracy for true/false rumour classification and RMSE for confidence relative to a reference confidence of 1.RMSE is described as the square root of the Brier score.
Baseline
The proposal plans to provide state-of-the-art baselines for both stance classification and rumour veracity, including a veracity baseline incorporating the best-performing stance system.
- Task A: Stance classification: Task A will provide code for a state-of-the-art RumourEval 2017 baseline and a later higher-performing RANLP entry.
- Task B: Veracity classification: Task B will provide an implementation of the RumourEval 2017 state-of-the-art baseline incorporating the best-performing stance classification system.