Source-linked AI summary
Turing at SemEval-2017 Task 8: Sequential Approach to Rumour Stance Classification with Branch-LSTM
Elena Kochkina, Maria Liakata, Isabelle Augenstein
TL;DR
Rumour stance classification identifies how Twitter users regard the truthfulness of rumours, and is presented as a first step towards verification. The paper processes conversation branches with an LSTM-based model and reports 0.784 test accuracy, setting the state of the art for rumour stance classification.
Problem
The task is to identify whether tweets in rumour conversations support, deny, query, or comment on a rumour, as a first step towards verification of early reports.
Source: §1 · p. 1Method
The system represents tweets with averaged word vectors and additional features, then uses an LSTM to process conversation branches and label their tweets.
Source: §4.1 · p. 3, §4.2 · p. 3Results
0.784 accuracy on the testing set; the system sets the state of the art for rumour stance classification.
Source: §7 · p. 4Takeaways & Limitations
The paper concludes that decomposing conversation trees into linear sequences can achieve state-of-the-art rumour stance classification on the test set.
Source: §7 · p. 4The model is unable to pick out denying tweets, and the authors say more labelled data would help improve its performance.
Source: §6 · p. 4
Abstract
from arXiv · showhide
This paper describes team Turing's submission to SemEval 2017 RumourEval: Determining rumour veracity and support for rumours (SemEval 2017 Task 8, Subtask A). Subtask A addresses the challenge of rumour stance classification, which involves identifying the attitude of Twitter users towards the truthfulness of the rumour they are discussing. Stance classification is considered to be an important step towards rumour verification, therefore performing well in this task is expected to be useful in debunking false rumours. In this work we classify a set of Twitter posts discussing rumours into either supporting, denying, questioning or commenting on the underlying rumours. We propose a LSTM-based sequential model that, through modelling the conversational structure of tweets, which achieves an accuracy of 0.784 on the RumourEval test set outperforming all other systems in Subtask A.
1 Introduction
The paper frames rumour stance classification as identifying how tweets in a conversation regard a rumour’s truthfulness. It proposes an LSTM approach that uses conversational structure, with stance detection positioned as a first step towards rumour verification.
1 Introduction
- Tweets are classified as supporting, denying, querying, or commenting on the rumour initiated by the source tweet.The conversation consists of a source tweet and nested replies.Source: §1 · p. 1
- Automatically detecting stance is presented as a first step towards verifying early reports during events that provoke public resonance and rumours.The cited studies report that later-false rumours tend to attract more denying tweets than later-confirmed rumours.Source: §1 · p. 1
- The paper introduces an LSTM-based approach that exploits the conversational structure of social-media threads for stance classification.Source: §1 · p. 1
2 Related Work
Prior rumour-stance work includes single-tweet classification and approaches that model tweet sequences or conversation structure. The paper situates its task in conversation threads, rather than target-conditioned single-tweet stance classification.
2 Related Work
- Earlier rumour-stance work treated classification as binary support or denial, and used tweets about the same rumour for training and testing.Source: §2 · p. 1
- A target-conditioned bidirectional LSTM reached state of the art on SemEval-2016 Task 6, but RumourEval addresses conversation threads instead.Source: §2 · p. 1
- Hawkes-process work classified temporal tweet sequences and found value in text and timing, while disregarding discourse structure.Source: §2 · p. 1
- Zubiaga et al. (2016a) modeled source tweets and replies as chains or trees with CRFs, outperforming the Hawkes-process approach.Source: §2 · p. 2
3 Dataset
The dataset comprises rumour-related Twitter conversations from breaking-news events, annotated for stance at the tweet level. Threads form trees that can be split into branches, and the stance labels are imbalanced towards commenting and supporting.
3 Dataset
- The dataset contains 325 conversation threads and 5,568 tweets across ten breaking-news events.The events include the Paris shootings in Charlie Hebdo, the Ferguson unrest, and the Germanwings plane crash.Source: §3 · p. 2
- A thread contains a source tweet and nested replies; it can be split into linear branches, each a chain beginning at a leaf with its direct parents and continuing to the source.Tweet depth is the number of its parents starting from the root node; branches 1 and 2 in Figure 1 have depth one, while branch 3 has depth three.Source: §3 · p. 2
- Commenting tweets make up 66% of the data and supporting tweets 18%, while denying and querying each account for 8%.The paper notes that the imbalance reflects the realistic scenario in which only a few users question a statement’s veracity.Source: §3 · p. 2
4 System Description
The system combines tweet-level lexical and surface features with an LSTM that processes conversation branches. It produces a stance label for each tweet while accounting for shared tweets across overlapping branches during training.
4.1 Features
- The lexical features include Google News word2vec embeddings and counts of negation and swear words.The authors report that Twitter-trained GloVe embeddings reduced performance on development and test sets compared with Google News vectors.Source: §4.1 · p. 2
- Tweet vectors average word embeddings and concatenate punctuation, attachment, similarity, length, and tweet-role features.The feature set performed best against word2vec alone and reduced feature combinations.Source: §4.1 · p. 3
4.2 Branch - LSTM Model
- Branch-LSTM processes a whole tweet branch as a sequence of tweet vectors, incorporating conversation structure.The input representation is the mean of word vectors concatenated with additional features.Source: §4.2 · p. 3
- The model records each time-step output to label every tweet, then uses dense ReLU layers, dropout, and softmax class probabilities.Zero-padding and masks handle varying branch lengths; a mask excludes repeated tweets from the loss because branches can overlap.Source: §4.2 · p. 3
- More complex word-sequence tweet encoders and bidirectional LSTMs showed no noticeable performance improvement in cross-validation.The complex tweet encoder took significantly longer to train.Source: §4.2 · p. 3
5 Experimental Setup
The system selected hyperparameters using development-set accuracy, then trained on the combined training and development sets and evaluated on the held-out test set.
5 Experimental Setup
- The organisers split the dataset into training, development, and test sets.Source: §5 · p. 3
- Table 3 reports development- and test-set accuracy, macro-averaged F1, and per-class F1 for supporting, denying, querying, and commenting.Source: Table 3 · p. 4
- Table 4 presents tweet counts and performance at each depth.Source: Table 4 · p. 4
- 100 hyperparameter trials optimized development-set accuracy before the selected settings were used to train on combined training and development data and evaluate on held-out test data.The search used the Tree of Parzen Estimators algorithm.Source: §5 · p. 4
- The search varied dense ReLU and LSTM layer counts, layer widths, batch size, L2 regularization, and training epochs.The search ran 100 combinations; the supplied passage gives the candidate values for each parameter.Source: §5 · p. 4
6 Results
The model handled the majority commenting class well but struggled with denying tweets, with errors concentrated in predictions of commenting and performance varying across tweet depths.
6 Results
- Denying was the most challenging underrepresented class; the model predicted commenting, the majority class, well but could not pick out denials.Source: §6 · p. 4
- Accuracy differed minimally between test and development sets, while macro-F differed significantly because the sets had different class balances.The authors note macro-F might improve if used to optimize hyperparameters.Source: §6 · p. 4
- Most denying instances were misclassified as commenting, with only one misclassified as querying and two as supporting.The test-set confusion matrix is shown in Table 5.Source: §6 · p. 4, Table 5 · p. 4
- Source tweets at depth zero were usually supporting and predicted very well, but supporting-tweet performance decreased at other depths; test branches reached depth 13, with most tweets at depths 0–3.The authors report no noticeable performance difference across tweet lengths.Source: §6 · p. 4
- The authors state that more labelled data would help improve model performance.Source: §6 · p. 4
7 Conclusions
The paper decomposes conversation trees into linear sequences and reports state-of-the-art rumour stance classification accuracy on the test set.
7 Conclusions
- The linear-sequence approach achieved 0.784 test accuracy and set the state of the art for rumour stance classification.Future work will explore other methods for modelling tree-structured conversations.Source: §7 · p. 4