Source-linked AI summary
Discourse-Aware Rumour Stance Classification in Social Media Using Sequential Classifiers
Arkaitz Zubiaga, Elena Kochkina, Maria Liakata, Rob Procter, Michal Lukasik, Kalina Bontcheva, Trevor Cohn, Isabelle Augenstein
TL;DR
The paper asks whether rumour stance classification can improve by modelling conversational structure instead of treating tweets independently. It compares sequential classifiers across eight breaking-news datasets and finds that sequential approaches outperform non-sequential baselines, with a reduced-feature LSTM performing best consistently.
Problem
Most prior rumour stance classifiers treat tweets as isolated inputs, leaving conversational discourse structure underexplored despite its relevance to classifying responses around breaking-news rumours.
Method
The paper compares Hawkes Processes, Linear CRF, Tree CRF and Branch LSTM using local and contextual features on tree-structured Twitter conversations.
Results
Sequential classifiers outperform non-sequential classifiers, while a reduced-feature LSTM performs best consistently across datasets and stance types.
Takeaways & Limitations
Conversational structure is useful for rumour stance classification, and LSTM can achieve strong consistent performance while using a limited feature set.
Takeaways & Limitations
Commenting tweets make up over 64% of the dataset, while supporting and denying tweets together account for less than 28%, creating a challenging class imbalance.
Abstract
from arXiv · showhide
Rumour stance classification, defined as classifying the stance of specific social media posts into one of supporting, denying, querying or commenting on an earlier post, is becoming of increasing interest to researchers. While most previous work has focused on using individual tweets as classifier inputs, here we report on the performance of sequential classifiers that exploit the discourse features inherent in social media interactions or 'conversational threads'. Testing the effectiveness of four sequential classifiers -- Hawkes Processes, Linear-Chain Conditional Random Fields (Linear CRF), Tree-Structured Conditional Random Fields (Tree CRF) and Long Short Term Memory networks (LSTM) -- on eight datasets associated with breaking news stories, and looking at different types of local and contextual features, our work sheds new light on the development of accurate stance classifiers. We show that sequential classifiers that exploit the use of discourse properties in social media conversations while using only local features, outperform non-sequential classifiers. Furthermore, we show that LSTM using a reduced set of features can outperform the other sequential classifiers; this performance is consistent across datasets and across types of stances. To conclude, our work also analyses the different features under study, identifying those that best help characterise and distinguish between stances, such as supporting tweets being more likely to be accompanied by evidence than denying tweets. We also set forth a number of directions for future research.
1. Introduction
The paper addresses rumour stance classification in breaking-news social media, where posts contribute to conversational threads rather than standing alone. It evaluates sequential classifiers that exploit discourse structure and finds that they outperform non-sequential approaches, with LSTM performing best consistently.
- Motivation: Social media rumours complicate breaking-news analysis because reports may be unverified when they emerge.The paper situates stance classification within broader systems for detecting, tracking and assessing rumour veracity.
- Task: Stance classification assigns each post an orientation toward earlier posts, including supporting, denying, querying or commenting.Aggregating these stances can contribute to determining a rumour’s likely veracity.
- Approach: Sequential classifiers model evolving conversational threads and are compared with classifiers that treat tweets in isolation.The study evaluates Hawkes Processes, Linear CRF, Tree CRF and LSTM, alongside non-sequential baselines.
- Evaluation: The study analyses performance across datasets, stance types and feature settings, including local and contextual features.It also examines classifier errors and which features characterise different stances.
- Findings: Sequential approaches substantially outperform non-sequential classifiers on macro-averaged F1, while LSTM performs best consistently across datasets and stance types.LSTM performs especially well with only local features; other classifiers need contextual features to achieve comparable but inferior performance.
2. Related Work
Earlier stance work often classified isolated posts or used simpler stance categories, while this paper focuses on richer rumour responses in breaking-news conversations. Its contribution is to examine tree-structured conversational context through a broader comparison of sequential classifiers and features.
- Prior stance work: Earlier stance studies commonly used binary or topic-oriented categories such as in favour versus against.Rumour stance classification extends this setting toward determining information veracity.
- Breaking-news setting: Breaking-news rumours require classifying newly emerging claims whose supporting evidence and authoritative review may be limited.The paper targets rumours associated with new events rather than only previously observed long-standing rumours.
- Task scope: The task includes querying and commenting alongside supporting and denying because social-media responses do not all directly resolve veracity.This broader response inventory captures appeals for information and contributions unrelated to resolution.
- Conversational structure: Prior conversational modelling often used linear reply sequences, whereas this work exploits the full tree structure of Twitter threads.The paper identifies detailed use of Twitter conversational structure as underexplored in stance classification.
- Paper contribution: The study broadens prior experiments by comparing Hawkes Processes, CRF variants and LSTM, adding contextual features and expanding result analysis.It also analyses which features characterise different stance types.
3. Research Objectives
The paper investigates whether the evolving structure of social-media conversations improves tweet-level stance classification. Its objectives cover classifier gains, contextual features, cross-event consistency, thread depth, errors and stance-characterising features.
- Overall objective: The central objective is to exploit conversational evolution while assigning each post its own stance value.Posts contribute to an accumulating discussion, so the paper studies classification beyond isolated tweets.
- Performance objectives: The study quantifies sequential-classifier gains over non-sequential baselines using local features only.This isolates the benefit of modelling conversational sequence rather than adding information from surrounding tweets.
- Performance objectives: It tests whether contextual features extracted from surrounding tweets improve classification performance.The motivation is that tweets are short and neighbouring posts may provide useful information.
- Robustness objectives: The experiments assess consistency across eight event datasets and classification performance at different conversation depths.Depth ranges from source tweets to nested replies, where more preceding conversation must be aggregated.
- Diagnostic objectives: The paper examines when and why classifiers succeed or fail and analyses features that distinguish stance types.Feature analysis serves both performance improvement and understanding of social-media responses.
4. Rumour Stance Classification
Rumour stance classification labels each post’s orientation toward a disputed rumour within a tree-structured conversational thread. The paper uses breaking-news discussions collected across eight English-language events, whose heavily comment-skewed distribution makes classification challenging.
- Task definition: Rumour stance classification determines the orientation each individual post expresses toward a rumour’s disputed veracity.The task is defined over conversational threads with a source tweet and nested replies.
- Task definition: Each tweet receives one of four labels: supporting, denying, querying or commenting.The labels capture both direct positions and responses seeking information or contributing without resolving veracity.
- Dataset: The dataset contains tree-structured conversations initiated by rumour source tweets and annotated replying tweets from eight English-language breaking-news events.The full dataset includes 4,519 tweets; the German event was excluded from these experiments.
- Dataset: Figure 1 illustrates the source tweet, nested replies, stance annotations and tweet depth within a rumour thread.Depth is measured by the number of steps from the source tweet to the current tweet.
- Dataset: Commenting tweets comprise over 64% of the dataset, creating class imbalance that varies across the eight events.Supporting and denying tweets together account for less than 28%, making the four-way task more challenging than earlier binary settings.
5. Classifiers
This section presents Hawkes Processes, CRFs, and branch-LSTM models for exploiting temporal or conversational structure in rumour stance classification. It contrasts tree-wide and branch-based representations, while addressing repeated tweets and class imbalance.
- The experiments compare Hawkes Processes, Linear CRF, Tree CRF, Branch LSTM, and non-sequential baseline classifiers.The sequential models exploit temporal sequences, linear branches, or complete conversation trees.
- 5.1. Hawkes Processes: Hawkes Processes model tweet occurrence over time with an intensity function whose base rate and previous-tweet influences determine event likelihood.Label interactions are encoded through a matrix, and parameters are learned by maximizing tweet-generation likelihood.
- 5.2. Conditional Random Fields (CRF): Linear CRF and Tree CRF: CRFs represent Twitter conversations as graphs whose output labels depend on each tweet’s features and neighboring labels.The study uses both linear-chain CRFs for branches and Tree CRFs for whole tree-structured conversations.
- 5.3. Branch LSTM: The branch-LSTM processes branches as sequences of tweet vectors, with each input vector formed by averaging the word vectors in a tweet.Its architecture includes recurrent layers followed by feed-forward and softmax layers, with dropout and L2 regularization.
- Repeated tweets created by branch extraction are excluded from the loss through masking, while class weights use inverse training frequencies to address imbalance.Per-word nested LSTM representations did not significantly improve over averaged word vectors.
- 5.4. Summary of Sequential Classifiers: Tree CRF builds one graph for the full conversation, whereas Hawkes Processes, Linear CRF, and LSTM split trees into linear branches.Branch decomposition simplifies the model but repeats source and other tweets across sequences; whole-tree modeling avoids that repetition.
6. Features
The study evaluates sequential classifiers using local, contextual, and Hawkes features. It first enables fair comparison with non-sequential classifiers, then adds conversational context and Hawkes features for further comparison.
- Experiments use local, contextual, and Hawkes features to study sequential classifiers for discursive stance classification.Local features support comparison across sequential classifiers, while contextual and Hawkes features extend the feature settings.
- Local features restrict inputs to the current tweet, making it possible to quantify whether mining sequences improves performance over non-sequential classifiers.
- Contextual features draw on other tweets in a conversation and are incorporated to further boost sequential-classifier performance.
- Hawkes features are described to enable comparison with the Hawkes process classifier.
- Table 2 lists the local, contextual, and Hawkes features and groups the local and contextual features into several subtypes.
7. Experimental Results
Sequential classifiers generally outperform non-sequential alternatives for rumour stance classification, with LSTM showing strong and consistent performance using only local features. Contextual features substantially help non-sequential classifiers but add little benefit to sequential models, while feature analysis reveals stance-specific signals and remaining difficulty distinguishing comments.
- Sequential classifier performance: Sequential classifiers outperform non-sequential classifiers overall, with LSTM slightly exceeding the CRF alternatives and CRFs outperforming MaxEnt.The differences between CRF and LSTM are statistically significant except for LF1, and CRF–MaxEnt differences are statistically significant.
- Feature combinations: The best local-feature combination is lexicon, content formatting and punctuation, achieving a macro-F1 score of 0.449.Combinations of local-feature subgroups improve on single subgroups, and LF123 is the strongest reported combination.
- Contextual features: Contextual features substantially improve non-sequential classifiers but provide little benefit to sequential classifiers, whose sequence modeling already incorporates surrounding context.For LSTM, relying only on local features is better because additional contextual features produce no improvement and may be harmful.
- Performance across events and depths: LSTM performs best across many events and remains consistently strong across tweet depths, despite not being the top classifier at any individual depth.LSTM is best in three of eight event cases, while sequential classifiers perform best in the majority of cases.
- Stance-specific behavior: LSTM handles support, deny and query best, while comments remain harder to distinguish because few features clearly characterize them.Comments are more likely to occur later in conversations, but early comments are difficult to classify from this feature alone.
- Stance-specific features: Supporting tweets tend to include links and resemble the source tweet more closely, querying tweets more often contain question marks, and denying tweets more often use negating words.Supporting links may indicate evidence, while negations also occur in other stance categories, complicating denial classification.
8. Conclusions and Future Work
The paper evaluates sequential classifiers for four-way rumour stance classification by exploiting conversational-thread structure, finding consistent advantages for LSTM and identifying directions for extending the system.
- The study classifies tweets as supporting, denying, querying, or commenting on a rumour’s truth value using conversational-thread structure rather than treating tweets in isolation.
- Four sequential classifiers are evaluated: Hawkes Process, Linear CRF, Tree CRF, and LSTM.
- Sequential modelling of Twitter interactions can produce substantial improvements over isolated-tweet stance classification.
- LSTM with a reduced feature set performs best among the sequential classifiers and remains consistent across datasets and stance types.The paper attributes this performance to native context handling and processing branches rather than entire trees.
- The work presents the first attempt described by the authors to aggregate Twitter conversational structure for tweet-level classifications and suggests applying sequential classifiers to other tasks.Suggested applications include sentiment analysis, tweet geolocation, language identification, and event detection.
- Future work targets features for difficult, infrequent denying and querying stances, whole-conversation LSTM modelling, and integration with rumour detection and veracity classification.
Appendix A.1. Local Features
Local features represent each tweet independently through lexical, formatting, punctuation, textual, and other surface-level properties.
- Local features are extracted from each tweet in isolation, without requiring other thread features.
- The local feature groups include lexicon, content formatting, punctuation, and tweet formatting.
- Lexical features include word embeddings, part-of-speech tags, negation counts, and swear-word counts.
- Content and punctuation features include tweet length, word count, question marks, and exclamation marks.
- Tweet formatting includes whether the tweet contains at least one URL.
Appendix A.2. Contextual Features
Contextual features capture relational, structural, and social properties that connect a tweet to its source, preceding tweet, thread, author, and conversation position.
- Relational features measure similarity between the current tweet and the source tweet, preceding tweet, or other tweets in the thread.These similarities are computed using Word2Vec representations.
- Structural features indicate whether a tweet is a leaf, the source tweet, or authored by the source user.
- Social and temporal features include favourites, retweets, author persistence, and elapsed time from the source tweet.
Appendix A.3. Hawkes Features
The Hawkes feature set represents tweet content and posting time for modelling temporal activity in conversations.
- Hawkes features include a bag-of-words representation of tweet content and the UNIX timestamp of posting.