Source-linked AI summary
Stance Classification in Rumours as a Sequential Task Exploiting the Tree Structure of Social Media Conversations
Arkaitz Zubiaga, Elena Kochkina, Maria Liakata, Rob Procter, Michal Lukasik
TL;DR
Prior stance-classification work treated tweets as separate units, leaving Twitter conversation structure unmodeled for this task. This paper applies linear-chain and tree CRFs to reply conversations; its reported results favor sequential methods on macro-F1, with tree CRF outperforming non-sequential classifiers. The study assumes rumours have already been identified or supplied by a human.
Problem
Earlier rumour-stance approaches treated each tweet as a separate unit, leaving conversational reply structure unused for tweet-level stance classification.
Source: §1 · p. 1Method
The paper compares linear-chain CRFs over conversation branches and tree CRFs over whole reply trees with non-sequential classifiers.
Source: §5.1 · pp. 5–6, §7 · p. 9Results
Tree CRF outperforms non-sequential classifiers, and exploiting conversational structure yields significant improvements in stance classification.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
The results support exploring sequential classifiers for tweet-level tasks that may benefit from whole-conversation structure.
Source: §7 · p. 10The approach assumes rumours have already been identified or entered by a human, rather than detecting them itself.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Rumour stance classification, the task that determines if each tweet in a collection discussing a rumour is supporting, denying, questioning or simply commenting on the rumour, has been attracting substantial interest. Here we introduce a novel approach that makes use of the sequence of transitions observed in tree-structured conversation threads in Twitter. The conversation threads are formed by harvesting users' replies to one another, which results in a nested tree-like structure. Previous work addressing the stance classification task has treated each tweet as a separate unit. Here we analyse tweets by virtue of their position in a sequence and test two sequential classifiers, Linear-Chain CRF and Tree CRF, each of which makes different assumptions about the conversational structure. We experiment with eight Twitter datasets, collected during breaking news, and show that exploiting the sequential structure of Twitter conversations achieves significant improvements over the non-sequential methods. Our work is the first to model Twitter conversations as a tree structure in this manner, introducing a novel way of tackling NLP tasks on Twitter conversations.
1 Introduction
Rumour stance classification assigns tweets a stance toward a rumour, and aggregating those stances may help assess its veracity. This work investigates whether conversational reply structure can improve tweet-level classification.
1 Introduction
- Aggregating stances across tweets may help assess a rumour’s veracity and flag highly disputed rumours that are likely false.The paper presents this as a potential benefit of the task.Source: §1 · p. 1
- Earlier tweet-level approaches leave conversational context and interaction structure unused, although replies can build on one another’s stances.A source tweet may receive supporting replies or replies offering counter-evidence.Source: §1 · p. 1
- The paper evaluates whether sequential structure in Twitter conversations improves stance classification over classifiers that treat each tweet alone.It compares linear branches and whole-tree models with non-sequential baselines.Source: §1 · pp. 1–2
- The proposed approach models conversation threads as either separate linear branches or whole trees and tests two CRF settings against non-sequential baselines.The comparison includes a non-sequential Maximum Entropy classifier.Source: §1 · p. 2
2 Related Work
Prior rumour-stance work includes binary classification, breaking-news approaches, and studies of stance or interaction structure in other settings. This paper’s related-work discussion distinguishes its four-way, tree-structured tweet classification from those lines of work.
2 Related Work
- Early rumour-stance work classified tweets as supporting or denying, while later approaches included rules and models for classifying new tweets about the same rumour.The cited rule-based method may not generalise to unseen rumours; the cited Hawkes Process work used temporal sequences and four stance classes.Source: §2 · p. 2
- Unlike SemEval-2016 Task 6, this work addresses rumours and breaking news and uses relations between tweets rather than treating them as isolated instances.Task 6 covered three-way stance classification for non-rumour targets.Source: §2 · p. 2
- Related work on breaking-news rumours includes four-way stance classification, though one cited approach used overlapping rumours for training and testing.The paper notes that Lukasik et al. (2016b) used Hawkes Processes to exploit temporal sequences while treating tweets as separate units.Source: §2 · pp. 2–3
- The paper adopts four classes—supporting, denying, querying, and commenting—to capture responses beyond clear support or opposition.Querying seeks more information, while commenting does not contribute to resolving veracity.Source: §2 · p. 3
- Earlier conversation-structure studies used linear reply sequences or forum-thread models, but did not exploit the full Twitter reply tree for stance classification.The paper also distinguishes its task from work on agreement or disagreement, which concerns whether posts share a view rather than their orientation toward a rumour.Source: §2 · p. 3
3 Stance classification using the conversational structure of Twitter threads
The task assigns each tweet in a conversation about a new, unseen rumour one of four stance labels. It uses the nested reply structure as context for classifying each tweet.
3 Stance classification using the conversational structure of Twitter threads
- Each tweet about a new, unseen rumour is classified as supporting, denying, querying, or commenting.The four-way scheme includes skeptical questions and comments that do not resolve the rumour’s veracity.Source: §3 · pp. 3–4
- The classification target is each tweet’s stance toward the rumour, not merely its agreement with another post.The broader scheme accounts for users who ask questions or comment without resolving veracity.Source: §3 · pp. 3–4
- A Twitter conversation is a nested reply tree rooted at a source tweet, with replies able to receive further replies.Tweet indices encode successive child relationships while traversing the conversation.Source: §3 · p. 4
4 Dataset
The study uses English tweets from eight PHEME breaking-news events, adapting reply annotations to the paper’s four stance labels. The event data are imbalanced toward commenting tweets.
4 Dataset
- The study uses the PHEME rumour dataset’s eight English-language breaking-news events, whose tweets have stance annotations and nested reply conversations.The dataset includes a ninth event in German, which this study excludes.Source: §4 · p. 4
- The dataset contains 4,519 tweets, and adapting its reply annotations to the stance labels changes only 24 tweets (0.53%).Replies marked as agreeing or disagreeing are mapped to support or denial relative to the rumour, using the source tweet’s stance.Source: §4 · p. 4
- Commenting is the dominant category, and its imbalance varies across the eight events, making the event-held-out evaluation more challenging.The authors describe this distribution as more imbalanced than in much prior binary stance work.Source: §4 · p. 4
- Figure 1 presents a rumour thread with stance labels attached to individual tweets.The example includes a source tweet and its replies.Source: Figure 1 · p. 5, §4 · p. 5
- Table 1 gives category distributions for the dataset’s eight events.The caption identifies the categories and event scope but provides no values in the supplied passage.Source: Table 1 · p. 5
5 Experiment Design
The experiments compare sequence models of Twitter conversations with tweet-level classifiers, using local tweet features and an event-held-out evaluation. They assess performance with micro- and macro-averaged F1 scores.
5.1 Classifiers
- Linear CRF models each branch as a sequence, while Tree CRF models the whole conversation; both are compared with tweet-level MaxEnt and four additional baselines.The additional baselines are Naive Bayes, SVM, Random Forests, and Majority.Source: §5.1 · p. 6
5 Experiment Design
- Eight-fold cross-validation trains on seven events and tests on the remaining event, then aggregates outputs across runs for micro-averaged evaluation.This setup simulates training on past events to classify tweets from new events.Source: §5.1 · p. 6
5.2 Features
- Four feature groups use tweet-local information: lexicon, content formatting, punctuation, and tweet formatting.Features include word embeddings, POS counts, negation and swear-word indicators, tweet length and word count, punctuation, URLs, pictures, and source-tweet status.Source: §5.2 · p. 7
5.3 Evaluation Measures
- Micro-averaged F1 is equivalent to accuracy, while macro-averaged F1 weights each category equally to account for class imbalance.The study uses both measures.Source: §5.3 · p. 7
6 Results
Conditional classifiers achieve the strongest macro-F1 results, with Tree CRF slightly outperforming Linear CRF and both significantly outperforming non-sequential Maximum Entropy. Tree CRF also performs best for supporting and querying tweets, outperforms Hawkes Processes on macro-F1 across the four comparable events, and performs best at seven of ten tweet depths.
6 Results
- Only MaxEnt, Linear CRF, and Tree CRF reach macro-F1 of at least 0.4; Tree CRF slightly outperforms Linear CRF, and both significantly outperform Maximum Entropy.The authors argue for macro-F1 because the dataset is imbalanced toward commenting tweets, allowing even the majority classifier to score well on micro-F1.Source: §6 · pp. 7–8, Table 2 · p. 8
- Tree CRF shows marked gains for supporting and querying tweets, performing best for both, while all classifiers struggle with denials; Tree CRF predicts comments for 75.8% of actual denials.The authors suggest that denials’ minority status and overlap with comments may contribute to the difficulty, and that collecting additional denying tweets may help.Source: §6 · p. 8, Table 3 · p. 8
- Across the four events with comparable results, Hawkes Processes perform better on micro-F1, while Tree CRF performs substantially better on macro-F1.Lukasik et al. (2016b) tested only four events; the authors emphasize macro-F1 for this imbalanced task because even a majority classifier achieves high micro-F1.Source: §6 · p. 8, Table 4 · p. 9
- Tree CRF has the best macro-F1 at seven of ten tweet depths; Linear CRF is best at depth 2, and Maximum Entropy at depths 4 and 5.Depths beyond 9 are omitted because few instances are available; micro-F1 shows little difference across classifiers except for CRF gains on source tweets, which tend to support rumours.Source: §6 · p. 9, Figure 3 · p. 9
7 Conclusions
The paper models tweet-level rumour stance using sequential classifiers over reply-tree conversations, evaluated on eight breaking-news datasets. The Tree CRF outperforms non-sequential classifiers, and the authors identify prior rumour identification as an assumption of their approach.
7 Conclusions
- The study compares a branch-based linear-chain CRF and a whole-tree CRF against Maximum Entropy and other baselines on eight breaking-news Twitter datasets.Both sequential classifiers model reply structure rather than treating each tweet as an isolated unit.Source: §7 · p. 9
- Tree CRF outperforms all non-sequential classifiers, with significant improvements from conversational transitions even when using only local features.The authors also report that modelling a branch’s linear sequence is useful and that the whole-tree view can further boost performance.Source: §7 · p. 10
- The authors describe this as the first attempt to aggregate Twitter-thread structure for tweet-level classification and suggest studying sequential classifiers for other tweet-level NLP tasks.Examples named include sentiment analysis and language identification.Source: §7 · p. 10
- The approach assumes rumours have already been identified or supplied by a human; future work includes testing contextual features and additional classifiers.The authors also propose developing a rumour-detection system whose output would feed into stance classification.Source: §7 · p. 10