Source-linked AI summary
Learning Reporting Dynamics during Breaking News for Rumour Detection in Social Media
Arkaitz Zubiaga, Maria Liakata, Rob Procter
TL;DR
Breaking-news social media contains unverified information that is difficult to distinguish from verified reports, while existing automatic detection has limited coverage of new rumours. The paper uses a sequential CRF to learn reporting dynamics from five Twitter events and classify rumours from tweet content with event context rather than querying replies. Across these datasets, the approach improves on non-sequential classifiers and the state-of-the-art reply-querying baseline, while the data remain restricted to highly retweeted tweets.
Problem
Existing automatic rumour detection has been limited, while unverified social-media information during breaking news is difficult to distinguish and may later prove false.
Method
A Linear-Chain CRF learns reporting dynamics during breaking news and classifies a tweet using event context and its content without requiring querying tweets.
Results
The sequential classifier substantially outperforms non-sequential classifiers and beats the state-of-the-art baseline that uses querying replies to detect rumours.
Takeaways & Limitations
Sequential context can substantially boost rumour-detection performance across five breaking-news datasets and reporting phases.
Takeaways & Limitations
The experiments are restricted to a subset of highly retweeted tweets.
Abstract
from arXiv · showhide
Breaking news leads to situations of fast-paced reporting in social media, producing all kinds of updates related to news stories, albeit with the caveat that some of those early updates tend to be rumours, i.e., information with an unverified status at the time of posting. Flagging information that is unverified can be helpful to avoid the spread of information that may turn out to be false. Detection of rumours can also feed a rumour tracking system that ultimately determines their veracity. In this paper we introduce a novel approach to rumour detection that learns from the sequential dynamics of reporting during breaking news in social media to detect rumours in new stories. Using Twitter datasets collected during five breaking news stories, we experiment with Conditional Random Fields as a sequential classifier that leverages context learnt during an event for rumour detection, which we compare with the state-of-the-art rumour detection system as well as other baselines. In contrast to existing work, our classifier does not need to observe tweets querying a piece of information to deem it a rumour, but instead we detect rumours from the tweet alone by exploiting context learnt during the event. Our classifier achieves competitive performance, beating the state-of-the-art classifier that relies on querying tweets with improved precision and recall, as well as outperforming our best baseline with nearly 40% improvement in terms of F1 score. The scale and diversity of our experiments reinforces the generalisability of our classifier.
1 Introduction
Breaking news streams make verified information difficult to distinguish from rumours, motivating systems that flag unverified posts. The paper proposes sequential rumour detection that learns reporting dynamics during events and avoids relying on querying tweets.
- Fast-paced social-media reporting makes it harder to distinguish verified information from rumours.
- Flagging unverified posts could help limit the spread of information later shown to be false and reduce associated harms.
- The paper introduces a journalist-developed bottom-up methodology for collecting diverse rumours and non-rumours, including stories not known beforehand.
- Linear-Chain CRFs learn event-level reporting dynamics and classify rumours using context alongside the tweet’s content, without requiring querying tweets.
- Experiments across five breaking-news datasets show consistent improvements across events and reporting phases, including early stages.
2 Background: Definition of Rumour
The paper defines rumours in breaking news as circulating stories that appear credible but remain difficult to verify and provoke skepticism or anxiety. Rapidly assessing whether information is unverified is important as events unfold.
- A rumour is information or a story passed between people whose truth has not been proven.
- In breaking news, a rumour is a credible-looking story of questionable veracity that is hard to verify and motivates seeking the truth.
- Quickly assessing whether information remains unverified is crucial while breaking news develops.
3 Related Work
Prior research has largely tracked rumours known in advance or classified reactions to them, leaving limited automatic detection of new rumours. This paper instead uses sequential context from breaking-news reporting to identify new rumours.
- Automatic rumour-detection research has been limited, with much prior work focused on rumours known a priori.
- The only identified approach for detecting new rumours uses skeptical or enquiring replies as evidence that a post is rumourous.
- The paper introduces a context-aware sequential classifier that examines reporting dynamics during breaking news to detect new rumours.
- Stance-classification studies categorize tweets discussing rumours as supporting, denying, or questioning rather than directly performing rumour detection.
4 Dataset
The dataset was built from five breaking-news events by collecting highly retweeted tweets and their replies, then having journalists annotate timeline entries as rumours or non-rumours. The resulting distributions varied substantially across events and over time.
- Data collection: Tweets were collected from Twitter’s streaming API after journalists identified newsworthy events likely to generate rumours.
- Events: The study covers five breaking-news events: Ferguson unrest, Ottawa shooting, Sydney siege, Germanwings crash, and Charlie Hebdo.
- Sampling and conversations: Tweets were sampled using a threshold of 100 retweets, and replies were collected to reconstruct complete conversations.
- Annotation: Journalists annotated time-sorted timeline tweets as rumours or non-rumours, consulting replies when context was needed while keeping annotation independent of that context.
- Dataset composition: 5,802 tweets were annotated: 1,972 rumours and 3,830 non-rumours, with substantially different ratios across the five events.
- Temporal distribution: Rumour prevalence followed different temporal patterns across events, so report earliness alone could not reliably indicate rumour status.
5 Rumour Detection Task
Rumour detection operates on an evolving timeline of tweets, assigning each tweet a rumour or non-rumour label to flag unconfirmed information. Rumour ratios can be examined across successive timeline deciles as events unfold.
- Figure 2 tracks rumour ratios for timeline deciles to show how they evolve during each event.
- The task takes an evolving timeline of tweets as input and labels each tweet R or NR.
- Flagged rumours warn users that information has not been confirmed and may later prove false.
- Performance is evaluated using precision, recall, and F1 for the rumour category.
6 Learning Sequential Dynamics for Rumour Detection
The approach models breaking-news tweets as sequential reports, using event context alongside tweet content rather than relying on public enquiry reactions. CRF is evaluated against non-sequential and enquiry-based baselines under cross-event testing.
- Hypothesis: The method addresses cases where a tweet alone lacks evidence by leveraging how the whole breaking-news event unfolds.Earlier rumours and non-rumours provide context for fitting the current tweet into the developing story.
- Sequential classifier: CRF models tweets as a sequence whose labels depend on tweet features and neighbouring label probabilities.Each tweet is represented as a vertex linked to its preceding tweet in the event.
- Sequential classifier: The classifier uses preceding event predictions as context, but early errors can propagate to subsequent tweets.It uses predictions rather than ground-truth annotations when classifying later tweets.
- Baselines: Maximum Entropy provides a non-sequential comparison by classifying tweets individually while ignoring preceding tweets.Naive Bayes, SVM, and Random Forests are additional non-sequential baselines.
- Baselines: The enquiry-based baseline labels a tweet as rumourous when at least one reply matches a curated enquiry regular expression.The reproduced approach uses replies responding to each tweet to identify enquiry posts.
- Evaluation: Evaluation uses five-fold cross-validation, training on four events and testing on a completely unknown event in each fold.Outputs from all five runs are aggregated using micro-averaged evaluation.
- Features: Features combine tweet content and social information, including word vectors and author experience or reputation indicators.The social features include tweet count, listed count, follow ratio, account age, and verification status.
7 Results
CRF generally outperforms non-sequential classifiers, especially with content-based or combined features, though performance varies across social-feature settings and event stages.
- 7.1 Comparison of Classifiers: CRF balances precision and recall better than the other classifiers with content-based features, achieving the highest F1 score.Naive Bayes has higher recall but substantially poorer precision.
- 7.1 Comparison of Classifiers: Social features alone perform significantly worse than content-based features, indicating that social features are insufficient by themselves.SVM obtains the best F1 score among social-feature classifiers, while CRF retains the best precision.
- 7.1 Comparison of Classifiers: CRF achieves the best F1 score with combined content and social features, improving 39.9% over Naive Bayes.The improvement mainly comes from higher recall, offset by a slight precision drop.
- 7.1 Comparison of Classifiers: Recall for the enquiry-based baseline is as low as 0.065, despite relatively good precision.Its manually curated regular expressions also match non-rumours and require substantial manual adaptation for broader coverage.
- 7.2 Consistency of the Sequential Classifier’s Performance: CRF outperforms the best non-sequential classifier in 43 of 50 event deciles.The seven exceptions are cases where another classifier performs better.
- 7.2 Consistency of the Sequential Classifier’s Performance: CRF has a cold-start problem in early event stages, especially for the Charlie Hebdo and Ottawa shootings.Its performance is better in the second decile than the first across all five events.
- 7.2 Consistency of the Sequential Classifier’s Performance: Event characteristics, rather than classifier choice, appear to drive shared performance increases and decreases across deciles.Charlie Hebdo and Sydney siege show notable late-stage performance drops.
8 Discussion
Across five breaking-news datasets, sequentially exploiting earlier-tweet context improves rumour detection and outperforms non-sequential and enquiry-based approaches. The experiments, however, are restricted to highly retweeted tweets, requiring at least 100 retweets.
- 8 Discussion: Sequential context produces significant improvements over standard non-sequential classifiers across five breaking-news datasets.CRF substantially outperforms Maximum Entropy and other non-sequential classifiers.
- 8 Discussion: CRF beats the Zhao et al. state-of-the-art baseline while classifying tweets from their content and earlier context without waiting for reactions.The baseline relies on querying reactions and has limited recall.
- 8 Discussion: The dataset annotation covers diverse rumours and non-rumours not necessarily known a priori, using journalists and tweet timelines.This bottom-up procedure differs from approaches that begin with preidentified rumour stories.
- 8 Discussion: Experiments are limited to tweets retweeted at least 100 times, so classifier input requires reaching that threshold.The paper identifies prompt identification of highly retweeted tweets as necessary for earlier rumour detection.
9 Conclusion
The approach uses preceding Twitter context with a sequential classifier to improve rumour detection during breaking news. Across five journalist-annotated datasets, it outperforms a state-of-the-art system while balancing precision and recall, and the datasets are publicly available for further research.
- The sequential classifier leverages context preceding a tweet to substantially boost rumour-detection performance.It processes the preceding context as a sequence rather than relying only on isolated tweet content.
- Across five breaking-news Twitter datasets annotated by journalists, the approach achieves superior and better-balanced precision and recall than the state-of-the-art system.The comparison system relies on manually curated regular expressions and performs well in precision but poorly in recall.
- The annotated datasets are publicly available to enable and encourage further research in rumour detection.