Source-linked AI summary
All-in-one: Multi-task Learning for Rumour Verification
Elena Kochkina, Maria Liakata, Arkaitz Zubiaga
TL;DR
Rumour verification is difficult because its pipeline subtasks were previously developed as separate components. The paper jointly trains veracity classification with auxiliary detection and stance tasks, finding that combining all three improves performance, while dataset composition affects outcomes.
Problem
Prior work largely studied rumour-resolution subtasks separately, leaving their interaction and integration into a complete verification system underexplored.
Method
The paper uses hard-parameter-sharing multi-task learning with veracity as the main task and rumour detection and stance classification as auxiliaries.
Results
Multi-task models improve accuracy and macro F-score over single-task learning, with adding the third task bringing further improvement across reported setups.
Takeaways & Limitations
Joint learning of the verification pipeline's tasks can outperform single-task, majority-baseline, and state-of-the-art veracity systems.
Takeaways & Limitations
The authors identify task dataset-size differences and dataset composition as boundaries requiring further investigation and improved training schedules.
Abstract
from arXiv · showhide
Automatic resolution of rumours is a challenging task that can be broken down into smaller components that make up a pipeline, including rumour detection, rumour tracking and stance classification, leading to the final outcome of determining the veracity of a rumour. In previous work, these steps in the process of rumour verification have been developed as separate components where the output of one feeds into the next. We propose a multi-task learning approach that allows joint training of the main and auxiliary tasks, improving the performance of rumour verification. We examine the connection between the dataset properties and the outcomes of the multi-task learning models used.
1 Introduction
Rumour resolution comprises interconnected subtasks, but prior work largely studied them separately. The paper proposes joint multi-task learning centered on veracity classification and reports improvements when auxiliary tasks are combined.
- Rumour resolution proceeds through detection, tracking, stance classification, and final veracity prediction.
- Prior research developed the four subtasks separately, leaving their interaction and integration into a complete system underexplored.
- The proposed formulation treats veracity classification as the main task and the remaining components as auxiliary tasks that may improve it.
- The study jointly trains task combinations using a sequential deep-learning architecture and compares them with single-task and state-of-the-art veracity systems.
- Using all three subtasks produces substantial improvements over single-task, majority, and state-of-the-art systems, and outperforms two-task combinations.
2 Related work
The related work presents rumour classification as a sequential pipeline and motivates multi-task learning through related subtasks and shared representations. Prior studies emphasize temporal context, while existing approaches often address tasks independently.
- Rumour classification is represented as detection, stance classification, and verification, with tracking collecting follow-up replies rather than assigning labels.
- Rumour detection distinguishes unverified information from non-rumours, after which detected rumours proceed to stance and veracity classification.
- Stance classification predicts whether posts support, deny, question, or comment on a rumour, and skepticism has been associated with later falsity.
- Sequential classifiers have substantially outperformed non-sequential classifiers for stance classification, while sequential detection improves recall over rule-based detection.
- Multi-task learning shares hidden layers across related tasks, potentially enlarging effective training data and reducing overfitting through shared representations.
- Auxiliary tasks can direct the main task toward features it might otherwise miss; stance is especially relevant because prior work links it to veracity.
3 Data
The study uses PHEME and RumourEval, Twitter-conversation datasets with different task annotations and event compositions. PHEME supports broader evaluation across events, while RumourEval contains rumours only and therefore omits detection.
- The experiments use PHEME and RumourEval, which contain Twitter conversation threads from multiple newsworthy events and different annotation levels.
- RumourEval contains 325 rumour threads and supports stance and veracity classification but not rumour detection.
- PHEME provides annotations for rumour detection, veracity, and tweet-level stance on a subset of threads.
- PHEME event sizes and class proportions vary substantially, with fewer rumours than non-rumours overall and true as the majority rumour class.
- Experiments evaluate the five largest events separately from all nine using leave-one-event-out cross-validation.
- The five largest events are more balanced, whereas four smaller events each center on a single rumour story.
4 Models
The models use branch-level tweet sequences with shared recurrent representations and task-specific outputs, supporting sequential baselines and joint veracity learning with stance or detection.
- The branchLSTM baseline splits conversations into linear branches and feeds each branch through an LSTM, dense ReLU layers, and softmax class predictions.
- Multi-task models use a shared LSTM layer followed by task-specific layers for different combinations of stance, detection, and veracity tasks.
- Stance classification produces predictions for each tweet, while detection and verification produce one output for each tweet branch.
- The sequential NileTMRG baseline passes stance-classification outcomes into veracity classification, using support patterns as veracity indicators.
- Tweet representations are formed by averaging 300-dimensional pre-trained Google News word embeddings after preprocessing.
5 Experiment setup
The experiments tune models with a development-set objective, evaluate under dataset-specific protocols, and compare veracity performance across single-task, multi-task, and baseline systems.
- Hyperparameters are selected with Tree of Parzen Estimators by minimizing macroF-based losses that weight all tasks equally.
- Table 3 compares sequential single-task, two-task and three-task multi-task models with Majority and NileTMRG baselines on veracity classification.
- RumourEval uses its provided training, development, and testing split, with final models retrained on combined training and development data before testing.
- PHEME uses leave-one-event-out cross-validation, creating a harder setup that better approximates verification of unseen rumours.
6 Results and Discussion
Multi-task learning consistently improves rumour-veracity performance over single-task and majority baselines, with the three-task setup generally strongest. Results also show that dataset composition and label distributions affect performance across events and tasks.
- All models improve over the majority baseline in macro F-score, while multi-task models improve over single-task learning in accuracy and macro F-score.Macro-averaged F-score is emphasized because class imbalance makes majority-baseline accuracy misleading and recognizing all classes is important.
- MTL3, combining stance, veracity, and detection, outperforms both MTL2 configurations and single-task veracity classification on the reported datasets.On RumourEval, MTL2 improves over both NileTMRG* and branchLSTM; on PHEME, scores increase from single-task models to MTL2 and then MTL3.
- Using nine PHEME events produces worse performance than using five, because the four added events are small and qualitatively different from the large news-breaking events.The five large events contain all rumour classes and non-rumours, whereas the additional events concern false or unverified rumours, highlighting the difficulty of leave-one-event-out evaluation.
- Per-event and per-class results analysis: On PHEME’s five largest events, multi-task models outperform single-task approaches for every event, while Ferguson is hardest because its class distribution differs from the others.Figure 4 compares macro-averaged F-scores across the events; the event-level pattern is consistent despite differing difficulty.
- Per-event and per-class results analysis: Per-class results show that all models predict true rumours best, while single-task models favor false over unverified detection and multi-task models show the opposite pattern.Ferguson is the only event with high performance on the unverified class because it is strongly dominated by unverified rumours.
- Analysis of data properties: The observed multi-task trends align with prior links between gains and dataset properties: lower kurtosis and relatively high entropy favor multi-task learning, and stance has higher average entropy than detection.The paper reports similar trends even though the auxiliary tasks are higher-level theme-related tasks rather than low-level linguistic tasks.
7 Conclusions and Future Work
The proposed multi-task rumour verification model improves veracity classification by jointly leveraging rumour detection and stance classification. The study also links multi-task outcomes to dataset label distributions and identifies training-schedule and task-hierarchy extensions for future work.
- Conclusions: Joint learning of rumour detection, stance classification, and verification further improves veracity classification beyond single-task and two-task approaches.The three-task combination also outperforms majority and state-of-the-art baselines.
- Conclusions: The outcomes of multi-task learning are linked to properties of the dataset’s label distribution, supporting earlier findings.
- Future Work: Future work will adapt training schedules for different dataset sizes so that no task dominates the model.
- Future Work: Future work will incorporate the hierarchy between tasks and investigate additional user and interaction features in private or shared layers.