Source-linked AI summary
Relevance Classification of Flood-related Twitter Posts via Multiple Transformers
Wisal Mukhtiar, Waliiya Rizwan, Aneela Habib, Yasir Saleem Afridi, Laiq Hasan, Kashif Ahmad
TL;DR
Disaster analytics faces noisy and irrelevant social media content, complicating the classification of flood-related tweets. The paper applies several transformers individually and in equal-weight late fusion; on the test set, the best two-model pair improved by 20% over the three-model fusion.
Problem
Social media posts contain noise and irrelevant information, while the task is to classify flood-related tweets as relevant or non-relevant.
Source: §1 · pp. 1–2Method
The framework preprocesses tweets, trains BERT, RoBERTa, and XLNet, and combines their classification scores through equal-weight late fusion.
Source: §3 · p. 2, §3.2 · p. 3, §3.3 · p. 4Results
20% improvement over the three-model fusion was obtained by the best-performing pair of models on the test set.
Source: §4 · p. 4Takeaways & Limitations
The authors propose late fusion of transformers for the RCTP subtask and aim to explore merit-based fusion methods in future work.
Source: §5 · p. 5
Abstract
from arXiv · showhide
In recent years, social media has been widely explored as a potential source of communication and information in disasters and emergency situations. Several interesting works and case studies of disaster analytics exploring different aspects of natural disasters have been already conducted. Along with the great potential, disaster analytics comes with several challenges mainly due to the nature of social media content. In this paper, we explore one such challenge and propose a text classification framework to deal with Twitter noisy data. More specifically, we employed several transformers both individually and in combination, so as to differentiate between relevant and non-relevant Twitter posts, achieving the highest F1-score of 0.87.
1. Introduction
Disaster response depends on access to relevant, timely information, but social media posts can contain noise and irrelevant content. This paper addresses that challenge through the Relevance Classification of Twitter Posts task, which classifies flood-related tweets as relevant or non-relevant.
1. Introduction
- Access to relevant and timely information is key to rapid, effective disaster response, yet the literature reports situations where such access may not be possible.Natural disasters can cause significant damage to infrastructure and human lives.Source: §1 · p. 1
- Social media has been explored for emergency communication and information dissemination, as well as disaster information collection and damage assessment.Source: §1 · p. 1
- The Relevance Classification of Twitter Posts task targets noisy, irrelevant social media content by classifying flood-related tweets as relevant or non-relevant.It was introduced as a sub-task in the DisasterMM challenge of MediaEval 2022.Source: §1 · pp. 1–2
2. Related Work
Prior disaster analytics work uses social media for communication, damage assessment, and other tasks, with flood analysis prominent in the literature. MediaEval has also addressed several distinct aspects of flood analysis over successive years.
2. Related Work
- Social media disaster analytics spans communication, damage assessment, response management, sentiment analysis, and identifying affected individuals’ needs.Examples include visual-media damage assessment and post-flood road-condition monitoring.Source: §2 · p. 2
- Floods are the most common disaster focus in the literature described, including work on passable-road detection and post-flood damage-severity assessment.Source: §2 · p. 2
- MediaEval’s flood-related tasks shifted across years from image retrieval in 2017 to multimodal classification in 2018, then flood-severity and event-recognition tasks in 2019 and 2020.Source: §2 · p. 2
3. Approach
The framework cleans Twitter text, obtains classification scores from BERT, RoBERTa, and XLNet, and combines model scores through late fusion. The models are treated equally in the fusion step.
3. Approach
- The framework has three stages: preprocessing, transformer training and classification, and late fusion of classification scores.Source: §3 · p. 2
3.1. Pre-processing
- Preprocessing removes usernames, URLs, emojis, punctuation, and stop words, then transforms the raw text for transformer use with TF.text.Source: §3.1 · p. 3
3.2. Classification via Transformers
- BERT, RoBERTa, and XLNet are trained to produce classification scores; their selection is motivated by performance on similar tasks.The models use distinct fine-tuning settings, including different learning rates, batch sizes, and epoch counts.Source: §3.2 · p. 3
3.3. Fusion
- Late fusion aggregates the individual model scores equally to produce the final classification score.The models are assigned equal weights in the current implementation.Source: §3.3 · p. 4
4. Results and Analysis
On the test set, the best-performing two-model fusion improved on the three-model fusion by 20%. Development-set results favored BERT among individual models and the BERT–XLNet pair among fusion methods.
4. Results and Analysis
- On the development set, BERT achieved the overall better individual-model results, while the BERT–XLNet pair achieved the better fusion results.The authors report that RoBERTa had a lower F1-score and suggest its less accurate predictions as a potential reason for the lower performance of fusing all three models.Source: §4 · p. 4, Table 1 · p. 4
- 20% improvement over the three-model fusion was obtained by the best-performing pair of models on the test set.The official test submissions included one three-model fusion and two pairwise fusions.Source: §4 · p. 4, Table 2 · p. 5
5. Conclusions
The paper presents its solutions for DisasterMM's RCTP subtask through a late-fusion framework incorporating several state-of-the-art transformers. Its current implementation treats models equally, while future work aims to use merit-based fusion to improve the final classification score.
5. Conclusions
- The proposed solution for DisasterMM's RCTP subtask is a late-fusion framework incorporating several state-of-the-art transformers.Source: §5 · p. 5
- In the current implementation, all models receive equal weights of 1.Source: §5 · p. 5
- Future work aims to employ merit-based fusion methods to improve the final classification score.Source: §5 · p. 5