Source-linked AI summary
Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages
Muhammad Imran, Prasenjit Mitra, Carlos Castillo
TL;DR
Crisis-related Twitter messages contain information useful for situational awareness, but noisy short texts and scarce human-annotated data hinder processing. The paper builds annotated corpora from 19 crises, trains classifiers and crisis-specific embeddings, and releases normalized lexical resources. The authors report that these resources support classification and language-processing research for crisis messages and humanitarian response.
Problem
Crisis-message processing requires human-annotated data to handle noisy, brief, informal tweets and classify their information.
Method
The paper collects and annotates crisis tweets, trains machine-learning classifiers and 300-dimensional word2vec embeddings, and creates normalized lexical resources for out-of-vocabulary terms.
Results
The corpora cover over 52 million crisis-related tweets from 19 crises, with annotations empirically evaluated using classifiers whose results are reported across eight disaster datasets.
Takeaways & Limitations
The released annotations, embeddings, and normalization resources are intended to support improved automatic processing of crisis-related messages and humanitarian organizations.
Abstract
from arXiv · showhide
Microblogging platforms such as Twitter provide active communication channels during mass convergence and emergency events such as earthquakes, typhoons. During the sudden onset of a crisis situation, affected people post useful information on Twitter that can be used for situational awareness and other humanitarian disaster response efforts, if processed timely and effectively. Processing social media information pose multiple challenges such as parsing noisy, brief and informal messages, learning information categories from the incoming stream of messages and classifying them into different classes among others. One of the basic necessities of many of these tasks is the availability of data, in particular human-annotated data. In this paper, we present human-annotated Twitter corpora collected during 19 different crises that took place between 2013 and 2015. To demonstrate the utility of the annotations, we train machine learning classifiers. Moreover, we publish first largest word2vec word embeddings trained on 52 million crisis-related tweets. To deal with tweets language issues, we present human-annotated normalized lexical resources for different lexical variations.
1. Introduction
The paper introduces human-annotated crisis-related Twitter corpora to support classification and informal-language normalization. It also provides classifiers, crisis-trained word embeddings, and normalized lexical resources.
- Contributions: Human annotations cover information categories and out-of-vocabulary terms with corrections and normalized forms.The category schemes incorporate input from formal crisis-response agencies, including UN OCHA.
- Contributions: The corpus contains around 50,000 annotated messages collected across countries and seasons, providing linguistic diversity for applied machine learning.The authors frame this diversity as useful for testing solutions beyond specific English dialects.
- Contributions: The authors train Naive Bayes, Random Forest, and Support Vector Machines classifiers on the annotated data.These classifiers target messages useful for humanitarian efforts and support further computational methods.
- Contributions: The paper provides human-annotated normalized lexical resources for different lexical variations.These resources address informal and out-of-vocabulary language in crisis-related tweets.
2. Crises Corpora Collection and Annotation
The corpora comprise crisis-related Twitter messages collected across 19 crises and annotated for information types and language variation. Annotation tasks use worker judgments and agreement-based category assignment.
- Data collection: Messages were collected from 19 crises occurring between 2013 and 2015 using the AIDR platform.AIDR supports Twitter streaming-API collection strategies based on keywords and geographic areas.
- Data annotation: The annotation scheme categorizes tweets by information types such as casualties, missing people, displacement, infrastructure damage, donations, advice, emotional support, and irrelevant content.The scheme uses a subset of annotations developed with input from UN OCHA.
- Data annotation: Each annotation task presents a tweet and category list for a paid worker or volunteer to select the most suitable category.Messages containing important information outside specific categories receive the “Other Useful Information” label.
- Data annotation: A category is assigned when three different volunteers or paid workers agree.This agreement rule finalizes the annotation task.
- Data collection: Table 1 records each crisis dataset’s type, name, year, message language, country, and tweet count.The caption identifies the table’s dimensions but does not report individual values here.
3. Classification of Messages
The paper frames crisis-message categorization as multiclass prediction and evaluates three standard classifiers after preprocessing Twitter text. Across eight disaster datasets, most classes reach acceptable AUC, though the smallest class performs worse.
- Classification task: Crisis-message categorization is modeled as learning a predictor from messages to one of several target classes.The classifiers evaluated are Naive Bayes, Support Vector Machines, and Random Forest.
- Preprocessing: Preprocessing removes stop-words, URLs, and user mentions, applies Lovins stemming, and uses unigram and bigram features.Information gain selects the top 1,000 features before classification.
- Evaluation: 10-fold cross-validation evaluates classifier performance using Area Under the ROC Curve across all classes in eight disaster datasets.The evaluation compares Support Vector Machines, Naive Bayes, and Random Forest models.
- Crisis word embeddings: The paper also trains crisis-related word embeddings using word2vec on 52 million Twitter messages.The embeddings use CBOW with negative sampling and 300-dimensional representations.
4. Twitter Text Normalization
The normalization pipeline identifies Twitter OOV words using lexical resources and probabilistic correction, then uses crowdsourcing to assign tags and corrected forms. It covers misspellings, abbreviations, slang, phonetic substitutions, and words without spaces.
- Normalization targets: Twitter normalization targets noisy OOV forms including misspellings, abbreviations, slang, phonetic substitutions, and words without spaces.The paper motivates normalization by Twitter’s brief, informal, noisy, and character-limited messages.
- Candidate identification: Initial vocabularies combine online resources for abbreviations, chat shortcuts, and slang with the 349,554-word SCOWL dictionary.SCOWL is used as an English spell-checking resource across most English dialects.
- Candidate identification: The MaxMind world cities database adds coverage for place names missing from SCOWL, containing 3,173,959 cities.The expanded resources are used to find OOV words in the dataset.
- Candidate correction: A language model ranks possible corrections by P(c|w), using word frequencies and restricting candidates to one edit-distance change.The model uses frequent words from Wiktionary, the British National Corpus, and SCOWL.
- Crowdsourced normalization: CrowdFlower workers assign each OOV a tag such as slang, location, organization, misspelling, or person name, and correct misspelled forms.The task presents a Twitter message containing one or more OOV words together with instructions and examples.
5. Related Work
The paper situates crisis-related Twitter processing within disaster-response research, where timely analysis of large streams can support humanitarian organizations.
- Related work: Crisis-related Twitter messages can contain information useful to humanitarian organizations when processed timely and effectively.The paper places this processing alongside machine learning and natural language processing techniques.
6. Conclusions
The paper contributes annotated crisis-message corpora, classifiers, normalized lexical resources, and word embeddings intended to support automatic processing of crisis-related Twitter messages.
- Conclusions: The corpora contain over 52 million crisis-related tweets collected during 19 crisis events.The data include annotations for topic categorization and OOV words with their normalizations.
- Conclusions: The authors build machine-learning classifiers to empirically validate the effectiveness of the annotated datasets.The classifiers address categorization of crisis-related messages.
- Conclusions: The paper provides word2vec embeddings trained on 52 million messages and human-annotated normalized lexical resources for different lexical variations.These resources are presented as tools for improving automatic NLP of crisis-related messages.