Source-linked AI summary
A large-scale COVID-19 Twitter chatter dataset for open scientific research -- an international collaboration
Juan M. Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, Katya Artemova, Elena Tutubalina, Gerardo Chowell
TL;DR
Researchers need openly available social-media data to study COVID-19 alongside biomedical and epidemiological evidence. This paper constructs and processes a large COVID-19 Twitter dataset, with daily collection totals illustrating its scale and supporting research on pandemic-related social dynamics.
Problem
Open scientific research needs additional social-media data to measure COVID-19’s social dynamics alongside biomedical and epidemiological analyses.
Method
The authors collect COVID-19 tweets through Twitter’s Stream API and collaborator datasets, then deduplicate, standardize, and preprocess them into shareable metadata.
Results
4,160,194 tweets were collected on March 13, 2020, including 751,804 clean tweets after processing.
Takeaways & Limitations
The dataset can support analyses of COVID-19-related symptoms, risk factors, spread patterns, and crisis communication using openly available Twitter data.
Abstract
from arXiv · showhide
As the COVID-19 pandemic continues its march around the world, an unprecedented amount of open data is being generated for genetics and epidemiological research. The unparalleled rate at which many research groups around the world are releasing data and publications on the ongoing pandemic is allowing other scientists to learn from local experiences and data generated in the front lines of the COVID-19 pandemic. However, there is a need to integrate additional data sources that map and measure the role of social dynamics of such a unique world-wide event into biomedical, biological, and epidemiological analyses. For this purpose, we present a large-scale curated dataset of over 152 million tweets, growing daily, related to COVID-19 chatter generated from January 1st to April 4th at the time of writing. This open dataset will allow researchers to conduct a number of research projects relating to the emotional and mental responses to social distancing measures, the identification of sources of misinformation, and the stratified measurement of sentiment towards the pandemic in near real time.
1. Introduction
COVID-19 created an urgent need for openly available data capturing social dynamics alongside biomedical and epidemiological research. Twitter offers a large source of crisis-communication data that can support monitoring pandemic-related patterns.
- Open data is needed to help the scientific community advance research during the COVID-19 pandemic.
- Twitter and Facebook provide abundant text data for research on COVID-19-related social dynamics.
- Twitter data can support analysis of symptoms, risk factors, virus origins, genetics, and spread patterns.
2. Methods
The dataset combines keyword-based Twitter collection with contributed tweet files, then standardizes, deduplicates, and preprocesses the records. It provides both full and retweet-free clean versions for different research uses.
- Collection expanded after March 12, 2020, when dedicated COVID-19 keywords replaced the initial three-keyword process.The Twitter Stream API provides only a one percent sample of the daily stream, and an additional 30+ million tweets were contributed from January 27 to March 27, 2020.
- The full dataset retains tweets and retweets to support tracing important tweets and their dissemination.
- A clean dataset removes retweets and is intended for NLP researchers, with frequent terms, bigrams, and trigrams also released.
- Contributed files were converted to the local format, deduplicated, and restricted to unique tweet identifiers.
- Preprocessing extracts shareable metadata, including tweet identifiers, collection dates, and collection times.
3. Data Validation and Control
The dataset is distributed through Zenodo as multiple files containing tweet data, statistics, and extracted lexical, emoji, hashtag, and mention summaries. Validation examples report daily tweet counts and frequent terms or symbols.
- The Zenodo repository contains 7 files describing the released dataset and its utilities.Tweet identifiers can be hydrated into JSON objects, although deleted tweets or accounts may no longer be downloadable.
- 4,160,194 tweets and 751,804 clean tweets were collected on March 13, 2020.
- The term covid19 appeared 1,767,060 times, while the bigram covid 19 appeared 1,467,434 times.
- The trigram coronavirus covid 19 appeared 52,143 times.
- Released summaries include daily frequencies for emojis, hashtags, and mentions, including face with tears of joy, #socialdistancing, and @whitehouse.
4. Re-use Potential
The resource supports reuse by providing preprocessing and parsing software alongside tweet identifiers that can be hydrated into tweet data. Researchers can use these tools to extract standardized metadata and clean records.
- The dataset includes frequent terms, bigrams, trigrams, emojis, hashtags, and mentions for downstream research use.
- The authors provide preprocessing, cleaning, and parsing software in a GitHub processing-code directory.
- Tweets must first be hydrated with tools such as SMMT or twarc before the parsing scripts extract tweet metadata.