Source-linked AI summary
COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter
Martin Müller, Marcel Salathé, Per E Kummervold
TL;DR
COVID-19 Twitter content is difficult to analyse at scale, motivating a domain-specific language model. The paper pretrained CT-BERT on COVID-19 tweets and found higher classification performance than BERT-LARGE across five datasets, with the largest gain on COVID-19-related data. The model’s evaluation was limited to categorisation tasks and included only one COVID-19-related dataset.
Problem
Large-scale COVID-19 Twitter content is difficult to classify, filter, and summarise manually or with existing NLP tools.
Method
CT-BERT was developed by domain-specific pretraining of a BERT-based transformer on COVID-19 Twitter content and evaluating it through downstream classification tasks.
Results
25.88% marginal improvement on the COVID-19-specific CC dataset was the largest gain among the five evaluated classification datasets.
Takeaways & Limitations
CT-BERT is expected to perform particularly well on COVID-19-related classification problems, especially text from social media platforms.
Takeaways & Limitations
The evaluation covered categorisation tasks and included only one COVID-19-related dataset.
Abstract
from arXiv · showhide
In this work, we release COVID-Twitter-BERT (CT-BERT), a transformer-based model, pretrained on a large corpus of Twitter messages on the topic of COVID-19. Our model shows a 10-30% marginal improvement compared to its base model, BERT-Large, on five different classification datasets. The largest improvements are on the target domain. Pretrained transformer models, such as CT-BERT, are trained on a specific target domain and can be used for a wide variety of natural language processing tasks, including classification, question-answering and chatbots. CT-BERT is optimised to be used on COVID-19 content, in particular social media posts from Twitter.
1 Introduction
COVID-19 Twitter content is valuable but too large and difficult to classify manually or with existing NLP tools. The paper therefore develops CT-BERT, a domain-specific transformer model based on BERT-LARGE.
- COVID-19 Twitter content provides news and public expression but is difficult to classify, filter, and summarise at scale.
- CT-BERT addresses this gap by adapting a general transformer model to specialised COVID-19 Twitter content through domain-specific pretraining.
- Transformer models learn general language representations through unsupervised objectives such as masked language modelling and next sentence prediction.
- These pretrained representations support supervised tasks including classification, question answering, and chatbots.
- BERT-LARGE was trained mainly on Wikipedia and a book corpus, which contain little information about specific subdomains.
2 Method
CT-BERT was pretrained on a large, cleaned corpus of COVID-19 tweets and evaluated through downstream classification datasets. The method used standard BERT inputs, checkpoints, and dataset-specific finetuning settings.
- CT-BERT was trained on 160M coronavirus-related tweets collected through Crowdbreaks between January 12 and April 16, 2020.
- Tweets were cleaned by removing retweet tags and pseudonymising usernames, URLs, and emoticons before training.
- Twitter messages were represented with a 30 000-word vocabulary and sequence length of 96 tokens, enabling training batches of 1024 examples.
- The study saved checkpoints and finetuned them on downstream classification tasks to assess domain-specific pretraining.
- The evaluation used five independent Twitter-related datasets, including publicly available and internal datasets with multi-class labels.
3 Results
CT-BERT improved downstream classification performance relative to BERT-LARGE across the evaluated datasets. Pretraining metrics improved over time, but their usefulness for selecting when to stop training was limited.
- The evaluation compared CT-BERT with BERT-LARGE using mean F1 scores on five classification datasets.
- Training used different epoch counts across SST-2, CC, SE, VC, and MVC according to dataset size and category balance.
- All metrics improved during CT-BERT pretraining, with the MLM loss showing the most notable improvement and reaching a final value of 1.48.
- Marginal performance improvement was computed for each dataset because initial performance varied widely across datasets.
- Intermediate checkpoints and repeated finetuning runs were used to examine whether decreasing pretraining loss translated linearly to downstream performance.
4 Discussion
CT-BERT improves classification performance across evaluated datasets, with the largest gains on COVID-19 and health-related Twitter content. The authors note scope limits and opportunities for further optimization and broader evaluation.
- Performance across datasets: CT-BERT improves significantly over BERT-LARGE on all five evaluated Twitter-based datasets, with the largest gains on health- and COVID-19-related data.The authors expect similar performance on other COVID-19-related classification problems, particularly social-media text.
- Performance across datasets: 25.88% ∆MP is the largest reported improvement and occurs on the COVID-19-specific CC dataset.Marginal improvement is also high on the vaccine-sentiment MVS dataset, while SST-2 and SemEval improve to a smaller extent.
- Pretraining dynamics: Most COVID-19-related marginal improvement occurred after 100k pretraining steps, whereas non-Twitter SST-2 reached final performance only after 200k steps.This pattern suggests out-of-domain text may require longer pretraining for a similar performance boost.
- Variability: Performance variance is dataset dependent but does not increase across pretraining checkpoints and is comparable to BERT-LARGE.SemEval was the most stable training set and SST-2 the least stable, although most of that difference falls within error margins.
- Scope and limitations: The evaluation used categorisation tasks, while other natural-language-understanding tasks such as named entity recognition and question answering were not tested.The authors expect CT-BERT may also support those tasks within the target domain, but this remains unverified.
- Scope and limitations: Downstream finetuning was intended to produce stable evaluation results rather than optimize CT-BERT, leaving room for downstream-task optimization.Epoch counts and learning rate were optimized for BERT-LARGE rather than CT-BERT.
6 Funding
The work received support from European Commission funding schemes and Google Cloud resources for COVID-19-related research.
- Funding: PK received European Commission funding through H2020-MSCA-IF-2017 and the MSCA-IF-EF-ST VACMA project.The grant agreement ID was 797876.
- Funding: MM and MS received support through the Versatile Emerging infectious disease Observatory grant under the European Commission’s Horizon 2020 framework.The grant agreement ID was 874735.
- Funding: The research received Cloud TPUs from Google’s TensorFlow Research Cloud and Google Cloud credits for COVID-19-related research.