Source-linked AI summary
BERTweet: A pre-trained language model for English Tweets
Dat Quoc Nguyen, Thanh Vu, Anh Tuan Nguyen
TL;DR
Existing language models were mainly trained on conventional text, while Tweets contain distinctive short, informal, and irregular language. BERTweet addresses this gap with RoBERTa-style pre-training on a large English Tweet corpus and outperforms strong baselines and previous state-of-the-art models across three Tweet NLP tasks.
Problem
Tweets differ from conventional text in length, grammar, and vocabulary, and no large-scale language model pretrained on English Tweets was available.
Method
BERTweet uses the BERT-base architecture and RoBERTa pre-training procedure on 850M English Tweets.
Results
BERTweet outperforms RoBERTa-base, XLM-R-base, and previous state-of-the-art models on POS tagging, NER, and text classification.
Takeaways & Limitations
BERTweet is publicly released as a strong baseline for future Tweet research and applications.
Abstract
from arXiv · showhide
We present BERTweet, the first public large-scale pre-trained language model for English Tweets. Our BERTweet, having the same architecture as BERT-base (Devlin et al., 2019), is trained using the RoBERTa pre-training procedure (Liu et al., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base (Conneau et al., 2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks: Part-of-speech tagging, Named-entity recognition and text classification. We release BERTweet under the MIT License to facilitate future research and applications on Tweet data. Our BERTweet is available at https://github.com/VinAIResearch/BERTweet
1 Introduction
Tweets differ from conventional text through short length, informal grammar, and irregular vocabulary, creating a gap for language models pretrained on formal corpora. BERTweet fills this gap as a large-scale Tweet language model and outperforms established baselines across three Tweet NLP tasks.
- Tweets use short, informal, and irregular language, including abbreviations, typographical errors, and hashtags.
- No large-scale language model pretrained specifically on English Tweets existed to address these characteristics.
- BERTweet is trained on 80GB containing 850M English Tweets and evaluated on POS tagging, NER, and text classification.
- BERTweet is presented as the first large-scale pre-trained language model for English Tweets.
- BERTweet outperforms RoBERTa-base, XLM-R-base, and previous state-of-the-art models on the three Tweet NLP tasks.
- The model is publicly released for use with fairseq and transformers as a baseline for Tweet analytics research and applications.
2 BERTweet
BERTweet uses the BERT-base architecture with RoBERTa-style pre-training on a large, processed corpus of English Tweets. The training pipeline tokenizes, normalizes, subword-segments, and optimizes 850M Tweets over 40 epochs.
- Architecture: BERTweet uses the BERT-base architecture and a masked language modeling objective, with pre-training based on RoBERTa.
- Pre-training data: The pre-training dataset contains 80GB of uncompressed text, 850M Tweets, and 16B word tokens.
- Pre-training data: The corpus combines general Twitter data with COVID-19 Tweets collected during early 2020.
- Pre-training data: Tweets are language-identified, Tweet-tokenized, emoji-converted, and normalized by replacing mentions and URLs with special tokens.
- Pre-training data: All 850M Tweets are segmented into subwords using fastBPE with a 64K vocabulary, averaging 25 subword tokens per Tweet.
- Optimization: BERTweet is optimized with Adam for 40 epochs over about four weeks using 8 V100 GPUs.
3 Experimental setup
The experiments compare BERTweet with RoBERTa and XLM-R on benchmark Tweet datasets covering POS tagging, NER, and text classification. Evaluation uses task-specific prediction layers, normalization variants, fine-tuning, and repeated runs.
- Tasks and datasets: BERTweet is evaluated on POS tagging, NER, and text classification using benchmark Tweet datasets.
- Tasks and datasets: The evaluation includes three POS datasets, two NER datasets, and sentiment and irony classification datasets.
- Data splits: The experiments use existing or specified training, validation, and test splits across the benchmark datasets.
- Normalization: Both soft normalization and hard lexical normalization are applied to Tweet datasets.
- Task models: POS tagging and NER use a linear layer over the first subword representation, while classification uses a layer over the pooled output.
- Fine-tuning: Each task and dataset is fine-tuned for up to 30 epochs with early stopping, and final test scores average five random-seed runs.
- Baselines: The main baselines are same-configuration RoBERTa-base and XLM-R-base, with larger models also evaluated as an explicitly unfair comparison.
4 Experimental results
BERTweet generally outperforms RoBERTa-base and XLM-R-base across Tweet NLP datasets, while larger models retain advantages on POS tagging and NER. It also improves prior state-of-the-art results and shows that lexical normalization generally does not improve pretrained-model performance.
- Experimental results: Tables 1–4 report BERTweet and baseline scores under soft and hard normalization strategies.The tables cover the experimental comparisons across the reported tasks and datasets.
- Normalization: Soft normalization scores are generally higher than corresponding hard normalization scores for each pretrained language model.Applying lexical normalization dictionaries generally does not improve downstream performance.
- Comparison with base models: BERTweet outperforms RoBERTa-base and XLM-R-base on all experimental datasets except Ritter11-T-POS, where XLM-R-base performs slightly better.The comparison covers the reported Tweet NLP experiments.
- Comparison with larger models: RoBERTa-large and XLM-R-large obtain better POS tagging and NER scores than BERTweet, whereas BERTweet performs better on two text classification datasets.The larger models use significantly larger model configurations.
- Comparison with prior state of the art: 14+% absolute improvement on WNUT17 novel and emerging entity recognition exceeds the previous state-of-the-art result.BERTweet also improves prior state-of-the-art by 5% on SemEval2017-Task4A and 4% on SemEval2018-Task3A.
- Data scale and domain specificity: BERTweet performs better than competitors despite RoBERTa and XLM-R using 2 and approximately 3.75 times more English data, respectively.The comparison supports the reported effectiveness of a large-scale, domain-specific Tweet language model.
5 Conclusion
The paper presents BERTweet as the first large-scale language model pretrained for English Tweets and demonstrates its usefulness across three downstream Tweet NLP tasks. It also extends BERTweet with cased and uncased COVID-19 Tweet variants.
- BERTweet is presented as the first large-scale language model pretrained for English Tweets.
- BERTweet outperforms RoBERTa-base and XLM-R-base and improves previous state-of-the-art performance on POS tagging, NER, and text classification.The text classification tasks include sentiment analysis and irony detection.
- 23M cased and uncased COVID-19 English Tweets are used for 40 additional pretraining epochs to create two BERTweet-COVID19 variants.