Source-linked AI summary
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
Francesco Barbieri, Jose Camacho-Collados, Leonardo Neves, Luis Espinosa-Anke
TL;DR
Social-media NLP lacks a unified evaluation framework and strong domain-specific baselines, making Twitter-model comparisons difficult. TweetEval standardizes seven tweet-classification tasks and compares generic, Twitter-only, and continued Twitter pretraining. Generic pretrained models perform strongly, while additional Twitter training can improve them, with task and dataset constraints affecting results.
Problem
Twitter NLP evaluation is fragmented, with no standardized protocol or strong domain-specific baseline set for comparing language models across tasks.
Method
TweetEval unifies seven heterogeneous English tweet-classification tasks and evaluates strong baselines plus language models using generic, Twitter-only, and continued Twitter pretraining.
Results
Using a pretrained language model may be sufficient, but continued training on in-domain Twitter data can improve results.
Takeaways & Limitations
TweetEval provides a common benchmark, reference baselines, and evidence for comparing language-model pretraining strategies in social-media NLP.
Takeaways & Limitations
The benchmark initially focuses on classification and leaves multi-label emotion, expanded emoji coverage, sequence tagging, multimodality, and code-switching for future work.
Abstract
from arXiv · showhide
The experimental landscape in natural language processing for social media is too fragmented. Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction. Therefore, it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domain-specific data. In this paper, we propose a new evaluation framework (TweetEval) consisting of seven heterogeneous Twitter-specific classification tasks. We also provide a strong set of baselines as starting point, and compare different language modeling pre-training strategies. Our initial experiments show the effectiveness of starting off with existing pre-trained generic language models, and continue training them on Twitter corpora.
1 Introduction
Social-media NLP is difficult because tweets are noisy, conversational, idiosyncratic, and constrained by platform-specific features. TweetEval addresses fragmented evaluation by standardizing seven English tweet-classification tasks and benchmarking strong models.
- Twitter text introduces challenges such as POS tagging, lexical normalization, and named entity recognition that are less prominent in generic NLP.
- The field lacks a unified evaluation framework for comparing language models across Twitter datasets and tasks.Existing benchmarks standardize evaluation in more ordinary textual genres.
- TweetEval provides a standardized English benchmark spanning seven heterogeneous tweet-classification tasks.The tasks are sentiment, emotion, offensive language, hate speech, stance, emoji, and irony detection.
- The benchmark applies unified train, validation, and test-split criteria and evaluates strong baselines and transformer models trained with different corpora.
2 TweetEval: The Benchmark
TweetEval unifies heterogeneous Twitter classification datasets, task formulations, preprocessing, and evaluation into one benchmark. Its tasks retain distinct social-media phenomena while using standardized dataset handling and reporting.
- Tasks: The benchmark covers sentiment, emotion, offensive language, hate speech, stance, emoji, and irony classification tasks drawn from shared-task datasets.
- Tasks: Emotion recognition converts an original 11-emotion multi-label dataset into four single-label classes: Anger, Joy, Sadness, and Optimism.Only tweets with a single emotion label are retained because such examples were scarce.
- Tasks: Emoji prediction uses one emoji per tweet, 20 labels, and a training and validation cap of 50,000 tweets imposed by Twitter distribution policies.More than 42% of tweets carry the three most frequent emoji, producing a highly difficult, skewed task.
- Tasks: Stance detection evaluates favorable, neutral, or negative positions toward five targets, with training and validation handled separately by target domain.
- Unification and evaluation: Across tasks, preprocessing anonymizes user mentions and removes line breaks and website links, while metrics mostly use macro-averaged F1 with defined task-specific exceptions.TweetEval also introduces TE, the average of dataset-specific metrics.
3 Language Models for Tweet Classification
The paper compares whether tweet classifiers benefit more from generic language-model pretraining, Twitter-only training, or continued Twitter-domain training. The comparison tests whether domain adaptation should preserve generic pretrained knowledge.
- Transformer language models are pretrained on unlabeled text and then fine-tuned on task-specific classification data.
- The study compares an existing pretrained language model, Twitter-only training from scratch, and continued Twitter training from a generic pretrained model.
- The comparison asks whether Twitter-specific models should rely exclusively on Twitter data or initialize from standard-corpus pretrained weights before Twitter training.
- Initializing from generic pretraining may reduce the expense of reproducing large-scale pretraining while enabling subsequent training on Twitter data.The passage identifies large pretraining corpora and reproduction cost as theoretical considerations.
4 Evaluation
The evaluation uses RoBERTa variants, Twitter data, and task-specific fine-tuning to compare language-model strategies on TweetEval. Generic RoBERTa performs strongly, while continued Twitter training is most effective overall, with task- and data-specific caveats.
- Experimental setting: The experiments compare RoBERTa-base, RoBERTa retrained on Twitter, and RoBERTa trained from scratch on 60M English tweets.RoBERTa is selected partly because it omits next-sentence prediction, which suits predominantly single-sentence tweets.
- Experimental setting: Classification fine-tuning adds a dense layer mapping RoBERTa’s final representation to task labels and updates all model parameters.
- Results: RoBERTa-base outperforms the Twitter-only model on most tasks, while continued training of RoBERTa-base on Twitter proves more effective overall.
- Results: 13 points versus 5 points: RoBERTa retrained on Twitter loses more validation-to-test F1 on irony than the Twitter-only model.Twitter-only training performs better only on irony detection, where it generalizes better according to the reported split gap.
- Evaluation caveats: 50,000 tweets per task limit Emoji Prediction relative to the original competition, whose training data was one order of magnitude larger.Different collection timespans also create topic-distribution differences between hate-speech validation and test splits.
- Tokenizer analysis: 6.8% is the largest wordpiece difference between RoBERTa-base and Twitter-only tokenizers, observed for hate detection.For most tasks, tokenizer differences are not significant despite the original tokenizer not being trained on Twitter text.
5 Conclusion
TweetEval provides a unified benchmark for seven heterogeneous social-media classification tasks, with strong baselines and language-model training-strategy analysis. The benchmark is a fixed classification setting whose datasets are subsamples and may not generalize to all settings.
- TweetEval combines seven heterogeneous social-media classification tasks into one benchmark with strong reference baselines.
- The analysis compares tokenization statistics across tasks using Twikenizer and two RoBERTa tokenizers.
- The initial benchmark fixes classification for reproducibility and accessibility, while recognizing that other tasks require different evaluation settings.
- Because the benchmark includes related tasks, it could support multitask-learning analyses that exploit their similarities.