Source-linked AI summary
XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond
Francesco Barbieri, Luis Espinosa Anke, Jose Camacho-Collados
TL;DR
Multilingual NLP has focused largely on standard benchmarks and clean corpora, leaving Twitter-specific multilingual modeling comparatively underexplored. XLM-T addresses this by pretraining XLM-R on Twitter data in over thirty languages and releasing unified eight-language sentiment benchmarks with supporting tooling. The results indicate that the domain-specific model is better suited to multilingual Twitter sentiment analysis and shows promise for zero-shot cross-lingual settings.
Problem
Multilingual model analyses have focused largely on standard benchmarks and clean corpora, while multilingual Twitter pretraining has been comparatively neglected.
Method
XLM-T pretrains an XLM-R-based multilingual language model on Twitter data, releases unified sentiment datasets in eight languages, and provides fine-tuning and evaluation code.
Results
XLM-T achieved the best overall zero-shot results in six of eight languages and outperformed XLM-R by 7.9 absolute points in Hindi.
Takeaways & Limitations
The domain-specific language model is more suited to multilingual Twitter sentiment analysis and shows potential for zero-shot cross-lingual use when language-specific training data is unavailable.
Takeaways & Limitations
Language-specific tokenizers could further reduce performance differences, but using them was outside the paper’s scope.
Abstract
from arXiv · showhide
Language models are ubiquitous in current NLP, and their multilingual capacity has recently attracted considerable attention. However, current analyses have almost exclusively focused on (multilingual variants of) standard benchmarks, and have relied on clean pre-training and task-specific corpora as multilingual signals. In this paper, we introduce XLM-T, a model to train and evaluate multilingual language models in Twitter. In this paper we provide: (1) a new strong multilingual baseline consisting of an XLM-R (Conneau et al. 2020) model pre-trained on millions of tweets in over thirty languages, alongside starter code to subsequently fine-tune on a target task; and (2) a set of unified sentiment analysis Twitter datasets in eight different languages and a XLM-T model fine-tuned on them.
1. Introduction
Multilingual language models have expanded NLP beyond monolingual systems, but Twitter has been surprisingly neglected as a source of multilingual pretraining data. XLM-T addresses this gap with a Twitter-specific multilingual model, unified multilingual sentiment datasets, and supporting code.
- Motivation: Twitter has been surprisingly neglected in the trend toward massive multilingual pretraining.The paper identifies Twitter’s uncurated and discursive nature as possible reasons for this neglect.
- Contributions: XLM-T provides a multilingual Twitter-specific language model based on XLM-R checkpoints.The framework includes baseline evaluations in multiple settings, including zero-shot transfer.
- Contributions: The paper introduces UMSAB, a unified sentiment analysis benchmark covering eight languages.It also reports results on TweetEval for comparison with standard English Twitter tasks.
- Contributions: The released framework includes starter code for analyzing, fine-tuning, and evaluating language models.XLM-T and its associated data are released for further research.
2. XLM-T: Language Models in Twitter
XLM-T is a framework built around multilingual and monolingual Twitter language models, unified datasets, and code for Twitter NLP workflows. It trains on 198M tweets and supports efficient fine-tuning, feature extraction, classification, and evaluation.
- Framework: The framework trains a multilingual Twitter language model and fine-tunes it for monolingual and multilingual applications.It also provides an interface and starter code for common Twitter NLP tasks.
- Interfaces and Workflows: Figures 1–3 document multilingual training-data distribution, tweet-embedding and similarity extraction, and model fine-tuning workflows.Figure 1 uses a log scale and marks unidentified Twitter-API languages as UNK.
- Pretraining Data: 198M tweets posted between May 2018 and March 2020 provide the language-model pretraining data.Tweets with fewer than three tokens or URLs were excluded, while language filtering was not applied.
- Released Models: The released toolkit integrates XLM-Twitter with monolingual language models of any nature.Examples include previously released English monolingual Twitter models.
- Fine-tuning: Adapter fine-tuning freezes the language model and trains one additional classification layer.The paper uses adapters for efficiency, reporting memory and speed benefits over full-model fine-tuning.
- Starter Code: The toolkit provides Python code for tweet embeddings, classification, fine-tuning, and evaluation.It also supports inference through Hugging Face pipelines and evaluation across selected or all UMSAB languages.
3. Evaluation
The evaluation tests XLM-Twitter across English Twitter tasks, multilingual sentiment settings, and cross-lingual embedding similarity. Results show strong multilingual transfer and broader gains from additional languages, while language-specific performance and multilingual versatility involve trade-offs.
- Monolingual Evaluation (TweetEval): XLM-Twitter improves over RoBERTa-base, XLM-R, and RoBERTa-Twitter on heterogeneous English TweetEval tasks, but underperforms monolingual Twitter-specific models.The evaluation covers emoji prediction, emotion recognition, hate speech, irony, offensive language, sentiment, and stance detection.
- Unified Multilingual Sentiment Analysis Benchmark (UMSAB): The unified benchmark contains eight balanced sentiment datasets with 3,033 instances per language and equal representation of positive, negative, and neutral labels.The datasets preserve original train/test splits and include typologically diverse language groupings.
- Zero-shot Cross-lingual Transfer: XLM-Twitter achieves the best overall zero-shot results in six of eight languages and outperforms XLM-R by 7.9 absolute points in Hindi.These results remain competitive despite diverse domains and the source language being unseen during training.
- Cross-lingual Transfer with Target Language Training Data: Adding training data from other languages improves macro-F1, especially for XLM-Twitter and challenging settings such as Hindi, which rises from 40.29 to 56.39.Selecting languages by validation accuracy beats using all languages in half of the cases, revealing a trade-off between multilingual versatility and optimal language selection.
- Qualitative Analysis: Embedding similarities indicate that topic overlap, alongside typological proximity, may influence multilingual representation quality.English and French are most similar, Hindi has clearly low resemblance, and Arabic remains similar to English despite differing in typology and script because both datasets use the same keywords.
4. Conclusions
The paper presents a multilingual Twitter language-model framework centered on XLM-Twitter and eight-language sentiment-analysis evaluation. Its conclusions highlight domain-specific utility, zero-shot cross-lingual potential, and trade-offs in multilingual training.
- The framework releases a multilingual language model trained on almost 200M tweets and supports research across over thirty languages.
- The unified sentiment-analysis benchmark covers eight languages and compares the Twitter-based model with a general-domain multilingual language model.
- Figure 5 measures cosine-distance similarity between English training tweets and test tweets in the other seven languages using averaged contextualized tweet embeddings.
- XLM-Twitter shows potential for multilingual social-media sentiment analysis and zero-shot cross-lingual settings without language-specific training data.
A. Full Experimental Results
The appendix reports monolingual, bilingual, and multilingual experiments, with macro-F1 averages across three runs and best-run results. Tables 5–7 organize these results by training and evaluation setting.
- The appendix includes full experimental results, standard deviations after three runs, and validation-selected best runs.
- Monolingual experiments: Table 5 reports monolingual experiments for XLM and XLM-Twitter, including average and best macro-F1 across three runs.
- Bilingual experiments: Table 6 reports bilingual experiments that initialize target-language adapters from the best-performing zero-shot source language.
- Multilingual experiments: Table 7 reports multilingual experiments using one multilingual dataset and evaluates each language plus the combined test sets.