RoBERTuito: a pre-trained language model for social media text in Spanish

• The authors publish around 500M Spanish tweets for training models or creating regional and thematic corpus subsets.Source: §1 · p. 1
• RoBERTuito’s weights are available through HuggingFace, reducing computation demands for researchers and making the model transparent, though not interpretable.Source: §1 · p. 1
• The paper establishes a benchmark for classification tasks involving Spanish user-generated text.Source: §1 · p. 1
• Domain-specific pre-trained models outperform general-language models on the corresponding user-generated-text tasks.Source: §1 · p. 1
• The uncased corpus performs better than cased and uncased, deaccented alternatives, while RoBERTuito is competitive on Spanish-English code-switching and English user-generated-text tasks.Source: §1 · pp. 1–2

• Transformer language models include general-purpose BERT, RoBERTa, and GPT, as well as models trained on scientific, medical, and social-media text.Source: §2 · p. 2
• Spanish models include BETO, RoBERTa-BNE, and BERTin, while multilingual alternatives include mBERT and XLM-R.Source: §2 · p. 2
• TwilBERT was the only specialized Spanish Twitter model known to the authors, but its training data were unavailable and its access was difficult.Source: §2 · p. 2

• The collection began with a May 2019 Spritzer sample, then included Spanish-language seed tweets and timelines from their users, yielding about 622M tweets from 432K users before filtering.Source: §3 · p. 2
• Filtering out tweets with fewer than six tokens produced a training corpus of around 500M tweets.Source: §3 · p. 2
• The authors note concerns about possible manipulation of the sampled stream and say the published data and model specifics allow inspection for suspected bias.Source: §3 · p. 2
• The collection retains non-Spanish tweets because only the initial sampled post had to be Spanish; an estimated 92% of the data is Spanish, 4% English, and 3% Portuguese.Source: §3 · p. 3

4. RoBERTuito

• Three versions preserve case, lowercase text, or lowercase and remove accents, each with a SentencePiece tokenizer capped at 30,000 tokens.Source: §4 · p. 3

4.1. Preprocessing

• Preprocessing limits character repetitions, replaces user handles with a special token, restructures hashtags, and converts emojis to text representations.Source: §4.1 · p. 3

4.2. Architecture and training

• The RoBERTa-base model uses 12 self-attention layers, 12 heads, hidden size 768, and masked-language training without next-sentence prediction.Source: §4.2 · p. 3
• Training used a 4K batch size, 600K steps for each model, and dynamic token masking with probability 0.15.Source: §4.2 · p. 3

5. Evaluation

• Evaluation covers Spanish-only, English-only, and Spanish-English code-mixed settings, including classification and token-labeling tasks.The code-switching evaluation uses POS tagging, named entity recognition, and sentiment analysis.Source: §5 · p. 3, §5.3 · p. 5

5.1. Spanish evaluation

• The Spanish benchmark covers sentiment, emotion, irony, and binary hate detection; sentiment data from different geographic variants were merged.EmoEvent contributes 8,409 Spanish tweets, while the Spanish HatEval subset has 6,600 instances.Source: §5.1 · pp. 3–4
• IroSVa links irony labels to a headline or topic and includes news comments and forum posts as well as tweets.The paper motivates irony detection by noting its relevance to social-network text and its potential connection to sentiment analysis.Source: §5.1 · p. 4

5.2. English evaluation

• English evaluation uses emotion, hate-speech, and sentiment tasks, comparing RoBERTuito with monolingual and multilingual models.The datasets include English portions of EmoEvent and HatEval and SemEval 2017 Task-4 for sentiment.Source: §5.2 · p. 4

5.3. Code-switching evaluation

• The Spanish-English LinCE evaluation tests POS tagging, named entity recognition, and sentiment on development data against multilingual and monolingual models.Because LinCE does not release test gold labels, the authors compare dev results with those reported by Winata et al. (2021).Source: §5.3 · pp. 4–5

5.4. Model fine-tuning

• For sentence-classification tasks, pretrained models were fine-tuned for 5 epochs with a triangular learning rate of 510−5; classifiers used the [CLS] token for sentence classification and the first corresponding sub-word for token classification.The best checkpoint at each epoch was selected based on each task’s metric; warm-up was 10% of the training steps.Source: §5.4 · p. 5

6. Results

• Across Spanish tasks, RoBERTuito configurations mostly outperform other models, especially for hate speech and sentiment.Uncased and deacc versions consistently outperform the cased model, with few differences between uncased and deacc.Source: §6 · p. 5, Table 2 · p. 6
• In English, RoBERTuito beats the evaluated multilingual models, performs similarly to RoBERTa and slightly above BERT, while BERTweet leads.The reported comparisons concern the three English classification tasks.Source: §6 · p. 5, Table 3 · p. 6
• On the LinCE leaderboard, RoBERTuito ranks first for Spanish-English sentiment Micro F1 and second for NER and POS.At 108 million parameters, RoBERTuito has about one-fifth the parameters of XLM-RLARGE, which leads NER and POS.Source: §6 · p. 5

6.1. Vocabulary efficiency

• RoBERTuito representations are more compact than BETO and RoBERTa-BNE on Spanish tasks and shorter on code-mixed tasks, but longer on English tasks.Within RoBERTuito, deacc has a slightly lower mean token length than uncased.Source: §6.1 · p. 5, Figure 1 · p. 5

7. Discussion

• On the evaluated Spanish social-media tasks, RoBERTuito outperforms BETO, RoBERTa, and BERTin.The authors note that the evaluation covers a small set of tasks.Source: §7 · p. 6
• The Spanish evaluation is limited by the lack of social-domain, Spanish-only sequence-labeling datasets for tasks such as NER and POS tagging.The authors describe the available Spanish evaluation as a small set of tasks.Source: §7 · p. 6
• On Spanish-English LinCE tasks, RoBERTuito performs better than mBERT and matches XLM-R, though the comparison is not completely fair because the models support different numbers of languages.Table 5 reports test results for the code-mixed LinCE tasks.Source: Table 5 · p. 6, §7 · p. 7
• The uncased and deaccented variants perform comparably, while the cased variant trails on most tasks except POS tagging and NER.The authors say stronger normalization yields no significant improvement and that keeping accent marks is neither beneficial nor harmful; they leave the reasons for variant differences for further investigation.Source: §7 · p. 6
• RoBERTuito remains competitive against monolingual models on the evaluated English social-media tasks.The authors attribute its multilingual features, particularly for code-switching, to a collection process that allowed other languages and regional variants into the training data.Source: §7 · p. 7

8. Conclusion

• RoBERTuito outperforms other available general-domain pre-trained models on the paper's Spanish social-media classification benchmark.The paper also reports good performance on Spanish-English code-switching tasks and competitive performance against monolingual English models in the social domain.Source: §8 · p. 7
• The model is provided in cased, uncased, and deaccented versions, with the uncased version slightly better than the cased one and similar to the deaccented version.The authors state that further research is needed to systematize the reasons for these results.Source: §8 · p. 7
• The authors make the pre-trained models and training corpus public, with the corpus potentially supporting models for Spanish subdomains such as regional variants or specific topics.They also state that the model weights are available through the HuggingFace model hub.Source: §8 · p. 7
• Future work includes expanding the benchmark to open-ended tasks and testing models for specific Spanish subdomains.Source: §8 · p. 7

11. Language Resource References

• The listed references include work on LinCE and code-switched part-of-speech tagging.Source: §11 · p. 8
• Other listed works concern SciBERT, cross-lingual representation learning, BERT, RoBERTa, and BERTweet.Source: §11 · p. 8
• The references also include Spanish pre-trained BERT and TWilBert, a pre-trained model for Spanish Twitter.Source: §11 · p. 8

Hyperparameters and training

• A larger learning rate in the first RoBERTuito prototype usually diverged near its peak, so the authors lowered it for the definitive training.Table 6 presents hyperparameters for the three versions, and Table 7 reports their masked-language-modeling cross-entropy loss and perplexity.Source: Hyperparameters and training · p. 9

Model comparison

• RoBERTuito has the smallest vocabulary among the compared models, which accounts for its parameter-count difference given that most models share a base architecture.Table 8 compares model size in parameters and vocabulary size.Source: Model comparison · p. 9, Table 8 · p. 9

Vocabulary efficiency

• For Spanish and code-mixed Spanish-English, RoBERTuito's uncased and deaccented forms have the shortest mean sentence representations; for English, BERTweet has the shortest.RoBERTuito has longer English sequences than its monolingual counterparts; Table 9 reports mean token counts, with shorter being better.Source: Vocabulary efficiency · p. 9, Table 9 · p. 9