Source-linked AI summary
Multilingual Offensive Language Identification with Cross-lingual Embeddings
Tharindu Ranasinghe, Marcos Zampieri
TL;DR
Offensive-language research is concentrated in English despite the need to identify harmful content in lower-resource languages. The paper transfers English-trained XLM-R representations to Bengali, Hindi, and Spanish, including Bengali’s cross-domain and cross-task setting, and reports favorable comparisons with recent shared-task systems.
Problem
Most offensive-language studies and datasets focus on English, leaving data scarcity in lower-resource languages such as Bengali, Hindi, and Spanish.
Method
The paper trains XLM-R on English offensive-language data and transfers its learned weights to other languages, using only XLM-R weights when Bengali requires a different task label space.
Results
XLM-R with transfer learning outperformed the tested methods and the best participant results from shared tasks on Bengali, Hindi, and Spanish.
Takeaways & Limitations
The results support using cross-lingual contextual embeddings and transfer learning for offensive-language identification across languages, domains, and task label structures.
Abstract
from arXiv · showhide
Offensive content is pervasive in social media and a reason for concern to companies and government organizations. Several studies have been recently published investigating methods to detect the various forms of such content (e.g. hate speech, cyberbulling, and cyberaggression). The clear majority of these studies deal with English partially because most annotated datasets available contain English data. In this paper, we take advantage of English data available by applying cross-lingual contextual word embeddings and transfer learning to make predictions in languages with less resources. We project predictions on comparable data in Bengali, Hindi, and Spanish and we report results of 0.8415 F1 macro for Bengali, 0.8568 F1 macro for Hindi, and 0.7513 F1 macro for Spanish. Finally, we show that our approach compares favorably to the best systems submitted to recent shared tasks on these three languages, confirming the robustness of cross-lingual contextual embeddings and transfer learning for this task.
1 Introduction
The paper addresses the scarcity of annotated resources for offensive-language identification beyond English by transferring knowledge from English to Bengali, Hindi, and Spanish. It evaluates cross-lingual contextual embeddings and transfer learning, including cross-domain and cross-task transfer for Bengali.
- Motivation: Offensive social-media content can harm users, motivating systems that identify posts for moderation or deletion.The passage links offensive content to suicide attempts, ideation, and mental-health problems.
- Research gap: Most offensive-language studies and datasets focus on English, while XLM-R had not yet been applied to this task.The paper frames cross-lingual contextual embeddings as a way to use English resources for other languages.
- Approach: The study applies cross-lingual contextual word embeddings to project predictions from English into Bengali, Hindi, and Spanish.The approach uses existing English data to address lower-resource languages.
- Approach: For Bengali, the study evaluates transfer across languages, domains, and tasks.The Bengali setting differs in both source domain and classification scheme, such as Twitter versus Facebook and binary versus three-way classification.
- Resources: The authors provide the code and English model for researchers applying the methodology to low-resource languages.These resources are stated to be freely available.
2 Related Work
Prior work has progressed from feature-engineered classifiers to neural and transformer models, but research remains concentrated on English. The paper therefore uses transfer learning and cross-lingual transformers to address data scarcity in Bengali, Hindi, and Spanish.
- Prior approaches: Offensive-language identification evolved from traditional classifiers to neural networks and transformer models.The cited progression includes Naive Bayes, support vector machines, LSTMs, bidirectional LSTMs, GRUs, ELMO, and BERT.
- Language coverage: Most studies focus on English because English corpora, lexicons, and pretrained models are more available.Research has also addressed languages including Arabic, Danish, Dutch, French, Greek, Italian, Portuguese, Slovene, and Turkish.
- Research gap: Multilingual competitions created opportunities for cross-lingual learning, but low-resource languages often lack the annotated data required by deep-learning methods.The paper addresses this scarcity by transferring from English to Bengali, Hindi, and Spanish.
3 Data
The study combines an English OLID dataset with Bengali, Hindi, and Spanish shared-task datasets. These datasets differ in source domain and label structure, enabling evaluation across comparable and mismatched settings.
- Datasets: The study uses datasets in English, Bengali, Hindi, and Spanish drawn from shared tasks held in 2019 and 2020.The shared-task origins enable comparison with competing approaches.
- English source data: OLID supplies English tweets with a three-level taxonomy for offensive-language identification, categorization, and target identification.Its first level distinguishes offensive from non-offensive content.
- Dataset alignment: OLID was selected because its hierarchical annotation model maps level A to labels in the three target datasets.The model covers multiple types of abusive content within one taxonomy.
- Dataset differences: Hindi and Spanish contain Twitter data with two labels, whereas Bengali contains Facebook data with three aggression-related classes.The Bengali dataset therefore differs from the other target datasets in both domain and label structure.
- Dataset overview: Table 1 summarizes dataset instance counts, sources, and labels, using F for Facebook and T for Twitter.The table’s source notation highlights the domain difference between Bengali and the Twitter datasets.
4 Methodology
The methodology fine-tunes XLM-R for text classification and transfers knowledge learned from English OLID to lower-resource languages. It tests both inter-language transfer and transfer across language, task, and domain boundaries.
- Transfer learning: The approach trains a classifier on English with XLM-R, saves model weights, and initializes training for a lower-resource language from those weights.This transfer-learning design uses a resource-rich language as the starting point.
- XLM-R classification: XLM-R represents each input sequence using the final hidden state of its [CLS] token, followed by a softmax classifier.The described XLM-R-large model accepts sequences of up to 512 tokens and has approximately 125M parameters.
- XLM-R classification: The model jointly fine-tunes XLM-R and the task-specific classifier parameters by maximizing the correct-label log-probability.The paper uses the XLM-R-large model for classification.
- Transfer-learning strategies: Inter-language transfer reuses both XLM-R and softmax weights from English for Hindi and Spanish.The target experiments use the same classification setup after English initialization.
- Transfer-learning strategies: Inter-task and inter-language transfer reuses only XLM-R weights for Bengali, allowing a new classifier for three target classes.The strategy omits the English softmax layer because Bengali has a different number of classes.
- Training setup: Models use an 0.8:0.2 training-validation split, with a learning rate of 1e−5 and 3 epochs selected for all languages.Training was performed on an Nvidia Tesla K80 GPU with manual tuning against validation results.
5 Results and Evaluation
Transfer learning with XLM-R cross-lingual embeddings achieved the strongest reported results across Bengali, Hindi, and Spanish, including Bengali data differing from English in domain and label structure. The approach also outperformed available shared-task systems and improved over multilingual BERT across the tested languages.
- The evaluation used organizer-provided test sets and compared weighted and macro F1 with the best systems from TRAC-2, HASOC, and HatEval.TRAC-2 reported macro F1, HatEval weighted F1, and HASOC both metrics.
- 0.8568 weighted F1 and 0.8580 macro F1 were achieved for Hindi with XLM-R transfer learning, exceeding the reported HASOC 2019 best system.The cited HASOC system scored 0.8149 macro F1 and 0.8202 weighted F1.
- 0.7513 macro F1 and 0.7591 weighted F1 were achieved for Spanish with XLM-R transfer learning, above the best HatEval systems’ 0.7300 macro F1.The compared HatEval systems used SVM classifiers with character and word n-grams, POS tags, lexicons, and embeddings.
- 0.7513 macro F1 and 0.7591 weighted F1 were achieved for Bengali despite its Facebook domain and three aggression labels differing from the English training data.The Bengali result outperformed the other tested models by a significant margin; the best TRAC-2 system scored 0.821 weighted F1.
- Bengali performance was substantially better for the non-aggressive class than for overtly or covertly aggressive classes.This class-specific pattern followed a trend observed among TRAC-2 participants.
- Across all experimental settings, transfer-learning models with cross-lingual embeddings outperformed the best available systems, while XLM-R improved over multilingual BERT in every tested language.
6 Conclusion
The paper applies cross-lingual contextual embeddings to project English offensive-language predictions onto Bengali, Hindi, and Spanish benchmark datasets. Its results support transfer learning across offensive-language subtasks, including Bengali data that differs from English in domain and label structure, while future work proposes broader multilingual evaluation.
- The study applies cross-lingual contextual word embeddings to project English predictions onto benchmarked Bengali, Hindi, and Spanish datasets.
- The authors report that XLM-R with transfer learning outperformed the other tested methods and the best systems from the three language competitions.
- The results support using OLID’s hierarchical annotation model to jointly model offensive-language subtasks including aggression and hate speech.
- Bengali results show high performance is possible with transfer learning across off-domain and off-task data despite non-corresponding English and Bengali label sets.
- Future work proposes evaluation with SOLID, additional OffensEval languages, and languages with fewer available resources.