Source-linked AI summary
SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)
Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, Çağrı Çöltekin
TL;DR
OffensEval-2020 followed OffensEval-2019 by extending its hierarchical offensive-language identification task with new semi-supervised and multilingual datasets. Across five languages and multiple subtasks, broad participation enabled comparison of approaches, with large-scale BERT-style Transformers emerging as the strongest systems.
Problem
OffensEval-2020 addressed the need to extend hierarchical offensive-language identification beyond the previous English-centered task to new datasets and languages.
Method
The task introduced a 9,089,140-tweet semi-supervised English dataset and multilingual datasets covering Arabic, Danish, Greek, and Turkish.
Results
528 teams registered, 145 submitted results, and 70 submitted system-description papers, while the best systems across languages and subtasks used large-scale BERT-style pretrained Transformers.
Takeaways & Limitations
The multilingual task and wide participation enabled cross-language comparison and analysis of approaches, including cross-lingual training explored by several participants.
Takeaways & Limitations
The reported task did not yet offer Subtasks B and C for all five languages and had not covered code-mixing, code-switching, or a wider range of social-media platforms.
Abstract
from arXiv · showhide
We present the results and main findings of SemEval-2020 Task 12 on Multilingual Offensive Language Identification in Social Media (OffensEval 2020). The task involves three subtasks corresponding to the hierarchical taxonomy of the OLID schema (Zampieri et al., 2019a) from OffensEval 2019. The task featured five languages: English, Arabic, Danish, Greek, and Turkish for Subtask A. In addition, English also featured Subtasks B and C. OffensEval 2020 was one of the most popular tasks at SemEval-2020 attracting a large number of participants across all subtasks and also across all languages. A total of 528 teams signed up to participate in the task, 145 teams submitted systems during the evaluation period, and 70 submitted system description papers.
1 Introduction
OffensEval-2020 extended the hierarchical OLID shared-task framework to multilingual offensive-language identification and added a large semi-supervised English dataset. The task attracted broad participation across languages and subtasks.
- OffensEval-2020 used the OLID three-level taxonomy for offensive-language identification, offense-type categorization, and offense-target identification.
- Over nine million English tweets were provided in a new large-scale semi-supervised training dataset.
- Arabic, Danish, Greek, and Turkish were added as new languages alongside English.The multilingual expansion enabled cross-lingual training and analysis by participating systems.
- The task used larger test datasets for all subtasks than OffensEval-2019.
- 528 teams signed up, 145 submitted official test runs, and 70 submitted system description papers.The 70 papers constituted an all-time record for a SemEval task.
- The paper reports task results and participating approaches across five languages, with future-work directions discussed in the conclusion.
2 Annotation Schema
The OLID annotation schema organizes offensive-language analysis hierarchically, moving from whether content is offensive to its targeting and target category.
- Level A distinguishes offensive text (OFF) from non-offensive text (NOT).OFF includes inappropriate language, insults, or threats; NOT is neither offensive nor profane.
- Level B distinguishes targeted offensive text (TIN) from untargeted profanity or swearing (UNT).TIN covers targeted insults or threats toward a group or individual.
- Level C identifies whether the offensive content targets an individual, group, or other entity.The categories are IND, GRP, and OTH, with OTH covering organizations, events, and issues.
3 Data
OffensEval-2020 standardized offensive-language datasets across five languages using the OLID schema, with English additionally supporting Subtasks B and C. The datasets varied in source, sampling, annotation, and scale.
- Dataset framework: Five languages—Arabic, Danish, English, Greek, and Turkish—followed the OLID annotation schema and shared preprocessing procedures.User mentions were replaced with @USER for anonymization.
- English: English included OLID and SOLID, a semi-supervised dataset containing 9,089,140 tweets.SOLID labels were produced through democratic co-training using OLID as a seed dataset and four models with different inductive biases.
- Arabic: Arabic data comprised 10,000 tweets collected with Arabic-language filtering and selection based on vocative particles to increase offensive-content prevalence.The collection used tweets containing at least two occurrences of the Arabic vocative particle yA.
- Danish: Danish data comprised 3,600 comments from Facebook, Reddit, and Ekstra Bladet, with abusive-term seeding limited to half the data.Participants were encouraged to use cross-validation because the training data lacked fixed training/development splits.
- Greek: Greek data comprised 10,287 tweets sampled from popular and trending hashtags, television and political content, pejorative terms, and “you are” keywords.Three annotators judged each tweet, majority labels above 66% were selected, and Fleiss’ κ was 0.78.
- Turkish: Turkish data contained over 35,000 tweets sampled uniformly from Twitter and filtered using frequent Turkish words.Most tweets had one annotator; Cohen’s κ on 5,000 doubly annotated tweets was 0.761.
4 Task Participation
OffensEval-2020 attracted broad participation across languages and subtasks, with teams predominantly using pretrained contextual representations and Transformer-based models.
- Participation: 528 teams registered, 145 submitted results, and 70 submitted system description papers.Six teams submitted for all five languages, while 96 focused on one language.
- Representations: Most teams used pretrained embeddings, especially contextualized Transformers and ELMo.The most popular Transformers were BERT, RoBERTa, and multilingual mBERT.
- Models: mBERT had language-processing issues, including absent Arabic word segmentation and incorrect handling of Danish å/aa mapping.These issues affect the suitability of multilingual representations for some languages.
- Representations: Teams also used word2vec, GloVe, word and character n-grams, sentiment and offensive-word lexicons, emoji priors, and sentiment analysis.Some representations were language-specific, such as Mazajak for Arabic.
- Models: Most teams used pretrained Transformer models, while CNNs, RNNs, GRUs, and SVMs also appeared, often within ensembles.Popular Transformer choices included BERT, RoBERTa, XLM-RoBERTa, ALBERT, and GPT-2.
5 English Track
The English track drew broad participation across three subtasks, with systems predominantly using BERT-based Transformers and many exploiting SOLID. Top systems achieved strong macro-averaged F1 scores, substantially exceeding OffensEval-2019 results on English subtasks A and C.
- Participation: 87 teams submitted to the English track, including 27 teams participating in all three subtasks.
- Data and preprocessing: Most teams applied preprocessing or text normalization, including emoji conversion, hashtag segmentation, abbreviation expansion, and removal of tweet-specific artifacts.
- Data and preprocessing: 58 teams used the weakly supervised SOLID dataset, while six tried it without benefiting and the remaining teams used only manually annotated data.
- Subtask B: 0.7462 was the best F1 score for English subtask B, achieved by Galileo’s ensemble model; top-ranking systems generally used BERT-based Transformers.
- Subtask C: 0.7145 was the best F1 score for English subtask C, achieved by Galileo, and all teams outperformed the majority class baseline.
- Cross-year comparison: 0.9223 vs. 0.8290 for subtask A and 0.7145 vs. 0.6600 for subtask C were the best English scores in 2020 versus 2019, respectively.The paper suggests that the much larger SOLID dataset helped models make more accurate predictions and that its weakly supervised annotation method is viable.
6 Arabic Track
The Arabic track attracted 108 registrations and 53 valid participants, with widespread text normalization and strong subtask A results. ALAMIHamza achieved the highest F1 score using BERT with a sigmoid classifier and emoji translation.
- Participation: 108 teams registered for the Arabic track, and 53 entered the competition with at least one valid submission.
- Preprocessing: Most teams performed preprocessing or normalization involving Arabic character forms, diacritics, non-Arabic characters, and related text variants.
- Results: 0.9017 was the best Arabic subtask A F1 score, achieved by ALAMIHamza; ALT followed closely with 0.9016 and Galileo with 0.8989.
- Best system: ALAMIHamza used BERT to encode Arabic tweets, followed by a sigmoid classifier, and translated the meaning of emojis.
7 Danish Track
The Danish track was a first for offensive-language identification and received 72 registrations with 39 official submissions. Every submitted system beat the majority baseline, while most reached F1 scores of at least 0.7.
- Participation: 72 teams registered for the Danish track, and 39 made official submissions.
- Preprocessing: Teams used embedding-model preprocessing such as BPE and WordPiece, along with emoji normalization, spelling correction, sentiment tagging, and hashtag segmentation.
- Results: All teams outperformed the majority class baseline, and all but one improved over the FastText baseline of F1 = 0.5148.
- Results: Most teams achieved an F1 score of 0.7 or higher, while the top-ranked JCT system was entirely non-neural.
- Best system: LT@Helsinki won using NordicBERT, orthographic-lengthening reduction, emoji sentiment scores, hashtag–username co-occurrences, and 10-fold hyperparameter tuning.
8 Greek Track
The Greek track was another first for the task, with 71 registrations and 37 official submissions. High-ranking systems primarily used large pretrained Transformers, and NLPDove achieved the highest F1 score.
- Participation: 71 teams registered for the Greek track, and 37 made official submissions.
- Preprocessing: Participants experimented with preprocessing and text normalization techniques, including replacing emojis with textual equivalents.
- Results: 0.852 was the highest Greek subtask A F1 score, achieved by NLPDove, followed by Galileo at 0.851 and KS@LTH at 0.848.
- Methods: Most high-ranking submissions used large-scale pretrained Transformers, especially BERT, alongside non-contextualized pretrained word embeddings.
- Best system: NLPDove fine-tuned mBERT with a domain-specific WordPiece vocabulary and extended-vocabulary embeddings for pretraining and fine-tuning.
9 Turkish Track
The Turkish track introduced an extended Turkish dataset and attracted substantial participation. Galileo led with a macro-averaged F1 of 0.8258, while multilingual approaches and broader task coverage shaped the competition.
- 86 teams registered for the Turkish track, and 46 submitted official test-dataset results.
- Galileo achieved the highest Turkish macro-averaged F1, 0.8258, followed by SU-NLP at 0.8167 and KUI-SAIL at 0.8141.The majority class baseline had an F1 score of 0.4435.
- Galileo used a language-agnostic system trained on data from all five languages, unlike the second- and third-ranked systems.
- OffensEval-2020 covered Arabic, Danish, English, Greek, and Turkish, with three subtasks for English and one top-level subtask for the other languages.
- Cross-language approaches proved effective and were used by some top-ranked systems across the multilingual task.
- Future extensions include offering English subtasks B and C for all five languages, adding under-represented languages, and studying code-mixing, code-switching, and additional platforms.
A Best-Performing Teams
The strongest systems relied mainly on pretrained Transformer models, while top-performing teams also used multilingual transfer, ensembles, multitask learning, augmentation, and language-specific processing.
- Galileo: Galileo ranked among the top systems across every language and English subtask, using multilingual XLM-RoBERTa models with multilingual fine-tuning.
- UHH-LT: UHH-LT ranked first on English subtask A with an ensemble combining different ALBERT models.
- LT@Helsinki: LT@Helsinki ranked first for Danish using NordicBERT, orthographic normalization, emoji sentiment scores, and hashtag-user co-occurrences.
- NLPDove: NLPDove ranked first for Greek and third for Danish using preprocessing, semi-supervised augmentation, cross-lingual transfer, and Translation Embedding Distance.
- ALAMIHamza: ALAMIHamza ranked first for Arabic using BERT to encode tweets followed by a sigmoid classifier and emoji-meaning translation.
- PGSG: PGSG used SOLID-based unsupervised MLM fine-tuning, a BERT-LSTM model, and a Noisy Student architecture for English subtask B.
- Rouges: Rouges fine-tuned XLM-RoBERTa sequentially across English, Turkish, Greek, Arabic, and Danish.
- NLP: NLP used hierarchical multitask learning in which outputs for subtasks A and B feed progressively into later layers for subtask C.
NTU
The supplied passages describe systems ranked third in English subtask C, Greek subtask A, and Turkish subtask A, highlighting feature augmentation and differing monolingual or hybrid designs.
- PRHLT-UPV: PRHLT-UPV ranked third on English subtask C by concatenating hand-crafted tweet features with BERT’s [CLS] representation.Features included tweet length, misspellings, punctuation, emoticons, and noun phrases.
- KS@LTH: KS@LTH ranked third for Greek after comparing monolingual BERT with cross-lingual XLM-RoBERTa.The team found BERT to perform slightly better.
- KUISAIL: KUISAIL ranked third for Turkish using a BERT-CNN model that combined BERTurk with a CNN.