Source-linked AI summary
KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media
Ali Safaya, Moutasem Abdullatif, Deniz Yuret
TL;DR
Offensive-language detection on multilingual social-media text requires models that can use textual context across languages. The paper combines pretrained BERT models with CNNs, trains ArabicBERT models, and reports top-four rankings for Arabic, Greek, and Turkish. The authors conclude that BERT-CNN outperformed BERT alone and that language-specific pretrained models were advantageous in their experiments.
Problem
The paper addresses offensive-language identification in Twitter data for OffensEval2020 Subtask-A across Arabic, Greek, and Turkish.
Method
The authors combine contextualized representations from pretrained BERT models with CNN layers and train ArabicBERT models because no pretrained Arabic BERT was available.
Results
The BERT-CNN system outperformed BERT alone, while language-specific pretrained models showed an advantage over multilingual models in the experiments.
Takeaways & Limitations
The proposed model achieved top-four rankings across all three studied languages in OffensEval2020.
Takeaways & Limitations
The model relies on the assumption that concatenating BERT’s last four hidden layers encodes more information than using the top layer alone.
Abstract
from arXiv · showhide
In this paper, we describe our approach to utilize pre-trained BERT models with Convolutional Neural Networks for sub-task A of the Multilingual Offensive Language Identification shared task (OffensEval 2020), which is a part of the SemEval 2020. We show that combining CNN with BERT is better than using BERT on its own, and we emphasize the importance of utilizing pre-trained language models for downstream tasks. Our system, ranked 4th with macro averaged F1-Score of 0.897 in Arabic, 4th with score of 0.843 in Greek, and 3rd with score of 0.814 in Turkish. Additionally, we present ArabicBERT, a set of pre-trained transformer language models for Arabic that we share with the community.
1 Introduction
The paper addresses offensive-language detection on Twitter across Arabic, Greek, and Turkish, using a BERT-CNN approach with minimal text preprocessing. The system ranked among the top four participants for each language.
- Existing solutions range from manual control to rule-based filtering, but they can be time-consuming or error-prone when full textual context is needed.
- OffensEval2020 Subtask-A focused on detecting offensive language in Twitter data across five languages, of which the authors studied Arabic, Greek, and Turkish.
- The proposed approach combines pre-trained bidirectional BERT representations with CNNs for text classification and uses minimal text preprocessing.
- The system ranked 4th in Arabic, 4th in Greek, and 3rd in Turkish among more than 40 participants.
2 Background
Prior offensive-language identification research has used lexical, linguistic, metadata, machine-learning, and deep-neural approaches. The paper builds on datasets and models including OLID, SVM, BiLSTM, and CNN, while also releasing source code.
- Offensive-language identification has employed lexical resources, linguistic features, metadata, and machine-learning models.
- The authors provide source code for the main model and additional experiments.
- Deep-neural approaches include CNNs, LSTMs, and their derivatives for offensive-language classification.
- OLID is a tweet dataset annotated for offensive content and was evaluated with models such as SVM, BiLSTM, and CNN.
3 Data
The task data consists of tweets labeled as offensive or non-offensive, with the training data split into training and development subsets. Preprocessing converts hashtags, applies Greek-specific normalization, and tokenizes text with language-specific BERT WordPiece tokenizers.
- The task dataset contains tweets annotated as Offensive or Non-offensive, with positive and negative tweet counts reported for each set.
- Because the provided training data lacked a development split, the authors allocated 90% for training and 10% for development.
- Hashtags were converted into raw words by splitting hashtag strings into separate tokens.
- For Greek text, preprocessing lowercased letters and removed Greek diacritics before tokenization with the corresponding language-specific BERT WordPiece tokenizer.
4 Model Description
The model feeds contextualized representations from BERT into a CNN classifier, using language-specific or multilingual pretrained models and ArabicBERT trained for this task. Its convolutional outputs are pooled and classified into binary labels.
- 4 Model Description: The proposed model feeds contextualized embeddings from BERT’s last four hidden layers into convolutional filters, then uses a dense layer for prediction.
- 4 Model Description: The experiments use GreekBERT, BERTurk, ArabicBERT, and multilingual BERT as pretrained language models.
- 4.3 ArabicBERT: Because no pretrained Arabic BERT model was available, the authors trained four Arabic BERT models from scratch and released them publicly.
- 4.3 ArabicBERT: ArabicBERT models of Large, Base, Medium, and Mini sizes were trained with masked language modeling and whole-word masking for 4M steps.
- 4 Model Description: BERT provides contextualized vector representations, while CNN serves as the classifier in the two-part architecture.
- 4.4 BERT-CNN Model Structure: For 64-token tweets, the last four BERT layers were concatenated into 768x4x64 representations and processed by 160 filters across five kernel sizes.
- 4.4 BERT-CNN Model Structure: Global max-pooling outputs were concatenated, flattened, passed through a dense layer and Sigmoid function, and converted into binary labels.
- 4.4 BERT-CNN Model Structure: The model was trained for 10 epochs with learning rate 2e-5, retaining the version with the best development-set macro-averaged F1-Score.
5 Experiments and Results
Experiments evaluated the proposed system and comparison models using macro-averaged F1 on shared data splits. Results indicate that combining BERT with CNN and using language-specific pre-trained models improved average performance.
- Evaluation setup: Macro-averaged F1-Score was used to evaluate submissions and comparison experiments on test data.All models used the same train/dev/test splits.
- Comparison models: The baseline used TF-IDF with an SVM and a 3000-feature Count Vectorizer.
- Comparison models: CNN-Text used randomly initialized 300-dimensional embeddings instead of pre-trained BERT, and the difference was significant.
- Comparison models: Two-layer BiLSTM models with hidden size 128 and randomly initialized 300-dimensional embeddings were also evaluated, but CNN-Text outperformed them on average.
- Results: Combining BERT with CNN improved average results over BERT alone, while language-specific pre-trained models outperformed multilingual ones.
6 Conclusion
The paper describes BERT-CNN for offensive speech identification and reports that combining BERT with CNN outperformed BERT alone. It also explains ArabicBERT pre-training and reports high rankings across OffensEval2020 languages.
- Conclusion: The paper describes and compares BERT-CNN with other models for identifying offensive speech in social media.
- Conclusion: Combining BERT with CNN yielded better results than using BERT alone.
- Conclusion: The paper explains ArabicBERT pre-training and reports strong average results with minimal text preprocessing.
- Conclusion: The team ranked among the four highest participating teams for every language in OffensEval2020.