Source-linked AI summary
Offensive Language Identification in Greek
Zeses Pitenis, Marcos Zampieri, Tharindu Ranasinghe
TL;DR
Online offensive content is difficult to moderate, while Greek has had fewer available language resources than English. This paper introduces and evaluates the Offensive Greek Tweet Dataset, a manually annotated Greek Twitter dataset, with the best reported model achieving a macro-F1 of 0.89.
Problem
Most prior offensive-language research focused on English because English has more available language resources, leaving Greek less represented.
Method
The paper constructs OGTD from manually annotated Greek tweets and evaluates classical machine-learning and deep-learning classifiers using textual and linguistic features.
Results
0.89 macro-F1 was achieved by the LSTM and GRU with Attention model, while Linear SVM achieved 0.80 among classical machine-learning approaches.
Takeaways & Limitations
OGTD provides a Greek dataset for offensive-language identification, and deep-learning models with word embeddings achieved better results than classical machine-learning models.
Abstract
from arXiv · showhide
As offensive language has become a rising issue for online communities and social media platforms, researchers have been investigating ways of coping with abusive content and developing systems to detect its different types: cyberbullying, hate speech, aggression, etc. With a few notable exceptions, most research on this topic so far has dealt with English. This is mostly due to the availability of language resources for English. To address this shortcoming, this paper presents the first Greek annotated dataset for offensive language identification: the Offensive Greek Tweet Dataset (OGTD). OGTD is a manually annotated dataset containing 4,779 posts from Twitter annotated as offensive and not offensive. Along with a detailed description of the dataset, we evaluate several computational models trained and tested on this data.
1. Introduction
Online offensive content—including racist, sexist, insulting, and threatening posts—has become a growing moderation challenge. NLP research addresses this problem by training computational systems on annotated data to identify offensive language and related categories.
- Online offensive content includes racist and sexist posts, insults, and threats directed at individuals or groups.
- Its increasing prevalence has made offensive content a growing issue for online communities, social media platforms, and authorities.
- NLP studies apply machine learning and deep learning systems to annotated data for offensive language identification.
- Researchers use different definitions of offensive language, with hate speech being the most studied category among them.
2. Related Work
Prior work has developed datasets and classifiers for abusive language, hate speech, aggression, and related phenomena, primarily in English and several other languages. These studies use varied labels, data sources, linguistic features, and classification methods.
- Research has addressed textual attacks, hate speech, aggression, cyberbullying, and other forms of abusive language.
- OGTD adopts a broader definition of offensiveness than studies focused on particular abusive-language types.
- The referenced hierarchical annotation model distinguishes targeted from general profanity and relates targets to hate speech and cyberbullying categories.
- Earlier offensive-language datasets used labels such as flame, okay, abusive, and clean with classifiers combining n-grams, linguistic or syntactic features, and embeddings.
- Hate-speech studies used manually annotated Twitter or web-comment datasets with logistic regression, linear SVM, word or character n-grams, sentiment lexicons, and part-of-speech features.
- Related resources also cover Dutch, German, Italian, Arabic, and other non-English settings.
3. The OGTD Dataset
OGTD was built from Greek Twitter data collected around trending topics and offensive keywords, then filtered, sampled, and manually annotated. The resulting dataset uses three annotation tags and agreement-based label selection.
- Collection: Tweets were collected during May and June 2019 from Greek trending hashtags, including entertainment programs and election-related discussions.
- Collection: The collection targeted diverse offensive tweets using Greek expletives and their plural forms, while accounting for ironic and insulting uses.
- Pre-processing and annotation: 49,154 tweets were collected, reduced to 46,218 after duplicate removal, and 5,000 were randomly sampled for annotation.
- Pre-processing and annotation: Three volunteers labeled each sampled tweet as Offensive, Not Offensive, or Spam using Greek annotation guidelines.
- Pre-processing and annotation: Labels with 100% agreement were accepted, while disagreements were resolved using majority agreement above 66%.
4. Methods
The study normalizes Greek text, combines TF/IDF and linguistic features for classical models, and evaluates several deep-learning architectures with Greek or multilingual representations. Across experiments, LSTM and GRU with Attention achieved the strongest performance, while unigram features generally outperformed bigrams and linguistic additions were mixed.
- Preprocessing: Greek tweets were lower-cased and normalized to remove accentuation differences before experimentation.The normalization aimed to create a uniform dataset and avoid ambiguity caused by mixed accented and unaccented writing.
- Feature representations: Classical models used TF/IDF unigram or bigram features, with additional experiments incorporating POS and dependency-relation tags.The feature pool was selected based on prior methodology and the dataset size.
- Deep learning: Six deep-learning architectures and BERT were evaluated, with LSTM and GRU with Attention outperforming the other classical and deep-learning models.The attention models achieved the best macro-F1 and the strongest precision, recall, and F1 results for both classes; multilingual BERT performed poorly relative to Greek word embeddings.
- Classical models: Linear SVM and SGDC performed best among individual classifiers with TF/IDF unigrams, while Naïve Bayes models traded class recall against precision.Linear classifiers led macro-F1; Bernoulli Naïve Bayes had the best Offensive-class recall but the lowest precision.
- Feature comparisons: TF/IDF bigrams generally reduced evaluation scores, whereas POS and dependency features produced mixed changes across classifiers.POS tags marginally improved Linear SVM recall for the Offensive class, but other classifiers often deteriorated.
- Overall results: The best OGTD model achieved F1-macro 0.89, compared with 0.80 for the best classical Linear SVM model.SGD achieved the best Offensive-class recall among classical approaches at 0.61.
5. Conclusion
The paper introduces OGTD as the first Greek dataset for offensive language identification and reports its expansion into a larger resource. Together, the dataset versions support Greek NLP development and cross-lingual research.
- 5. Conclusion: OGTD v1.0 is the first Greek dataset for offensive language identification and contains 4,779 manually annotated tweets.The tweets cover topics popular among Greek users and were annotated by volunteers using guidelines from the English OLID dataset.
- 5.1. Ongoing - OGTD v2.0 and OffensEval 2020: OGTD v2.0 combines the original data with 5,508 newly annotated tweets, producing 10,287 instances for OffensEval 2020 training.The expansion was motivated by the need for a larger Greek dataset for the competition.
- 5.1. Ongoing - OGTD v2.0 and OffensEval 2020: OGTD v1.0 and v2.0 can be combined with English, Danish, and Turkish datasets annotated under compatible guidelines to test cross-lingual learning.The resources also facilitate development of NLP language resources for Greek.