Source-linked AI summary
SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets
Parth Patwa, Gustavo Aguilar, Sudipta Kar, Suraj Pandey, Srinivas PYKL, Björn Gambäck, Tanmoy Chakraborty, Thamar Solorio, Amitava Das
TL;DR
Code-mixed text challenges conventional NLP systems that rely heavily on monolingual resources, motivating a shared benchmark and annotated datasets. SentiMix runs a shared task for Hinglish and Spanglish with word-level language and sentence-level sentiment annotations. The task attracted 89 submissions, with best F1 scores of 75.0% for Hinglish and 80.6% for Spanglish, while BERT-like models and ensembles were prominent among successful approaches.
Problem
Code-mixed text challenges conventional NLP systems that rely heavily on monolingual resources, including for phonetic typing and word-level mixing.
Method
SentiMix runs a shared task and releases Hinglish and Spanglish corpora annotated with word-level language identification and sentence-level sentiment labels.
Results
75.0% F1 for Hinglish and 80.6% F1 for Spanglish were the best reported performances among 89 submissions.
Takeaways & Limitations
BERT-like models were the most successful among participants, while ensembles performed better than their individual models.
Abstract
from arXiv · showhide
In this paper, we present the results of the SemEval-2020 Task 9 on Sentiment Analysis of Code-Mixed Tweets (SentiMix 2020). We also release and describe our Hinglish (Hindi-English) and Spanglish (Spanish-English) corpora annotated with word-level language identification and sentence-level sentiment labels. These corpora are comprised of 20K and 19K examples, respectively. The sentiment labels are - Positive, Negative, and Neutral. SentiMix attracted 89 submissions in total including 61 teams that participated in the Hinglish contest and 28 submitted systems to the Spanglish competition. The best performance achieved was 75.0% F1 score for Hinglish and 80.6% F1 for Spanglish. We observe that BERT-like models and ensemble methods are the most common and successful approaches among the participants.
1 Introduction
Social media is multilingual, and code-mixing creates challenges that conventional monolingual NLP resources do not adequately address. SentiMix responds with Hinglish and Spanglish corpora annotated for word-level language identification and sentence-level sentiment.
- 1 Introduction: Code-mixing is common in multilingual social media and creates challenges across multiple NLP tasks.These include language identification, parsing, translation, and semantic processing.
- 1 Introduction: Transliteration and word-level inflection patterns are represented in the Hinglish and Spanglish corpora.Hinglish captures phonetic Latin-script typing, while Spanglish captures code-mixed inflections.
- 1 Introduction: SentiMix releases Hinglish and Spanglish tweets with word-level language identification and three sentence-level sentiment labels.The corpora contain 20K and 19K tweets, respectively.
- 1 Introduction: 89 submissions participated across the two contests, with 75.0% F1 for Hinglish and 80.6% F1 for Spanglish as the best results.The overview reports 61 Hinglish submissions and 28 Spanglish submissions.
2 Related Work
Prior work studied code-mixed language processing, sentiment analysis, and shared tasks, but code-mixed text remains more difficult to process than monolingual text. Existing efforts span classifiers, representations, retrieval, language identification, and named entity recognition.
- 2 Related Work: Code-mixed text is more complicated to process than monolingual text, especially with nonstandard social-media spelling and capitalization.The difficulty increases as the amount of code-mixing increases.
- 2 Related Work: Earlier sentiment studies used SVM classifiers, skip-gram representations, lexicon-based vectors, embeddings, and hybrid systems.These approaches classified sentiment in tweets or text messages.
- 2 Related Work: Mixed-script information retrieval and CALCS workshops advanced research on code-mixed retrieval, language identification, and named entity recognition.The cited workshops provided shared-task settings for these problems.
3 Task Description
SentiMix defines a shared task for sentiment analysis of code-mixed tweets and releases two datasets with tweet-level sentiment and word-level language labels. Submissions are evaluated and ranked on held-out test data using weighted F1.
- 3 Task Description: Each tweet is assigned Positive, Negative, or Neutral sentiment and word-level language marking in the Hinglish and Spanglish datasets.The task addresses sentiment analysis while retaining token-level language information.
- 3 Task Description: Participants first accessed training and validation data, then ranked on a previously unseen test set.Only the first three test submissions per participant counted to reduce test-set over-fitting.
- 3 Task Description: 20k Hinglish and approximately 19k Spanglish labeled tweets are released with both tweet-level sentiment and word-level language labels.The language annotations also support other NLP tasks.
- 3 Task Description: Weighted F1 is computed across positive, negative, and neutral test examples for evaluation and ranking.Classwise F1 scores are weighted by support because class sizes differ.
- 3 Task Description: Precision and recall are also calculated for each class to analyze false positives and false negatives.These metrics supplement the ranking metric with class-level error information.
4 Dataset
The datasets combine language identification and sentiment annotations for Hinglish and Spanglish tweets collected from Twitter and prior CALCS resources. Annotation procedures differ by language, and the corpora have distinct class-balance characteristics.
- 4 Dataset: The datasets label tweets as Positive, Negative, or Neutral, with examples defining praise, criticism, and factual or non-polar content.These labels are applied at the tweet level.
- 4.1 Hinglish: Hinglish tweets were crawled using a 10,786-token Hindi list and annotated with HIN, ENG, or O language labels.Around 60 bilingual or multilingual annotators labeled sentiment, retaining tweets only when two annotations matched.
- 4.1 Hinglish: Hinglish is relatively balanced, with Neutral as the majority class across splits and average CMI values of 25.32, 25.53, and 25.13.Inter-annotator agreement was 55%.
- 4.2 Spanglish: Spanglish data came from Spanish-English CALCS workshops and was annotated using eight language-identification classes.The classes include English, Spanish, mixed, ambiguous, foreign words, named entities, other, and unrecognizable words.
- 4.2 Spanglish: Spanglish sentiment annotation used independent positive and negative sliders with strengths from 1 to 5.Subtracting negative from positive strength yielded Positive, Negative, or Neutral sentiment.
- 4.2 Spanglish: Spanglish annotation used three Amazon Mechanical Turk annotators and quality-control questions within ten-tweet assignments.Two tweets were quality-control items and eight were annotation items.
- 4.2 Spanglish: 56% of the Spanglish corpus is Positive, while Negative and Neutral account for around 16% and 27%, respectively.The imbalance resulted from extending an originally mostly positive corpus rather than sentiment-oriented crawling.
5 Baseline
The baseline fine-tunes multilingual BERT separately for Hinglish and Spanglish, using short sequence limits and AdamW optimization. The surrounding materials include labeled code-mixed examples with language and sentiment tags.
- Baseline model: The baseline uses pre-trained multilingual BERT trained on 104 languages with a 110K-token WordPiece vocabulary.The pretraining data used exponentially smoothed weighting to balance low-resource language representation.
- Fine-tuning: The model is fine-tuned separately for Hinglish and Spanglish using maximum lengths of 56 and 40 tokens, respectively.Training runs for three epochs with AdamW and η = 2e−5.
- Labeled data: The released examples pair word-level language labels with sentence-level sentiment labels for code-mixed tweets.The sentiment categories are positive, neutral, and negative.
6 Participation and Top Performing Systems
The task attracted many submissions across both language contests, with participating systems spanning transformer, recurrent, convolutional, embedding, translation, and ensemble approaches. The listed top systems combine these techniques in varied ways.
- Participation: 61 teams submitted systems for Hinglish and 28 submitted systems for Spanglish, with 16 teams entering both contests.The organizers received 33 system description papers.
- Hinglish systems: Top Hinglish systems included XLM-R with adversarial examples, multilingual BERT embeddings, BERT fine-tuning, and CNN–self-attention ensembles.Other listed systems used word and character embeddings with attention-based BiLSTM or combined ULMFiT with BERT.
- Method diversity: The system overview covers transformer and recurrent models alongside word, character, and multilingual embeddings.The listed methods show substantial architectural diversity among participants.
- Spanglish systems: Top Spanglish systems augmented data through machine translation, used Facebook Research XLM embeddings, and optimized a weighted loss based on code-mixing complexity.The described classifier used CNN or a linear classifier after the embeddings.
7 Results and Analysis
Across the two contests, transformer-based models and ensembles were prominent among the strongest approaches. Results varied by task and sentiment class, while participants also explored several less-established techniques.
- Competition results: 75.0% F1 was the highest Hinglish result, while 80.6% F1 was the highest Spanglish result.The top 15 systems ranged from 75% to 68.6% F1 for Hinglish and from 80.6% to 71.0% for Spanglish.
- Competition results: 44 Hinglish participants and 22 Spanglish teams beat their respective baselines.17 Hinglish participants and 6 Spanglish teams did not beat the baseline.
- Class-wise analysis: Results were much better for the positive class than for the neutral and negative classes due to data imbalance.The Hinglish and Spanglish result tables report precision, recall, and F1 for each class and weighted-average F1 rankings.
- Technique analysis: Transformer, BERT, and related language models gave the best results among the summarized technique groups.The overview mentions BERT, XLNet, and XLM-R systems across the participants.
- Technique analysis: Ensemble methods performed better than the individual models in the reported cases.Examples include teams combining CNN, self-attention, and LSTM-based models.
- Additional approaches: Participants also explored capsule networks with biGRU, phoneme-based generative morphemes, pseudo-labeling, vector gating, and adversarial methods.The paper identifies capsule networks as promising and morpheme-based approaches as potentially useful for spelling-variant code-mixed data.
8 Conclusion and Future Work
SentiMix drew substantial participation and achieved strong reported results, while its data are intended to support broader research on code-mixing. The authors identify data scarcity and propose expanding future shared tasks across languages and annotations.
- 75.0% F1 for Hinglish and 80.6% F1 for Spanglish were the best reported performances.
- 61 teams submitted Hinglish systems and 28 teams submitted Spanglish systems.
- BERT-like models were the most successful among participants.
- The released data can serve researchers studying code-mixing beyond the task’s sentiment-analysis focus.
- Properly annotated code-mixed data remains scarce, motivating future events with more languages and annotations for NER, emotion recognition, and translation.
A Participants
Table 6 lists the teams participating in SentiMix-2020 that submitted system description papers, together with their corresponding references, in alphabetical order.
- Table 6 covers teams that participated in SentiMix-2020.
- Each listed team is associated with a submitted system description paper and its corresponding reference.
- The teams are sorted alphabetically.