Source-linked AI summary
A Sentiment Analysis Dataset for Code-Mixed Malayalam-English
Bharathi Raja Chakravarthi, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, John P. McCrae
TL;DR
Sentiment analysis lacks resources for Malayalam-English code-mixed social-media text, whose language mixing complicates existing approaches. The paper constructs and annotates a YouTube-comment corpus, reports annotation agreement above 0.8, and provides benchmark results for this setting.
Problem
Resources for sentiment analysis of Malayalam-English code-mixed text are unavailable, despite the prevalence and complexity of code-mixed social-media content.
Method
The authors collected Malayalam movie-trailer YouTube comments, filtered and preprocessed code-mixed text, annotated sentiment, and evaluated benchmark machine-learning and deep-learning systems.
Results
Krippendorff’s alpha exceeded 0.8, and the paper reports benchmark results from traditional machine-learning and deep-learning models on the corpus.
Takeaways & Limitations
The released gold-standard corpus supports research on Malayalam-English code-mixed sentiment analysis and related code-mixed language research.
Takeaways & Limitations
Some comments were difficult to annotate because they compared movies or expressed opinions about multiple movie aspects in one sentence.
Abstract
from arXiv · showhide
There is an increasing demand for sentiment analysis of text from social media which are mostly code-mixed. Systems trained on monolingual data fail for code-mixed data due to the complexity of mixing at different levels of the text. However, very few resources are available for code-mixed data to create models specific for this data. Although much research in multilingual and cross-lingual sentiment analysis has used semi-supervised or unsupervised methods, supervised methods still performs better. Only a few datasets for popular languages such as English-Spanish, English-Hindi, and English-Chinese are available. There are no resources available for Malayalam-English code-mixed data. This paper presents a new gold standard corpus for sentiment analysis of code-mixed text in Malayalam-English annotated by voluntary annotators. This gold standard corpus obtained a Krippendorff's alpha above 0.8 for the dataset. We use this new corpus to provide the benchmark for sentiment analysis in Malayalam-English code-mixed texts.
1. Introduction
Social-media users often express opinions in code-mixed and nonnative-script text, creating challenges for sentiment analysis. The paper addresses the lack of Malayalam-English resources by introducing an annotated corpus.
- Motivation: Code-mixing combines more than one language in the same conversation, often alongside Roman-script usage on social media.These practices complicate language identification and existing NLP tasks.
- Research gap: Few datasets exist for Malayalam-English code-mixed text, so the authors collected YouTube comments from Malayalam movie trailers.The proposed dataset supplies resources for sentiment analysis in this setting.
- Forms of code-mixing: ‘Super’ and ‘family’ illustrate intra-sentential code-mixing, while ‘familyaayi’ combines English vocabulary with Malayalam morphology through intra-word switching.The examples show that mixing can occur both between languages within a sentence and inside a word.
- Sentiment examples: The corpus examples include positive comments recommending a movie and negative comments expressing dislike of a trailer and poor dialogue delivery.These examples demonstrate how sentiment is identified from code-mixed viewer opinions.
2. Related Work
Prior sentiment-analysis research includes lexicon-based and machine-learning approaches across monolingual languages. The related work also discusses deep-learning and representation-learning approaches for dynamic user-generated data.
- Existing resources: Sentiment analysis has been studied extensively for English, Russian, German, Norwegian, and Indian-language monolingual corpora.
- Approaches: Traditional sentiment-analysis approaches include lexicon-based methods and machine-learning models.The passage identifies these as two established approaches.
- Deep learning: Lexicon-based and traditional machine-learning methods are described as inefficient for dynamic user-generated data, motivating deep-learning approaches.
- Representation learning: Transfer learning and pretrained representations such as GloVe and Word2Vec are presented among the approaches discussed for sentiment analysis.
3. Corpus Creation and Annotation
The paper constructs a Malayalam-English code-mixed YouTube corpus, filters and annotates comments using a five-category sentiment schema, and evaluates annotation agreement. The resulting dataset contains 7,743 sentences and reports high Krippendorff’s alpha agreement.
- Annotation scheme: Each sentence was annotated by at least three annotators using positive, negative, mixed-feelings, neutral, and not-in-intended-language categories.The annotation schema distinguishes emotional polarity, mixed sentiment, neutrality, and language eligibility.
- Annotators: Six Malayalam-English bilingual volunteers annotated the corpus; four were female, two were male, and all were postgraduates.Five annotators were educated in English-medium schools and one in Malayalam-medium schooling.
- Corpus statistics: 7,743 distinct sentences and 70,075 tokens form the code-mixed sentiment dataset, with 19,992 unique tokens and an average of ten tokens per sentence.The corpus includes 6,739 comments and is split into 1,348 test, 674 validation, and remaining training instances.
- Corpus statistics: 2,811 comments are positive and 1,903 are neutral, making positive polarity the most frequent category and neutral polarity the second-largest.The complete dataset is categorized into positive, negative, neutral, mixed-feeling, and non-Malayalam groups.
- Agreement: 0.890 nominal alpha and 0.911 interval alpha quantify the inter-annotator agreement obtained for the annotation.Krippendorff’s alpha was selected because it accommodates missing data, varying sample sizes, categories, raters, and measurement levels.
4. Difficult Examples
The difficult examples involve comparisons and comments expressing different attitudes toward multiple movie aspects. The annotators resolve these cases by interpreting the specific sentiment expressed rather than relying on the sentiment of a referenced film or standard.
- Sources of difficulty: Ambiguous comments compare movies or express opinions about different movie aspects in the same sentence, making sentiment annotation difficult.These cases require deciding how multiple evaluative cues combine into one category.
- Movie comparisons: A comparison suggesting that Helen resembles the positively reviewed Tamil film Aadai is labeled mixed feeling rather than positive.The annotators do not transfer Aadai’s favorable reception to the Malayalam movie under discussion.
- Movie comparisons: A comment comparing Helen with the worldwide Hollywood standard is labeled positive, unlike the comparison with Aadai.The annotation distinguishes the evaluative implication of the comparison target.
- Multiple aspects: The example stating that a good story like the trailer would be sufficient illustrates another comment requiring interpretation of aspect-level sentiment.The passage provides the Malayalam comment and its English gloss but does not state a final category.
- Multiple aspects: A comment praising a trailer while doubting the story is labeled positive because enjoyment of the trailer and optimism toward the movie dominate.The example contains both appreciation and doubt but is resolved as positive.
5. Benchark Systems
The paper benchmarks traditional machine-learning and deep-learning models for five-class Malayalam-English code-mixed sentiment classification using TF-IDF or pretrained embeddings. Among the reported results, 1DConv performs best on macro-averaged precision, recall, and F1-score, while SVM and BERT show class-specific weaknesses.
- Benchmark setup: Six traditional classifiers use TF-IDF features trained only on the newly annotated dataset.The models are Logistic Regression, SVM, Decision Tree, Random Forest, Multinomial Naive Bayes, and K-nearest neighbours.
- Benchmark setup: Four deep-learning baselines—DME, CDME, 1DConv, and BERT—are evaluated with precision, recall, and F-score.DME and CDME combine fastText and word2vec embeddings trained on the code-switched dataset.
- Results: 1DConv achieves the best macro-averaged precision, recall, and F1-score among the reported deep-learning models.Its convolutional filters capture standout features using neighbouring-word context.
- Results: SVM labels all examples as non-Malayalam, producing recall 1.00 and precision around 0.13 for that class.Other than SVM, LR, DT, and RF show considerable macro-average scores, while MNB and KNN trade recall for higher macro-averaged precision.
- Results: BERT fails to identify the mixed-feeling class in the five-class sentiment task.The test set contains positive, negative, neutral, mixed-feeling, and non-Malayalam examples.
6. Conclusion
The paper presents a Malayalam-English YouTube-comment corpus annotated for sentiment analysis and provides agreement measurements, baseline results, and community access. The corpus is intended to support research on code-mixed sentiment analysis and broader code-mixed research.
- Corpus contribution: The contribution is a Malayalam-English code-mixed corpus of YouTube comments annotated for sentiment analysis.
- Research utility: The annotation project aims to enable research on code-mixed sentiment analysis and provide useful data for code-mixed research.
- Resources and evaluation: The paper reports Krippendorff’s alpha, baseline results, and corpus availability to the research community.