Source-linked AI summary
Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text
Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, John P. McCrae
TL;DR
Tamil-English code-mixed sentiment analysis lacks annotated data despite the prevalence of multilingual social-media comments. The paper constructs and annotates a 15,744-comment YouTube corpus, measures annotator agreement, and benchmarks sentiment classifiers. The dataset establishes a substantial resource and baseline, although annotation includes difficult borderline examples and the experiments have a skewed movie-trailer distribution.
Problem
Tamil-English code-mixed sentiment analysis lacks annotated data for a low-resource language, limiting available evidence for analyzing social-media sentiment.
Method
The paper creates and annotates the TamilMixSentiment YouTube-comment corpus, measures agreement with Krippendorff’s alpha, and evaluates sentiment-classification models.
Results
15,744 comments were annotated with Krippendorff’s α of 0.6; logistic regression and random forest performed comparatively best, while deep learning did not yield higher automatic-metric scores.
Takeaways & Limitations
The corpus provides a substantial annotated Tanglish resource and baseline for further code-mixed sentiment research.
Takeaways & Limitations
The corpus is skewed toward positive sentiment because comments were collected from movie trailers, and some examples are difficult for annotators to classify.
Abstract
from arXiv · showhide
Understanding the sentiment of a comment from a video or an image is an essential task in many applications. Sentiment analysis of a text can be useful for various decision-making processes. One such application is to analyse the popular sentiments of videos on social media based on viewer comments. However, comments from social media do not follow strict rules of grammar, and they contain mixing of more than one language, often written in non-native scripts. Non-availability of annotated code-mixed data for a low-resourced language like Tamil also adds difficulty to this problem. To overcome this, we created a gold standard Tamil-English code-switched, sentiment-annotated corpus containing 15,744 comment posts from YouTube. In this paper, we describe the process of creating the corpus and assigning polarities. We present inter-annotator agreement and show the results of sentiment analysis trained on this corpus as a benchmark.
1. Introduction
The paper addresses limited annotated resources for Tamil-English code-mixed sentiment analysis by introducing a gold-standard YouTube-comment corpus and benchmarking classification models. TamilMixSentiment contains 15,744 annotated comments, with Krippendorff’s α of 0.6, and logistic regression and random forest achieve the best reported results.
- Motivation: Tamil-English code-mixed sentiment analysis lacks sufficient annotated data, especially for the relatively low-resource language Tamil.
- Dataset: TamilMixSentiment is a gold-standard dataset of YouTube video comments in Tanglish annotated for sentiment analysis.
- Dataset: The corpus includes Roman-script Tamil-English mixing, Tamil-script comments with English expressions, and inter-sentential, intra-sentential, and tag switching.
- Experiments: The study evaluates ten classification models to establish baselines for future comparisons.
- Experiments: Logistic regression and random forest produce the best results among the evaluated models.
2. Related Work
The related-work context identifies broad growth in code-switched resources but a continuing shortage of sentiment datasets and lexicons, particularly for Tamil-English. The paper responds by creating a Tanglish sentiment dataset and reporting baseline model results.
- Resource gap: Code-switched datasets and lexicons for sentiment analysis remain limited in number, size, and availability.
- Existing resources: Existing code-mixed resources include English-Hindi, English-Spanish, Chinese-English, and English-Bengali datasets, while Tamil-English sentiment annotation had no previous dataset available.
- Paper response: The paper creates a sentiment dataset for Tanglish using voluntary annotators and presents baseline results with several models.
3. Corpus Creation and Annotation
The corpus was built from YouTube comments, filtered for Tamil-English code-mixing, and annotated using a multi-annotator sentiment scheme. Quality control, dataset splitting, and Krippendorff’s alpha supported the released corpus and agreement analysis.
- Corpus Collection: 184,573 Tamil YouTube comments were collected and filtered using comment-level language identification to create a code-mixed corpus.The source comments came from movie trailers released in 2019.
- Annotation Setup: Each sentence was annotated by at least three annotators under a sentiment schema that added an Other language category.Guidelines were provided in English and Tamil.
- Annotation Setup: Eleven native Tamil-speaking volunteers participated, while poor-quality contributions were removed using response-delay, uniform-label, and error-count criteria.Annotators could withdraw, and additional annotators were used when someone did not agree to participate.
- Dataset Split: 15,744 sentences were randomly split into 11,335 training, 1,260 validation, and 3,149 testing examples without k-fold cross-validation.The models were applied to this fixed train-validation-test subset.
- Inter-Annotator Agreement: Krippendorff’s alpha was selected because it handles incomplete annotation coverage and disagreement severity across sentiment classes.The study used nominal and interval metrics for agreement calculation.
4. Difficult Examples
The section presents comments whose sentiment is difficult to assign because their wording supports multiple interpretations. These ambiguities produced disagreements among annotators.
- Several comments are ambiguous because they can express appreciation, disappointment, observation, or no clear sentiment.Examples concern movie trailers, actor comparisons, and recurring movie features.
- Comments about trailers may describe similarities without revealing whether the speaker enjoyed or disliked them.
- Actor comparisons may be interpreted either as disappointment about the hero or positive appreciation for the villain.
- Questions and remarks designated as neutral by the instructions can still be interpreted as neutral, positive, or negative when they concern actors and movies.
- Annotator disagreements were found for sentences with these ambiguous interpretations.
5. Benchmark Systems
The paper applies several machine-learning algorithms to YouTube posts in code-mixed Tamil-English as a simple sentiment-classification baseline.
- Various machine-learning algorithms were applied to determine sentiment in code-mixed Tamil-English YouTube posts.
5.1. Experimental Settings
The experimental settings evaluate several baseline classifiers using text features and report performance through precision, recall, and F-score tables.
- Logistic Regression uses L2 regularization with TF-IDF features consisting of up to 3-grams.
- SVM uses L2 regularization and the same features as Logistic Regression.
- KNN classification is evaluated with 3, 4, 5, and 9 neighbors using uniform weights.
- Decision Tree classification recursively splits nodes to define rules, with maximum depth 800 and minimum sample splits of 5.The reported criteria are Gini and entropy.
- Performance is reported in separate tables for precision, recall, and F-score.
5.1.5. Random Forest (RF):
The paper evaluates Random Forest and Multinomial Naive Bayes as additional classifiers using vectorized text features.
- Random Forest generates trees randomly without defining rules and uses the same features as the Decision Tree.
- Multinomial Naive Bayes is evaluated with α=1 using TF-IDF vectors.
5.1.7. 1DConv-LSTM:
This section describes the evaluated neural and embedding-based models for sentiment classification on code-mixed data, including BERT and dynamic meta-embedding.
- 5.1.7. 1DConv-LSTM:: The evaluated neural architecture combines embedding, dropout, 1D convolution with ReLU, max-pooling, and LSTM layers.Its embeddings are randomly initialized.
- 5.1.7. 1DConv-LSTM:: BERT is a bidirectional language representation model pretrained on unlabelled text and fine-tuned for classification.The paper explores its use for assigning code-mixed data to sentiment categories.
- 5.1.7. 1DConv-LSTM:: Dynamic meta-embedding combines Word2Vec and FastText embeddings trained from the dataset for supervised learning.The paper evaluates this embedding ensemble on the code-mixed data.
5.2. Experiment Results and Discussion
The experiments find that all classifiers perform poorly overall, with logistic regression, random forest, and decision trees comparatively stronger. Performance varies with the imbalanced sentiment distribution, while the movie-trailer source skews the data toward positive comments and leaves room for richer modeling.
- 5.2. Experiment Results and Discussion: All classification algorithms perform poorly on the code-mixed dataset, while logistic regression, random forest, and decision trees fare comparatively better across sentiment classes.SVM shows worse diversity, and deep learning does not improve the three automatic metrics.
- 5.2. Experiment Results and Discussion: 67% of the 15,744 sentences belong to the Positive class, whereas Neutral and Mixed feeling classes receive disastrous precision, recall, and F-measure scores.Negative and Other language classes perform better than Neutral and Mixed feeling classes.
- 5.2. Experiment Results and Discussion: Movie-trailer comments produce more positive sentiment because viewers are more likely to be interested in the movies, skewing the overall distribution.The authors identify this collection setting as a source of imbalance.
- 5.2. Experiment Results and Discussion: The dataset supports further research, including models that incorporate linguistic information or hierarchical meta-embedding.The authors also identify the resource as a starting point for code-mixed research.
6. Conclusion
The paper concludes that it contributes a substantial sentiment-annotated corpus for under-resourced code-mixed Tanglish and establishes baselines across sentiment classes. The authors expect this resource to support further code-mixed research.
- 6. Conclusion: The authors present what they describe as the most substantial corpus for under-resourced code-mixed Tanglish with sentiment-polarity annotations.The corpus uses gold-standard annotated data collected through Google forms from voluntary annotators.
- 6. Conclusion: The paper establishes baselines and reports precision, recall, and F-score results for each sentiment class.It also reports high inter-annotator agreement in terms of Krippendorff α.
- 6. Conclusion: The authors expect this resource to enable researchers to address new problems in code-mixed research.