Source-linked AI summary
Analyzing Roles of Classifiers and Code-Mixed factors for Sentiment Identification
Soumil Mandal, Dipankar Das
TL;DR
Roman-script English–Bengali code-mixing makes sentiment identification challenging, and the authors note a lack of prior work on this setting. They compare English-feature and code-mixed-feature supervised classifiers on movie reviews, finding higher accuracy with code-mixed training features. They also analyze misclassified snippets and discuss challenges to improving accuracy.
Problem
The authors identify English–Bengali code-mixed sentiment identification in Roman script as an unstudied setting to their knowledge, where mixed grammar and shared script make accurate sentiment analysis challenging.
Source: §1 · pp. 1–2Method
The paper compares supervised classifiers trained on English features and code-mixed features using English and English–Bengali movie-review datasets, then analyzes misclassified snippets.
Source: §1 · pp. 2–3, §2 · p. 3Results
84.45% accuracy on English reviews, 59.00% on code-mixed reviews with English-trained classifiers, and 72.50% on code-mixed reviews with code-mixed-feature training were reported.
Source: §7 · p. 14Takeaways & Limitations
The authors conclude that similar resources, including n-grams and lexicons, are effective for code-mixed analysis, and that SVM tends to perform well when training and test data are similar in type.
Source: §7 · p. 14The relatively small code-mixed training set left Bengali sentiment words such as “ghyam” unidentified, and negation placement also contributed to misclassifications.
Source: §6.3 · p. 13
Abstract
from arXiv · showhide
Multilingual speakers often switch between languages to express themselves on social communication platforms. Sometimes, the original script of the language is preserved, while using a common script for all the languages is quite popular as well due to convenience. On such occasions, multiple languages are being mixed with different rules of grammar, using the same script which makes it a challenging task for natural language processing even in case of accurate sentiment identification. In this paper, we report results of various experiments carried out on movie reviews dataset having this code-mixing property of two languages, English and Bengali, both typed in Roman script. We have tested various machine learning algorithms trained only on English features on our code-mixed data and have achieved the maximum accuracy of 59.00% using Naive Bayes (NB) model. We have also tested various models trained on code-mixed data, as well as English features and the highest accuracy of 72.50% was obtained by a Support Vector Machine (SVM) model. Finally, we have analyzed the misclassified snippets and have discussed the challenges needed to be resolved for better accuracy.
1 Introduction
The paper evaluates supervised classifiers on English–Bengali movie reviews written in Roman script, where code-mixing and shared-script use pose challenges for sentiment identification. It compares English-feature training with code-mixed-feature training and analyzes misclassified snippets.
1 Introduction
- Code-mixing and use of Roman script by speakers of languages with non-Roman alphabets are described as growing social-media practices, while resources and systems for handling such data are lacking.The experiments use English and Bengali.Source: §1 · pp. 1–2
- The study addresses English–Bengali code-mixed sentiment identification in Roman script, a setting the authors say had not previously been studied to their knowledge.The review datasets cover the same topic: movies and films.Source: §1 · p. 2
- 72.50% accuracy was achieved by LSVC trained on code-mixed features; English-trained classifiers reached 59.00% on code-mixed reviews, with MNB performing best in that experiment.On English reviews, LSVC reached 84.45%.Source: §1 · p. 2
- The authors calculate evaluation parameters, inspect misclassified snippets, and discuss challenges and possible steps for improving accuracy.The paper’s later sections present the datasets, classifiers, features, experiments, error analysis, and future work.Source: §1 · p. 3
2 Data Sets
The experiments use separate English and English–Bengali code-mixed movie-review datasets. The code-mixed data is smaller and includes Roman-script Bengali and English in either grammatical arrangement.
2 Data Sets
- The study uses two datasets: English and English–Bengali code-mixed movie reviews.The authors note that the code-mixed collection is comparatively smaller because Bengali is a low-resource language, especially on social media.Source: §2 · p. 3
2.1 English Data
- The English dataset contains 5,331 positive and 5,331 negative snippets; training uses 4,000 per class and testing uses 1,200 per class, with no snippets shared.Table 1 shows the English data distribution.Source: §2.1 · p. 3, Table 1 · p. 3
2.2 Code-Mixed Data
- The code-mixed dataset has 800 positive and 800 negative snippets; 600 per class are used for training and the remaining 200 per class for testing, with no overlap.Collected from Twitter and Facebook between June 15 and December 1, 2016, reviews use Roman script, and either language may supply the grammar.Source: §2.2 · p. 4, Table 2 · p. 4
3 Supervised Classifiers Used
The paper tests supervised Naïve Bayes, linear-model, and support-vector classifiers using scikit-learn defaults. It excludes unsupervised systems from the experiments.
3 Supervised Classifiers Used
- The experiments use supervised classifiers; the authors report that unsupervised sentiment identification had not produced satisfactory results and that supervised models are necessary for special code-mixed features.Implementations use scikit-learn with default parameters.Source: §3 · p. 5
3.1 Naïve Bayes (NB)
- Three Naïve Bayes variants are tested: Gaussian, Bernoulli, and Multinomial, with the family based on Bayes’ theorem and a naïve feature-independence assumption.The variants differ mainly in their assumptions about prob(x/y).Source: §3.1 · p. 5
3.2 Linear Model (LM)
- The linear-model classifiers are Logistic Regression and Stochastic Gradient Descent, with the latter based on a partial-fit method.The paper describes linear models as representing a continuous response as a function of predictor variables.Source: §3.2 · p. 5
3.3 Support Vector Machine (SVM)
- The SVM classifiers are Linear Support Vector Machine, using one-versus-rest, and NuSVC, using one-versus-one.The paper characterizes SVMs as large-margin rather than probabilistic classifiers.Source: §3.3 · p. 5
4 Features
The feature set combines part-of-speech information, n-grams, and sentiment lexicons. The paper describes preprocessing and the intended role of these feature types.
4 Features
- The feature section introduces feature selection as important for supervised-model performance and covers POS tags, n-grams, and lexicon-based features.The subsequent subsections describe these feature families.Source: §4 · p. 5, §4.1 · p. 5, §4.2 · p. 6
4.1 Part Of Speech (POS)
- POS tagging follows punctuation removal with an NLTK regex tokenizer; the authors note that adjectives can contribute strongly to sentiment.The experiments use the NLTK POS tagger.Source: §4.1 · pp. 5–6
4.2 N-Grams
- The experiments use unigrams, bigrams, and trigrams to capture context, including how a preceding negation can reverse a word’s sentiment.The paper illustrates the n-grams with the mixed-language phrase “Movie ta khub bhalo.”Source: §4.2 · p. 6
4.3 SWN 3.0
- SentiWordNet 3.0 is used to help distinguish emotion-bearing words from non-emotion words.The authors use the version available in NLTK.Source: §4.3 · p. 6
4.4 SO-CAL
- SO-CAL is described as a semantic-orientation calculator designed mainly for online product reviews, with adjective, adverb, noun, verb, and intensification dictionaries.Source: §4.4 · p. 6
4.5 NRC emotion lexicon
- The NRC emotion lexicon associates about 14,000 English words with eight basic emotions and positive or negative sentiment.Source: §4.5 · p. 6
5 Experimental Setup & Analysis
Three experiments compare English and English-Bengali code-mixed movie-review data, varying the training and test language and the feature sets. The strongest reported code-mixed result comes from training and testing on code-mixed data.
5.1 English Data
- On English train–English test data, SVMs outperformed other classifier families; LSVC achieved 84.45% accuracy versus NuSVC’s 84.37%, while Naïve Bayes ranked above linear models.Maximum accuracy rose from 72.80% for fe01 to 78.16% for fe02; adding bigrams (fe03) increased accuracy by 1.65% on average, and adding SWN (fe05) improved it by 2.49% on average.Source: §5.1 · pp. 7–8
5.2 Code-Mixed Data
- 59.00% accuracy: MNB performed best when English-trained classifiers were tested on code-mixed data, down from 84.45% in English testing.In this setting, adding nouns, adverbs, and verbs to adjective features raised accuracy by an average of 3.67%; bigrams and trigrams brought little improvement.Source: §5.2 · pp. 8–9, Table 4 · p. 8
- 72.50% accuracy: LSVC performed best when trained and tested on code-mixed data.The experiment used code-mixed unigrams, bigrams, trigrams, and lexicons; adding SWN improved accuracy by an average of 9.35%.Source: §5.2 · pp. 9–10, Table 5 · p. 9
- In code-mixed training, adding SWN improved accuracy by an average of 9.35%, with the authors attributing the gain to many sentiment-bearing English words in the dataset.The code-mixed feature set includes both English and Bengali features, unlike the English-only features used in the English-trained experiment.Source: §5.2 · pp. 9–10
6 Error Analysis
Error analysis compares the strongest systems across English and code-mixed test settings, then examines why code-mixed snippets are misclassified. The examples highlight vocabulary coverage, training-data size, negation, idioms, and word formation as distinct challenges.
6 Error Analysis
- 84.45% for LSVC, 59.00% for MNB, and 72.50% for LSVC were the best accuracies for English→English, English→code-mixed, and code-mixed→code-mixed testing, respectively.The reported systems are LSVC trained on fe07 for English testing, MNB trained on fe07 for code-mixed testing, and LSVC trained on fe13 for code-mixed testing.Source: §6 · p. 10, Table 6 · p. 11, Table 7 · p. 11
6.1 English Train – English Test
- On English train–English test, LSVC (SVM based) performed best with 84.45% accuracy, ahead of NuSVC (SVM based) at 84.37%; TP and TN indicated LSVC was overall quite balanced.Source: §6.1 · p. 10
6.2 English Train - Code-Mixed Test
- With English-only training, classifiers missed sentiment-bearing Bengali words such as kharap and baje, while English words such as patience or satisfactorily could mislead predictions.The authors attribute the majority of misclassifications in this setting to classifiers’ inability to identify important code-mixed words.Source: §6.2 · pp. 11–12
6.3 Code-Mixed Train – Code-Mixed Test
- Code-mixed training corrected some exp2 errors, but small training data left ghyam unidentified, while negation order and idioms remained sources of errors.The examples include na preceding bhalo but following kharap, and the idiom jeno teno prokare.Source: §6.3 · pp. 12–13
6.4 Others
- English roots with Bengali suffixes, such as time(ta) and movie(ta), could evade identification because conventional stemming does not always handle these forms.The authors also note that a POS tagger may assign an English-looking word such as money its usual meaning even when it means something else in a code-mixed bigram.Source: §6.4 · p. 14
7 Conclusion & Future Work
English-feature-trained classifiers reached 84.45% accuracy on English data but 59.00% on code-mixed data, while code-mixed-feature training reached 72.50%. The authors propose expanding data and resources and exploring additional models and features.
7 Conclusion & Future Work
- 84.45% accuracy on English data fell to 59.00% on code-mixed data for English-feature-trained classifiers, while code-mixed-feature training reached 72.50%.Source: §7 · p. 14
- SVM tended to perform well when training and test data were similar, while NB tended to perform better when they differed slightly.Source: §7 · p. 14
- Including SWN in experiment 3 boosted accuracy, consistent with users often using English sentiment words in code-mixed text.Source: §7 · p. 14
- The authors aim to collect larger, cleaner code-mixed corpora spanning more topics and platforms; gathering and cleaning those data is a challenge.Source: §7 · p. 14
- Potential resources include a Bengali SentiWordNet in Roman script and smaller NLP tools such as negation, stop-word, intensifier, and idiom dictionaries.Source: §7 · p. 14
- Future experiments could test language identification followed by POS tagging and polarity detection, other classifiers and ensembles, and additional features such as capitalization and context.Source: §7 · pp. 14–15