Source-linked AI summary
Sentiment Analysis of Code-Mixed Indian Languages: An Overview of SAIL_Code-Mixed Shared Task @ICON-2017
Braja Gopal Patra, Dipankar Das, Amitava Das
TL;DR
The paper addresses limited sentiment-analysis work on code-mixed Indian-language data, especially Bengali-English. It presents a shared task and dataset for Hindi-English and Bengali-English; top systems outperform random baselines on both pairs, with the highest overall macro average f-score on Hindi-English.
Problem
Prior code-mixed sentiment analysis included Hindi-English work, but almost no work addressed Bengali-English, despite multilingual code-mixing in Indian user-generated text.
Source: §1 · pp. 1–2Method
The paper presents a shared task using annotated Hindi-English and Bengali-English code-mixed datasets, evaluated with macro average f-score.
Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 5Results
Top systems outperformed random baselines on both language pairs; the best HI-EN macro average f-score was 0.569.
Source: §4.3 · p. 7, §5 · p. 8Takeaways & Limitations
The authors plan to expand the dataset and propose aspect-based sentiment analysis and stance detection as future tasks for code-mixed data.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Sentiment analysis is essential in many real-world applications such as stance detection, review analysis, recommendation system, and so on. Sentiment analysis becomes more difficult when the data is noisy and collected from social media. India is a multilingual country; people use more than one languages to communicate within themselves. The switching in between the languages is called code-switching or code-mixing, depending upon the type of mixing. This paper presents overview of the shared task on sentiment analysis of code-mixed data pairs of Hindi-English and Bengali-English collected from the different social media platform. The paper describes the task, dataset, evaluation, baseline and participant's systems.
1 Introduction
Mining social-media text is difficult because it is noisy, and India’s multilingual communication also produces code-mixed text. This shared task evaluates sentiment identification for Hindi-English and Bengali-English, where prior work on Bengali-English sentiment was nearly absent.
- Sentiment analysis of Bengali-English code-mixed text had almost no prior work, while Hindi-English sentiment analysis had been studied.Source: §1 · p. 2
- The shared task evaluated sentiment identification on Hindi-English and Bengali-English code-mixed data from social media.Source: §1 · p. 2
- Forty participants registered, but nine teams submitted predictions; eight submitted HI-EN runs and seven submitted BN-EN runs.Source: §1 · p. 2
- Systems were ranked by macro average f-score using training and test data with language and sentiment tags.Source: §1 · p. 2
2 Related Work
Related work covers code-mixed language identification, POS tagging, and named-entity recognition, while sentiment analysis remains difficult amid noisy text, limited resources, and small datasets. Prior code-mixed sentiment work focused on Hindi-English, with little work on Bengali-English.
- Code-mixed language identification, POS tagging, and named-entity recognition had been addressed in prior work and shared tasks.Source: §2 · pp. 2–3
- Sentiment analysis is difficult because code-mixed text is noisy, requires language identification and POS tagging, lacks sentiment-annotated lexicons, and has small datasets.Source: §2 · p. 3
- Prior code-mixed sentiment analysis included Hindi-English work using sub-word representations in an LSTM, while Bengali-English sentiment work was nearly absent.Source: §2 · p. 3
3 Dataset and Evaluation
The section introduces the dataset and evaluation procedure, describing a newly developed resource for code-mixed Indian-language research. It outlines tweet collection and annotation, then defines overall and two-way evaluation.
3 Dataset and Evaluation
- The authors developed a dataset because no code-mixed dataset for Indian languages was available for research.Source: §3 · p. 3
3.1 Dataset
- Twitter4j collected Hindi-English and Bengali-English tweets using common language words, mostly sentiment words in Romanized form; incomplete, spam, and language-ineligible tweets were rejected.Source: §3.1 · pp. 3–4
- A manually developed dictionary automatically assigned word-level language tags, followed by manual positive, negative, or neutral sentiment annotation and correction of language-tag errors.Source: §3.1 · p. 4
3.2 Evaluation
- Systems were ranked by macro average f-score, which averages class-specific scores equally; evaluation included overall three-class scores and two-way positive, negative, and neutral classification.Source: §3.2 · p. 5
4 System Description and Comparison
The section describes the baseline and participant systems for the HI-EN and BN-EN sentiment task, then compares their results. IIIT-NBP achieved the highest overall macro average F-scores, while the authors discuss feature choices and dataset size as relevant to the observed results.
4.1 Baseline
- The baseline randomly assigns sentiment labels to test instances and is evaluated using the same evaluation techniques as the submitted systems.Source: §4.1 · p. 5
4.3 Results and Discussion
- IIIT-NBP achieved the maximum overall macro average f-score, 0.569 for HI-EN and 0.526 for BN-EN; the random baseline beat CEN@AMRIT and SVNIT on HI-EN.Source: Table 2 · p. 6, §4.2 · p. 6, §4.3 · p. 7
4.2 System Descriptions
- IIIT-NBP used GloVe embeddings, word and character n-gram TF-IDF features, and linear SVM or an ensemble; KS used n-grams, English and Bengali sentiment lexicons, and multinomial Naive Bayes.Source: §4.2 · p. 7
- BIT Mesra submitted only for HI-EN, using sentiment-tagged emoji and unigram/bigram SVM or Naive Bayes; AMRITA used 1–5-grams with linear-kernel SVM.Source: §4.2 · p. 7
- CFIL combined fastText embeddings, CNN subword features, and a Bi-LSTM; Subway submitted only for HI-EN and used manually polarity-tagged bigrams and trigrams with Naive Bayes.Source: §4.2 · p. 7
4.3 Results and Discussion
- Two-way maximum f-scores were 0.707/0.666/0.663 for HI-EN and 0.641/0.677/0.621 for BN-EN (positive/negative/neutral); HI-EN scored better, attributed to more instances.Source: §4.3 · pp. 7–8
- Most teams used n-gram features, which the authors associate with better macro average f-score; KS used sentiment lexicons only for BN-EN, while BITMesra removed hashtags despite their cited importance (Patra et al., 2016).Source: §4.3 · p. 8
- The deep-learning-based CFIL system did not perform as well as machine-learning systems, which the authors suggest may reflect insufficient training data.Source: §4.3 · p. 8
5 Conclusion and Future Work
The paper reports the ICON 2017 HI-EN and BN-EN sentiment task results, with IIIT-NBP attaining the highest overall macro average f-scores. It proposes expanding the dataset and exploring aspect-based sentiment analysis and stance detection.
- IIIT-NBP achieved maximum macro average f-scores of 0.569 for HI-EN and 0.526 for BN-EN, using word and character n-grams with SVM.Source: §5 · p. 8
- The authors plan to enhance the current dataset and include more data pairs in a future version of the shared task.Source: §5 · p. 8
- The authors identify aspect-based sentiment analysis and stance detection as possible future tasks for code-mixed data.Source: §5 · p. 8