Source-linked AI summary
Towards Sub-Word Level Compositions for Sentiment Analysis of Hindi-English Code Mixed Text
Ameya Prabhu, Aditya Joshi, Manish Shrivastava, Vasudeva Varma
TL;DR
Sentiment analysis for Hindi-English code-mixed social-media text lacks suitable annotated resources and must handle severe spelling and structural variation. The paper introduces an annotated dataset, compares sentiment-analysis methods, and uses sub-word representations in an LSTM; Subword-LSTM achieves an F-score of 0.658 versus 0.511 for Char-LSTM.
Problem
Hindi-English code-mixed sentiment analysis is limited by the lack of a suitable annotated dataset and by highly noisy, sparse text with spelling and structural variation.
Method
The paper constructs and annotates a Hi-En code-mixed dataset, compares sentiment-analysis methods, and introduces an LSTM using sub-word-level representations.
Results
Subword-LSTM obtains an F-score of 0.658, compared with 0.511 for Char-LSTM, while Multinomial Naive Bayes outperforms SVM on snippets.
Takeaways & Limitations
Subword-LSTM interprets sentiment through morpheme-like structures and produces results significantly better than baselines on noisy Hi-En code-mixed data.
Takeaways & Limitations
Further work should examine scaling recurrent neural networks and using larger datasets, as well as deeper architectures for noisy social-web data.
Abstract
from arXiv · showhide
Sentiment analysis (SA) using code-mixed data from social media has several applications in opinion mining ranging from customer satisfaction to social campaign analysis in multilingual societies. Advances in this area are impeded by the lack of a suitable annotated dataset. We introduce a Hindi-English (Hi-En) code-mixed dataset for sentiment analysis and perform empirical analysis comparing the suitability and performance of various state-of-the-art SA methods in social media. In this paper, we introduce learning sub-word level representations in LSTM (Subword-LSTM) architecture instead of character-level or word-level representations. This linguistic prior in our architecture enables us to learn the information about sentiment value of important morphemes. This also seems to work well in highly noisy text containing misspellings as shown in our experiments which is demonstrated in morpheme-level feature maps learned by our model. Also, we hypothesize that encoding this linguistic prior in the Subword-LSTM architecture leads to the superior performance. Our system attains accuracy 4-5% greater than traditional approaches on our dataset, and also outperforms the available system for sentiment analysis in Hi-En code-mixed text by 18%.
1 Introduction
The paper frames Hindi-English code-mixed sentiment analysis as important but difficult because romanized social-media text is noisy and suitable annotated resources are scarce. It proposes sub-word representations in LSTMs alongside a dataset and comparative evaluation of sentiment-analysis methods.
- Motivation: Hindi-English code mixing is widespread in multilingual India and creates opportunities for sentiment, opinion, and linguistic analysis.The paper notes India has 22 official languages and over 375 million people online, with Hindi use increasing on the internet.
- Related work: The paper situates its contribution against prior Hindi sentiment resources and code-mixed approaches, while noting that existing shared tasks do not meet sentiment-analysis requirements.Earlier work included fallback models, sentiment lexicons, and lexicon lookup for Hindi or code-mixed text.
- Challenges: Romanized code-mixed social-media text contains contractions, non-standard spellings, and nongrammatical constructions that produce many word variations.Hindi is phonetically typed, while Roman-script English does not preserve phonetics, increasing variation in online Hindi words.
- Contributions: The paper contributes a manually annotated Hi-En code-mixed sentiment dataset and evaluates state-of-the-art sentiment-analysis techniques on it.The contributions include dataset creation, annotation, analysis, and empirical comparison of methods.
- Contributions: Sub-word representations are introduced for LSTMs to model morpheme-like information instead of relying only on character- or word-level representations.The paper investigates whether linguistic priors can be encoded through sub-word features for sentiment analysis.
2 Dataset
The authors construct a sentiment-annotated Hi-En code-mixed dataset from public Facebook comments, filtering the data to retain suitable Roman-script, single-sentence examples. The dataset captures grammatical and spelling variation characteristic of romanized Hindi-English text.
- Data collection: Comments were collected from public Facebook pages for Salman Khan and Narendra Modi, whose pages had 31 million and 34 million likes respectively.The pages were selected for their large Indian audiences and varied code-mixed sentiment comments.
- Preprocessing: The preprocessing removed non-Roman-script comments, comments longer than 50 words, complete English sentences, and multi-sentence comments.Single sentences were retained because different sentences could express different sentiment polarities.
- Dataset characteristics: The dataset contains short, grammatically unclear sentences and multiple spellings caused by Hindi romanization and flexible sentence ordering.These variations are identified as major challenges in Hi-En code-mixed data.
- Dataset contribution: Unlike several English datasets and a prior Hi-En code-mixed dataset without sentiment labels, this resource includes sentiment polarity annotations.The constructed dataset is described as comparable in size with several existing datasets.
- Availability: The dataset and code are freely available for download to support further research.
3 Learning Compositionality
The paper argues that sub-word representations offer a better linguistic unit than words or individual characters for noisy code-mixed sentiment analysis. It generates these representations with character convolutions, pools them, and models their sequence with an LSTM before classification.
- Motivation: Large vocabulary growth in the dataset challenges word-level methods that assume a small vocabulary relative to data size.The paper specifically identifies Word2Vec, N-grams, and Word-RNNs as unsuitable under this condition.
- Character-level models: Character-level models avoid fixed-vocabulary limits but may represent arbitrary character combinations lacking semantic information.The paper contrasts meaningless combinations such as “Cat − C + B = Bat” with semantically interpretable character groups.
- Sub-word level representations: Sub-word representations combine characters into meaningful lexical units that can carry semantic weight and generate new lexical structures.Examples include “Un + Holy = Unholy” and “Cat + s = Cats.”
- Sub-word level representations: The proposed method applies one-dimensional convolutions to character inputs to produce sub-word-level, morpheme-like feature maps.Each filter processes a window of character embeddings, adds a bias, applies a nonlinearity, and produces a feature map.
- LSTM sentiment modeling: An LSTM models relationships among pooled sub-word features, retains relevant history, and passes the resulting representation to a fully connected sentiment classifier.The architecture propagates useful information through memory before calculating final sentiment polarity.
- Evaluation: The experiments qualitatively and quantitatively evaluate the proposed architectural claims.The paper presents the architecture schematically before describing the experiments.
4 Experiments
The experiments compare methods suited to sparse, noisy Hi-En code-mixed data and evaluate the proposed Subword-LSTM against character-level and conventional approaches. Results favor subword representations and show that learned features can capture sentiment despite misspellings.
- Method comparison: The study compares state-of-the-art methods that can generalize to code-mixed data, including Sharma et al.’s Hi-En sentiment system.The evaluation also considers approaches that may not suit code-mixed text because required linguistic tools, reliable embeddings, or surface features are unavailable.
- Method comparison: Sparse data favors unigrams over bigrams and bag of words over tf-idf, while lexicon lookup performs poorly because misspellings cause incorrect transliterations.These observations contrast with trends reported for English text and are attributed to sparsity and noisy spelling variation.
- Subword-LSTM evaluation: 0.658 F-score for Subword-LSTM exceeds 0.511 for Char-LSTM on the Hi-En dataset.The comparison uses the same broad LSTM setting, with sub-word features replacing direct character-level representations.
- Subword-LSTM evaluation: The hypothesis is cross-validated on SemEval’13 Twitter data using 7,800 training tweets and 1,368 development tweets without extra features or heuristics.Tweets are lowercased and tokenized before character-by-character model input.
- Qualitative analysis: Figure 4 visualizations show filters identifying sentiment-bearing word sections despite misspellings, while severe morpheme stretching affects sentiment polarity.Different filters learn mappings from different parts of comments, including sentiment-bearing phrases such as “party” and “gift.”
5 Conclusion
The paper presents Sub-Word LSTM for sentiment analysis in noisy Hindi-English code-mixed text, where existing methods face tool and sparsity limitations. It reports significantly better results than baselines while identifying larger datasets and RNN scaling as directions for further study.
- Sub-Word LSTM learns sentiment from morpheme-like structures in noisy Hindi-English code-mixed data.The approach is introduced specifically for sentiment learning in this setting.
- Existing sentiment-analysis methods may be unsuitable because Hi-En code-mixed text lacks NLP tools and unsupervised word representations suffer from sparsity.
- Sub-Word LSTM produces significantly better results than the baselines.
- Further work should examine how RNN scaling and larger datasets affect the results.