Twitter Sentiment Analysis via Bi-sense Emoji Embedding and Attention-based LSTM

1 INTRODUCTION

• Emoji meanings and sentiments can vary with context, including sarcasm where an emoji’s sentiment opposes the text.A single emoji embedding can therefore confuse a sentiment model when its signal conflicts with the text.Source: §1 · pp. 1–2
• The proposed bi-sense scheme gives each emoji separate positive- and negative-sense vectors, which an attention-based LSTM uses alongside word embeddings.The attention mechanism selects the emoji sense according to context.Source: §1 · p. 2
• Because existing Twitter sentiment datasets often exclude emojis and few emoji tweets have sentiment labels, the authors construct a dataset using weak labels and a manually labeled subset.Vader generates weak labels; manually labeled tweets are used for fine-tuning and testing.Source: §1 · p. 2
• The experiments report that the bi-sense embeddings with attention-based LSTMs outperform state-of-the-art sentiment analysis models.The authors also visualize the learned emoji semantics and sentiments.Source: §1 · p. 2

2.1 Sentiment Analysis

• Multimodal sentiment methods commonly enforce similar sentiment across modalities, but Twitter text and emojis can diverge through irony, jokes, or sarcasm [15].Textual sentiment work ranges from lexical features to neural embedding approaches.Source: §2.1 · p. 2

2.2 Emojis and Sentiment Analysis

• Prior work assigns emoticon labels, smooths noisy labels, uses emoji features [31], ranks emoji sentiment [32], learns emoji embeddings [12], or pretrains on emoji prediction [15].The surveyed approaches include both lexical methods and learned representations.Source: §2.2 · p. 3
• Lexical and dictionary approaches provide limited understanding of emoji semantics, while emoji-labeled sentiment datasets remain scarce because studies often filter emojis [15].Source: §2.2 · p. 3

3 METHODOLOGY

• The workflow comprises bi-sense embedding initialization, self-selected senti-emoji embedding generation, and sentiment classification with the proposed attention-based LSTMs.The two mechanisms are Word-guide Attention-based LSTM and Multi-level Attention-based LSTM.Source: §3 · p. 3

3.1 Bi-sense Embedding

• Each emoji receives positive- and negative-sense vectors by training separate tokens with fastText on sentiment-specific contexts.Text sentiment is initialized with Vader; the authors select fastText partly for computational efficiency and report only its initialization results.Source: §3.1 · p. 3

3.2 Word-guide Attention-based LSTM

• WATT conditions attention over emoji senses on the current word, then concatenates the weighted senti-emoji vector with that word as LSTM input.The attention function is a fully connected layer with ReLU activation; the final LSTM output feeds a sigmoid sentiment classifier trained with binary cross-entropy.Source: §3.2 · p. 4

3.3 Multi-level Attention-based LSTM

• MATT first derives a tweet-level senti-emoji vector, then uses additional attention to distribute emoji information across the LSTM steps.It concatenates the attention vector with each original word embedding and uses the same network configuration and binary cross-entropy loss as WATT.Source: §3.3 · p. 4

4.1 Data Collection and Annotation

• The dataset contains 1,492,065 tweets and 55 frequently used emojis; 4,183 manually labeled tweets came from samples with sentiment scores between 0.60 and 0.70.Tweets with absolute scores over 0.70 retained automatic labels; manual examples with annotator disagreement or neutral labels were discarded, and no neutral class was included.Source: §4.1 · pp. 4–5
• About 74% was the average positive ratio across emojis, and most emojis appeared in both positive- and negative-sentiment tweets.The authors cite strong-positive emojis appearing in negative tweets at about 5% of occurrences as one example supporting sentiment-specific emoji representations.Source: §4.1 · p. 5, Table 2 · p. 5

4.2 Sentiment Analysis

• Over 82.4% accuracy was achieved by models using bi-sense emoji embeddings, while ATT-E-LSTM reached F1@84.6% and accuracy@82.0% on HA-Sentiment.MATT-BiE-LSTM and WATT-BiE-LSTM improved performance with attention, and the authors report that the proposed models outperform baseline models; results followed fine-tuning with manually annotated samples.Source: Table 3 · p. 6, §4.2 · p. 6
• BiE-LSTM significantly outperformed T-LSTM and E-LSTM, and all bi-sense models significantly outperformed models without emoji features or with word-emoji embeddings.MATT-BiE-LSTM and WATT-BiE-LSTM had similar performance despite computing attention differently; the authors suggest extending the approach to multi-sense embeddings.Source: §4.2 · p. 6

4.3 Qualitative Analysis

• In Figure 3, MATT-BiE-LSTM focused more on sentiment-relevant words than ATT-E-LSTM, including “happy” and “lovely” rather than conflicting or irrelevant words.The authors report that word-emoji attention assigned more weight to “no” and “pressure” in one example and to sentiment-irrelevant words in others.Source: Figure 3 · p. 7, §4.3 · p. 7

4.4 Bi-sense Emoji Embedding Visualization

• In Figure 4, positive- and negative-sense emoji embeddings form separate clusters, and their differences preserve similar cluster patterns.The authors interpret the separated clusters as evidence of semantic differences across positive- and negative-sentiment contexts.Source: Figure 4 · p. 7, §4.4 · pp. 7–8

• The proposed approach achieves state-of-the-art performance on sentiment analysis.Source: §5 · p. 8
• Bi-sense emoji embeddings represent complex semantic and sentiment information, while attention-based LSTMs selectively attend to correlated emoji senses and fuse them with word embeddings.Source: §5 · p. 8
• The authors plan to extend the attention-based LSTM and bi-embedding framework to tasks involving multi-sense embeddings, including word-sense embedding.Source: §5 · p. 8