Source-linked AI summary
Performance Evaluation of Sentiment Analysis on Text and Emoji Data Using End-to-End, Transfer Learning, Distributed and Explainable AI Models
Sirisha Velampalli, Chandrashekar Muniyappa, Ashutosh Saxena
TL;DR
The paper examines sentiment analysis for tweets and emoji data, addressing limited emoji-focused evidence and applications of distributed learning and explainable AI. It uses sentence embeddings with neural classifiers, distributed training, and SHAP, reporting about 98% text accuracy, 70% accuracy on unseen emojis, and roughly 15% lower runtime without reduced accuracy.
Problem
Few researchers have studied emoji sentiment analysis, and the authors report no prior application of distributed machine learning and explainable AI to emoji datasets.
Method
The study uses USE and SBERT embeddings with standard fully connected and LSTM neural networks, distributed parameter-server training, and SHAP explanations.
Results
Text classification accuracy was around 98%, unseen-emoji validation accuracy fell to 70%, and distributed training reduced runtime by roughly 15% without compromising accuracy.
Takeaways & Limitations
USE and SBERT performed well for text embeddings but poorly at identifying semantic relationships between emojis not seen during training.
Abstract
from arXiv · showhide
Emojis are being frequently used in todays digital world to express from simple to complex thoughts more than ever before. Hence, they are also being used in sentiment analysis and targeted marketing campaigns. In this work, we performed sentiment analysis of Tweets as well as on emoji dataset from the Kaggle. Since tweets are sentences we have used Universal Sentence Encoder (USE) and Sentence Bidirectional Encoder Representations from Transformers (SBERT) end-to-end sentence embedding models to generate the embeddings which are used to train the Standard fully connected Neural Networks (NN), and LSTM NN models. We observe the text classification accuracy was almost the same for both the models around 98 percent. On the contrary, when the validation set was built using emojis that were not present in the training set then the accuracy of both the models reduced drastically to 70 percent. In addition, the models were also trained using the distributed training approach instead of a traditional singlethreaded model for better scalability. Using the distributed training approach, we were able to reduce the run-time by roughly 15% without compromising on accuracy. Finally, as part of explainable AI the Shap algorithm was used to explain the model behaviour and check for model biases for the given feature set.
I. INTRODUCTION
Sentiment analysis determines whether opinions are positive, negative, or neutral and supports applications such as trend, review, and market analysis. This work focuses on sentiment analysis involving emojis in tweets and a Kaggle emoji dataset.
- Sentiment analysis determines whether an opinion is positive, negative, or neutral.
- The general framework includes collecting and preprocessing data, transforming it into a computationally suitable form, and labeling it for training.
- Sentiment analysis algorithms are categorized as rule-based, automatic, or hybrid approaches.
- Social media users increasingly use emojis to express or enhance emotions in text.
- The study performs sentiment analysis on emoji-containing tweets and a Kaggle emoji dataset.
- Tweet sentiment analysis is applied in decision support and recommendation systems, motivating efforts to improve its accuracy.
II. PROBLEM STATEMENT AND OBJECTIVES
The paper addresses the limited use of emoji data for sentiment analysis and the lack of distributed machine learning and explainable AI studies on emoji datasets. Its objectives span emoji and tweet classification, embedding-based modeling, scalable training, and bias examination.
- Only a few researchers have studied emojis for sentiment analysis, and the authors report no prior application of distributed machine learning and explainable AI to emoji datasets.
- The study analyzes sentiment in tweets and a Kaggle emoji dataset.
- Universal Sentence Encoder and SBERT are used to generate sentence embeddings.
- Standard fully connected neural networks and LSTM neural networks are used to improve classification accuracy.
- The models are trained with a distributed approach instead of a traditional single-threaded approach for scalability.
- Explainable AI is used to explain model behavior and check for bias based on the given feature set.
III. RELATED WORK
Prior work addressed sentiment analysis across several modalities, but emoji-focused sentiment research remained limited. The paper positions sentence embeddings as an alternative to keyword-based representations because they are designed to capture semantic relationships.
- Earlier research addressed sentiment analysis using text, images, emoticons, audio, or video, while relatively few studies focused on emojis.
- BOW and TF-IDF methods were described as keyword-based representations that do not capture semantic relationships between similarly meaningful words.
- The paper uses S-BERT and Universal Sentence Encoders to generate fixed-length sentence embeddings without manually padding sentences of different lengths.
- The authors state that these end-to-end models do not require manual data cleaning.
- The paper reports improved LSTM results with 98% accuracy after using the sentence-embedding approach.
IV. METHODOLOGY
The methodology combines Twitter and Kaggle emoji data with sentence embeddings and neural classifiers. The workflow splits the datasets, generates fixed-length embeddings with S-BERT and Universal Sentence Encoder, and evaluates standard and LSTM neural networks.
- Twitter and Kaggle data are used for sentiment analysis of tweets and emoji records.The Twitter collection contains 2252 tweets, while the emoji-only dataset contains 855 records.
- S-BERT and Universal Sentence Encoder transform the datasets into sentence embeddings before model training.
- The tweet dataset is divided into 80% training and 20% validation data, and the emoji-only dataset is divided into 80% training and 20% test data.
- The generated embeddings have fixed lengths of 1024 for S-BERT and 512 for Universal Sentence Encoder.
- Standard fully connected and LSTM neural networks are trained and evaluated on the resulting embedding datasets.
- Figure 1 presents the paper's overall methodology.
A. Embedding Models: SBERT and USE
The paper uses SBERT and USE to generate sentence embeddings, with SBERT producing semantically meaningful fixed-sized vectors through contextual sentence representations. LSTM networks are described as recurrent models that retain sequence information through memory cells and gates.
- SBERT and USE: SBERT is a modification of pretrained BERT that derives semantically meaningful sentence embeddings using siamese and triplet network structures.Its fixed-sized vectors support efficient semantic similarity search and clustering.
- SBERT and USE: SBERT produces fixed-sized vectors for input sentences that can be compared using cosine similarity or distance measures.
- LSTM: LSTM networks are recurrent neural networks that learn order dependence in sequence prediction problems.
- LSTM: An RNN processes sequences one element at a time while retaining a state containing information from earlier elements.
- LSTM: LSTM maintains a cell state and carry, using forget, input, and output gates to process sequence information and produce predictions.
C. Distributed Training
The paper applies transfer learning to integrate sentence embeddings with a Keras Sequential DNN and uses parameter-server distributed training to scale model training across machines.
- Distributed Training: The parameter-server strategy is a data-parallel method that scales model training across multiple machines.The cluster contains workers and parameter servers, with variables created on parameter servers and updated by workers.
- Distributed Training: Workers independently read and update shared variables by default without synchronizing with one another.
D. Explainable AI
The paper uses explainable AI to inspect neural-network decisions, identify potential bias or discrimination, and assess model strengths and weaknesses through feature-level explanations.
- Explainable AI: SHAP assigns an importance value to each feature for a particular prediction using a game-theoretic explanation approach.The method connects Shapley-value credit allocation with local explanations of machine-learning outputs.
- Explainable AI: The paper applies XAI to examine model behaviour and check for bias or discrimination against users.
V. EXPERIMENTS
The experiments evaluate sentiment classification for text, emoji-only data, and tweets containing both, repeating the comparisons across SBERT and USE embeddings with Standard and LSTM neural networks.
- Experimental Design: The experiments classify only text, only emojis, and tweets containing both modalities.
- Experimental Design: The experiments repeat each classification setting using both SBERT and USE embedding models.
- Experimental Design: Tables I and II report results for SBERT embeddings with Standard and LSTM neural networks.
- Experimental Design: Tables III and IV report results for USE embeddings with Standard and LSTM neural networks.
A. Distributed Training
The study uses distributed training and SHAP-based explainable AI to improve scalability and examine model behavior and potential bias.
- Distributed training used a parameter server strategy with 5 threads.
- SHAP values were examined for text and emoji tweets to investigate model biases.
- Figure 3 presents the SHAP algorithm as the explainable AI approach.
VI. RESULTS & ANALYSIS
The models performed strongly on text and seen emojis but poorly on unseen emojis, while distributed training preserved accuracy and the examined feature set showed no observed bias.
- 98% text classification accuracy was achieved by both models, improving on the 78% accuracy reported in prior work.
- 100% accuracy was achieved on seen emojis, but accuracy fell to 70% for unseen emojis.The unseen emojis were all wrongly classified.
- Universal and S-BERT sentence embeddings performed well for text but poorly at identifying semantic relationships between emojis.
- Distributed training with 5 threads took 5.88 seconds consistently while maintaining the same model accuracy as standalone training.
- The examined text and emoji tweets were correctly classified by sentiment, and no model biases were observed for the given feature set.
VII. CONCLUSION AND FUTURE WORK
The work evaluates sentiment analysis on tweets and emojis using sentence embeddings, neural models, distributed training, and SHAP explainability. Text accuracy was around 98%, unseen-emoji accuracy fell to 70%, and distributed training reduced runtime by roughly 15% without compromising accuracy.
- The study analyzes Twitter and Kaggle emoji sentiment datasets using Universal Sentence Encoder and SBERT embeddings.
- The embeddings train fully connected and LSTM neural networks for sentiment classification.
- Text classification accuracy was around 98%, whereas validation on unseen emojis reduced accuracy to 70%.
- Distributed training reduced runtime by roughly 15% without compromising accuracy.
- SHAP was used to explain model behavior and check for biases in the given feature set.