PaperlayerView exact revision on arXiv

Source-linked AI summary

Twitter Sentiment Analysis: Lexicon Method, Machine Learning Method and Their Combination

Olga Kolchyna, Tharsis T. P. Souza, Philip Treleaven, Tomaso Aste

arXiv:1507.00955v3cs.CLcs.IRcs.LGstat.MEstat.ML

TL;DR

The paper examines how lexicon-based and supervised machine-learning methods classify sentiment in Twitter messages, where specialized signals and feature choices matter. It compares these approaches on SemEval-2013 data and finds that combining a lexicon score with machine learning yields more precise classifications, while cost-sensitive SVM improves performance for highly unbalanced data.

  • Problem

    The paper addresses how lexicon-based and machine-learning methods can be implemented and compared for Twitter sentiment classification.

  • Method

    The study evaluates lexicon combinations and supervised classifiers on SemEval-2013 task 2-B, using a lexicon sentiment score as an additional machine-learning feature.

  • Results

    The combined approach outperforms the two individual approaches, and cost-sensitive SVM yields a 7% increase in performance over a standard SVM on the benchmark Twitter dataset.

  • Takeaways & Limitations

    Enhancing Twitter lexicons with emoticons, abbreviations, and social-media slang improves lexicon classification, while combining lexicon and machine-learning signals produces more precise classifications.

  • Takeaways & Limitations

    Reported preprocessing effects are not uniform: studies disagree on whether POS tagging improves sentiment-classification results, and but-clause reversal fails for some constructions.

Abstract

from arXiv · show

This paper covers the two approaches for sentiment analysis: i) lexicon based method; ii) machine learning method. We describe several techniques to implement these approaches and discuss how they can be adopted for sentiment classification of Twitter messages. We present a comparative study of different lexicon combinations and show that enhancing sentiment lexicons with emoticons, abbreviations and social-media slang expressions increases the accuracy of lexicon-based classification for Twitter. We discuss the importance of feature generation and feature selection processes for machine learning sentiment classification. To quantify the performance of the main sentiment analysis methods over Twitter we run these algorithms on a benchmark Twitter dataset from the SemEval-2013 competition, task 2-B. The results show that machine learning method based on SVM and Naive Bayes classifiers outperforms the lexicon method. We present a new ensemble method that uses a lexicon based sentiment score as input feature for the machine learning approach. The combined method proved to produce more precise classifications. We also show that employing a cost-sensitive classifier for highly unbalanced datasets yields an improvement of sentiment classification performance up to 7%.

1 Introduction

The paper motivates Twitter sentiment analysis by the scale and commercial value of social-media opinions, then introduces lexicon, machine-learning, and combined approaches. It evaluates these methods on SemEval-2013 Twitter data and reports that combining lexicon scores with machine learning outperforms either approach alone.

  • Motivation: Twitter’s large, diverse stream of opinions makes it useful for analyzing brands, products, politicians, well-being, and social events.The paper links this information to understanding collective behavior and possible market or unemployment predictions.
  • Approaches: The study presents lexicon-based and machine-learning approaches for Twitter sentiment classification.It describes both methods step by step and evaluates their adoption for Twitter messages.
  • Evaluation: A combined method uses a lexicon sentiment score as a machine-learning feature and outperforms the two individual approaches.The evaluation uses the SemEval-2013 competition dataset, task 2-B.

2 Sentiment Analysis Methodology: Background

Sentiment analysis builds on text categorization and commonly represents messages with bag-of-words features. Lexicon and machine-learning methods differ in how they assign sentiment, while newer work incorporates linguistic, social, and visual context and ensemble strategies.

  • Background: Bag-of-words represents documents as vectors of independent words, commonly called unigrams.The representation is described as simple and capable of high performance.
  • Core methods: Lexicon methods assign polarity scores to lexicon-matched unigrams, whereas machine-learning classifiers use unigrams or N-grams as features.The lexicon approach computes overall text polarity from unigram polarities.
  • Sentiment features: Research has examined part-of-speech patterns and emoticons as signals for detecting sentiment polarity.Adverb-adjective combinations and emoticons are cited as specific lines of investigation.
  • Context: Advanced algorithms incorporate message context, authors, social relations, network structure, and visual content alongside textual information.The cited work includes joint clustering of tweets, users, and features and content-plus-context features.
  • Alternative methods: Ensemble methods and feature-subset search have been explored to improve sentiment classification beyond individual base learners.Reported approaches include Bagging, Boosting, Random Subspace, Bayesian Model Averaging, and genetic algorithms.

3 Data Pre-processing for Sentiment Analysis

Twitter preprocessing converts noisy social-media text into classification features through tagging, normalization, tokenization, and polarity handling. The section also emphasizes that preprocessing choices involve trade-offs and may not improve performance uniformly.

  • Preprocessing: Preprocessing is used before sentiment extraction to improve text-classification quality and reduce computational complexity.The procedure includes multiple transformations of the incoming text.
  • Part-of-speech tagging: POS tagging extracts noun, pronoun, adverb, adjective, verb, interjection, and intensifier patterns from text.The paper notes disagreement about whether POS tagging improves classification.
  • Part-of-speech tagging: POS tagging has mixed reported effects: some studies find improvements, while others report decreased performance.The paper does not identify a common conclusion across studies.
  • Stemming and lemmatisation: Stemming maps related word forms to roots, reducing bag-of-words dimensionality but risking over-stemming and under-stemming errors.Over-stemming lowers precision, under-stemming lowers recall, and the overall effect depends on the dataset and algorithm.
  • Negation handling: Negation handling reverses or shifts sentiment polarity according to negation terms and their scope.The simplest approach reverses words between a negation and the following punctuation mark; contextual valence shifters also include intensifiers and diminishers.
  • But-clauses: But-clauses are handled by assigning opposite orientations before and after polarity-shifting conjunctions.The rule fails for constructions such as “Not only he is smart, but also very kind,” where “but” is not contrary.
  • Tokenisation into N-grams: Twitter tokenization is difficult because emoticons, URL links, and abbreviations are not easily separated as ordinary text.N-grams combine neighboring words, but larger n-grams can increase statistical sparseness.
  • Tokenisation into N-grams: N-gram effects vary by classifier and dataset: bigrams improved Naive Bayes on Yahoo-Science from 65% to 70% break-even point, while trigrams performed poorly.The paper reports no unique optimal n-gram size and a decrease in 28 of 48 Rocchio experiments using bigrams.

4 Sentiment Computation with Lexicon-Based Approach

The lexicon-based approach assigns sentiment from polarity-bearing words or phrases using a dictionary, with preprocessing, lexicon lookup, and score aggregation forming its core pipeline. Lexicon construction and composition affect performance, and the resulting score maps texts to sentiment classes.

  • Lexicon-based approach: Lexicon-based sentiment analysis requires a dictionary whose words or phrases have assigned polarity values.Examples include Opinion Lexicon, SentiWordNet, AFINN, LoughranMcDonald, NRC-Hashtag, and General Inquirer.
  • Sentiment computation: The pipeline preprocesses text, checks each token against the lexicon, adds matched polarities, and computes the final sentiment score.Unmatched words receive polarity zero, while the final score divides sentiment-bearing word scores by their count.
  • Sentiment score: Averaging produces sentiment scores from -1 to 1, where 1 is strongly positive, -1 strongly negative, and 0 neutral.The paper’s example obtains a score of 0.616 and labels the sentence as positive.
  • Lexicon construction: Lexicon quality strongly affects classification, and lexicons can be manually constructed, trained from labelled data, or expanded through bootstrapping.Manual construction is time-consuming; trained-data construction uses labelled sentences, while bootstrapping adds related adjectives or lexical relations.
  • Lexicon construction: Manually constructed lexicons can encode polarity classes and intensity, as illustrated by MPQA’s 8,222 subjective expressions.The corpus includes POS tags, polarity classes, and intensity.

5 A Machine Learning Based Approach

Machine learning sentiment classification treats Twitter messages as labelled examples, extracts features, and trains classifiers to assign sentiment classes. The paper discusses common algorithms and the need to include neutral messages.

  • Machine learning approach: Supervised sentiment classification learns from previously labelled positive, negative, or neutral texts to classify unseen examples.The process extracts features that model differences between classes and infers a classification function.
  • Algorithms: The paper identifies Support Vector Machines, Naive Bayes, and Decision Trees as popular algorithms for text classification.Prior studies reported differing strengths among these classifiers.
  • Sentiment classes: Twitter sentiment classification includes positive, negative, and neutral classes because short messages often state facts rather than opinions.The neutral class is therefore incorporated into the classification process.

2. Feature generation.

Feature generation represents Twitter text through attributes that capture lexical and sentiment patterns. Common features include n-grams and message-level indicators such as negations, length, punctuation, and part-of-speech counts.

  • Feature generation: N-gram features represent either whether a token sequence is present or how frequently it occurs in each text.Presence-based vectors use binary values, while frequency-based vectors use occurrence counts.
  • Feature generation: Additional features count positive and negative words, negations, message length, exclamation marks, parts of speech, and comparative or superlative adjectives.These attributes supplement lexical n-gram representations with message-level information.

3. Feature selection.

Feature selection reduces the rapidly expanding text feature space by retaining features with predictive value. The process searches, evaluates, and stops according to predefined or performance-based criteria before classifier training.

  • Feature selection: Feature-space dimensionality grows with dataset size, making it impractical to calculate every n-gram feature.Removing redundant and irrelevant features can reduce computation, speed prediction, and help avoid overfitting.
  • Selection procedure: Feature selection searches for a subset of features, evaluates that subset, and applies a stopping criterion.Search may proceed forward, backward, or randomly; evaluation can use Chi-squared, Information Gain, Odds Ratio, Probability Ratio, Document Frequency, or Term Frequency.
  • Learning algorithm: After feature generation and selection, the resulting representation is used to train a classification algorithm.The paper notes that choosing the best classifier remains difficult because methods have different advantages and disadvantages.
  • Learning algorithm: SVMs classify linearly separable two-class data by maximizing the margin between the separating hyperplane and closest training examples.For three sentiment classes, the described approach uses positive-negative, negative-neutral, and positive-neutral pairwise classifications.
  • Evaluation: Accuracy, error rate, precision, recall, and F-score provide alternative evaluation measures depending on the task.For unbalanced datasets, precision and recall may be preferable, while F-score balances precision and recall when they trade off.

6 Application of Lexicon and Machine Learning Methods for Twitter Sentiment Classification

The paper applies lexicon-based and machine-learning sentiment methods to SemEval-2013 Twitter message classification and describes their preprocessing and automatic lexicon-generation procedures. The workflow includes tokenization, Twitter-oriented tagging, polarity scoring, and handling of negations and neutral words.

  • Benchmark application: The case study evaluates sentiment methods on SemEval-2013 Task 2-B, which classifies messages as positive, negative, or neutral.For mixed-sentiment messages, the stronger sentiment is selected.
  • Benchmark application: The study also applies its algorithm to 42,803,225 retail-related Twitter messages to investigate relationships with stock returns and volatility.This is presented as a separate large-scale application.
  • Pre-processing: Preprocessing uses WEKA, extracts unigrams and bigrams, applies a default stop-word list, and selects ArkTweetNLP because it was trained on Twitter data.ArkTweetNLP provides tags for Twitter-specific elements including hashtags, at-mentions, retweets, emoticons, and abbreviations.
  • Pre-processing: The implementation filters tagged tokens to common nouns, verbs, adjectives, adverbs, interjections, emoticons, abbreviations, foreign words, and possessive endings.These categories are retained after part-of-speech tagging.
  • Negation handling: Negation handling reverses the sentiment score of words between a negation and clause-level punctuation, with exceptions for phrases and negative rhetorical questions lacking negation scope.Examples include “not only” and questions whose negation appears within the first three words.
  • Automatic lexicon generation: The lexicon maps sentiment scores to [-1;1], removes words in the neutral polarity range [0.4; 0.6], and illustrates these scores with examples such as GOOD, BAD, and LIKE.LIKE is treated as neutral because its uses can express preference, comparison, or no overall polarity.

Lexicons Combinations.

The study compares lexicon combinations for Twitter sentiment analysis and constructs an EMO lexicon containing emoticons, abbreviations, and social-media slang.

  • Existing public lexicons often omit emoticons and social-media slang, while preprocessing may remove these expressions as typographical symbols.
  • The manually constructed EMO lexicon contains emoticons, abbreviations, and slang words commonly used to express emotions on social media.
  • The study evaluates combinations built around the opinion lexicon baseline, including combinations with EMO and automatically created lexical resources.

Sentiment Score Calculation.

The paper calculates tweet sentiment with simple-average and logarithmic scores, then uses these scores and generated features for classification. The logarithmic score produces more distinct sentiment classes, while selected lexicon and manually constructed features improve classifier performance.

  • Sentiment Score Calculation: The logarithmic sentiment score is normalized to range from -1 to 1, with -1 most negative and 1 most positive.
  • Sentiment Score Calculation: Logarithmic scoring makes positive, negative, and neutral classes more defined and facilitates setting classification thresholds compared with simple averaging.
  • Sentiment Score Calculation: K-means clustering assigns sentiment labels from Simple Average and Logarithmic scores and compares predictions with tweet labels.
  • Feature Generation: Feature generation combines unigram presence, lexicon sentiment, elongated-word counts, emoticon indicators, negation, part-of-speech, punctuation, and token-count features.
  • Feature Selection: 1826 generated features were reduced to 528 using Information Gain and Ranker selection, retaining features with information gain above zero.
  • Feature Validation: Adding lexicon sentiment and manually constructed features increased Naive Bayes accuracy by 7%, Decision Trees by over 9%, and SVM by 4.5%.

Training the Model, Validation and Testing.

The study trains supervised classifiers on labeled SemEval-2013 Twitter data and evaluates them on an unseen test set. Cost-sensitive SVM performs best among the reported models and exceeds the competition winner by four points.

  • Training: A binary feature matrix from labeled SemEval-2013 Task 2-B tweets trains Naive Bayes, Support Vector Machines, and other classifiers.
  • Validation and Testing: Models are tested on an unseen SemEval-2013 Task 2-B set using F-score between positive and negative classes for three-class classification.
  • Results: Cost-sensitive SVM achieved the best reported F-measure at 73%, compared with 66% for SVM, 64% for Naive Bayes, and 62% for Decision Trees.
  • Results: The cost-sensitive SVM result scored 4 points higher than the winner of the SemEval-2013 competition.

7 Conclusion

The paper reviews lexicon-based and machine-learning approaches for Twitter sentiment analysis, showing that expressive social-media signals improve lexicons and that cost-sensitive classification benefits imbalanced data.

  • The paper reviews lexicon-based and machine-learning methods for sentiment analysis.
  • Lexicon-based approach: OL + EMO outperforms both traditional OL and the larger OL + EMO + AUTO lexicon on a benchmark Twitter dataset.The result supports incorporating emoticons, abbreviations, and social-media slang, while larger lexicons can reduce performance through polarity ambiguity and increased complexity.
  • Machine-learning approach: 7% increase in performance over a standard SVM is achieved by a cost-sensitive SVM on the benchmark Twitter dataset.The improvement applies to highly unbalanced datasets.
  • Machine-learning approach: A lexicon sentiment score is used as an input feature for training machine-learning classifiers.Feature ranking by information gain placed the lexicon feature at the top of the list.
Loading 1507.00955v3…