Source-linked AI summary
DepecheMood: a Lexicon for Emotion Analysis from Crowd-Annotated News
Jacopo Staiano, Marco Guerini
TL;DR
Emotion lexica are scarce and limited in coverage compared with sentiment lexica, motivating a resource for finer-grained affective analysis. The paper automatically builds DepecheMood from crowd-sourced emotion annotations on Rappler news using distributional semantics and evaluates it in unsupervised recognition tasks. The authors report high coverage and precision with improvements over state-of-the-art unsupervised approaches, even using naïve classification and regression strategies.
Problem
Emotion analysis remains less developed than sentiment analysis because available emotion lexica are fewer and limited in coverage.
Method
The authors combine Rappler news articles and Mood Meter crowd votes with word-document representations, using matrix multiplication and normalization to derive word-level emotion scores.
Results
The resulting DepecheMood lexicon contains 37k entries and improves over state-of-the-art unsupervised approaches in emotion classification and regression, even with naïve strategies.
Takeaways & Limitations
Crowd-sourced social-media affective data can be harvested to build high-coverage, high-precision resources for emotion recognition from text.
Takeaways & Limitations
HAPPINESS receives at least three times more votes than other emotions, with possible causes including audience, dataset, and social-network mood biases.
Abstract
from arXiv · showhide
While many lexica annotated with words polarity are available for sentiment analysis, very few tackle the harder task of emotion analysis and are usually quite limited in coverage. In this paper, we present a novel approach for extracting - in a totally automated way - a high-coverage and high-precision lexicon of roughly 37 thousand terms annotated with emotion scores, called DepecheMood. Our approach exploits in an original way 'crowd-sourced' affective annotation implicitly provided by readers of news articles from rappler.com. By providing new state-of-the-art performances in unsupervised settings for regression and classification tasks, even using a naïve approach, our experiments show the beneficial impact of harvesting social media data for affective lexicon building.
1 Introduction
Emotion analysis extends sentiment analysis by distinguishing individual emotions that a positive-versus-negative division can miss. The paper addresses limited emotion-lexicon coverage by automatically building and evaluating a scored resource from crowd-sourced news annotations.
- Motivation: Individual emotions can distinguish negative reactions such as sadness and anger, which a simple positive-versus-negative model treats alike.The paper notes that anger may be more relevant than sadness for buzz monitoring in a product-failure example.
- Background: Lexical resources associate words with prior polarity and commonly serve as baselines or features in sentiment-analysis systems.Prior polarity describes whether a word out of context evokes something positive or negative.
- Applications: High-precision, high-coverage affective lexica support graded text modification, compositional scoring, and evocative copywriting.These applications use word-level scores to vary existing text, compose sentence-level affect, or select emotionally evocative names.
- Contribution: The paper automatically produces a high-coverage, high-precision emotion lexicon with numerical emotion scores using distributional semantics and crowd-sourced annotations from rappler.com.The resource is evaluated in unsupervised classification and regression settings for emotion recognition.
2 Related Work
Related resources provide sentiment or emotion information at different levels of coverage, precision, granularity, and annotation scale. Emotion lexica are fewer and generally smaller than sentiment lexica.
- Sentiment Lexica: Sentiment lexicon construction involves a trade-off between resource coverage and precision.Sentiment lexica commonly list words with prior polarities for sentiment-analysis systems.
- Sentiment Lexica: SentiWordNet assigns numerical positive and negative scores to WordNet senses, from which prior-polarity variants can be computed.Its entries use lemma#pos#sense-number, while prior-polarity forms can use lemma#PoS.
- Sentiment Lexica: ANEW prioritizes precision with manually assigned valence scores for 1k words, whereas crowdsourced Warriner ratings cover 14k frequent English words.The comparison illustrates different coverage and annotation trade-offs among sentiment resources.
- Emotion Lexica: Emotion lexica are less numerous and lower-coverage than sentiment lexica, with WordNetAffect providing 900 annotated synsets and 1.6k words.Other listed resources include AffectNet, Fuzzy Affect Lexicon, EmoLex, and Affect database, which contains emotion scores rather than only labels.
3 Dataset Collection
The authors collected news text and implicit affective annotations from Rappler’s Mood Meter to construct document-level emotion data. The corpus contains 13.5 million words across 25.3 thousand documents, but happiness is strongly overrepresented.
- Corpus: 13.5 M words across 25.3 K documents formed the Rappler corpus, averaging 530 words per document.Articles were harvested as of June 3rd 2013.
- Crowd Annotations: Rappler’s Mood Meter let readers vote on the emotion elicited by each story, producing percentages for eight affective dimensions.The collected votes were organized into a document-by-emotion matrix, MDE.
- Crowd Annotations: The document-by-emotion matrix records voting percentages rather than a single emotion label for each document.The paper contrasts this crowd-based multi-score setup with earlier author-provided single-label annotations.
- Dataset Bias: HAPPINESS received at least three times more votes than the other emotions on average.The paper considers audience culture, predominantly positive news, and positive self-presentation on social networks as possible explanations, without resolving the bias.
4 Emotion Lexicon Creation
The lexicon is derived by combining document-level emotion votes with word-document representations, then normalizing and scaling the resulting word-emotion scores. The resulting DepecheMood resource contains 37k entries.
- Representation: Words were lemmatized and part-of-speech tagged, then restricted to lemma#PoS entries also present in WordNet.This aligns the representation with resources such as SWN-prior and WordNetAffect.
- Representation: Raw frequencies, normalized frequencies, and tf-idf were tested as alternative word-document weights.The resulting matrices are denoted MW D,f, MW D,nf, and MW D,tfidf.
- Lexicon Construction: Matrix multiplication MDE · MW D sums word weights with document-level emotion weights to produce a raw word-by-emotion matrix, MW E.This composition merges words with emotions through their document associations.
- Normalization: Column-wise normalization reduces happiness overrepresentation, and row-wise scaling makes each word’s emotion weights sum to one.The resulting scores indicate how strongly each word contributes to the considered affective dimensions.
5 Experiments
Experiments evaluate DepecheMood on SemEval 2007 Affective Text headlines using label mapping and naïve regression and classification procedures. The resource achieves broad headline coverage and improves over competing systems across the reported emotion-recognition evaluations.
- Dataset: The SemEval 2007 Affective Text dataset contains 1,000 news headlines evaluated in regression and classification settings.The regression task supplies numerical emotion scores, while classification supplies binary emotion labels.
- Preprocessing: The evaluation maps Rappler Mood Meter dimensions to SemEval 2007 emotions before transforming headlines to lemma#PoS format.The mapping includes suboptimal cases.
- Coverage: Only one test headline contains exclusively words absent from DepecheMood, indicating high coverage on the evaluation data.Coverage statistics are reported per headline alongside other sentiment and emotion lexica.
- Regression: For regression, headline emotion scores are computed by averaging DepecheMood scores for all present lemma#PoS items.This naïve approach is compared using Pearson correlation with the best performance of other systems for each emotion.
- Regression: For all five aligned emotions, DepecheMood improves over the best-performing systems in Pearson correlation.DISGUST is excluded because it has no alignment with the resource’s labels.
- Classification: In classification, the naïve method improves over the best systems for three of five emotions, matches one, and fails to outperform one.Normalized frequencies provide the best results among the tested word-by-document matrix representations.
6 Conclusions
The paper presents DepecheMood as an automated emotion lexicon built from crowd-sourced affective annotations on a social news network. Its experiments report high coverage and precision and improvements over state-of-the-art unsupervised approaches, even with naïve regression and classification strategies.
- Contribution: DepecheMood is built automatically by harvesting crowd-sourced affective annotations from a social news network.The authors frame this as a way to use social-media information for emotion recognition from text.
- Findings: The experiments indicate high coverage and high precision for the lexicon.These findings are reported alongside improvements over state-of-the-art unsupervised approaches.
- Findings: The resource improves over state-of-the-art unsupervised approaches even with naïve classification and regression strategies.The conclusion identifies affective information from social media as useful for building emotion-recognition resources.
- Future work: Future work includes singular value decomposition and studying perceived mood effects on virality and reader engagement.The proposed analyses would use similar-word propagation and social-media interaction signals such as tweets, likes, reshares, and comments.