Source-linked AI summary
Emotion Intensities in Tweets
Saif M. Mohammad, Felipe Bravo-Marquez
TL;DR
Emotion-intensity detection lacked suitable annotated data because most prior work used categorical labels and direct fine-grained ratings were difficult to keep consistent. The paper builds four tweet datasets with BWS annotations, analyzes hashtags and predictive features, and studies relations between emotions. It reports that hashtags often convey more intense emotion, while combined word-embedding and lexicon features achieve the best regression result, with average r = 0.66.
Problem
Most emotion detection work used categorical labels, while suitable annotated data for reliable fine-grained intensity detection was lacking.
Method
The paper creates four emotion-intensity tweet datasets using BWS and crowdsourcing, then evaluates hashtag effects, regression features, and cross-emotion predictiveness.
Results
Hashtags often convey more intense emotion, and combined word-embedding and lexicon features obtain the best overall regression result, with average r = 0.66.
Takeaways & Limitations
BWS yields fine-grained emotion-intensity datasets, while affect lexicons and word embeddings are useful for determining intensity.
Takeaways & Limitations
The study targets emotion felt by the tweet’s speaker; reader-evoked and mentioned-entity emotion are left for future work.
Abstract
from arXiv · showhide
This paper examines the task of detecting intensity of emotion from text. We create the first datasets of tweets annotated for anger, fear, joy, and sadness intensities. We use a technique called best--worst scaling (BWS) that improves annotation consistency and obtains reliable fine-grained scores. We show that emotion-word hashtags often impact emotion intensity, usually conveying a more intense emotion. Finally, we create a benchmark regression system and conduct experiments to determine: which features are useful for detecting emotion intensity, and, the extent to which two emotions are similar in terms of how they manifest in language.
1 Introduction
The paper addresses emotion intensity detection in tweets, a gap left by predominantly categorical emotion datasets. It introduces BWS-based datasets and regression experiments for four emotions.
- Motivation: Emotion intensity captures the degree of anger, sadness, or another emotion conveyed by language.Applications can use intensity to prioritize significant frustration or anger over minor inconvenience.
- Motivation: Intensity annotation is difficult because respondents face greater cognitive load and consistency is hard to maintain.The challenge applies both across annotators and within one annotator’s repeated responses.
- Approach: Best–Worst Scaling asks annotators to identify the highest- and lowest-intensity items in small tuples, producing comparative information efficiently.For 4-tuples, each best and worst choice reveals the order of five of six item pairs.
- Contributions: The paper creates the first tweet datasets annotated for anger, joy, sadness, and fear intensity.Intensity is represented as a real-valued score between 0 and 1, with 1 indicating the highest amount and 0 the lowest.
- Contributions: The contributions include hashtag-impact analysis, the open-source AffectiveTweets regression system, and cross-emotion feature-predictiveness experiments.The paper also provides data for the WASSA-2017 shared task, in which twenty-two teams participated.
- Scope: The study limits its target to emotion felt by the tweet’s speaker, leaving reader-evoked and mentioned-entity emotion for future work.
2 Related Work
Prior work largely treated emotion detection as categorical classification and offered limited regression data. BWS provided a more reliable fine-grained alternative, but had mainly been applied to words.
- Emotion annotation: Most emotion annotation work assigns discrete binary labels such as joy–nojoy or fear–nofear.
- Emotion annotation: Only limited prior work provided degree-of-emotion scores, including a SemEval-2007 effort using newspaper headlines and direct 0-to-100 ratings.Direct fine-grained ratings can produce disagreement, temporal inconsistency, and scale-region bias.
- Best–Worst Scaling: BWS had been shown to produce more reliable fine-grained scores than rating scales, but NLP applications had focused mainly on words.Its use for larger textual units such as sentences remained unclear.
- Emotion regression: Automatic emotion classification has been studied for tweets, whereas emotion regression had little prior work beyond three SemEval submissions.
3 Data
The paper constructs emotion-rich tweet datasets by querying emotion-associated terms, annotating four-tweet tuples with BWS, and converting comparisons into normalized intensity scores.
- Tweet selection: For each emotion, the authors select 50 to 100 query terms associated with different intensity levels.Terms include examples such as angry, frustrated, annoyed, and peeved for anger.
- Tweet selection: Tweets are collected through emotion query terms, with retweets and URLs discarded and sampling capped per term and tweeter–term combination.These caps reduce skew toward particular tweeters or query terms.
- Tweet selection: The master set includes hashtag-query tweets, hashtag-removed copies, and other query-term tweets to study hashtag effects.The master set contains 7,097 tweets, including 1,030 hashtag-query tweets and 1,030 matched hashtag-removed copies.
- BWS annotation: Each emotion set is annotated with 2 × N randomly generated 4-tuples using random maximum-diversity selection.Each item appears in eight tuples, while no item pair appears in more than one tuple.
- BWS annotation: BWS scores equal the percentage chosen most intense minus the percentage chosen least intense, then are transformed from −1 to 1 into 0-to-1 scores.
- Dataset splits: The Tweet Emotion Intensity Dataset is partitioned into training, development, and test sets at approximately 50%, 5%, and 45%.Hashtag-query tweets and their hashtag-removed copies remain in the same partition.
4 Reliability of Annotations
The study evaluates BWS annotation quality through reproducibility rather than standard agreement, finding generally high reliability across emotion datasets, with anger somewhat harder to assess.
- Annotation quality: BWS disagreement can be informative when tuple items have similar intensities, so standard inter-annotator agreement is unsuitable.The authors instead assess whether independent annotations produce similar rankings and scores.
- Reliability measure: 100 repeated random splits estimate split-half reliability from the three annotations assigned to each 4-tuple.Each split produces two intensity-score and ranking sets whose correlations are averaged.
- Results: 0.84–0.88 Pearson and Spearman correlations indicate high reproducibility for fear, joy, and sadness.These values are reported for both correlation measures across the three datasets.
- Interpretation: SHR is expected to be lower here than prior word-level sentiment results because this study uses three annotations and whole-sentence emotion judgments.Prior work reported SHR of 0.98 using eight annotations per tuple for words.
- Results: Anger has slightly lower correlations, indicating that speaker anger degrees are relatively more difficult to determine from tweets.The reported reliability values are lower-bound estimates because they use half the available annotations.
5 Impact of Emotion Word Hashtags on Emotion Intensity
The paper compares tweets with and without emotion-word hashtags to measure their effect on perceived emotion intensity. Hashtags usually increase intensity, but their effect depends on the surrounding tweet and emotion.
- Study design: Emotion-word hashtags had no systematically established intensity effect before this study, motivating matched hashtag-present and hashtag-removed tweet pairs.The dataset includes HQT–NQT pairs specifically for this investigation.
- Overall effect: Removing an emotion-word hashtag lowered intensity in a majority of matched cases, although some cases rose or showed no change.The scatter plot compares HQT intensity on the x-axis with NQT intensity on the y-axis; positions relative to the diagonal encode these outcomes.
- Overall effect: 0.58 versus 0.47 was a statistically significant difference between average HQT and NQT scores.The significance was tested with a Wilcoxon signed-rank test at the 0.05 level.
- Magnitude: 0.17 average drops exceeded 0.08 average rises when hashtag removal changed intensity.The drop and rise correspond to 17% and 8% of the total 0–1 intensity range, respectively.
- Context dependence: Hashtag effects depend on context: underspecified tweets show marked increases, while already explicit emotion often yields only small changes.The authors also report emotion-specific patterns, such as lower fear scores after outward-judgment hashtags.
6 Automatically Determining Tweet Emotion Intensity
The paper benchmarks regression models for tweet emotion intensity and examines feature usefulness, moderate-to-high intensity prediction, and cross-emotion transfer. Affect lexicons, especially NRC-Hash-Emo, are strong predictors, while emotion similarities vary and can be asymmetric.
- 6.1 Supervised Regression and Ablation: AffectiveTweets extracts n-grams, word embeddings, and affect-lexicon features for Weka regression models evaluated against human intensity scores using Pearson correlation.Models are trained on official training data, tuned on development data, and evaluated on test sets.
- 6.1 Supervised Regression and Ablation: 0.66 average Pearson correlation is the best overall result, obtained by combining word-embedding and affect-lexicon features.This improves significantly over the 0.63 average from combining all lexicons.
- 6.1 Supervised Regression and Ablation: NRC-Hash-Emo is the most predictive single lexicon, while Twitter-specific entries, intensity scores, and emotion labels tend to improve prediction.NRC-Aff-Int’s fine-grained word–emotion association scores outperform NRC-EmoLex; sadness reaches 0.71 correlation, versus about 0.65 for anger, fear, and joy.
- 6.1.1 Moderate-to-High Intensity Prediction: In the ≥0.5 intensity subset, performance is generally lower, but lexicons remain crucial and embeddings and word n-grams become more competitive.NRC-Hash-Emo remains the source of the most predictive lexicon features, while SentiStrength appears less useful.
- 6.2 Similarity of Emotion Pairs: Cross-emotion transfer correlations range from r = 0.23 to r = 0.65, with negative emotions positively correlated and joy negatively correlated with them.Fear-trained models predict sadness at r = 0.63, close to the r = 0.65 same-emotion upper bound, and correlations are asymmetric.
- 6.2 Similarity of Emotion Pairs: Combining fear and sadness training data yields a sadness correlation of 0.67, exceeding the score obtained using sadness training data alone.The authors identify domain adaptation as a possible avenue for further gains.
7 Conclusions
The paper introduces a tweet emotion-intensity dataset and shows how best–worst scaling, hashtags, affect lexicons, and cross-emotion correlations support fine-grained intensity detection.
- 7 Conclusions: The paper creates the first emotion-intensity dataset for tweets.It covers anger, fear, joy, and sadness intensities.
- 7 Conclusions: Best–worst scaling improves annotation consistency and produces fine-grained emotion-intensity scores.The paper applies BWS to tweets and hashtag-removed tweet versions.
- 7 Conclusions: Emotion-word hashtags often affect intensity, usually conveying a more intense emotion.The analysis compares original tweets with hashtag-removed versions.
- 7 Conclusions: Affect lexicons, especially those with fine word–emotion association scores, are useful for determining tweet emotion intensity.The paper evaluates these features in a benchmark regression system.
- 7 Conclusions: Emotion pairs show asymmetric correlations in how their intensities manifest in language.The paper gives fear as strongly indicative of sadness, while sadness is only moderately indicative of fear.
A.1 Best–Worst Scaling Questionnaire used to Obtain Emotion Intensity Scores
The questionnaire asks annotators to compare four tweets when assigning fear intensity.
- A.1 Best–Worst Scaling Questionnaire used to Obtain Emotion Intensity Scores: The questionnaire presents four tweets and asks which speaker is most fearful and which is least fearful.These best–worst judgments are used to obtain fear-intensity scores.
Degree Of Fear In English Language Tweets
The task asks annotators to judge the degree of fear expressed or felt by speakers in English-language tweets.
- Degree Of Fear In English Language Tweets: Fear intensity ranges from no fear to extreme fear, and annotators judge the degree of fear from tweet content.The questionnaire frames the task around the speaker’s likely fear.
- Degree Of Fear In English Language Tweets: Annotators identify which of four speakers is likely to be the MOST fearful.
- Degree Of Fear In English Language Tweets: Annotators identify which of four speakers is likely to be the LEAST fearful.
Important Notes
The fear-intensity task asks annotators to compare speakers based on the speaker’s own fear, using intuitive judgments without overthinking.
- Annotators judge the fear level of the speaker, not the fear experienced by someone else mentioned in the text.
- When multiple speakers appear equally fearful, annotators may select any one of them.
- Instructions encourage respondents to rely on instinct rather than overanalyzing each comparison.
EXAMPLE
The paper illustrates fear-intensity annotation with multi-speaker comparisons and describes supporting tools, feature processing, and dataset exploration. It also notes that the package’s additional lexicon and distant-supervision filters were still under development.
- EXAMPLE: In the example, Speaker 3 is labeled most fearful because the tweet includes the hashtag #terrified, while Speaker 1 is labeled least fearful.
- EXAMPLE: Respondents rated questionnaire instructions 4.2/5 for clarity and the task 3.5/5 for ease, indicating that the task required non-trivial thought.
- EXAMPLE: Emotion-word hashtags were used to create manually annotated emotion data, although query terms were often removed in related classification datasets to erase obvious cues.
- EXAMPLE: AffectiveTweets provides filters for lexicon, input-lexicon, SentiStrength, and embedding-based tweet feature vectors.
- EXAMPLE: Additional filters for creating affective lexicons from tweets and supporting distant supervision were still under development.
- EXAMPLE: The interactive visualization displays emotion partitions and lets users filter components by emotion or train, development, and test split.