Source-linked AI summary
WASSA-2017 Shared Task on Emotion Intensity
Saif M. Mohammad, Felipe Bravo-Marquez
TL;DR
Emotion intensity in tweets is difficult to annotate and predict beyond binary emotion categories. The paper creates BWS-based intensity datasets and a shared-task benchmark, whose best system reached a Pearson correlation of 0.747. The resulting resources support finer-grained study of emotional language.
Problem
Most emotion annotation uses binary labels, while fine-grained rating scales can be inconsistent and affected by scale-region bias.
Method
The paper uses crowdsourced Best–Worst Scaling to create anger, fear, joy, and sadness intensity datasets and evaluates systems in a shared task.
Results
0.747 Pearson correlation was achieved by the best system among 22 participating teams on gold test-set annotations.
Takeaways & Limitations
The dataset and shared task improve understanding of how language conveys more or less intense emotions and add dimensionality to categorical emotion models.
Takeaways & Limitations
The task focuses on intensity felt by the tweet’s speaker; intensity evoked in readers or felt by mentioned entities is left for future work.
Abstract
from arXiv · showhide
We present the first shared task on detecting the intensity of emotion felt by the speaker of a tweet. We create the first datasets of tweets annotated for anger, fear, joy, and sadness intensities using a technique called best--worst scaling (BWS). We show that the annotations lead to reliable fine-grained intensity scores (rankings of tweets by intensity). The data was partitioned into training, development, and test sets for the competition. Twenty-two teams participated in the shared task, with the best system obtaining a Pearson correlation of 0.747 with the gold intensity scores. We summarize the machine learning setups, resources, and tools used by the participating teams, with a focus on the techniques and resources that are particularly useful for the task. The emotion intensity dataset and the shared task are helping improve our understanding of how we convey more or less intense emotions through language.
1 Introduction
The paper introduces WASSA-2017, the first shared task for predicting the intensity of emotion felt by a tweet’s speaker. It presents BWS-annotated datasets and analyzes participating systems, with the best system reaching a Pearson correlation of 0.747.
- Emotion intensity is the degree or amount of an emotion, distinct from arousal’s calming-to-exciting dimension.
- Automatically detecting speaker emotion intensity has applications in commerce, public health, intelligence gathering, and social welfare.
- WASSA-2017 introduced a shared task for predicting a real-valued 0–1 intensity score for an emotion felt by a tweet’s speaker.
- BWS and crowdsourcing produced training, development, and test datasets for anger, fear, joy, and sadness intensity.
- 22 teams participated, and Prayas achieved a Pearson correlation of 0.747 with gold annotations.
- Top systems commonly combined neural models, ensembles, dense representations, and affect-lexicon features, especially larger lexicon sets.
- The dataset and shared task support finer-grained understanding of emotional expression and add dimensionality to traditionally categorical emotion models.
2 Related Work
Prior emotion annotation largely used binary labels, while direct fine-grained ratings suffer from inconsistency and scale-region bias. The paper motivates BWS as a more reliable way to obtain real-valued intensity rankings for whole tweets.
- Most emotion annotation work assigns discrete binary labels such as joy–nojoy and fear–nofear.
- Direct fine-grained rating scales are difficult for annotators because scores vary across annotators, over time, and by preferred scale region.
- In BWS, annotators select the highest- and lowest-property items from an n-tuple, commonly a 4-tuple.
- Each best–worst judgment on a 4-tuple reveals the ordering of five of its six item pairs.
- BWS annotations can be converted into real-valued association scores, with reliable scores obtainable from 2N 4-tuples for N items.
- Empirical work finds BWS more reliable than rating scales for fine-grained scores, and this paper extends it from words to whole tweets.
3 Data
The dataset constructs emotion-intensity tweets and obtains fine-grained scores through balanced Best–Worst Scaling annotations. Reliability analyses support the reproducibility of the resulting rankings, especially for fear, joy, and sadness.
- Dataset overview: The Tweet Emotion Intensity Dataset covers anger, fear, joy, and sadness and summarizes the dataset-construction approach.The source summary explicitly excludes separate analyses of emotion-pair similarities, cross-emotion training transfer, and hashtag effects.
- Tweet collection: 50–100 emotion-associated query terms at different intensity levels were used to collect tweets through the Twitter API.Terms included examples such as angry, frustrated, annoyed, devastated, and heartbroken.
- Tweet collection: 7,097 tweets formed the master set, including hashtag-query tweets, copies with query hashtags removed, and other query-term tweets.The dataset included 1,030 HQT tweets, 1,030 paired NQT tweets, and 5,037 QT tweets.
- Best–Worst Scaling: Best–Worst Scaling presented four tweets at a time and asked annotators to select the highest- and lowest-intensity speakers.Each item appeared in eight 4-tuples, no pair appeared in more than one tuple, and every tuple received three independent annotations.
- Scoring: Intensity scores equal the percentage selected as most intense minus the percentage selected as least intense, then linearly transformed to 0–1.The original BWS scores range from −100 to 100 before transformation.
- Reliability of annotations: 0.84–0.88 Pearson and Spearman split-half correlations indicate high reproducibility for fear, joy, and sadness; anger was slightly lower.Because reliability was computed using half the annotations, these values are lower bounds for the three-annotation results.
4 Task Setup
The shared task asked systems to predict real-valued emotion intensity scores and ranked submissions by average Pearson correlation across four emotions. The competition provided standardized data partitions, evaluation tools, and flexible resource usage.
- The Task: Systems predicted a real-valued intensity score from 0 to 1 for each tweet and emotion, where 0 and 1 represent the lowest and highest amounts.Submissions were uploaded through a CodaLab competition website with leaderboard reporting.
- Data partitioning: The dataset used about 50% training, 5% development, and 45% test tweets for each emotion.Paired No-Query-Term and Hashtag-Query-Term tweets remained in the same partition.
- Competition schedule: Participants received training and development data more than two months before the two-week official evaluation period.Development gold scores were released two weeks before evaluation, and combined training-development data could be used for final modeling.
- Resources: Participants could use manually created or automatically generated affect lexicons, software packages, and a provided baseline Weka system.The rules allowed systems to be built from scratch or from available resources consistent with fair competition.
- Evaluation: The bottom-line metric was the average Pearson correlation across anger, fear, joy, and sadness predictions against gold ratings.Additional metrics included Spearman correlation and correlations restricted to test instances with gold intensity scores ≥0.5.
- Evaluation: Spearman results were largely in line with Pearson results, so the paper reports Pearson while the official leaderboard shows both.The official evaluation script calculated both correlations and checked submission format.
5 Baseline System for Automatically Determining Tweet Emotion Intensity
The baseline evaluates tweet emotion-intensity regression using textual, embedding, and affect-lexicon features, then compares individual and combined feature sets with Pearson correlation.
- The baseline uses AffectiveTweets to calculate feature vectors and trains Weka regression models on emotion-intensity-labeled tweets.
- Features include word and character n-grams, averaged Word2Vec word embeddings trained on ten million English tweets, and affect-lexicon information.The word embeddings use 400 dimensions and a window size of 5.
- 0.55 average Pearson correlation: word embeddings significantly outperform word or character n-grams, whose results are around 0.48.Random intensity guesses are expected to produce correlations close to 0.
- 0.66 average Pearson correlation: combining word embeddings with lexicon features gives the best overall feature-combination result.Combining all lexicons reaches 0.63, and the increase from 0.63 to 0.66 is statistically significant.
- NRC-Hash-Emo is the most predictive single lexicon, while lexicons with Twitter-specific entries or fine-grained intensity scores tend to perform better.Correlations are around 0.65 for anger, fear, and joy, and 0.71 for sadness.
- Correlations are generally lower on tweets with gold intensity scores ≥0.5 because systems cannot rely on coarse low-versus-high distinctions.The restricted 0.5-to-1 range is described as a harder task than the full intensity range.
6 Official System Submissions to the Shared Task
The shared task attracted 22 teams using supervised regression systems with diverse neural, ensemble, embedding, and affect-lexicon features. Prayas ranked first overall, while performance varied across emotions and test subsets.
- 6.1 Results: 22 teams submitted systems, and Prayas achieved the highest full-test score with r avg. = 0.747.The competition metric averages Pearson correlations across the four emotions.
- 6.1 Results: Seven teams exceeded the publicly available SVM-based benchmark, which scored r avg. = 0.66.
- 6.1 Results: On instances with gold intensity scores ≥0.5, Prayas remained first with r avg. = 0.571, although scores were lower overall than on the full test set.
- 6.2 Machine Learning Setups: Systems mapped tweets to feature vectors and trained regression models using training data plus external resources such as large tweet corpora and affect lexicons.
- 6.2 Machine Learning Setups: Common representations included word and sentence embeddings, while regression methods included SVR, neural networks, random forests, gradient boosting, AdaBoost, and least-squares regression.
- 6.3 Prayas: Rank 1: Prayas combined three models, including an embedding-plus-lexicon feed-forward network, a four-task network with shared early layers, and an LSTM-CNN architecture.
- 6.3 Prayas: Rank 1: Top systems commonly used ensembles and substantially more affect lexicons than lower-performing systems.
7 Conclusions
The paper introduced the first shared task and dataset for tweet-based emotion-intensity detection, using best–worst scaling and crowdsourcing. Results showed strong participation and a best Pearson correlation of 0.747, while follow-on datasets broadened the scope of affect annotation.
- Contributions: The paper conducted the first shared task on detecting the intensity of emotion felt by a tweet speaker.
- Contributions: Best–worst scaling and crowdsourcing were used to create the emotion-intensity dataset, and affect lexicons helped determine intensity.
- Results: 22 teams participated, with the best system reaching Pearson correlation 0.747 against the gold annotations.
- Future Work: Ongoing work extends emotion-intensity datasets beyond anger, fear, sadness, and joy and adds valence, arousal, and dominance annotations in three languages.
8 Appendix
The appendix includes the best–worst scaling questionnaire used to obtain fear annotations.
- Questionnaire: The appendix presents the BWS questionnaire used for obtaining fear annotations.
Degree Of Fear In English Language Tweets
The fear-intensity task asks annotators to infer how fearful a tweet’s speaker is. Because numerical scoring is difficult, annotators compare four tweets by selecting the most and least fearful speakers.
- Task Definition: Fear intensity ranges from zero fear to extreme fear and is inferred from what a person says.
- Annotation Procedure: Annotators compare four tweets and identify which speaker is likely to be the MOST fearful.
- Annotation Procedure: Annotators also identify which of the four speakers is likely to be the LEAST fearful.
Important Notes
The fear annotation task focuses on the speaker’s fear and uses instinctive judgments when selecting among speakers or resolving ties.
- The task measures the fear level of the speaker, not fear attributed to someone else mentioned in the tweet.
- When multiple speakers seem equally fearful, respondents may select any one of them.
- Respondents are instructed not to over-think their answers and to rely on instinct.
EXAMPLE
The paper illustrates fear-intensity annotation with a four-speaker comparison and describes post-evaluation visualization and Weka-based feature-processing tools for exploring and modeling the dataset.
- EXAMPLE: Speaker 3 is labeled the most fearful because the speaker reports being terrified about medical-test results.
- EXAMPLE: Speaker 1 is labeled the least fearful in the example, where the utterance expresses anger rather than fear.
- Annotation survey: 4.2/5 clarity and 3.5/5 ease-of-task scores indicate clear instructions but a task requiring non-trivial thought.
- Visualization: The interactive visualization was released after evaluation, so shared-task participants could not use it while developing systems.
- Visualization: The visualization links partition tables, intensity-score histograms, and tweet details through interconnected filtering.
- AffectiveTweets: AffectiveTweets provides Weka filters for sparse linguistic features, affective lexicons, SentiStrength scores, and aggregated pre-trained embeddings.
- AffectiveTweets: The sparse-feature filter adds a NEG prefix within heuristic negation scopes defined by 28 negator words ending at punctuation or sentence boundaries.
- AffectiveTweets: After feature creation, users can apply Weka regression or classification algorithms, with additional filters under development.