PaperlayerView exact revision on arXiv

Source-linked AI summary

Word Affect Intensities

Saif M. Mohammad

arXiv:1704.08798v2cs.CL

TL;DR

Existing manually created emotion lexicons generally provide only coarse associations, while automatic lexicons offer fine-grained scores with lower accuracy. This paper constructs the NRC Affect Intensity Lexicon with Best–Worst Scaling and finds highly reliable repeat annotations, while identifying applications and scope boundaries for the resource.

  • Problem

    Existing manual lexicons for basic emotions provide coarse association categories, while fine-grained human intensity annotation is difficult because annotator judgments can be inconsistent.

  • Method

    The paper uses Best–Worst Scaling to create real-valued affect-intensity scores for anger, fear, joy, and sadness across close to 6,000 English words.

  • Results

    Repeat annotations yielded Spearman rank correlations of 0.92 and Pearson correlation of 0.91 with the original scores, indicating reliable and discriminative intensity estimates.

  • Takeaways & Limitations

    AIL supports automatic emotion analysis, affect-composition research, natural-language generation, and evaluation of automatic word–emotion lexicons.

  • Takeaways & Limitations

    BWS annotation quality cannot be assessed with standard inter-annotator agreement because disagreement between similarly intense items is a useful signal.

Abstract

from arXiv · show

Words often convey affect -- emotions, feelings, and attitudes. Further, different words can convey affect to various degrees (intensities). However, existing manually created lexicons for basic emotions (such as anger and fear) indicate only coarse categories of affect association (for example, associated with anger or not associated with anger). Automatic lexicons of affect provide fine degrees of association, but they tend not to be accurate as human-created lexicons. Here, for the first time, we present a manually created affect intensity lexicon with real-valued scores of intensity for four basic emotions: anger, fear, joy, and sadness. (We will subsequently add entries for more emotions such as disgust, anticipation, trust, and surprise.) We refer to this dataset as the NRC Affect Intensity Lexicon, or AIL for short. AIL has entries for close to 6,000 English words. We used a technique called best-worst scaling (BWS) to create the lexicon. BWS improves annotation consistency and obtains reliable fine-grained scores (split-half reliability > 0.91). We also compare the entries in AIL with the entries in the NRC VAD Lexicon, which has valence, arousal, and dominance (VAD) scores for 20K English words. We find that anger, fear, and sadness words, on average, have very similar VAD scores. However, sadness words tend to have slightly lower dominance scores than fear and anger words. The Affect Intensity Lexicon has applications in automatic emotion analysis in a number of domains such as commerce, education, intelligence, and public health. AIL is also useful in the building of natural language generation systems.

1. Introduction

The paper introduces a manually created lexicon that assigns fine-grained affect-intensity scores to nearly 6,000 English words across four emotions. It uses Best–Worst Scaling to improve annotation consistency and reports highly reliable repeat annotations.

  • Motivation: Existing manual emotion lexicons are largely categorical, while words can convey different intensities of the same emotion.Fine-grained intensity annotation is difficult because responses can vary across and within annotators.
  • Method: Best–Worst Scaling uses comparative judgments to derive real-valued association scores and rank items by affect intensity.Annotators identify the highest- and lowest-intensity items in tuples; the resulting comparisons support score calculation.
  • Contribution: AIL provides scores from 0 to 1 for anger, fear, joy, and sadness across close to 6,000 English words.A score of 1 represents the highest intensity and 0 the lowest intensity for a given emotion.
  • Reliability: Repeat annotations produced Spearman rank correlations of 0.92 and Pearson correlation of 0.91 with the original scores.These results indicate that the BWS scores are both discriminative and reliable.
  • Applications: AIL entries are compared with the NRC VAD Lexicon, and the lexicon is positioned for emotion analysis and natural-language generation applications.The paper also makes AIL freely available for non-commercial research.

2. Related Work

Prior work includes sentiment, categorical-emotion, and automatic word–emotion lexicons, but fine-grained annotation remains difficult. The paper situates BWS as a less familiar comparative-annotation approach for constructing such resources.

  • Emotion scope: The project focuses on anger, fear, joy, and sadness, four emotions commonly included among proposed basic emotions.Psychologists disagree about which emotions and how many should count as basic.
  • Existing lexicons: Sentiment lexicons are widespread, whereas manually created categorical-emotion lexicons are comparatively limited.Examples include WordNet Affect and the crowdsourced NRC Emotion Lexicon.
  • Annotation challenges: Prior systems commonly use coarse categorical annotations because humans struggle to provide consistent fine-grained scores.Annotators may differ in scale use and may assign different scores to the same word at different times.
  • Best–Worst Scaling: BWS has been used in several NLP datasets and for valence, arousal, and dominance annotation, but remains relatively unfamiliar outside choice modeling and marketing research.The paper applies BWS to affect-intensity lexicon construction.
  • Automatic methods: Automatic methods assign real-valued word–emotion association scores, but AIL can be used to evaluate their quality and examine their correlation with intensity.The supplied passage frames AIL as a reference resource for automatic lexicons.

3. NRC Affect Intensity Lexicon

The NRC Affect Intensity Lexicon was constructed by selecting emotion-associated terms and collecting comparative best–worst judgments for four emotions. Scores were aggregated from these judgments into affect-intensity values, with quality controls and distributions examined afterward.

  • 3.1. Term Selection: Terms were selected from the NRC Emotion Lexicon and from words co-occurring with emotion-word hashtags in tweets.The tweet-derived terms included commonly used English and social-media language, including deliberate spelling variations.
  • 3.2. Annotating for Affect Intensity with Best–Worst Scaling: For each emotion, annotators chose the highest- and lowest-intensity word from randomly generated 4-tuples.There were 2 × N distinct 4-tuples, each word appeared in eight tuples, and no two tuples shared more than two items.
  • 3.2. Annotating for Affect Intensity with Best–Worst Scaling: Crowdsourcing produced annotations from native English speakers in the United States, with gold questions used to filter unreliable annotators.Annotators falling below 70% accuracy on gold questions were refused further annotation and had their responses discarded.
  • 3.2. Annotating for Affect Intensity with Best–Worst Scaling: 47,135 best–worst response pairs were obtained, with a median of four annotations per item.Each word was involved in 8 × 4 = 32 best–worst judgments under the specified task design.
  • 3.3. Annotation Aggregation: Scores were computed as most-intensity selection proportions minus least-intensity selection proportions, then linearly transformed from -1–1 to 0–1.The resulting 0–1 scale represents least to most emotion intensity.
  • 3.4. Distribution of Scores: Anger-intensity scores were displayed in 0.05-wide bins, and the score distribution was approximately normal, with similar shapes for other emotions.Figure 1 uses colors from gray to orange to indicate increasing affect intensity.

4. Relationships of the Basic Emotions with Valence, Arousal, and Dominance

The paper relates basic-emotion word intensities to valence, arousal, and dominance scores. Joy words are more positive, while negative-emotion words span broad and overlapping VAD ranges.

  • Joy words have much higher average valence than anger, fear, and sadness words.
  • Anger and fear words have very similar average VAD profiles.
  • Sadness words have the lowest average valence, followed by fear and anger.
  • Arousal increases from joy to sadness to anger and fear, with joy words most passive on average.
  • Negative-emotion words occupy wide, markedly overlapping ranges of dominance and arousal that also overlap with joy words.
  • The three negative emotions span wide ranges of valence, arousal, and dominance, so valence alone cannot determine the precise basic emotion.

5. Reliability of the Annotations

The paper evaluates BWS annotation quality through reproducibility rather than standard inter-annotator agreement. This is because disagreement on closely matched items can be a useful signal in BWS.

  • Standard inter-annotator agreement is unsuitable for BWS because disagreement on close-intensity items can produce the desired similar scores.
  • Reproducibility across independent annotation splits is used to assess whether the resulting intensity scores are consistent.

6. Applications and Future Work

The Affect Intensity Lexicon supports automatic emotion analysis across multiple domains and provides resources for literary analysis and evaluating automatic intensity methods.

  • AIL supports automatic emotion analysis in commerce, education, intelligence, and public health.
  • AIL has been used in shared tasks on emotion intensity in tweets, including by teams that placed first.
  • The lexicon can identify high-intensity words in digital humanities and literary analysis.
  • AIL provides gold intensity scores for evaluating automatic methods of determining word affect intensity.

7. Conclusions

The paper introduces a crowdsourced lexicon of fine-grained word-affect intensities for four emotions, created with best-worst scaling. Repeat annotations show that the resulting scores are both discriminative and reliable, while the lexicon is intended for broader affect analysis and future expansion.

  • The NRC Affect Intensity Lexicon captures real-valued word-affect intensities for anger, fear, joy, and sadness.
  • Best-worst scaling produces fine-grained scores and addresses annotation-consistency issues associated with traditional rating scales.
  • 0.92 split-half reliability and 0.91 Pearson correlation show close agreement between repeat and original annotations.
  • The lexicon supports automatic emotion analysis and understanding how constituent-word affect impacts sentence affect.
  • The authors plan to add entries for disgust, trust, surprise, and anticipation, and make the lexicon freely available.

9. Appendix

Figure 5 presents scatter plots for the four basic emotions in valence-dominance space.

  • Figure 5 shows scatter plots for anger, fear, joy, and sadness in valence-dominance space.
Loading 1704.08798v2…