Source-linked AI summary
EmoTxt: A Toolkit for Emotion Recognition from Text
Fabio Calefato, Filippo Lanubile, Nicole Novielli
TL;DR
Available sentiment tools classify polarity rather than specific emotions, motivating a toolkit for emotion recognition from text. EmoTxt combines text features with supervised classifiers and supports custom training from annotated data; the authors report comparable performance across datasets.
Problem
Freely available sentiment tools classify positive, negative, or neutral polarity, but the paper identifies no such tool supporting recognition of specific emotions.
Source: §1 · p. 1Method
EmoTxt combines n-gram and lexical features with supervised SVM classifiers, and lets users train custom emotion classifiers from manually labeled texts.
Source: §2 · p. 1, §3.2 · p. 2Results
The authors report comparable EmoTxt performance across different datasets.
Source: §4 · p. 2Takeaways & Limitations
The toolkit releases emotion-detection models and supports custom classifiers trained on manually annotated data.
Source: §4 · p. 2The authors plan validation on gold standards from additional sources to further assess the approach’s generality and robustness.
Source: §4 · p. 2
Abstract
from arXiv · showhide
We present EmoTxt, a toolkit for emotion recognition from text, trained and tested on a gold standard of about 9K question, answers, and comments from online interactions. We provide empirical evidence of the performance of EmoTxt. To the best of our knowledge, EmoTxt is the first open-source toolkit supporting both emotion recognition from text and training of custom emotion classification models.
1. Introduction
Existing sentiment tools classify positive, negative, or neutral polarity but do not recognize specific emotions; EmoTxt is presented as an open-source toolkit for emotion recognition and custom classifier training.
1. Introduction
- Existing freely available sentiment tools classify positive, negative, or neutral polarity, but none supports recognizing specific emotions such as joy, love, and anger.The paper identifies sentiment analysis as relevant to business, social well-being, politics, security, and software engineering.Source: §1 · p. 1
- EmoTxt is presented as the first open-source toolkit for recognizing emotions from text.Source: §1 · p. 1
- Researchers can use EmoTxt to detect emotions in text or train a custom emotion classifier from manually annotated data.Source: §1 · p. 1
2. System Description
EmoTxt provides released emotion classifiers and supports training custom classifiers from hand-labeled texts. Its system accepts text collections in CSV format and combines n-gram and lexical features with preprocessing choices described below.
2. System Description
- EmoTxt accepts a CSV collection with one identified text per line and returns a CSV containing each text ID and its predicted label.The toolkit is implemented in Java, distributed under the MIT license, and releases models trained on the authors’ gold-standard datasets.Source: §2 · p. 1
- Users can train a new classifier from hand-labeled texts, with the training approach independent of the classification framework.The approach learns presence or absence of each given emotion from the provided labels.Source: §2 · p. 1
- The feature set combines tf-idf-weighted unigrams and bigrams with lexical cues for emotion, politeness, sentiment polarity, and uncertainty.Emotion features use WordNet Affect associations and category-specific tf-idf; politeness is measured with Danescu et al., sentiment scores with SentiStrength, and uncertainty with De Smedt et al..Source: §2 · p. 1
- Texts are tokenized with Stanford NLP, while HTML tags, code fragments, and URLs are removed before feature extraction.Source: §2 · p. 1
- EmoTxt does not stem or lemmatize text because inflected forms may convey sentiment information.This differs from Ortu et al..Source: §2 · p. 1
3. Evaluation
The evaluation uses gold-standard emotion annotations from Stack Overflow and Jira, partitions each dataset for training and held-out testing, and tunes six binary SVM classifiers on the training data.
3.1. Dataset
- The emotion labels follow Shaver et al.’s tree-structured discrete framework, whose six top-level categories are love, joy, anger, sadness, fear, and surprise.Each successive level refines the emotion granularity.Source: §3.1 · p. 1
- The Stack Overflow gold standard contains 4,800 questions, answers, and comments annotated for presence or absence of six basic emotions by three raters per post.Twelve raters contributed; disagreements were resolved by majority vote, and observed agreement ranged from .86 for joy to .98 for surprise.Source: §3.1 · p. 1
- The second evaluation dataset is Ortu et al.’s gold standard of 4,000 Jira comments labeled for love, joy, anger, and sadness.Source: §3.1 · p. 2
- Both gold sets are split into stratified 70% training and 30% test partitions, with final models evaluated on unseen held-out data.Source: §3.1 · p. 2
3.2. Model Training and Classification Performance
- The evaluation trains six binary linear SVM classifiers and selects C by accuracy in 10-fold cross-validation within the training partition before testing on held-out data.The parameter search uses C values from 0.01 to 8; test-set performance is reported in Table II, but its values are not supplied here.Source: §3.2 · p. 2, §3.1 · p. 2, Table II · p. 2
4. Conclusions
EmoTxt is an open-source toolkit trained and tested on gold-standard Stack Overflow and Jira data. The authors report comparable performance across datasets and plan validation on additional sources.
4. Conclusions
- The authors report comparable EmoTxt performance across different datasets.Source: §4 · p. 2
- EmoTxt releases classification models for emotion detection and supports training classifiers from manually annotated data.Source: §4 · p. 2
- The training features are independent of the theoretical model used to label the data.Source: §4 · p. 2
- Future work is planned to validate EmoTxt on gold standards from other sources to further assess generality and robustness.Source: §4 · p. 2