PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep learning for affective computing: text-based emotion recognition in decision support

Bernhard Kratzwald, Suzana Ilic, Mathias Kraus, Stefan Feuerriegel, Helmut Prendinger

arXiv:1803.06397v6cs.CL

TL;DR

Text-based emotion recognition is difficult, while affective computing aims to support decisions by inferring emotional states. The paper customizes recurrent neural networks and introduces sent2affect, finding consistent gains over traditional machine learning across benchmark datasets.

  • Problem

    Text-based emotion recognition is challenging because computational routines must handle complex emotional language, while benchmark datasets remain scarce and heterogeneous.

  • Method

    The paper customizes recurrent neural networks with bidirectional processing, dropout, and weighted loss functions, and introduces sent2affect by transferring knowledge from sentiment analysis to emotion recognition.

  • Results

    Pre-trained bidirectional LSTMs consistently outperform traditional machine-learning baselines, with improvements up to 23.2 % in F1-score, 11.6 % in MSE, and 5.6 %–6.6 % from sent2affect.

  • Takeaways & Limitations

    Tailored deep-learning architectures and cross-task transfer learning improve affective computing performance across categorical and dimensional emotion models.

  • Takeaways & Limitations

    Affective-computing datasets remain fairly small, limiting the performance of bidirectional LSTMs and other deep neural architectures.

Abstract

from arXiv · show

Emotions widely affect human decision-making. This fact is taken into account by affective computing with the goal of tailoring decision support to the emotional states of individuals. However, the accurate recognition of emotions within narrative documents presents a challenging undertaking due to the complexity and ambiguity of language. Performance improvements can be achieved through deep learning; yet, as demonstrated in this paper, the specific nature of this task requires the customization of recurrent neural networks with regard to bidirectional processing, dropout layers as a means of regularization, and weighted loss functions. In addition, we propose sent2affect, a tailored form of transfer learning for affective computing: here the network is pre-trained for a different task (i.e. sentiment analysis), while the output layer is subsequently tuned to the task of emotion recognition. The resulting performance is evaluated in a holistic setting across 6 benchmark datasets, where we find that both recurrent neural networks and transfer learning consistently outperform traditional machine learning. Altogether, the findings have considerable implications for the use of affective computing.

1. Introduction

Affective computing seeks to recognize emotions in written language so decision support can reflect individuals’ emotional states. This paper addresses the limited use of deep learning by adapting recurrent networks and introducing task-based transfer learning for text emotion recognition.

  • Text-based emotion recognition is difficult because computational systems must interpret complex and ambiguous language.
  • Written materials are a prominent basis for decision-making, motivating this work’s focus on unimodal English text analysis.
  • Prior affective-computing research largely used traditional machine learning, leaving recurrent neural networks and transfer learning comparatively overlooked.
  • The proposed recurrent networks use bidirectional processing, dropout regularization, and weighted loss functions to address imbalanced emotion labels.Without loss weighting, the network can resemble a majority-class vote.
  • sent2affect pre-trains a network on sentiment analysis, replaces its output layer, and tunes it for emotion recognition.The transfer crosses tasks rather than merely transferring knowledge from another dataset.
  • A holistic benchmark comparison is motivated by the scarcity of widely accepted datasets for fair text-emotion benchmarking.

2. Background

The background distinguishes emotion theories, heterogeneous benchmark datasets, and computational approaches for inferring affect from text. It also motivates transfer learning and highlights comparability challenges caused by varied annotations, domains, and affective dimensions.

  • Affective computing identifies explicit emotional states, whereas sentiment analysis measures subjective polarity along positivity and negativity dimensions.
  • 2.1. Affect theory: Emotion theories represent affect either as mutually exclusive categories or as continuous dimensions with intensity scores.
  • 2.2. Datasets for benchmarking: Benchmark datasets differ in size, domain, linguistic style, annotation source, affective dimensions, and outcome structure.
  • 2.2. Datasets for benchmarking: Emotion labels are often highly unbalanced, causing classifiers to overlook infrequent classes.
  • 2.2. Datasets for benchmarking: Categorical emotion models require single-label classification, whereas dimensional models require regression to predict scores for each item and emotion.
  • 2.3. Computational methods: Text-emotion recognition includes lexicon-based methods and machine-learning approaches, with dictionary methods limited by manual labeling costs and narrow predefined dimensions.
  • 2.4. Transfer learning: sent2affect transfers inductive knowledge across tasks by using sentiment analysis as the source task for emotion recognition.

3. Methods

The methods compare traditional machine learning with deep recurrent neural networks for inferring affective states from narrative materials, optionally using sent2affect transfer learning.

  • The deep-learning pipeline can optionally incorporate sent2affect transfer learning.
  • The pipeline infers affective states from narrative materials using either feature-engineered traditional machine learning or deep recurrent neural networks.

3.2. Baselines from traditional machine learning

Traditional machine-learning baselines represent text with fixed-size bag-of-words features weighted by tf-idf, then apply random forest or support vector models. The section contrasts this feature-engineered setup with recurrent networks that process word sequences directly.

  • Traditional machine learning represents documents with fixed-size bag-of-words vectors whose term frequencies are weighted by tf-idf.The tf-idf value combines term frequency with inverse document frequency to measure relative term importance within a corpus.
  • The baseline predictive models are random forest and support vector machines, including support vector regression for numerical scores.
  • Recurrent neural networks process words in sequences of arbitrary length and therefore avoid the need for handcrafted features such as bag-of-words with tf-idf.
  • The basic recurrent architecture contains an embedding layer, a recurrent layer, and a final dense prediction layer.
  • The basic architecture performed almost identically to a majority-class vote, so the authors focus on architectural improvements rather than reporting exact results.

3.3. Proposed deep neural networks for affective computing

The proposed deep neural networks customize recurrent architectures for small, open-domain, imbalanced affective-computing datasets. They combine dropout, bidirectional processing, and weighted loss functions to address overfitting, language complexity, and class imbalance.

  • Dropout regularizes the recurrent network to address severe overfitting on small affective-computing datasets.
  • Bidirectional processing uses forward and backward LSTM layers to better handle complex, open-domain language.
  • A weighted loss function treats each class equally to reduce bias toward majority labels in severely imbalanced affective-computing targets.
  • Without appropriate imbalance handling, a naïve loss function can optimize toward the majority class and produce performance similar to a majority vote.
  • Over- and undersampling yielded only marginal improvements in the authors’ experiments, motivating the weighted-loss alternative.
  • For a sample x_i with ground-truth label y_i and prediction output p_ik, the weighted loss multiplies the sample’s error by a class-dependent weight w_i.
  • The weight w_i depends only on the ground-truth label y_i; K denotes the number of classes and N the number of samples.

3.4. Sent2affect approach to transfer learning across tasks

Sent2affect transfers knowledge from sentiment analysis to emotion recognition by pre-training a recurrent network, replacing its task-specific output layer, and fine-tuning it on affective-computing data.

  • The transfer is across related but different tasks: sentiment analysis infers positive or negative polarity, whereas affective computing addresses affective dimensions or emotional states.
  • The formal procedure optimizes network parameters on source task S and dataset D_S, replaces the task-specific prediction layer, then fine-tunes on target task T and dataset D_T.
  • The source corpus contains about 100,000 samples labeled for positive or negative sentiment.
  • Sent2affect pre-trains a network for sentiment analysis, replaces its output layer, and tunes the network for emotion recognition.
  • Most network weights benefit from transfer learning, while the new prediction layer is trained for the affective-computing output.

3.5. Model estimation

The models use standardized text preprocessing, tuned classical baselines, and reproducible deep-learning training procedures.

  • Text preprocessing: Documents are tokenized, lowercased, and stripped of punctuation, numbers, and stop words before stemming inflected words to base forms.The preprocessing uses NLTK and produces bag-of-words representations.
  • Data splitting: Datasets without designated test sets are divided randomly into 80% training and 20% test data.
  • Baseline estimation: Random forest and support vector classifiers are manually or grid-search optimized over their principal hyperparameters.The support vector search covers linear, radial-basis-function, and sigmoid kernels.
  • Deep-learning estimation: Deep models use either pretrained GloVe or learned embeddings, Adam optimization, validation-error early stopping, and averages over 10 independent runs.

4. Evaluation

Across classification, regression, and transfer-learning evaluations, recurrent neural networks—especially BiLSTMs with pretrained embeddings—and sent2affect improve on traditional baselines across benchmark datasets.

  • Classification according to categorical emotion models: Classification models are compared using weight-averaged F1, sensitivity, and specificity, with random forest and support vector machines as traditional baselines.Deep networks avoid the baselines’ bag-of-words tf-idf feature engineering.
  • Classification according to categorical emotion models: 1.6%–23.2% improvements over traditional baselines make the bidirectional recurrent model the only architecture that consistently wins across all classification datasets.The BiLSTM outperforms the unidirectional variant in four of five experiments.
  • Classification according to categorical emotion models: Sensitivity and specificity show the largest improvements on less noisy datasets, while the election-tweet baseline slightly exceeds the best BiLSTM in sensitivity, 57.1 versus 56.9.Transfer learning is subsequently used to strengthen performance on this challenging dataset.
  • Regression according to dimensional affect models: Regression evaluates emotional-intensity prediction with mean squared error, where lower values indicate better performance.Table 4 notes that the first task has higher errors because its outcome scale differs.
  • Regression according to dimensional affect models: Deep learning yields up to 11.6% improvement over traditional machine learning, and the pretrained-embedding BiLSTM is best in all seven regression experiments.The BiLSTM also outperforms the unidirectional LSTM in every experiment.
  • Transfer learning via sent2affect: sent2affect transfers representations learned from sentiment analysis by replacing and fine-tuning the final layer for emotion recognition.It is evaluated against a naïve BiLSTM and embedding-only transfer learning on election and general tweet datasets.

5. Discussion

The paper finds that customized deep-learning architectures improve text-based emotion recognition across varied applications, while dataset heterogeneity and small sample sizes constrain comparability and deep-learning performance.

  • 5.1. Comparison: Customized networks outperformed baselines in all experiments, with gains of up to 23.2%.The holistic evaluation covered multiple datasets rather than relying on a single benchmark.
  • 5.1. Comparison: Customized regularization enabled networks to learn from datasets with approximately 1,000 observations, although improvements tended to be larger with less language noise.Word embeddings also improved performance by reducing high-dimensional one-hot representations.
  • 5.1. Comparison: Bidirectional LSTMs usually performed better than simple LSTMs, while sent2affect added improvements of up to 6.6%.sent2affect transfers knowledge from sentiment analysis to emotion recognition by changing the output layer.
  • 5.3. Further use cases: Potential applications span management, marketing, finance, politics, education, healthcare, and social-media misinformation decision support.The paper presents these as use cases or opportunities for deep-learning-based affective computing.
  • 5.4. Implications for management and practice: The paper recommends customized architectures even for fairly small datasets, while emphasizing that practitioners need a thorough understanding of available tools.It notes that deep-learning use cases outside academia remain scarce.
  • 5.5. Implications for research: Comparability and generalizability remain constrained because datasets differ in size, annotation schemes, affective dimensions, and textual sources.The paper suggests standardized annotations and larger datasets to support benchmarking and effective deep-learning use.

6. Conclusion

The conclusion frames text-based affective computing as decision-support technology and presents tailored deep-learning models as a response to the difficulty of emotion recognition.

  • 6. Conclusion: Affective computing infers individual and collective emotional states from text to support decision-making in an anthropomorphic way.
  • 6. Conclusion: Bidirectional processing, dropout regularization, and weighted loss functions address characteristics of emotion-recognition datasets and tasks.The weighted loss functions are intended to cope with dataset imbalances.
  • 6. Conclusion: 23.2% maximum F1-score gains for classification and 11.6% maximum MSE gains for regression were observed over traditional machine-learning baselines.The experiments covered categorical and dimensional emotion models.
  • 6. Conclusion: Sent2affect, which transfers from sentiment analysis to emotion recognition, produced further performance improvements between 5.6% and 6.6%.

Appendix A. Recurrent Neural Networks

The recurrent architecture processes word embeddings sequentially, maintains hidden representations of the text, and uses LSTM mechanisms to preserve information across long dependencies.

  • Appendix A. Recurrent Neural Networks: The method groups emotional-state inference into classification of emotion labels and regression of affective intensity scores.
  • Appendix A. Recurrent Neural Networks: The architecture combines an embedding layer, recurrent LSTM layer, and final dense prediction layer.The dense layer differs according to whether the model predicts an affective category or emotional intensity.
  • Embedding layer: Word embeddings replace one-hot word encodings with dense, semantically organized numerical representations.Words with similar meanings are optimized to have shorter embedding distances.
  • Recurrent layer: The recurrent layer iterates through words and accumulates textual information in a hidden state that passes information forward.The recurrent process is illustrated as an unrolled sequence computation.
  • Recurrent layer: LSTMs add an information-storing cell with explicit update rules, while BiLSTMs process the word sequence in both directions.The final hidden state of the LSTM accumulates the complete document.
  • Dense layer: The recurrent output feeds the final prediction, producing either a classification label or a continuous regression score.

Appendix A.1. Dense layer for affect prediction

The dense layer maps the recurrent representation to the requested affective output, using softmax for categorical emotions and an affine transformation for intensity scores.

  • Appendix A.1. Dense layer for affect prediction: The dense layer connects every input neuron to every output neuron through trainable coefficients before applying an activation function.
  • Classification: Softmax converts classification inputs into values from 0 to 1 that sum to one across the k outputs.These values are interpreted as probabilities for the k emotional classes.
  • Classification: The predicted single-class emotion is the output with the highest softmax probability.The paper expresses this choice with an arg max over the k class probabilities.
  • Regression: Regression uses an affine transformation to aggregate the learned representation into one numerical score for affective intensity.
Loading 1803.06397v6…