Source-linked AI summary
Deep Learning for Human Affect Recognition: Insights and New Developments
Philipp V. Rouast, Marc T. P. Adam, Raymond Chiong
TL;DR
Human affect recognition is important for more natural human-computer interaction but remains difficult because affective data are high-dimensional, variable, and often sparsely labeled. This paper reviews 950 studies from 2010–2017, including 233 deep-learning studies, and finds broad use of deep networks across spatial, temporal, and multimodal feature learning. Deep architectures are widely represented in state-of-the-art systems, although limited labeled data and unreliable or imbalanced datasets remain important boundaries.
Problem
Human affect recognition is difficult because real-world affective inputs are highly variable and high-dimensional, while ambitious tasks have relatively few labeled examples.
Method
The paper comprehensively reviews and classifies 950 studies, then analyzes 233 deep-learning studies by spatial, temporal, and joint multimodal feature representation.
Results
95% of 150 studies comparing shallow and deep architectures reported improvements from deep learning over conventional approaches.
Takeaways & Limitations
Deep learning is applied across spatial, temporal, and multimodal representation learning and is generally well suited to continuous, multimodal prediction of spontaneous affective displays in the wild.
Takeaways & Limitations
Limited labeled data, unreliable ground truth, and imbalanced datasets constrain generalization and the training of large deep models.
Abstract
from arXiv · showhide
Automatic human affect recognition is a key step towards more natural human-computer interaction. Recent trends include recognition in the wild using a fusion of audiovisual and physiological sensors, a challenging setting for conventional machine learning algorithms. Since 2010, novel deep learning algorithms have been applied increasingly in this field. In this paper, we review the literature on human affect recognition between 2010 and 2017, with a special focus on approaches using deep neural networks. By classifying a total of 950 studies according to their usage of shallow or deep architectures, we are able to show a trend towards deep learning. Reviewing a subset of 233 studies that employ deep neural networks, we comprehensively quantify their applications in this field. We find that deep learning is used for learning of (i) spatial feature representations, (ii) temporal feature representations, and (iii) joint feature representations for multimodal sensor data. Exemplary state-of-the-art architectures illustrate the progress. Our findings show the role deep architectures will play in human affect recognition, and can serve as a reference point for researchers working on related applications.
1 INTRODUCTION
Affect recognition has become central to more user-centered human-computer interaction, while research has shifted from unimodal and controlled settings toward spontaneous, multimodal behavior. This review documents the growing use of deep learning and maps its applications across sensor modalities and feature types.
- Affective displays convey important information through speech, facial expressions, gestures, and other channels, motivating affect recognition for more natural human-computer interaction.
- Since 2010, affect recognition research has increasingly addressed realistic, spontaneous displays and multimodal settings represented by EmotiW and AVEC competitions.
- 950 studies reveal a trend toward using deep neural networks for affect recognition.
- 233 deep-learning studies identify spatial, temporal, and joint multimodal feature representations as the main application areas.
- The review illustrates deep neural network applications across visual, auditory, and physiological sensor data, while surveying relevant databases and open research directions.
2 DEEP LEARNING FOR AFFECT RECOGNITION
Visual and auditory affect recognition are difficult because real-world inputs vary substantially and are high-dimensional. Traditional shallow approaches address dimensionality through projection, but the review motivates deep architectures as an alternative for challenging affect-recognition problems.
- Visual and auditory affect recognition involve highly varying input functions and high-dimensional examples such as images and audio files.
- The curse of dimensionality makes conventional computational and statistical methods less effective as the number of data dimensions increases.
- Shallow methods commonly project high-dimensional data into lower-dimensional spaces before modeling, but deep architectures began receiving attention for affect recognition from 2010.
2.1 Deep learning
Deep learning uses hierarchical computational layers to learn increasingly abstract representations, with specialized architectures exploiting spatial or temporal structure. Its development has been supported by improved training methods, computing power, and larger datasets, although recurrent models remain difficult to train on long sequences.
- Deep architectures commonly contain three or more computational layers, distinguishing them from traditional shallow models represented by two computational layers.
- Deep neural networks learn feature representations through weighted layers of nonlinear processing units, with fully-connected networks linking adjacent layers densely.
- Greater depth supports feature reuse and hierarchical abstraction, with higher-level representations generally becoming more invariant to local input changes.
- Training advances, including unsupervised pre-training, improved optimizers, suitable architectures, and GPU-based computing, helped overcome difficulties such as vanishing gradients.
- CNNs exploit spatial structure through shared kernels and sparse connectivity, while RNNs share parameters across time to model sequential dynamics.
- RNN training becomes considerably more difficult on long sequences because gradients tend to vanish or explode, motivating gated architectures such as LSTMs and GRUs.
2.2 The notion of affect in affect recognition
Affect can be represented categorically as discrete states or dimensionally in a continuous space, and the literature has not reached consensus between these views. Dimensional models offer greater flexibility for representing affective complexity but do not eliminate practical labeling challenges.
- Affect representation remains unresolved because affect and emotion are subjective, with categorical and dimensional models forming the two dominant views.
- Categorical models represent affect as discrete basic states, commonly including anger, happiness, surprise, disgust, sadness, and fear.
- Dimensional models represent affect as points in a multidimensional space using concepts such as intensity and positivity.
- Dimensional representations address the restrictiveness of discrete categories and the complexity of affective states, including difficulties with labeling and labeler agreement.
2.3 Frontiers in affect recognition
Affect recognition research is moving toward real-life scenarios, where natural settings, temporal dynamics, multimodal fusion, and limited labeled data create major challenges. These conditions motivate more complex models and multimodal systems.
- Research has shifted from lab-based, acted databases toward real-life scenarios and increasingly complex models using available sensor data.
- Natural settings: Uncontrolled settings introduce visual variation such as occlusions, illumination changes, spontaneous behavior, poses, and rigid movements, alongside noisy and interrupted audio.
- Temporal dynamics: Temporal dynamics provide contextual information for distinguishing similar short-term displays and assessing the importance of specific segments.
- Multimodal sensor fusion: Multimodal fusion reflects how humans express and sense affect, with audiovisual fusion generally accepted to improve model robustness and accuracy.
- Limited availability of labeled data: Limited labeled data make training large deep neural networks difficult, requiring advanced regularization to avoid overfitting and motivating transfer learning.
2.4 Towards learning deep models of affect
Deep models address affect recognition’s spatial, temporal, and multimodal representation challenges by learning features directly from high-dimensional sensor data. This reduces reliance on work-intensive handcrafted features and supports downstream classification or regression.
- Affect recognition requires representations that disentangle spatial, temporal, and cross-modal variation in high-dimensional multimodal sensor data.
- Deep models are motivated by the limitations of handcrafted features, whose design is work-intensive and error-prone.
- Learning spatial feature representations: CNNs learn spatial feature representations from images, short-term image sequences, and audio segments.
- Learning temporal feature representations: RNNs learn temporal feature representations from audio, image sequences, and physiological measurements.
- Learning joint feature representations for multimodal data: Deep neural networks learn joint feature representations from multiple unimodal representations for feature-level fusion.
- Derived deep features can feed simple classifiers and regressors, including logistic regression and support vector regression.
3 THE STATE OF THE ART
The review measures deep-learning adoption and usage across affect-recognition research through a two-stage literature search. Deep learning expanded sharply after 2010, and the 233 deep-learning studies most often addressed FER and spatial representations, while temporal and joint multimodal learning remained important areas.
- The review aims to measure deep-learning adoption and identify the specific functions DNNs fulfill in human affect recognition systems.
- The two-stage search covers studies since 2010 using directly body-given cues, excluding affect communicated through generic media, music, or text.
- 950 studies were identified in Stage 1 and manually classified according to shallow or deep model usage.
- 25% year-on-year average growth occurred in affect-recognition studies between 2010 and 2016.
- 52% of studies employed deep learning in 2017, following a 119% average year-on-year increase in published deep-learning studies.
- 233 deep-learning studies were classified by deep-learning function and affect-recognition modality, with studies often using multiple modalities or functions.
- FER attracted almost twice as many DNN studies as SER, while EEG was the most frequent physiological-signal basis.
- Spatial representation learning was the most common DNN application, temporal learning was active in SER, and joint representations supported early multimodal feature fusion.
3.1 Learning spatial feature representations
The review finds that deep learning is applied to spatial feature learning across visual, speech, and physiological affect signals, with CNNs dominating reported approaches. Deep spatial representations generally outperform handcrafted features, although combining both can remain beneficial.
- Visual affect recognition: 56% of FER studies use architectures with six or fewer convolutional and fully-connected layers, often favoring smaller models over transfer learning.These models are specifically designed for FER and aim to reduce overfitting on relatively small datasets.
- Regularization: Data augmentation typically expands image or sequence examples by factors of 10–30, with reported accuracy gains such as 79% to 89%.Dropout is also widely used to reduce overfitting, reported in 57% of fully-connected layers and 12% of convolutional layers.
- Speech affect recognition: In speech emotion recognition, 90% of direct comparisons found deep spatial features outperform handcrafted low-level descriptors.CNNs can learn features from raw spectral representations without handcrafted feature engineering.
- Visual affect recognition: 93% of 103 studies reporting direct comparisons found deep spatial features more accurate than handcrafted spatial features for FER.CNNs learn hierarchical representations from low-level to higher-level features.
- Feature complementarity: Deep and handcrafted spatial features are often complementary in affect recognition, with several challenge-winning studies combining them through score-level or model-level fusion.This pattern differs from settings where deep features more consistently replace handcrafted features.
- Architectures and modalities: CNNs account for 91% of reviewed spatial-feature studies and are increasingly applied directly to raw speech and physiological data.The review contrasts this with spectrogram-based representations.
3.2 Learning temporal feature representations
Deep learning models represent temporal dynamics in affective sensor data through aggregated spatio-temporal features, recurrent sequence modeling, or local 3D convolution. Across modalities, reviewed evidence favors deep temporal representations over handcrafted alternatives, while CNNs and RNNs serve different temporal scales.
- Approaches: Temporal feature learning captures variation in spatial sensor characteristics across sequences, using fully-connected DNNs, RNNs, or CNNs.The review labels these approaches T1, T2, and T3, respectively.
- Evidence: 94% of FER studies comparing deep and handcrafted temporal features report better performance for deep features.
- Approaches: RNNs model global temporal variation from sequential frame-level spatial features, supporting arbitrary-length sequences and frame-level or sequence-level recognition.
- Approaches: 3D CNNs provide local spatio-temporal modeling, but are typically limited to very short sequences.
- Takeaways: 92% of 73 comparative studies support higher accuracies for deep temporal features than handcrafted temporal features.
- Takeaways: CNNs are suited to local temporal modeling, whereas RNNs are useful for global temporal modeling of affect.
3.3 Learning joint feature representations
Joint multimodal learning addresses fusion of heterogeneous affective signals through feature-level DNN or RNN architectures. Audiovisual fusion dominates the literature, but evidence remains mixed on whether deep feature fusion surpasses decision-level fusion.
- Fusion challenge: Joint multimodal learning must handle high dimensionality, differing temporal resolutions, and differing temporal dynamics across sensor modalities.
- Fusion strategies: Feature-level fusion concatenates independently extracted features so a model can learn correlations between modalities, while decision-level fusion combines independent model outputs.
- Architectures: Feature-level fusion uses fully-connected DNNs for joint representations without temporal context, or RNNs when temporal context and intermodal lags matter.
- Applications: In three AVP studies, ECG helped predict valence but not arousal in feature-level fusion.
- Applications: Joint feature learning is most commonly applied to audiovisual fusion.
- Evidence: 69% of 16 comparative studies find deep feature-level fusion more accurate than simple decision-level fusion, leaving no consensus.
3.4 Databases and competitions
The review emphasizes that database properties shape affect-recognition research and that competitions provide clearer comparisons than ordinary cross-study evaluations. Public, increasingly large databases and deep-learning competition entries mark the field’s recent development.
- Databases: Only 11% of 233 reviewed studies used private databases, while 77 different public databases were used overall.
- Databases: Database differences include modalities, subjects, examples, acquisition, elicitation, annotation, and affect labels, all of which affect algorithm design.
- Affect models: 75% of 77 public databases use categorical models, 18% dimensional models, and 5.2% both; reviewed studies show the same categorical predominance.
- Recent databases: AffectNet and EmotioNet each contain around 1M labeled images, while newer databases increasingly use automated labeling or unlabeled examples.
- Evaluation: The review avoids direct accuracy comparisons across studies because datasets, splits, and evaluation statistics differ; organized competitions provide clearer comparisons.
- Competitions: Since 2015, all winning competition entries have used deep learning, with EmotiW evaluated by accuracy and AVEC by correlation coefficients.
- Competitions: Recognition accuracy on AFEW increased from 41% in 2013 to 60.3% in 2017.
4 DISCUSSION
The review finds deep neural networks central to modern affect recognition, while limited labeled data and unresolved representation choices constrain progress. It also highlights hybrid features, multimodal integration, attention, and interpretability as important directions.
- Deep learning impact: 95% of 150 studies comparing shallow and deep architectures reported improvements from deep learning over conventional approaches.The review also finds DNNs in most state-of-the-art affect recognition systems.
- Open challenges: Limited labeled data remains a major obstacle because unreliable ground truth, class imbalance, and small datasets hinder generalization and large-model training.Unsupervised and semi-supervised learning are promising, but labeled fine-tuning remains necessary.
- Feature representations: Handcrafted features remain important because hybrid models can exploit their complementarity with deep features, despite DNN features often outperforming them directly.Challenge-winning systems have fused handcrafted and deep features at score or model level.
- System integration: End-to-end multimodal systems are a future direction because CNN, LSTM, and fusion components are often trained separately, although joint training requires more examples.CNN-plus-LSTM combinations are the most common multi-component design, appearing in 21 studies.
- Interpretability: Interpretability efforts find that higher CNN layers learn concepts similar to Action Units, while LSTM cells learn representations resembling established prosodic features.These findings may provide an additional perspective for interdisciplinary emotion research.
- Affect representations: The categorical-versus-dimensional representation of affect remains unresolved, with labels and prediction requirements strongly influencing the choice.Categorical models appear in 82% of reviewed studies, including 92% of spatial feature-learning studies.