PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-Temporal Recurrent Neural Network for Emotion Recognition

Tong Zhang, Wenming Zheng, Zhen Cui, Yuan Zong, Yang Li

arXiv:1705.04515v1cs.CV

TL;DR

Emotion recognition from EEG and facial signals requires modeling affective information distributed across spatial and temporal volumes. STRNN addresses this with spatial and temporal recurrent layers plus sparse projection, and experiments report competitive performance on public EEG and facial-expression datasets.

  • Problem

    EEG and facial emotion recognition use external affective signals, but the paper seeks a unified model for their shared spatial-temporal structure.

  • Method

    STRNN hierarchically combines multidirectional spatial RNNs, bidirectional temporal RNNs, and sparse projection to model dependencies and select salient regions.

  • Results

    89.50% accuracy was achieved by STRNN, about 4% higher than SRNN or TRNN, with lower deviation in the reported comparison.

  • Takeaways & Limitations

    STRNN provides a unified spatial-temporal dependency model for EEG-based and facial emotion recognition and achieves state-of-the-art performance on the reported datasets.

Abstract

from arXiv · show

Emotion analysis is a crucial problem to endow artifact machines with real intelligence in many large potential applications. As external appearances of human emotions, electroencephalogram (EEG) signals and video face signals are widely used to track and analyze human's affective information. According to their common characteristics of spatial-temporal volumes, in this paper we propose a novel deep learning framework named spatial-temporal recurrent neural network (STRNN) to unify the learning of two different signal sources into a spatial-temporal dependency model. In STRNN, to capture those spatially cooccurrent variations of human emotions, a multi-directional recurrent neural network (RNN) layer is employed to capture longrange contextual cues by traversing the spatial region of each time slice from multiple angles. Then a bi-directional temporal RNN layer is further used to learn discriminative temporal dependencies from the sequences concatenating spatial features of each time slice produced from the spatial RNN layer. To further select those salient regions of emotion representation, we impose sparse projection onto those hidden states of spatial and temporal domains, which actually also increases the model discriminant ability because of this global consideration. Consequently, such a two-layer RNN model builds spatial dependencies as well as temporal dependencies of the input signals. Experimental results on the public emotion datasets of EEG and facial expression demonstrate the proposed STRNN method is more competitive over those state-of-the-art methods.

I. INTRODUCTION

The paper introduces STRNN as a unified framework for EEG and facial emotion recognition, modeling spatial and temporal dependencies while selecting salient emotion regions. It targets limitations of separate signal-specific approaches by organizing both modalities as spatial-temporal volumes.

  • I. INTRODUCTION: STRNN unifies EEG-based and facial emotion recognition in one end-to-end deep network framework.The framework learns spatial dependencies in multi-electrode or image contexts and long-term temporal information in sequences.
  • I. INTRODUCTION: A quad-directional spatial RNN scans each temporal slice from multiple angles to learn spatial dependencies and discriminative contextual sequences.The spatial layer traverses spatial regions in different orders before producing features for temporal modeling.
  • I. INTRODUCTION: A bi-directional temporal RNN processes the spatial-feature sequences forward and backward to capture long-term temporal dependencies.The hierarchical RNN layers act as memory units that retain past inputs and model distant correlations.
  • I. INTRODUCTION: Sparse projection adaptively weights hidden states to identify salient, co-occurring spatial regions and more discriminative time slices.The projection operates globally across spatial or temporal domains and is intended to improve discriminative ability.
  • I. INTRODUCTION: EEG electrodes and facial-expression images are represented as spatial-temporal volumes so both tasks can be modeled jointly.EEG signals are spatially organized according to electrode coordinates, while facial signals provide image contexts over time.

II. RELATED WORK

Prior work applies handcrafted descriptors, conventional classifiers, CNNs, and RNNs to EEG or facial emotion recognition. The paper positions STRNN as an end-to-end approach that simultaneously integrates spatial and temporal dependencies across both tasks.

  • II. RELATED WORK: EEG emotion recognition has used descriptors such as high-order crossings and differential entropy with classifiers including SVM and GSCCA.These approaches represent established signal-processing and classification pipelines for EEG-based emotion recognition.
  • II. RELATED WORK: The 3DCNN-DAP method extracts video representations with 3DCNNs and strong spatial structural constraints on dynamic action parts.It exemplifies prior efforts to incorporate spatial structure into facial-video representation learning.
  • II. RELATED WORK: Facial emotion recognition has increasingly adopted CNNs and RNNs to learn robust features from video sequences.RNNs are used for sequential data and can scan image regions into ordered sequences to capture contextual dependencies.
  • II. RELATED WORK: STRNN is proposed to integrate spatial and temporal dependencies end to end for EEG-based and video-based facial emotion recognition.This unified formulation extends sequential modeling beyond separate modality-specific approaches.

III. STRNN MODEL FOR EMOTION RECOGNITION

STRNN stacks spatial and temporal recurrent processing to model dependencies in spatial-temporal emotion signals. Its spatial states are projected before temporal modeling and classification.

  • STRNN framework: STRNN accepts spatial-temporal data, including multi-channel EEG sequences and video volumes, and stacks spatial and temporal RNN layers.The framework is designed for inputs traversable in predefined spatial or temporal orders.
  • Spatial recurrent modeling: Four directional spatial RNNs traverse each time slice from complementary angles to capture contextual relationships among neighboring spatial elements.The spatial graph connects contiguous elements, while each direction recurrently reuses learned parameters.
  • Spatial projection: Sparse projection is applied to spatial hidden states to detect salient emotion regions and summarize representations across traversal directions.The projected states reduce the spatial representation to Kp hidden states before concatenation.
  • Temporal recurrent modeling: Concatenated spatial representations form a temporal sequence that a bidirectional temporal RNN traverses forward and backward to model emotion dynamics.The temporal hidden states are also projected before producing the output vector.
  • Classification and training: The temporal output is mapped to emotion classes and passed to softmax, while cross-entropy training includes sparsity terms for spatial and temporal projections.The sparsity terms encourage high weights for discriminative hidden states and low weights for others; optimization uses BPTT.

IV. EXPERIMENTS

The experiments evaluate STRNN on EEG and facial-expression emotion-recognition datasets against existing methods.

  • The experiments compare STRNN with other state-of-the-art methods on EEG and facial-expression emotion-recognition datasets.

A. Datasets and feature extraction

The study uses SEED EEG recordings and CK+ facial-expression sequences, with frequency-band differential entropy for EEG and pretrained visual features plus rotation augmentation for CK+.

  • Datasets: SEED contains three emotion categories from 15 subjects recorded through 62 electrodes at 1000 Hz.
  • Datasets: CK+ provides 327 facial-expression image sequences labeled with seven emotions.
  • Feature extraction: EEG features are differential entropy descriptors extracted across five frequency bands and 62 channels using 1-second nonoverlapping Hanning windows.
  • Feature extraction: CK+ images use pretrained feature extraction and rotations of 7°, −7°, 12° and −12° to augment the data to 1635 samples.

B. EEG Signals Based Emotion Recognition

On SEED, STRNN uses four-direction electrode scanning, spatial and temporal projection, and achieves higher and more stable emotion-recognition performance than the compared methods.

  • Evaluation protocol: SEED evaluation uses nine sessions for training and six sessions for testing across 30 experiments involving 15 subjects.
  • Model configuration: STRNN scans EEG electrodes from four directions to model interactions among spatially adjacent electrodes.
  • Overall results: 89.50% accuracy is achieved by STRNN, compared with 86.08% for DBN using differential-entropy features from all 62 channels.
  • Frequency-band analysis: STRNN accuracies exceed 80% on Delta, Theta, Alpha and Beta bands, with Beta highest and Gamma lower.
  • Frequency-band analysis: STRNN has lower recognition deviations than DBN on all bands except Gamma, indicating greater stability across subjects and experiments.
  • Confusion-matrix analysis: All three SEED emotion categories exceed 85.0% recognition accuracy, with positive and negative emotions easier than neutral.

C. Video Emotion Recognition

On CK+, STRNN was evaluated with subject-disjoint 10-fold cross-validation against several state-of-the-art methods. It achieved 95.4% accuracy, with expression-level accuracies above 90%.

  • The CK+ evaluation used 10-fold cross-validation with subject identities separated across subsets.Each run trained on nine subsets and tested on the remaining subset, with the average performance reported.
  • 95.4% accuracy was achieved by STRNN on the CK+ facial-expression dataset, outperforming the compared state-of-the-art methods.The compared methods used varying cross-validation protocols, including 10-fold, 5-fold, and 4-fold validation.
  • Each of the seven facial expressions achieved recognition accuracy above 90%.Anger, happiness, sadness, and surprise reached 97.73%, 97.06%, 96.00%, and 96.47%, respectively.
  • Contempt, disgust, and fear obtained recognition rates of 93.75%, 91.94%, and 91.67%, respectively.Higher confusions occurred among contempt and angry, fear, or sadness expressions.

V. CONCLUSIONS AND DISCUSSIONS

The paper concludes that STRNN unifies EEG-based and facial emotion recognition by hierarchically modeling spatial and temporal dependencies. Sparse projections further select salient emotion regions and support state-of-the-art performance on both datasets.

  • V. CONCLUSIONS AND DISCUSSIONS: STRNN combines a multi-direction SRNN layer with a bi-direction TRNN layer to learn spatial and temporal emotion dependencies hierarchically.The spatial layer models co-occurring variations, while the temporal layer models time-dependent variation.
  • V. CONCLUSIONS AND DISCUSSIONS: The framework handles both EEG-signal-based and face-image-based human emotion recognition.For EEG, electrode coordinates define spatial scanning and DE feature sequences are sliced to represent temporal variation.
  • V. CONCLUSIONS AND DISCUSSIONS: Sparse projection is imposed on spatial and temporal hidden states to select salient emotion regions and increase model discriminant ability.The conclusion links sparse projection to salient-region selection and discriminative performance.
  • V. CONCLUSIONS AND DISCUSSIONS: Experiments on SEED and CK+ demonstrate state-of-the-art performance for the proposed STRNN method.The reported scope covers both EEG and facial-expression datasets.

1) Comparisons of STRNN with SRNN and TRNN:

On SEED, the full STRNN outperformed versions containing only temporal or only spatial recurrent modeling. The reported comparison supports the effectiveness of the hierarchical spatial-temporal structure.

  • 1) Comparisons of STRNN with SRNN and TRNN:: The ablation compares STRNN with SRNN-only and TRNN-only networks by replacing the other recurrent layer with a fully connected layer.This isolates the contribution of spatial and temporal recurrent modeling.
  • 1) Comparisons of STRNN with SRNN and TRNN:: 89.50% accuracy was achieved by STRNN on SEED, about 4% higher than the SRNN-only or TRNN-only variants, with lower deviation.The comparison attributes the improvement to the hierarchical structure of spatial and temporal RNN layers.

2) Comparisons of STRNN with non-sparse STRNN:

The study compares sparse STRNN with a non-sparse variant on SEED and CK+. Sparse STRNN achieved accuracies more than one percentage point higher than non-sparse STRNN, while also enabling salient-region detection.

  • 2) Comparisons of STRNN with non-sparse STRNN:: More than one percentage point higher accuracy was obtained by sparse STRNN than by non-sparse STRNN on both SEED and CK+.The comparison verifies the effectiveness of sparse constraints in improving STRNN performance.
  • 2) Comparisons of STRNN with non-sparse STRNN:: Sparse constraints improved performance and achieved salient emotion-region detection.The reported benefit concerns both recognition accuracy and identification of salient regions.

3) Salient emotion detection:

The STRNN visualizes hidden-state weights to identify salient facial regions, highlighting expression-relevant areas and sparse projection patterns.

  • Salient emotion detection: Weights from the multi-direction SRNN layer are mapped back onto 2D facial images to visualize salient regions.Projection-matrix coefficient magnitudes are averaged across spatial traversing directions before mapping.
  • Salient emotion detection: The visualization highlights regions around the mouth, eyes, and nose that are crucial for perceiving facial expressions.
  • Salient emotion detection: Black facial regions indicate that most projection-matrix values are near zero, supporting the sparsity induced by l1-norm loss terms.
Loading 1705.04515v1…