Source-linked AI summary
HetEmotionNet: Two-Stream Heterogeneous Graph Recurrent Neural Network for Multi-modal Emotion Recognition
Ziyu Jia, Youfang Lin, Jing Wang, Zhiyang Feng, Xiangheng Xie, Caijie Chen
TL;DR
Emotion recognition with physiological signals faces challenges in combining spatial-spectral-temporal features and modeling multimodal heterogeneity and correlation. HetEmotionNet uses two graph-recurrent streams to address these challenges, and experiments report state-of-the-art performance on DEAP and MAHNOB-HCI.
Problem
Emotion-recognition methods face challenges in fully using complementary spatial-spectral-temporal features and simultaneously modeling heterogeneity and correlation among modalities.
Source: §1 · p. 2Method
HetEmotionNet uses spatial-temporal and spatial-spectral graph streams, with GTN, GCN, and GRU components for modality heterogeneity, correlation, and domain dependencies.
Source: §1 · p. 3, §4.2 · p. 5Results
Takeaways & Limitations
The authors present HetEmotionNet as a general framework for multimodal physiological signals.
Source: §6 · p. 8
Abstract
from arXiv · showhide
The research on human emotion under multimedia stimulation based on physiological signals is an emerging field, and important progress has been achieved for emotion recognition based on multi-modal signals. However, it is challenging to make full use of the complementarity among spatial-spectral-temporal domain features for emotion recognition, as well as model the heterogeneity and correlation among multi-modal signals. In this paper, we propose a novel two-stream heterogeneous graph recurrent neural network, named HetEmotionNet, fusing multi-modal physiological signals for emotion recognition. Specifically, HetEmotionNet consists of the spatial-temporal stream and the spatial-spectral stream, which can fuse spatial-spectral-temporal domain features in a unified framework. Each stream is composed of the graph transformer network for modeling the heterogeneity, the graph convolutional network for modeling the correlation, and the gated recurrent unit for capturing the temporal domain or spectral domain dependency. Extensive experiments on two real-world datasets demonstrate that our proposed model achieves better performance than state-of-the-art baselines.
1 INTRODUCTION
Physiological signals are presented as objective indicators of emotion, but existing approaches face challenges in combining spatial-spectral-temporal features and modeling modality heterogeneity alongside correlation. HetEmotionNet addresses these challenges with two graph-based streams and outperforms state-of-the-art models on two benchmark datasets.
1 INTRODUCTION
- Physiological signals are described as objective emotion indicators because they cannot be disguised, unlike facial expressions or sound.The introduction contrasts these signals with multimedia cues that may not reflect a person's real emotional state.Source: §1 · p. 1
- Existing approaches often focus on spatial-temporal or spatial-spectral features, leaving their complementarity underused; image-like maps can also introduce noise through zero-filling.Graph representations preserve electrode functional connectivity without that artificial zero-filling.Source: §1 · p. 2
- Prior multimodal methods often model heterogeneity or correlation separately, although physiological modalities differ in signal attributes and can be correlated within and across modalities.The paper identifies simultaneous modeling of both properties as a challenge.Source: §1 · pp. 2–3
- The proposed two-stream graph framework fuses spatial-spectral-temporal features while modeling modality heterogeneity and correlation.Its streams use graph transformers, graph convolutions, and recurrent units.Source: §1 · p. 3
- The model outperforms all state-of-the-art models in experiments on two benchmark datasets.The introduction reports extensive experiments but does not provide the dataset-specific scores here.Source: §1 · p. 3
2 RELATED WORK
Prior emotion-recognition work extracts spatial-temporal or spatial-spectral features and uses multimodal signals to model heterogeneity or correlation. HetEmotionNet targets the remaining joint fusion of domains and modality relationships.
2 RELATED WORK
- Prior methods extract either spatial-temporal features or spatial-spectral features, while SST-EmotionNet combines all three domains using image-like maps that can introduce noise,,,,.Examples include recurrent or convolutional-recurrent models for spatial-temporal features and frequency-band representations for spatial-spectral features.Source: §2 · p. 3
- Multimodal methods commonly model heterogeneity or correlation separately, using modality-specific feature branches or approaches that learn shared or correlated representations,,, [30].The cited examples include separate EEG/PPS feature extraction and GSCCA-based correlation learning.Source: §2 · p. 3
- The paper identifies joint fusion of multi-domain features, modality heterogeneity, and correlation as the gap addressed by HetEmotionNet.This is the stated motivation for its unified modeling approach.Source: §2 · p. 3
3 PRELIMINARIES
The preliminaries define typed heterogeneous graphs and graph sequences for representing multimodal physiological signals. Emotion recognition is framed as mapping spatial-temporal and spatial-spectral sequences to an emotion class.
3 PRELIMINARIES
- A heterogeneous graph has nodes and edges with type mappings, and qualifies as heterogeneous when the total number of node and edge types exceeds two.The notation assigns node types from A and edge types from R.Source: §3 · p. 3
- In the heterogeneous emotional network, each node represents a physiological-signal channel, node types correspond to modalities, and edges represent connections between channels.The network is also represented using node features and an adjacency matrix.Source: §3 · p. 3
- A heterogeneous graph sequence stacks graphs across timesteps or frequency bands, and the task maps spatial-temporal and spatial-spectral sequences to emotion classifications.The sequence lengths correspond to the number of timesteps and frequency bands, respectively.Source: §3 · pp. 3–4
4 METHODOLOGY
HetEmotionNet represents physiological channels as modality-typed graphs in separate spatial-temporal and spatial-spectral streams. GTN, GCN, and GRU components model heterogeneity, channel relationships, and sequence dependencies before stream fusion and classification.
4.1 Heterogeneous Graph Sequence Construction
- Each node is a modality-typed channel; temporal amplitude features or per-band differential entropy features are stacked into separate graph sequences using channel relationships.Channel relationships are determined from mutual information between channel pairs.Source: §4.1 · pp. 4–5, Figure 5 · p. 4
4.2 Heterogeneous Graph Recurrent Neural Network
- GTN models modality heterogeneity by learning meta-paths from adjacency matrices; each GT-layer softly constructs graph structures and multiplies them to represent second-order neighbor relationships.Stacked GT-layers learn higher-level relationships, while multi-channel convolution supports multiple meta-paths.Source: §4.2 · p. 5
- GCN aggregates information from neighboring nodes to capture spatial correlations, then GRU processes the graph sequence to capture timestep or frequency-band dependencies.Graph convolutions are applied across the sequence with shared parameters.Source: §4.2 · pp. 5–6
4.3 Fusion and Classification
- Features from the spatial-temporal and spatial-spectral streams are fused and passed through classification layers, with cross-entropy used as the loss function.The streams extract their respective domain features before fusion.Source: §4.3 · p. 6
5 EXPERIMENTS
Experiments compare HetEmotionNet on DEAP and MAHNOB-HCI and ablate its datasets, streams, modalities, and graph-recurrent components. The reported comparisons favor the full model and show distinct performance effects for the tested variants.
5.1 Datasets
- The DEAP and MAHNOB-HCI datasets contain multimodal physiological signals collected during multimedia stimulation, including EEG and peripheral physiological signals.DEAP includes 32 participants and 32-channel EEG plus 8-channel PPS; MAHNOB-HCI includes 27 participants and 32-channel EEG plus 6-channel PPS.Source: §5.1 · p. 7
5.2 Baseline Models and Settings
- For both datasets, the experiments use baseline reduction and differential-entropy features in four frequency bands; DEAP trials are split into 1-second non-overlapping samples, and evaluation uses 10-fold cross-validation.The implementation passage also specifies a threshold of 5 for dividing valence and arousal into two categories.Source: §5.2 · p. 7
5.3 Results Analysis and Comparison
- HetEmotionNet achieves the best performance on DEAP and MAHNOB-HCI; on MAHNOB-HCI, its valence and arousal accuracies are 93.95% and 93.90%, versus baseline performance between 72.84% and 90.06%.The DEAP results also report the model has the lowest standard deviation among the compared models.Source: Table 2 · p. 7, Table 1 · p. 7, §5.3 · p. 7
5.4 Ablation Studies
- Removing GTN, GCN, or GRU reduces performance; the reported ablations associate these components with modeling modality heterogeneity, capturing channel correlation, and capturing temporal or spectral dependency, respectively.Figure 6 presents the component ablations.Source: §5.4 · p. 8, Figure 6 · p. 8
- HetEmotionNet-noPPS performs better than HetEmotionNet-noEEG, while the multimodal HetEmotionNet-noGTN outperforms both single-modality variants; the full model achieves the best results.The modality-ablation variants remove GTN, as the text says it models heterogeneity across modalities.Source: §5.4 · p. 8, Figure 7 · p. 8
- HetEmotionNet-noSS performs better than HetEmotionNet-noST, while the full model outperforms both variants, supporting the reported benefit of combining spatial-spectral-temporal features.The variants remove the spatial-spectral or spatial-temporal stream, respectively; Figure 8 presents the two-stream ablation.Source: §5.4 · p. 8, Figure 8 · p. 8
6 CONCLUSION
HetEmotionNet uses two streams to extract spatial-spectral-temporal features from multimodal physiological signals. Experiments on DEAP and MAHNOB-HCI indicate state-of-the-art performance on both datasets.
6 CONCLUSION
- HetEmotionNet combines spatial-temporal and spatial-spectral streams, each using GTN for modality heterogeneity, GCN for correlation, and GRU for temporal or spectral dependency.The paper describes it as a general framework for multimodal physiological signals.Source: §6 · p. 8
Supplementary Material
The supplementary material lists a notation table and a table describing frequency-band division for physiological signals across modalities.
Supplementary Material
- Table S.1 is identified as a table of notations and explanations.Source: Table S.1 · p. 11
- Table S.2 describes frequency-band division strategies across modalities, using differential-entropy features extracted with a Hanning-window Fourier transform in four bands.The table caption attributes the strategies to [14,34].Source: Table S.2 · p. 11