Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing

Siddharth Siddharth; Tzyy-Ping Jung; Terrence J. Sejnowski

Source-linked AI summary

Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing

Siddharth Siddharth, Tzyy-Ping Jung, Terrence J. Sejnowski

arXiv:1905.07039v1May 16, 2019cs.LGcs.HCeess.SPstat.ML

TL;DR

Deep-learning advances in vision have not translated adequately to bio-sensing affective computing, partly because bio-sensing datasets are smaller and less consistent. This study applies deep-learning feature extraction to bio-sensing and video data from four emotion datasets, evaluates individual modalities and fusion, and reports improved prior results, new benchmarks, and cross-dataset evaluations.

Problem
Deep-learning research has translated less adequately to bio-sensing affective computing than to vision, where large and more consistent datasets are easier to obtain.
Method
The study extracts deep-learning features from multiple bio-sensing and video modalities, evaluates them separately and in fusion, and applies the framework across four datasets.
Results
The methods outperform previously reported results across the four datasets, while multimodal fusion improves classification in almost all cases.
Takeaways & Limitations
Off-the-shelf deep-learning features can support affective computing across heterogeneous bio-sensing datasets and can be fused across modalities.

Abstract

from arXiv · show

In recent years, the use of bio-sensing signals such as electroencephalogram (EEG), electrocardiogram (ECG), etc. have garnered interest towards applications in affective computing. The parallel trend of deep-learning has led to a huge leap in performance towards solving various vision-based research problems such as object detection. Yet, these advances in deep-learning have not adequately translated into bio-sensing research. This work applies novel deep-learning-based methods to various bio-sensing and video data of four publicly available multi-modal emotion datasets. For each dataset, we first individually evaluate the emotion-classification performance obtained by each modality. We then evaluate the performance obtained by fusing the features from these modalities. We show that our algorithms outperform the results reported by other studies for emotion/valence/arousal/liking classification on DEAP and MAHNOB-HCI datasets and set up benchmarks for the newer AMIGOS and DREAMER datasets. We also evaluate the performance of our algorithms by combining the datasets and by using transfer learning to show that the proposed method overcomes the inconsistencies between the datasets. Hence, we do a thorough analysis of multi-modal affective data from more than 120 subjects and 2,800 trials. Finally, utilizing a convolution-deconvolution network, we propose a new technique towards identifying salient brain regions corresponding to various affective states.

I. INTRODUCTION

The study addresses the limited translation of deep learning from vision to bio-sensing affective computing by analyzing multimodal bio-sensing and video data across four datasets. It evaluates individual modalities, feature fusion, and dataset differences to assess whether deep-learning methods improve emotion classification.

Research gap: Deep-learning research has advanced vision-based affective computing more rapidly than bio-sensing research.Bio-sensing data are harder to collect and less standardized across devices than image and video data.
Motivation: Multiple bio-sensing modalities may provide independent and complementary information for emotion classification.Modalities can differ in their performance for arousal and valence, motivating feature fusion.
Study scope: The framework evaluates deep-learning and traditional feature-fusion methods on four publicly available multimodal datasets.The datasets include DEAP, AMIGOS, MAHNOB-HCI, and DREAMER, with differing preprocessing and affective-label scales.
Contributions: The study reports improved results over previous studies, evaluates multimodal fusion and transfer across datasets, and examines deep learning with limited data.It also presents a technique for identifying salient brain regions associated with affective states.
Dataset variability: The datasets differ in recording systems, trial durations, baseline availability, and modalities, complicating cross-dataset generalization.These inconsistencies include variable trial lengths and differences in EEG channel counts.

MAHNOB-HCI Dataset

The MAHNOB-HCI materials and related-work context highlights heterogeneous sensing configurations and prior multimodal benchmarks. The study uses complete datasets and disjoint evaluation partitions while comparing separate modalities with fused features.

Dataset and modalities: MAHNOB-HCI uses EEG, ECG, GSR, respiration, and skin-temperature recordings in the reported related work.The dataset is part of a broader comparison of sensing modalities and evaluation methods.
Dataset limitations: The datasets differ in trial duration, baseline availability, stimulus type, and EEG acquisition systems.Such differences affect the comparability and generalizability of traditional feature-based algorithms.
Evaluation design: The study uses complete datasets with disjoint training, validation, and test partitions, then evaluates modalities separately and in combination.Valence, arousal, liking, and emotion are classified because dominance is not consistently reported across datasets and prior studies.

III. RESEARCH METHODS

The research methods section describes feature extraction procedures for each bio-sensing modality and for frontal videos.

Method overview: The study extracts features from each bio-sensing modality and from frontal videos.This section frames the subsequent modality-specific feature extraction methods.

A. EEG feature extraction

The study extracts EEG features from preprocessed signals using traditional spectral measures and conditional-entropy measures of interactions between EEG channels.

EEG data were bandpass-filtered and artifact-corrected before conversion to the frequency domain for feature extraction.
Traditional PSD features were computed for theta, alpha, and beta bands using overlapping half-second windows averaged over each trial.
Conditional-entropy features quantify information about the uncertainty in one EEG signal given another through mutual-information estimation.
Mutual information was estimated from Parzen-window density approximations, using Gaussian windows for the density function.
496 conditional-entropy features were calculated for DEAP and MAHNOB-HCI, compared with 91 for AMIGOS and DREAMER because channel counts differ.

3) EEG-PSD images-based Deep Learning features:

The proposed EEG representation converts band-specific PSD topographies into RGB heat maps, then uses a pretrained convolutional network to extract features and support brain-region analysis.

PSD values from theta, alpha, and beta bands were interpolated into two-dimensional heat maps that preserve EEG electrode topography.
Theta, alpha, and beta heat maps were assigned red, green, and blue color maps, respectively, and combined into one RGB image.
The combined image encodes spatial and spectral information through color-dependent interactions among the three EEG bands.
A pretrained VGG-16 network resized the RGB image to 224×224×3 and extracted 4,096 features from its penultimate layer.
Successive EEG-PSD images can be processed with a reverse deep-learning network to identify salient brain regions activated for affective responses.

B. ECG/PPG-based feature extraction

ECG and PPG processing extracts heart-rate and heart-rate-variability features from filtered signals by detecting peaks and analyzing inter-beat intervals.

Both ECG and PPG were processed with the same two feature-extraction methods across the four datasets for measurement consistency.
Signals were filtered with a 0.25-second moving-average window and scaled between 0 and 1 before peak detection.
A minimum 0.5-second interval between successive peaks was used to remove false positives during detection.
Heart rate was represented by the total number of detected peaks per minute, while successive peak differences produced inter-beat intervals for HRV.

2) Extracting deep-learning-based spectrogram features of ECG/PPG:

The study extracts ECG/PPG and GSR features in frequency and time domains, converting low-frequency signals into spectrogram images for deep-learning representation.

ECG/PPG spectrogram features: ECG/PPG signals are converted into 0–5 Hz spectrogram images to extract deep-learning-based frequency-domain features.The frequency range is restricted because ECG/PPG signals are low-frequency.
GSR features: GSR features are extracted using both time-domain and frequency-domain methods.The time-domain features capture signal peaks and statistical trends, while the frequency-domain features use spectrogram representations.
GSR features: GSR spectrograms cover 0–2 Hz, and their VGG-16 features are reduced to 30 components before concatenation with time-domain features.PCA reduces the deep-learning representation before fusion with the statistical GSR features.

D. Frontal video-based feature extraction

Frontal videos are processed through automated facial landmark detection, normalized geometric features, and deep-learning representations of face images.

Video preprocessing: Frontal video trials are sampled at one frame per second for facial-feature extraction.The first frame from each second is selected, while extreme image regions are excluded.
Facial points localization based features: Chehra detects 49 facial landmarks automatically, which are used to calculate 30 distance-based facial features.The features include distances among facial regions such as the eyebrows, eyes, nose, and lips.
Facial points localization based features: Facial features are normalized by detected-face height and width to remove variation caused by camera distance.Across each trial, the method calculates the mean, 95th percentile, and standard deviation of the features.
Deep-learning face features: Deep networks pretrained on the VGG-Faces dataset are applied to resized face regions to extract image-based representations.The selected face regions are resized to 224×224×3 before processing.
Dynamics of the EEG/Face features using deep-learning: A temporal method represents each second of a trial with EEG-PSD or face-region images before sending reduced features to an LSTM network.The approach produces 60×N features, where N is the trial length in seconds.

IV. EVALUATION

The evaluation compares individual and fused modalities across four datasets, including combined-dataset and transfer-learning settings. Deep-learning representations improve class separation over traditional EEG-PSD features in the reported visualization.

Evaluation design: The study evaluates deep-learning and traditional feature extraction, modality fusion, individual modalities, combined datasets, and transfer learning.Single-dataset evaluations use leave-one-subject-out testing, while combined-dataset and transfer-learning evaluations use 80/20 splits with 10-fold cross-validation.
Dataset characteristics: The four datasets differ in emotion-class distributions, trial durations, baseline availability, and EEG acquisition systems.These differences motivate evaluations across datasets and between datasets.
Visualizing class-separability using the traditional vs. deep-learning features: The EEG-PSD visualization includes two-class valence and four-class emotion representations.The reported comparison contrasts traditional EEG-PSD features with VGG features derived from combined RGB heat-map images.
Visualizing class-separability using the traditional vs. deep-learning features: VGG features from combined EEG-PSD images separate valence and emotion classes better than direct EEG-PSD features in t-SNE visualizations.Traditional EEG-PSD features form subject-specific clusters and do not separate valence or emotion classes as effectively.

B. Evaluating individual modality performance

Individual modalities produced above-chance affective classification across four datasets, with EEG generally strongest among biosignals and frontal video often highly accurate. Multi-modal fusion usually improved accuracy and exceeded prior results or established benchmarks.

Individual modalities: Individual-modality results exceeded previously reported results across multiple categories and all four datasets.CNN-based features contributed most to the improvement, and every modality performed substantially above chance.
Individual modalities: EEG was the best-performing bio-sensing modality, while cardiac and GSR features also performed well despite having fewer channels.Cardiac features used PPG for DEAP and ECG for the other datasets.
Individual modalities: Frontal video achieved high affective-classification accuracy for three datasets and often surpassed bio-sensing modalities.The comparison concerns individual modality performance.
Evaluation setup: Valence, arousal, and liking used two classes with 50% chance accuracy, whereas emotion used four classes with 25% chance accuracy.Reported values in the modality tables are mean percentage accuracy followed by mean F1-score.
Multi-modal fusion: Combining features from multiple modalities increased classification accuracy in almost all cases.Bio-sensing fusion improved accuracy in many cases across all four datasets, while EEG-plus-face fusion also exceeded individual modalities.
Multi-modal fusion: EEG-plus-face LSTM features outperformed the best previously reported DEAP accuracy, while multi-modal methods set new benchmarks for AMIGOS, MAHNOB-HCI, and DREAMER.The LSTM used time-varying EEG and face-video features for DEAP; it could not be applied to AMIGOS or MAHNOB-HCI because trial lengths varied widely, and DREAMER lacked video.

D. Evaluating the classification performance using combining datasets and transfer learning

The study combined datasets and used transfer learning to test whether deep-learning features remained effective across differences in channels, trial lengths, image resolution, and cardiac modalities. Performance on held-out datasets stayed above chance and generally exceeded prior studies, although some modalities declined under cross-dataset conditions.

Transfer learning: Transfer-learning evaluation tested whether deep-learning features were independent of EEG channel count, trial length, video resolution, and cardiac modality.The authors trained an ELM using data from more than one dataset and evaluated it on datasets whose trials were not used for training.
Transfer learning: Held-out-dataset results were consistently above chance and better than many previous studies, especially for ECG/PPG and GSR modalities.The evaluation used combined datasets and transfer learning.
Limitations: Some modalities showed slight performance decreases across datasets, attributed to varying trial lengths and restricting features to VGG-based representations common across datasets.The authors contrasted this restriction with additional features such as conditional entropy, HRV, face localization, and related methods.
Salient regions: The salient-brain-region analysis identified high frontal-lobe activation for low/high valence and arousal states in DEAP.This finding is reported in the accompanying brain-region figure.

E. Identifying the salient brain regions that contribute towards processing various emotions

A convolution-deconvolution network was applied to EEG-PSD images to identify brain regions with salient activity for affective states. The analysis highlighted distributed arousal processing and produced single-image representations spanning subjects and trials.

Network and representation: A reverse VGG convolution-deconvolution network was added to the pretrained VGG feature extractor to investigate brain regions contributing to emotional responses.The reverse network was placed before the final max-pooling step.
Implications: The study reports that deep-learning methods extracted more meaningful features and performed better than traditional features across the evaluated modalities.The proposed features were used for the classification analyses and the brain-region investigation.
Network and representation: RGB-combined EEG images from every second of low/high valence and arousal trials were convolved and deconvolved to estimate salient regional variations.Theoretically, areas with the greatest cross-trial variation represented regions most receptive to the affective state.
Findings: Arousal processing was much more widely distributed across the brain than valence in the high-versus-low difference image.The regional organization was interpreted according to the EEG 10-20 system.
Findings: The method represents salient areas across the brain, subjects, and trials with a single image for each affective measure.The authors present this as a starting point for investigating emotion generation and processing using EEG.