Source-linked AI summary
Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG
Birgit Nierula, Karam Tomotaki-Dawoud, Mert Akguel, Mustafa Tevfik Lafci, David Przewozny, Anna Hilsmann, Peter Eisert, Sebastian Bosse
TL;DR
HMDs occlude upper-face cues, limiting conventional facial emotion recognition in VR. The paper fuses lower-face video with upper-face EMG using a synchronized 20-participant dataset and evaluates seven emotions. Multimodal fusion outperforms unimodal baselines, while performance remains bounded by dataset size and posed, static emotion elicitation.
Problem
HMD occlusion removes upper-face information needed by conventional facial emotion recognition, while physiological measures do not capture emotional expression as a communication cue.
Method
The paper combines lower-face video with seven-channel upper-face EMG in a late-fusion architecture using kernelized EMG representations and a synchronized dataset.
Results
The best multimodal configuration achieved a 10-point macro-F1 gain over the corresponding image-only baseline, with greatest gains for Happiness and Surprise.
Takeaways & Limitations
Upper-face EMG provides complementary information under HMD occlusion, supporting multimodal facial emotion recognition in VR.
Takeaways & Limitations
The dataset includes only 20 participants, and instructed imitation of static high-intensity expressions may not transfer directly to naturalistic VR interactions.
Abstract
from arXiv · showhide
Head-mounted displays (HMDs) fundamentally limit emotion recognition in virtual reality (VR): by occluding the upper face, they render conventional image-based facial expression analysis incomplete, particularly for applications requiring real-time affective assessment. We address this challenge by fusing lower-face video with facial electromyography (EMG) from the occluded upper face to classify seven emotional categories (six basic emotions plus neutral). We introduce a synchronized multimodal dataset from 20 participants, pairing lower-face video with seven-channel upper-face EMG elicited by validated emotion stimuli. Under subject-independent test, our proposed late-fusion architecture merging convolutional visual embeddings with RBF-kernel EMG representations achieves 51% macro-F1, outperforming both image-only (41%) and EMG-only (43%) baselines. These results demonstrate that upper-face EMG provides robust complementary information under HMD-induced visual occlusion and establish a foundation for multimodal emotion recognition in naturalistic VR environments. This approach facilitates affect-adaptive applications, including communication training and therapeutic interventions. The dataset will be shared upon request under an ethical-use agreement.
1. INTRODUCTION
VR can elicit authentic emotional responses useful for communication training, but HMDs occlude upper-face cues needed for conventional facial emotion recognition. The study therefore combines visible lower-face video with upper-face EMG to classify seven emotional states.
- VR elicits authentic emotional responses while preserving control over scenarios for applications such as healthcare and law-enforcement training.
- Facial expressions are primary non-verbal channels for conveying emotional states, making assessment valuable for feedback and self-awareness.
- Physiological measures capture arousal and valence but do not reveal emotional expression as a non-verbal communication cue.
- HMDs occlude eyes, eyebrows, and forehead, constraining camera-based recognition of subtle distinctions such as surprise versus fear.
- Facial EMG complements visual FACS analysis by directly measuring muscle activity underlying facial action units.
- The proposed approach fuses lower-face computer vision with upper-face EMG and evaluates seven emotions, including six basic emotions and neutral.
2. Related Work
Prior VR emotion-recognition work uses physiological signals, optical work-arounds, reconstructed facial regions, and specialized datasets, but synchronized external video with upper-face EMG remains unavailable. This work targets discrete multi-class emotion recognition under HMD occlusion through multimodal late fusion.
- VR emotion-recognition datasets and systems commonly measure physiological responses, lower-face imagery, headset signals, or non-facial modalities rather than synchronized external video and upper-face EMG.
- Optical approaches use peri-ocular cameras, visor-mounted photo-reflective sensors, or reconstruction methods to address upper-face occlusion.
- Existing facial-EMG VR systems show informative upper-face signals but often classify isolated muscle actions rather than multi-muscle emotions.
- Multimodal fusion has improved robustness over unimodal streams in VR physiology and emotion-recognition pipelines.
- Real-time affect inference can support adaptive interaction, including regulating game difficulty using physiological signals.
- The paper contributes synchronized lower-face video and upper-face EMG, seven-class late-fusion evaluation, and direct-versus-kernelized EMG comparison.
3. Dataset
The dataset synchronizes multi-view lower-face video with seven-channel upper-face EMG during controlled VR emotion-expression trials. It uses validated stimuli, subject-specific recordings, and synchronized streams for multimodal analysis.
- The dataset pairs lower-face video with seven-channel upper-face EMG under HMD occlusion.
- 20 volunteers participated, with 10 female participants and mean age 27.4 years.
- The EmteqPRO mask recorded seven facial EMG channels covering bilateral frontalis, orbicularis oculi, zygomaticus major, and central corrugator supercilii.
- Three synchronized monochrome cameras recorded lower-face video at 2448×2048 pixels and 22 fps from 0° and ±30° viewpoints.
- 35 validated NimStim images provided five exemplars for each of seven emotion categories in randomized trials.
- EMG and video streams were synchronized with Lab Streaming Layer, while cameras shared a common host trigger and VR event markers.
4. Method: Multimodal Two-Stream Classification
The method processes lower-face video and upper-face EMG in separate streams, maps both to 128-dimensional embeddings, and classifies their concatenation under HMD occlusion. EMG kernelization represents inter-muscle structure before late fusion.
- Multimodal Architecture: The late-fusion architecture combines lower-face video and upper-face EMG to classify seven emotions under HMD occlusion.
- EMG Processing: EMG signals undergo filtering, epoching, clipping, rectification, and temporal binning before feature representation.
- EMG Feature Extraction: The RBF kernel matrix captures non-linear correlations among seven EMG sensors within each 1 s window and is flattened into a 784-D vector.
- EMG Feature Extraction: Direct post-processed EMG uses a 28-D representation with one RMS value per channel and time bin, without kernelization.
- Visual Processing: Three temporally aligned camera frames are paired with each EMG window, yielding 35,649 multimodal samples processed through YOLO-Pose crops and a CK+-initialized ResNet.
- Visual Processing: CK+ augmentation adds black patches, noise patches, and synthetic VR-headset overlays to the upper face to encourage lower-face focus under occlusion.
- Fusion Head: The two branches project to 128-D embeddings, concatenate them, and feed a fully connected fusion head producing seven emotion logits.
- Training and Evaluation: Training uses cross-entropy with Adam for 30 epochs, with model selection by validation macro-F1 and a subject-disjoint test set.
5. Results
Subject-independent results show that multimodal late fusion improves emotion recognition under HMD occlusion, with kernelized EMG providing the strongest complement to lower-face video. Performance gains are clearest for emotions involving occluded upper-face action units, although cross-subject variability and validation–test gaps remain substantial.
- Baselines: OpenFace achieved approximately 16% accuracy with F1 below 0.1, showing that conventional image-only FER struggled under HMD occlusion.Occlusion-augmented CK+ pretraining without target-domain fine-tuning also transferred poorly, reaching F1 approximately 0.08.
- Unimodal models: 46% accuracy and F1 = 0.43 made direct post-processed EMG the strongest unimodal representation, outperforming image-only and kernel-based EMG baselines.The representation used 28-D temporally structured features.
- Fusion performance: 50% accuracy and F1 = 0.51 were achieved by ResNet-50 plus kernel EMG, the strongest reported multimodal test configuration.Late fusion combined visual and EMG representations, while the final evaluation used held-out identities.
- Generalization: For the best ResNet-50 plus kernel-EMG model with CLAHE, test-subject F1 ranged from 0.36 to 0.66, underscoring substantial cross-subject variability.The paper motivates larger participant pools and stronger regularization to improve generalization.
- Generalization: Validation F1 reached approximately 0.95 for kernel-EMG fusion but held-out test F1 plateaued around 0.50, indicating possible overfitting.Validation sampled the same 18 identities used for training, whereas testing was subject-disjoint.
- Per-class effects: Up to 32 points of per-class accuracy improvement occurred for Sadness, while Surprise improved by 19 points with upper-face EMG.The gains were largest for emotions relying on occluded upper-face action units and smaller for mouth-dominant or neutral expressions.
6. Discussion
The study finds that multimodal fusion of lower-face video and upper-face EMG improves emotion recognition under HMD occlusion, while cross-subject generalization remains challenging. It also contributes a synchronized VR dataset and identifies limitations involving dataset diversity, ecological validity, and temporal modeling.
- Multimodal fusion: OpenFace-3 collapsed to near-chance performance on lower-face-only inputs, demonstrating that full-face methods do not transfer directly to HMD-occluded VR settings.
- EMG representations: Facial EMG remained a robust unimodal alternative, while its kernelized representation provided stronger fusion benefits than compact direct features.The compared representations were 28-D RMS values and 784-D kernel embeddings.
- Generalization: Validation F1 near 0.95 contrasted with subject-disjoint test F1 near 0.50, reflecting overfitting to intra-subject consistencies and difficult cross-subject affect recognition.
- Dataset contribution: The dataset contains roughly 35k paired image–EMG samples from 20 participants across seven emotions, using subject-exclusive splits.It is presented as the first synchronized external-video and upper-face-EMG dataset collected during VR exposure.
- Evaluation protocol: The evaluation used a fixed subject-disjoint test set with two identities and training on the remaining 18 participants rather than full LOSO or K-fold identity cross-validation.This pragmatic design prioritized compute efficiency and broad ablation coverage while preserving subject independence in testing.
- Limitations: The study’s scope is constrained by 20 participants, limited demographic diversity, posed static expressions, and independently classified temporal windows.The authors recommend larger datasets, naturalistic validation, and temporal sequence modeling.
- Multimodal fusion: Multimodal fusion substantially outperformed unimodal baselines, with the greatest gains for Happiness and Surprise.The strongest configuration produced a 10-point macro-F1 gain over the corresponding image-only baseline.
- Future applications: The proposed architecture is described as computationally lightweight and potentially suitable for real-time affect-adaptive VR scenarios, although deployment was not evaluated.
ETHICAL IMPACT STATEMENT
The ethical impact statement addresses biometric-data safeguards, privacy, bias, interpretive uncertainty, and regulatory obligations. It also describes controlled dataset access intended to reduce misuse.
- Data protection: Participants provided written informed consent for biometric and camera-data collection, analysis, and distribution, and data were stored and analysed pseudonymously.
- Privacy: Emotion recognition can reveal intimate psychological states without explicit awareness, motivating consent and transparent data-management tools.
- Bias: Cultural, demographic, and individual biases may produce unfair or inaccurate interpretations of emotional states.Real-time visualization of data flow and decisions is proposed as one possible mitigation.
- Interpretation: Because the system lacks situational context, verbal content, and interaction history, its outputs should be treated as probabilistic indicators requiring human judgment.
- Regulation: Biometric emotion-recognition systems may fall within the EU AI Act’s high-risk AI scope in employment or healthcare contexts.Potential obligations include conformity assessments, human oversight, and transparency.
- Data sharing: Dataset access requires acceptance of an ethical-use and data-privacy-compliant agreement, with synchronized recordings, metadata, and a reference identity split included.
Declaration on Generative AI
The authors used commercial LLMs for grammar and spelling checks and reviewed and edited the resulting content.
- Commercial LLMs were used for grammar and spelling checks, after which the authors reviewed and edited the content.