Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG

• VR elicits authentic emotional responses while preserving control over scenarios for applications such as healthcare and law-enforcement training.Source: §1 · p. 1
• Facial expressions are primary non-verbal channels for conveying emotional states, making assessment valuable for feedback and self-awareness.Source: §1 · p. 1
• Physiological measures capture arousal and valence but do not reveal emotional expression as a non-verbal communication cue.Source: §1 · pp. 1–2
• HMDs occlude eyes, eyebrows, and forehead, constraining camera-based recognition of subtle distinctions such as surprise versus fear.Source: §1 · p. 2
• Facial EMG complements visual FACS analysis by directly measuring muscle activity underlying facial action units.Source: §1 · p. 2
• The proposed approach fuses lower-face computer vision with upper-face EMG and evaluates seven emotions, including six basic emotions and neutral.Source: §1 · p. 2

• VR emotion-recognition datasets and systems commonly measure physiological responses, lower-face imagery, headset signals, or non-facial modalities rather than synchronized external video and upper-face EMG.Source: §2 · pp. 2–3
• Optical approaches use peri-ocular cameras, visor-mounted photo-reflective sensors, or reconstruction methods to address upper-face occlusion.Source: §2 · p. 2
• Existing facial-EMG VR systems show informative upper-face signals but often classify isolated muscle actions rather than multi-muscle emotions.Source: §2 · p. 3
• Multimodal fusion has improved robustness over unimodal streams in VR physiology and emotion-recognition pipelines.Source: §2 · p. 3
• Real-time affect inference can support adaptive interaction, including regulating game difficulty using physiological signals.Source: §2 · p. 3
• The paper contributes synchronized lower-face video and upper-face EMG, seven-class late-fusion evaluation, and direct-versus-kernelized EMG comparison.Source: §2 · p. 3

• The dataset pairs lower-face video with seven-channel upper-face EMG under HMD occlusion.Source: §3 · p. 4
• 20 volunteers participated, with 10 female participants and mean age 27.4 years.Source: §3 · p. 4
• The EmteqPRO mask recorded seven facial EMG channels covering bilateral frontalis, orbicularis oculi, zygomaticus major, and central corrugator supercilii.Source: §3 · p. 4
• Three synchronized monochrome cameras recorded lower-face video at 2448×2048 pixels and 22 fps from 0° and ±30° viewpoints.Source: §3 · p. 4
• 35 validated NimStim images provided five exemplars for each of seven emotion categories in randomized trials.Source: §3 · p. 5
• EMG and video streams were synchronized with Lab Streaming Layer, while cameras shared a common host trigger and VR event markers.Source: §3 · p. 5

Multimodal Architecture

• The late-fusion architecture combines lower-face video and upper-face EMG to classify seven emotions under HMD occlusion.Source: §4 · p. 5

EMG Processing

• EMG signals undergo filtering, epoching, clipping, rectification, and temporal binning before feature representation.Source: §4.1 · p. 6

EMG Feature Extraction

• The RBF kernel matrix captures non-linear correlations among seven EMG sensors within each 1 s window and is flattened into a 784-D vector.Source: §4.2 · p. 6
• Direct post-processed EMG uses a 28-D representation with one RMS value per channel and time bin, without kernelization.Source: §4.2 · p. 6

Visual Processing

• Three temporally aligned camera frames are paired with each EMG window, yielding 35,649 multimodal samples processed through YOLO-Pose crops and a CK+-initialized ResNet.Source: §4.3 · p. 6
• CK+ augmentation adds black patches, noise patches, and synthetic VR-headset overlays to the upper face to encourage lower-face focus under occlusion.Source: §4.3 · p. 6

Fusion Head

• The two branches project to 128-D embeddings, concatenate them, and feed a fully connected fusion head producing seven emotion logits.Source: §4.4 · pp. 6–7

Training and Evaluation

• Training uses cross-entropy with Adam for 30 epochs, with model selection by validation macro-F1 and a subject-disjoint test set.Source: §4.5 · p. 7

Baselines

• OpenFace achieved approximately 16% accuracy with F1 below 0.1, showing that conventional image-only FER struggled under HMD occlusion.Occlusion-augmented CK+ pretraining without target-domain fine-tuning also transferred poorly, reaching F1 approximately 0.08.Source: §5 · pp. 7–8

Unimodal models

• 46% accuracy and F1 = 0.43 made direct post-processed EMG the strongest unimodal representation, outperforming image-only and kernel-based EMG baselines.The representation used 28-D temporally structured features.Source: §5 · p. 8

Fusion performance

• 50% accuracy and F1 = 0.51 were achieved by ResNet-50 plus kernel EMG, the strongest reported multimodal test configuration.Late fusion combined visual and EMG representations, while the final evaluation used held-out identities.Source: §5 · pp. 7–8, Table 2 · p. 8

Generalization

• For the best ResNet-50 plus kernel-EMG model with CLAHE, test-subject F1 ranged from 0.36 to 0.66, underscoring substantial cross-subject variability.The paper motivates larger participant pools and stronger regularization to improve generalization.Source: §5 · pp. 8–10
• Validation F1 reached approximately 0.95 for kernel-EMG fusion but held-out test F1 plateaued around 0.50, indicating possible overfitting.Validation sampled the same 18 identities used for training, whereas testing was subject-disjoint.Source: §5 · pp. 9–10

Per-class effects

• Up to 32 points of per-class accuracy improvement occurred for Sadness, while Surprise improved by 19 points with upper-face EMG.The gains were largest for emotions relying on occluded upper-face action units and smaller for mouth-dominant or neutral expressions.Source: §5 · p. 10

Multimodal fusion

• OpenFace-3 collapsed to near-chance performance on lower-face-only inputs, demonstrating that full-face methods do not transfer directly to HMD-occluded VR settings.Source: §6 · p. 11

EMG representations

• Facial EMG remained a robust unimodal alternative, while its kernelized representation provided stronger fusion benefits than compact direct features.The compared representations were 28-D RMS values and 784-D kernel embeddings.Source: §6 · p. 11

Generalization

• Validation F1 near 0.95 contrasted with subject-disjoint test F1 near 0.50, reflecting overfitting to intra-subject consistencies and difficult cross-subject affect recognition.Source: §6 · p. 11, Limitations · p. 12

Dataset contribution

• The dataset contains roughly 35k paired image–EMG samples from 20 participants across seven emotions, using subject-exclusive splits.It is presented as the first synchronized external-video and upper-face-EMG dataset collected during VR exposure.Source: §6 · p. 11

Evaluation protocol

• The evaluation used a fixed subject-disjoint test set with two identities and training on the remaining 18 participants rather than full LOSO or K-fold identity cross-validation.This pragmatic design prioritized compute efficiency and broad ablation coverage while preserving subject independence in testing.Source: Limitations · pp. 11–12

Limitations

• The study’s scope is constrained by 20 participants, limited demographic diversity, posed static expressions, and independently classified temporal windows.The authors recommend larger datasets, naturalistic validation, and temporal sequence modeling.Source: Limitations · p. 12

Multimodal fusion

• Multimodal fusion substantially outperformed unimodal baselines, with the greatest gains for Happiness and Surprise.The strongest configuration produced a 10-point macro-F1 gain over the corresponding image-only baseline.Source: §6 · p. 11, §6.1 · p. 12

Future applications

• The proposed architecture is described as computationally lightweight and potentially suitable for real-time affect-adaptive VR scenarios, although deployment was not evaluated.Source: §6.1 · p. 12

Data protection

• Participants provided written informed consent for biometric and camera-data collection, analysis, and distribution, and data were stored and analysed pseudonymously.Source: ETHICAL IMPACT STATEMENT · p. 13

Privacy

• Emotion recognition can reveal intimate psychological states without explicit awareness, motivating consent and transparent data-management tools.Source: ETHICAL IMPACT STATEMENT · p. 13

Bias

• Cultural, demographic, and individual biases may produce unfair or inaccurate interpretations of emotional states.Real-time visualization of data flow and decisions is proposed as one possible mitigation.Source: ETHICAL IMPACT STATEMENT · p. 13

Interpretation

• Because the system lacks situational context, verbal content, and interaction history, its outputs should be treated as probabilistic indicators requiring human judgment.Source: ETHICAL IMPACT STATEMENT · p. 13

Regulation

• Biometric emotion-recognition systems may fall within the EU AI Act’s high-risk AI scope in employment or healthcare contexts.Potential obligations include conformity assessments, human oversight, and transparency.Source: ETHICAL IMPACT STATEMENT · p. 13

Data sharing

• Dataset access requires acceptance of an ethical-use and data-privacy-compliant agreement, with synchronized recordings, metadata, and a reference identity split included.Source: Dataset Availability and Risk-Mitigation · p. 13

• Commercial LLMs were used for grammar and spelling checks, after which the authors reviewed and edited the content.Source: Declaration on Generative AI · p. 13