PaperlayerView exact revision on arXiv

Source-linked AI summary

MUPA$^{2}$E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment

Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz, Randy Gomez

arXiv:2608.15999v1cs.AI

TL;DR

Existing multimodal emotion-recognition systems often use separate modality-specific pipelines, so MUPA2E tests a shared asymmetric-attention backbone for facial video and EEG. Merged fusion at stride 30 achieved 70.07% test accuracy on the original padded recordings, falling to 62.71% after cropping trials to 20 seconds.

  • Problem

    Existing EEG–facial-video fusion pipelines apply modality-specific preprocessing and feature extraction before combining representations, motivating a shared multimodal approach.

  • Method

    MUPA2E processes facial video and EEG through a single shared asymmetric-attention backbone, evaluating unimodal and fused configurations under subject-independent testing.

  • Results

    70.07% test accuracy was achieved by merged fusion at stride 30 on original padded recordings, while 62.71% resulted after cropping all recordings to 20 seconds.

  • Takeaways & Limitations

    The results support unified processing of neural and visual signals while showing that duration-related cues materially affect emotion-recognition performance.

  • Takeaways & Limitations

    Uneven recording durations introduced a potential classification cue, so the original padded-sample accuracy is less controlled than the 20-second cropped assessment.

Abstract

from arXiv · show

Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPA\textsuperscript{2}E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video--EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride~$30$ achieves the highest validation performance and a test accuracy of $70.07\%$. Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of $20$ seconds yielded a test accuracy of $62.71\%$, providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets.

1 Introduction

MUPA²E addresses the need for passive emotion assessment by unifying facial video and EEG within a shared asymmetric-attention backbone. The framework compares unimodal and fused configurations under a stratified, subject-independent protocol.

  • Motivation: Automatic emotion recognition aims to infer affective states without explicit self-report, enabling systems that sense and respond to users’ emotions.Self-report instruments such as PANAS and the Self-Assessment Manikin are widely used for affective labeling but require explicit user participation.
  • Complementary modalities: EEG and facial video provide complementary emotion-related information, reflecting internal cortical processing and external facial expression, respectively.Combining the modalities can provide nonredundant discriminative information and improve upon unimodal baselines.
  • Limitations of prior work: Prior multimodal approaches commonly use modality-specific facial and physiological representations before applying fusion strategies.This design introduces architectural choices associated with separate modality-specific processing pipelines.
  • Proposed framework: MUPA²E processes facial video and EEG through a single shared asymmetric-attention backbone for multimodal emotion assessment.The framework evaluates unimodal facial video, unimodal raw multichannel EEG, and fused video–EEG configurations within one common architecture.
  • Evaluation: The three configurations are assessed using a stratified, subject-independent evaluation protocol.The compared settings are unimodal facial video, unimodal EEG as a raw multichannel waveform, and multimodal fusion of both modalities.

2 Related Work

Prior work has examined emotion recognition across EEG, peripheral physiological signals, and facial video, with multimodal studies combining neural and behavioral information to capture complementary affective responses. These findings motivate unified architectures that process EEG and facial video together.

  • Physiological and behavioral modalities: Emotion recognition research spans EEG, peripheral physiological signals, and facial video paired with subjective affective ratings.
  • Physiological and behavioral modalities: EEG captures central nervous system correlates of affective states with high temporal resolution and neural dynamics unavailable from peripheral biosignals.
  • Facial-video modeling: Facial video encodes affect-related cues through facial muscle activity, eye movements, head motion, and dynamic appearance changes.
  • Multimodal fusion: Combining EEG and facial information has supported continuous emotion detection and multimodal fusion, with feature-level fusion improving over unimodal baselines in multi-class classification.

3 Methodology · 3.1 Signal Representation · Facial video

The framework converts facial video and EEG into token sequences for a shared asymmetric-attention backbone, supporting unimodal and multimodal evaluation. Facial-video processing folds temporal frames into channels, adds Fourier positional features, and partitions spatial tokens into contiguous segments.

  • 3 Methodology: The shared asymmetric-attention backbone processes facial-video and EEG token sequences under a common architecture for unimodal and multimodal configurations.Each trial’s stored segments are temporally sorted, concatenated, and zero-padded when fewer segments than the maximum are available.
  • 3.1 Signal Representation: Each facial-video trial is represented as a sequence of retained 224 × 224 RGB frames, with temporal information folded into the channel dimension.This transformation is termed axis folding.
  • Facial video: K = 6 Fourier frequency bands and fmax = 10 encode each spatial position p ∈ [−1, 1]2 before classification.The representation incorporates geometric information through positional encoding.
  • Facial video: 26 positional features per token are added because the input has D = 2 spatial axes and uses D(2K + 1) = 26.The positional-feature count follows directly from the stated spatial dimensionality and Fourier-band count.
  • Facial video: N = H×W = 50176 spatial tokens are formed by flattening the facial-video spatial axes and concatenating data channels with positional features.The resulting token matrix combines visual content and spatial information per token.
  • Facial video: S = 32 contiguous spatial segments partition the token sequence, each containing ns = N/S = 1568 tokens.The segmentation is applied after spatial flattening into the full token sequence.
  • 3.1 Signal Representation: EEG unimodal processing uses the raw multichannel waveform directly, without hand-crafted feature extraction.Temporal trial segments are concatenated along the time axis.

EEG waveform

The EEG waveform is flattened into time-indexed tokens enriched with Fourier positional features, then partitioned into 32 contiguous temporal segments for processing.

  • EEG waveform: 13 positional features are added to each flattened EEG waveform token using six Fourier bands over normalized time.The time axis is flattened into L_e tokens, with waveform channels and positional features concatenated at each step.
  • EEG waveform: 32 contiguous temporal segments partition the token sequence, with the final segment zero-padded when necessary.Each segment has length n_s = ⌈L_e/32⌉.

3.2 Multimodal Channel Fusion

Multimodal channel fusion projects EEG into a 2D spatial representation and combines it with video along the channel dimension for joint processing by a shared 2D backbone. EEG normalization differs between merged and per-channel configurations.

  • Fusion mechanism: EEG is projected into the 2D spatial domain and fused with the video tensor along the channel dimension.This enables joint processing of both modalities through a shared 2D backbone.
  • EEG preprocessing: EEG preprocessing applies z-score normalization followed by min-max rescaling to [0, 1].Merged fusion normalizes the full Ce × Le amplitude grid jointly, whereas per-channel fusion applies normalization separately.
  • Spatial projection: All Ce EEG channels are treated jointly as a Ce × Le amplitude grid and projected to a single image via bilinear interpolation.

Merged · Per-channel

The merged and per-channel configurations project EEG data for fusion with axis-folded video, then apply the same tokenization and positional encoding pipeline. Per-channel processing preserves independent EEG-channel projections before stacking.

  • Merged: Each EEG channel is independently projected into a 224 × 224 image using bilinear interpolation.The independently projected channel images are then stacked along the channel dimension.
  • Merged: The projected EEG channel images are stacked along the channel dimension for multimodal processing.
  • Per-channel: The EEG image is concatenated with the axis-folded video tensor along the channel dimension.
  • Per-channel: The fusion uses ne = 1 for merged EEG and ne = Ce for per-channel EEG.
  • Per-channel: The fused tensor is flattened into N = 50176 spatial tokens and augmented with 26 Fourier positional features.
  • Per-channel: The token sequence is partitioned into S = 32 segments, each containing ns = 1568 tokens.

3.3 Asymmetric Attention

All configurations use a shared asymmetric-attention backbone with one cross-attention block and eight self-attention rounds. A shared latent state coordinates processing across segments, while each segment state aggregates information only from its corresponding token subset.

  • Backbone architecture: All configurations share an asymmetric-attention backbone of depth 1 with one cross-attention block and a feed-forward sublayer.The backbone then applies self-attention rounds, each followed by a feed-forward sublayer.
  • Backbone architecture: 8 self-attention rounds follow the initial cross-attention block, with each round containing a feed-forward sublayer.
  • Latent state: A shared latent state e(0) with d0 = 128 is used across S = 32 segments.The shared state is the basis for the cross-attention module operating across configurations.
  • Segment aggregation: Each segment state aggregates information exclusively from its corresponding token subset.

Cross-attention · Self-attention

The framework uses asymmetric cross-attention to summarize token sequences for each segment, followed by repeated self-attention for global segment-level information exchange. Regularized attention blocks aggregate the final segment states to predict binary valence.

  • Cross-attention: Cross-attention uses a single query vector for each segment.The query is e(0) ∈ R^(B×1×d0).
  • Cross-attention: Each cross-attention query attends to a key-value sequence spanning ns ≫ 1 tokens.The keys and values are ˜Ts ∈ R^(B×ns×C′).
  • Cross-attention: 8-head cross-attention processes all S segments in parallel.Each head has dimension 16.
  • Cross-attention: After cross-attention, segment states are stacked into E ∈ R^(B×S×d0).This creates a segment-level representation for subsequent processing.
  • Self-attention: R = 8 self-attention rounds enable global information exchange across all segments.Self-attention also uses 8 heads with head dimension 16.
  • Self-attention: Pre-layer normalization, residual connections, and 0.10 dropout regularize attention and feed-forward sublayers.The dropout rate is applied uniformly to both sublayer types.
  • Self-attention: Final segment states are averaged over S and passed to a linear head predicting binary valence.The classes are positive versus negative.

3.4 Augmentation & Regularization

Augmentation was applied independently to video and EEG, with temporally consistent video transformations and channel-wise EEG masking before projection. The framework also used regularization and configurable dropout/masking schemes during training.

  • Video augmentation: Video augmentation included TrivialAugment, AugMix, additive noise, center cropping, and spatial masking, with one random seed shared across frames.This shared seed preserved temporal consistency within each video sample.
  • EEG augmentation: EEG augmentation used additive noise and temporal masking independently per channel before 2D projection in multimodal configurations.Temporal masking could target the beginning, end, or center of each waveform.
  • Regularization: Regularization included label smoothing, alongside dropout probabilities for attention and feed-forward sublayers.The supplied passage identifies label smoothing and the two dropout locations but truncates the remaining regularization details.
  • Masking configuration: Masking configurations used square 32×32 Cutout on video frames, while temporal masking size was specified as a fraction of total EEG waveform length.Notation x1–x2 indicates sampling a probability p ∼U(x1, x2) per sample and applying the transform with probability p.

4 Experimental Evaluation & Results

The evaluation compares unimodal and fused video–EEG models for binary affective-state classification on DMER. Merged fusion at stride 30 performs best on the original padded recordings, but cropping trials to 20 seconds substantially reduces test accuracy.

  • Validation results: 70.84% average validation performance is achieved by merged video–EEG fusion at stride 30, the highest result across unimodal and multimodal configurations.Its validation accuracy, precision, and F1 are 70.31, 72.68, and 69.52, respectively.
  • Duration confound: 68.56% accuracy is achieved by a simple duration-based rule because 62.16% of positive trials last 30 seconds versus 25.04% of negative trials.This performance is close to the proposed model’s 70.07% accuracy, indicating that padding may provide class information.
  • Unimodal results: 69.81% average validation performance makes EEG the strongest unimodal reference, while video remains relatively stable across stride settings.Video average scores range from 68.72 at stride 10 to 69.34 at stride 15.
  • Test performance: 70.07% held-out test accuracy is obtained by the validation-selected merged-fusion configuration, which uses 1.89M parameters and 3.13 GFLOPs.The configuration remains compact at inference.
  • Duration-controlled evaluation: 62.71% test accuracy remains after cropping all recordings to 20 seconds and retraining the merged-fusion configuration.The decrease from the original padded-sample result provides a more controlled estimate with duration-related cues removed.

5 Conclusion

MUPA2E unifies facial video and EEG processing in a shared asymmetric-attention backbone, supporting unimodal and fused configurations without separate modality-specific backbones. Merged fusion reached 70.07% test accuracy under padded samples, while cropping recordings to 20 seconds reduced hold-out accuracy to 62.71%, indicating duration contributed to the original performance.

  • Unified framework: MUPA2E processes facial video and EEG through a single shared asymmetric-attention backbone for video-only, EEG-only, and fused configurations.The framework does not require separate modality-specific feature-extraction backbones.
  • Performance: 70.07% test accuracy was achieved by merged multimodal fusion at stride 30 under the original padded-sample setting.EEG provided the strongest unimodal performance, while merged fusion achieved the highest validation performance.
  • Duration control: 62.71% hold-out accuracy followed cropping all recordings to 20 seconds, indicating that recording duration contributed to the original performance.The cropped setting removed variation in recording duration as a potential cue.
Loading 2608.15999v1…