PaperlayerView exact revision on arXiv

Source-linked AI summary

Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition

Philip H. Lee, Shreeram Suresh Chandra, John H. L. Hansen

arXiv:2609.31399v1cs.LGeess.SP

TL;DR

EEG–speech emotion recognition remains challenging because EEG artifacts can undermine fusion despite EEG’s complementary internal signal. EmoSpeechBrain combines differential-attention EEG encoding with gated EEG–speech alignment, achieving state-of-the-art accuracy on EAV and PME4 while outperforming unimodal baselines.

  • Problem

    EEG–speech MER remains comparatively underexplored because artifacts can corrupt fusion, while omitting EEG forfeits nonredundant internal affect information.

    Source: §I · p. 1

  • Method

    EmoSpeechBrain uses differential attention to suppress shared EEG noise and a cross-modal Gating Alignment Module to align and gate EEG and speech features.

    Source: §I · p. 2, 2) Differential Transformers Layer · p. 4, D. Gating Alignment Module · p. 4

  • Results

    Across EAV and PME4, EmoSpeechBrain achieves state-of-the-art MER accuracy and significantly outperforms unimodal speech and EEG variants.

    Source: §I · p. 2, A. Modality Ablation · p. 6, §VIII · p. 8

  • Takeaways & Limitations

    Ablations support that effective MER requires information-rich EEG representations and an explicit mechanism to bridge the modalities.

    Source: §VIII · p. 8

    The comparisons do not establish superiority over prior EEG–speech fusion systems, which were not directly evaluated under matched protocols.

    Source: §VII · pp. 7–8

Abstract

from arXiv · show

Multimodal emotion recognition (MER) increasingly pairs EEG with speech, treating internal neural signals and external vocal expression as informative views of affect. In practice, naive fusion underperforms the stronger single modality, because EEG artifacts inject noise that corrupts the shared representation. We introduce EmoSpeechBrain, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion. Its EEG encoder uses differential attention, taking the difference between two attention maps to cancel shared noise and isolate discriminative neural activity. An attention-based gating adapter aligns both modalities in a shared space and weights each one's contribution to the prediction. On two datasets - PME4 and EAV, EmoSpeechBrain improves MER accuracy by up to 12.9% over other state-of-the-art (SOTA) EEG encoders, and surpasses unimodal speech and EEG baselines by up to 13.1% and 23.1%. These results show that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.

I. Introduction

Speech and other expressive signals offer an indirect view of affect, while EEG provides a complementary internal correlate. EmoSpeechBrain addresses the challenge of combining them by suppressing EEG artifacts before aligning neural features with speech.

I. Introduction

  • Expressive modalities capture outward emotion that can be shaped or suppressed, whereas EEG provides a physiological correlate less subject to deliberate control.The introduction motivates combining EEG with speech as a more robust recognition approach.Source: §I · p. 1
  • EEG artifacts can corrupt cross-modal fusion, yet excluding EEG forfeits nonredundant internal physiological information about affect.The paper frames the problem as suppressing artifacts before fusing the preserved emotional information with speech.Source: §I · p. 1
  • EmoSpeechBrain combines differential-attention EEG encoding with a cross-modal Gating Alignment Module (GAM) to align informative EEG and speech features.Differential attention cancels shared noise across attention maps; GAM uses attention-based gating to align signal rather than artifacts.Source: §I · p. 2, Figure 1 · p. 2
  • The paper reports that its multimodal approach beats unimodal speech and EEG baselines on PME4 and EAV, with all gains statistically significant.Its contribution list also describes ablations of the model components and frequency bands.Source: §I · p. 2

II. Related Works

Prior MER work combines expressive modalities, while biosignals and EEG offer complementary routes to physiological emotion information. The paper introduces EmoSpeechBrain’s alignment approach and its frozen WavLM-Large speech encoder.

A. Common Emotion Recognition Modalities

  • Speech-and-text MER systems combine modalities, and later systems add facial expressions, but facial and vocal expressions can be voluntarily masked [23].The related-work discussion describes these approaches as capturing complementary information.Source: A. Common Emotion Recognition Modalities · p. 2

B. Biosignals for Emotion Recognition

  • EEG is presented as a noninvasive, inexpensive alternative for capturing emotion-correlated neural activity, unlike costly, cumbersome fMRI acquisition.The related-work discussion notes that EEG has gained attention as a biosignal for MER.Source: B. Biosignals for Emotion Recognition · p. 3

C. Unimodal EEG Emotion Recognition

  • Artifact-ridden EEG makes cross-modal fusion challenging, motivating a robust EEG encoder and lightweight alignment mechanism.Unimodal EEG emotion-recognition results motivate integrating EEG with speech.Source: C. Unimodal EEG Emotion Recognition · p. 3

III. Main Method

  • EmoSpeechBrain’s method comprises a speech encoder, EEG encoder, and GAM that selects relevant multimodal features and cross-aligns EEG and speech.The section overview introduces the speech encoder before detailing the EEG encoder and alignment module.Source: §III · p. 3

A. Speech Encoder

  • WavLM-Large is frozen to extract frame-level speech embeddings, which a learnable linear projection maps to the final speech representation.The projection maps the embeddings to the speech feature dimension used by the model.Source: A. Speech Encoder · p. 3

B. EEG Encoder

The EEG encoder targets motion artifacts and background noise to extract emotional representations. Its local feature extraction uses temporal and spatial convolutions, with softshrink suppressing small-magnitude noise.

B. EEG Encoder

  • The EEG encoder adapts a transformer to bypass artifact components and isolate neural signatures of emotion from noisy EEG recordings.The passage identifies motion artifacts and background noise as challenges in raw EEG.Source: B. EEG Encoder · p. 3

1) Local Feature Extraction Layer

  • Two successive convolutional layers extract local EEG features: temporal convolution captures patterns across time, and spatial convolution captures dependencies across channels.Each layer is followed by batch normalization and softshrink activation.Source: 1) Local Feature Extraction Layer · p. 3
  • Softshrink adaptively suppresses small-magnitude noise temporally and spatially, while the final EEG features match the speech features in temporal and feature dimensions.The temporal and spatial filters use kernels of sizes (1, kt) and (ks, 1), respectively.Source: 1) Local Feature Extraction Layer · p. 3

2) Differential Transformers Layer

The differential transformer layer uses paired attention maps and their difference to form sparse attention, then combines multiple heads and projects their outputs. Layer normalization, a feed-forward network, and residual connections produce the final EEG embedding.

2) Differential Transformers Layer

  • Six differential transformer layers capture global dependencies and have been empirically shown to capture more meaningful information from noisy data.Source: 2) Differential Transformers Layer · p. 3
  • The query and key projections are split into two halves while the value projection remains the same size, enabling each query-key pair to compute two attention maps.The projections use learnable weight matrices.Source: 2) Differential Transformers Layer · p. 4
  • Differencing the two attention maps through a learnable scalar λ yields a sparse attention map that enables focus on more relevant concepts.λ controls the degree of sparsity in the resulting attention map.Source: 2) Differential Transformers Layer · p. 4
  • Multiple heads compute differential attention in parallel; their outputs are concatenated and projected to jointly attend to complementary patterns across time.The differential-attention formulation is extended to multiple heads as in.Source: 2) Differential Transformers Layer · p. 4
  • Layer normalization, a softshrink feed-forward network, and residual connections produce the final EEG embedding with the same dimensional size as the speech input.The passage states that residual connections maintain gradient flow and layer normalization stabilizes training.Source: 2) Differential Transformers Layer · p. 4

C. Bidirectional Cross-Modal Attention Layer

The bidirectional attention layer lets EEG and speech embeddings attend to each other, then refines the resulting modality-specific representations for downstream fusion.

C. Bidirectional Cross-Modal Attention Layer

D. Gating Alignment Module

The Gating Alignment Module combines EEG and speech in a joint sequence, uses attention and a sigmoid-gated bottleneck to suppress irrelevant information, then restores modality-specific streams with shared context.

D. Gating Alignment Module

  • GAM concatenates EEG and speech embeddings along time and applies self-attention for direct cross-modal interaction.The joint representation captures feature interactions across both modalities.Source: D. Gating Alignment Module · p. 4
  • A sigmoid-gated bottleneck suppresses emotionally irrelevant information and provides soft feature selection over cross-modal interactions.The bottleneck compresses the feature dimension to F/2 before projecting back to F.Source: D. Gating Alignment Module · p. 4
  • The gated sequence is split into EEG and speech streams matching their original dimensions.The split divides the temporal dimension in half.Source: D. Gating Alignment Module · p. 4
  • Each stream adds its GAM output to preserve modality-specific features while incorporating shared cross-modal context before classification.Source: D. Gating Alignment Module · p. 4

E. Final Classification Layer

The classification layer combines the refined EEG and speech representations, pools their joint sequence, and maps it to an emotion prediction.

E. Final Classification Layer

IV. Experimental Setup

The experiments use paired EEG and speech from PME4 and EAV, with preprocessing, subject-held-out evaluation, significance testing, and frequency-band ablation specified.

A. Dataset Selection

  • PME4 contains audio, video, EEG, and EMG from 11 acting students and covers seven emotion categories.EEG was recorded at 5 kHz and downsampled to 1 kHz; audio was recorded at 44.1 kHz.Source: A. Dataset Selection · p. 5
  • EAV provides synchronized EEG and audio from 42 subjects, and experiments use its speaking-only portion because listening segments lack audio.The speaking-only subset provides 100 trials per subject with EEG, audio, and video jointly available.Source: A. Dataset Selection · p. 5

B. Dataset Preprocessing

  • EEG was filtered, downsampled to 200 Hz, segmented into non-overlapping two-second epochs, and normalized per channel; speech was resampled to 16 kHz and segmented into matching epochs.All available EEG channels were retained, and each speech epoch was normalized by peak absolute value.Source: B. Dataset Preprocessing · p. 5

C. Training Configurations

  • Models use leave-one-subject-out evaluation, with accuracy and F1-score averaged across test subjects; paired two-tailed t-tests assess comparisons at p < 0.05.The same protocol and test subjects are used for main results and component and frequency ablations.Source: C. Training Configurations · p. 5
  • Frequency ablation removes one EEG band at a time from test data without retraining and reports relative balanced-accuracy drops against the unaltered data.The removed bands are delta, theta, alpha, beta, and gamma.Source: C. Training Configurations · p. 5
  • EmoSpeechBrain uses Adam with a 1 × 10^-5 learning rate, 3 × 10^-4 weight decay, batch size 8, and early stopping for up to 50 epochs.Training uses a single NVIDIA RTX 4090 GPU; baselines follow their original setups.Source: C. Training Configurations · p. 5

D. Baselines

EmoSpeechBrain is evaluated against task-specific EEG encoders and EEG foundation models using the same frozen speech encoder. It achieves the highest reported performance on PME4 and EAV, with statistically significant improvements over all baselines.

D. Baselines

  • All baselines use the same frozen WavLM-Large speech encoder, while EEG encoders and cross-modal fusion components remain unfrozen during training.The comparison includes task-specific architectures and large-scale EEG foundation models.Source: A. Model Performance · pp. 5–6, D. Baselines · p. 5

A. Model Performance

  • EmoSpeechBrain achieves 0.734±0.012 balanced accuracy and 0.709±0.014 weighted F1 on PME4, and 0.799±0.043 balanced accuracy and 0.785±0.067 weighted F1 on EAV.It surpasses the strongest PME4 baseline, CBraMod, by over 10%; improvements over every baseline are statistically significant (p < 0.05).Source: A. Model Performance · p. 6, Table 1 · p. 6
  • Task-specific EEG encoders and foundation models underperform EmoSpeechBrain on both datasets; the authors attribute its advantage to jointly shaping EEG representations toward speech features.The proposed EEG Encoder and GAM support more discriminative cross-modal representations during fine-tuning.Source: A. Model Performance · p. 6

VI. Ablation Studies

Neither speech nor EEG alone matches the full model, and removing architectural components degrades performance. The ablations support combining both modalities with differential attention and explicit cross-modal fusion.

A. Modality Ablation

  • The modality comparison retains the proposed EEG encoder for EEG-only evaluation, but removes the speech branch and fusion layer; speech-only replaces fusion with a single-modality classification head.The remaining configurations are kept largely identical to the proposed method.Source: A. Modality Ablation · p. 6
  • Speech-only accuracy was 0.603 ± 0.022 on PME4 and 0.642 ± 0.010 on EAV; EEG-only accuracy was 0.503±0.030 and 0.562±0.032, respectively.Both unimodal variants show statistically significant degradation compared with the full model, supporting complementary emotional information from neural and acoustic signals.Source: A. Modality Ablation · p. 6

B. Architectural Ablation

  • Removing any component significantly degrades performance and cross-modal alignment; removing GAM yields 0.621 ± 0.023 accuracy on PME4 and 0.742±0.049 on EAV.Removing both GAM and differential attention gives the worst overall results: 0.613 ± 0.012 accuracy on PME4 and 0.721 ± 0.044 on EAV.Source: B. Architectural Ablation · p. 7

C. Frequency Ablation

Differential attention is less affected than vanilla attention by removing beta and gamma bands, while showing sensitivity to lower-frequency bands. The authors conclude that EmoSpeechBrain achieves SOTA accuracy across two datasets, but note that its fusion strategy has not been directly compared with prior EEG–speech systems.

C. Frequency Ablation

  • Removing beta and gamma reduced balanced accuracy 15.5–21.1%/14.8–20.6% for vanilla attention versus 7.2–9.1%/6.6–8.8% for differential attention on PME4/EAV.The authors suggest vanilla attention depends more on noisy high-frequency bands, whereas differential attention also uses lower-frequency bands.Source: C. Frequency Ablation · p. 7
  • Removing alpha and theta lowers differential attention’s balanced accuracy by 4.8–6.0% on PME4 and 4.9–6.1% on EAV, while affecting vanilla attention by at most 0.5%.The authors suggest this wider use of EEG may add information that speech lacks, given speech captures arousal better than valence.Source: C. Frequency Ablation · p. 7

VII. Limitations

  • Comparisons focus mainly on the EEG encoder, use the same fixed speech encoder, and do not directly compare EmoSpeechBrain with prior EEG–speech fusion systems.The authors cite limited prior work and differences in datasets, preprocessing, and evaluation protocols; future work will compare speech encoders and fusion methods under the same LOSO protocol.Source: §VII · pp. 7–8

VIII. Conclusion

  • EmoSpeechBrain achieves SOTA MER accuracy across two datasets; the ablations support information-rich EEG representations and an explicit mechanism for bridging modalities.The conclusion identifies the differential attention-based EEG encoder and GAM as the framework’s components.Source: §VIII · p. 8
Loading 2609.31399v1…