PaperlayerView exact revision on arXiv

Source-linked AI summary

Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models

Xiutian Zhao, Luqi Sun, Björn Schuller, Berrak Sisman

arXiv:2608.17102v1cs.CLeess.ASeess.IV

TL;DR

It remains unclear whether multimodal foundation models use shared internal components or modality-specific pathways to recognize emotion in speech and faces. Across three models, the paper identifies and intervenes on emotion-sensitive decoder neurons, finding partial emotion-aligned overlap and bidirectional causal transfer across modalities.

  • Problem

    It remains unclear whether speech and facial emotion recognition in multimodal foundation models relies on shared internal components or modality-specific pathways.

  • Method

    The study identifies acoustic and visual emotion-sensitive decoder neurons from speech and facial emotion tasks and tests them through within- and cross-modal interventions.

  • Results

    Across three models, emotion-matched acoustic and visual neurons overlap sparsely, show similar layer distributions, and causally transfer emotion-specific effects bidirectionally across modalities.

  • Takeaways & Limitations

    Speech and facial affect processing in these models partially converges onto sparse decoder-level components that can be localized and manipulated without training.

Abstract

from arXiv · show

Modern multimodal foundation models (MFMs) have made rapid progress on tasks requiring integrated perception across speech, vision, and language, including emotion recognition. However, it remains unclear whether they recognize speech and facial emotion through shared affective functional units or modality-specific pathways. We explore emotion-sensitive neurons (ESNs), sparse decoder neurons selectively associated with emotion categories, in three MFMs: Gemma-4-12B-it, MiniCPM-o-4.5, and Qwen2.5-Omni-7B. Using speech emotion recognition and facial expression recognition as complementary probes, we identify acoustic and visual ESNs. Visual ESNs are causally meaningful: deactivating them selectively impairs recognition of the associated facial emotion, whereas steering their activations selectively enhances recognition of that emotion relative to other emotion categories. Acoustic and visual ESNs further show emotion-matched overlap and similar layer-wise distributions, indicating partial structural alignment between affective representations across speech and faces. Finally, cross-modal interventions reveal bidirectional causal transfer: ESNs identified from one modality produce emotion-specific effects when applied to the other. Our findings provide one of the first cross-modality activation-level analyses of affective functional units in MFMs, suggesting that speech and facial emotion recognition partially converge onto sparse decoder-level components that can be localized and manipulated without training.

I. INTRODUCTION

This study investigates whether speech and facial emotion recognition in multimodal foundation models relies on related, causally meaningful emotion-sensitive neurons. It identifies visual ESNs, compares them with acoustic ESNs, and tests their functional sharing through cross-modal interventions.

  • Motivation: Emotion perception is multimodal, but behavioral success does not reveal whether models use related internal components for the same emotion across speech and faces.Speech and facial behavior are emphasized as practically important affective channels, motivating analysis beyond task-level performance.
  • Method: The study defines ESNs as sparse decoder MLP units selectively associated with affective input categories and testable through intervention.Deactivation probes whether suppressing a neuron set disrupts recognition, while steering probes whether amplifying it enhances the corresponding emotion.
  • Visual ESNs: Visual ESNs are identified from correctly recognized facial-expression examples and causally tested during facial emotion recognition.This extends prior causal analysis of speech-derived acoustic ESNs to vision-derived emotion-sensitive neurons.
  • Cross-modal alignment: Acoustic and visual ESNs show sparse but consistently stronger neuron-level overlap for matched than mismatched emotions, alongside broadly similar layer-wise distributions.These structural comparisons assess whether speech and facial emotion representations converge onto related decoder components.
  • Cross-modal causality: Cross-modal causal interventions transfer acoustic ESN masks to facial recognition and visual ESN masks to speech recognition to test functional sharing beyond structural overlap.The study presents this as a cross-modality analysis of affective functional units in multimodal foundation models.

II. RELATED WORK · III. METHOD

Prior work has established neuron-level interpretability and emotion-related representations across vision, language, speech, and multimodal systems, but ESN analyses have largely remained single-modality. The method applies one activation-based probing pipeline to speech and facial emotion recognition, testing sparse ESNs through deactivation and steering.

  • II. RELATED WORK: Neuron-level specialization has been studied as a route to interpretable transformer-based models across vision, language, and multimodal systems.This literature examines individual units or sparse unit groups that encode human-interpretable concepts.
  • II. RELATED WORK: Affective modeling has examined emotional and paralinguistic representations or control in speech systems through style embeddings and continuous affective methods.
  • II. RELATED WORK: Probing and dissection studies have analyzed phonetic, speaker, prosodic, and broader acoustic concepts encoded inside audio and multimodal models.
  • II. RELATED WORK: Recent work has identified emotion-sensitive neurons in large audio-language models for speech emotion recognition and emotional voice conversion, while primarily studying one modality.
  • III. METHOD: The shared pipeline collects decoder MLP activations from correctly recognized speech and facial emotion examples before selecting sparse emotion-sensitive neurons contrastively.
  • III. METHOD: Causal evaluation tests selected emotion-sensitive neurons by deactivating them and steering their activations in speech emotion recognition and facial expression recognition.

A. Emotion-Conditioned Activation Collection

The study collects emotion-conditioned activations by running unintervened MFMs on correctly answered speech and facial emotion-recognition examples. It records decoder MLP gate activity over valid token positions and normalizes neuron firing frequency by modality and emotion.

  • A. Emotion-Conditioned Activation Collection: Only correctly recognized examples from multiple-choice speech and facial emotion-recognition tasks are retained for emotion-sensitive neuron identification, reducing noise from model failures.The protocol is applied separately to each task modality and emotion category.
  • A. Emotion-Conditioned Activation Collection: Decoder MLP modules are instrumented to record activated SwiGLU gate outputs for each layer, neuron, and valid token position.A binary mask excludes padding and instruction-only tokens when counting activations.
  • A. Emotion-Conditioned Activation Collection: For each modality-emotion pair, normalized activation probability measures how frequently a neuron fires across valid positions.The calculation uses positive firing counts divided by the number of valid positions.

B. Emotion-Sensitive Neuron Identification

Emotion-sensitive neurons (ESNs) are identified with Contrastive Activation Margin, which assigns neurons to emotions based on firing frequency and activation-probability margins. The method selects the top 0.5% of decoder MLP neurons for each modality–emotion pair and compares them with equal-sized random masks.

  • ConAct-based identification: ConAct assigns each neuron to the emotion for which it fires most frequently and scores it by the margin between the highest and second-highest activation probabilities.The procedure is defined separately for each modality, layer, and neuron.
  • ConAct-based identification: For each modality–emotion pair, decoder MLP neurons are ranked by their emotion-specific ConAct scores, and the top fraction is selected as the ESN set.The selected set for emotion e is denoted I(q,e).
  • Modality-specific ESNs: 0.5% of all decoder MLP neurons are selected as visual ESNs in facial-expression recognition, with the same strategy used for acoustic ESNs in speech-emotion recognition.I(SER,e) denotes A-ESNs, while I(FER,e) denotes V-ESNs.
  • Control masks: Random control masks contain the same number of decoder MLP neurons, selected uniformly without using activation statistics.These masks provide a control for the activation-statistics-based ESN selection procedure.

C. Causal Interventions: Deactivation and Steering · IV. EXPERIMENT SETUP

The study tests emotion-sensitive neurons through deactivation and steering applied to decoder MLP gate activations. These complementary interventions assess emotion-selective causal effects against same-size random-mask controls within and across modalities.

  • C. Causal Interventions: Deactivation and Steering: Interventions modify decoder MLP gate activations before the standard SwiGLU computation.The modified gate vector is used in down proj(˜g_l,t ⊙ up proj(x_l,t)).
  • C. Causal Interventions: Deactivation and Steering: For deactivation, selected ESN gate activations are set to zero while all other neuron activations remain unchanged.The mask assigns r_l,n = 0 for selected neurons and r_l,n = 1 otherwise.
  • C. Causal Interventions: Deactivation and Steering: Deactivation and steering provide complementary loss-of-function and gain-of-function tests of ESN causal roles.Deactivation suppresses selected neurons, whereas steering enhances their activations.
  • C. Causal Interventions: Deactivation and Steering: For steering, selected ESN activations are increased multiplicatively while unselected neurons retain their original scaling.The steering mask uses s_l,n(α) = 1 + α for selected neurons and 1 otherwise.
  • C. Causal Interventions: Deactivation and Steering: Unless otherwise stated, steering uses a multiplicative gain of α = 0.5.The modified gate vector is ˜g^steer_l,t = g_l,t ⊙ s_l(α).
  • C. Causal Interventions: Deactivation and Steering: An ESN set is considered causally emotion-relevant when its effects are emotion-selective and exceed those of same-size random-mask controls.The protocol is applied to both within-modality validation and cross-modal transfer.

A. Datasets and Models … D. Evaluation Metrics

The study evaluates speech and facial emotion recognition across three open-source multimodal foundation models using controlled datasets, deterministic inference, sparse decoder-neuron interventions, and emotion-specific metrics. Its evaluation spans five shared emotions and distinguishes matched-emotion effects from cross-emotion effects relative to the unintervened model.

  • A. Datasets and Models: Speech emotion recognition uses MSP-Podcast, while facial expression recognition uses AffectNet, with five shared categories: anger, fear, happiness, neutral, and sadness.Evaluation samples include 150 utterances per emotion for SER and 300 facial images per emotion for FER.
  • A. Datasets and Models: The evaluation covers Gemma-4-12B-it, MiniCPM-o-4.5, and Qwen2.5-Omni-7B, which accept acoustic and visual inputs and expose decoder MLP modules for activation intervention.All models use the same multiple-choice question-answering protocol, with randomized emotion-option order and option-index outputs to reduce position and label bias.
  • B. Prompting and Decoding: Inference uses deterministic greedy decoding with temperature 0, a 20-token maximum, and lightweight postprocessing to extract predicted option indices.Randomness remains in dataset sampling, option ordering, and random-mask construction; fixed seeds control sampling and ordering unless otherwise specified.
  • C. Neuron Selection and Intervention Settings: Unless stated otherwise, ConAct selects the top r = 0.5% of decoder MLP neurons for each emotion, and steering applies a multiplicative gain with α = 0.5.These settings define the default sparse emotion-sensitive neuron selection and intervention configuration.
  • D. Evaluation Metrics: Baseline emotion-recognition performance is reported as unweighted average recall over the five emotion categories.Intervention effects are measured against the original unintervened model under the same evaluation protocol.
  • D. Evaluation Metrics: Self-Emotion measures post-intervention accuracy on test examples matching the mask emotion, averaged across target emotions.For each target mask, Avg. Cross-Emotion averages accuracies over all non-target emotion subsets before averaging across target emotions.
  • D. Evaluation Metrics: The Self-Cross Gap compares matched- and non-matched-emotion recognition, with negative values indicating selective degradation under deactivation and positive values indicating selective improvement under steering.Table changes are reported relative to the corresponding unmasked UAR under the same evaluation protocol.

V. RESULTS · A. Mono-Modal Causal Validation of ESNs

Mono-modal causal validation shows that ESNs identified from speech or facial expressions selectively influence recognition of their matched emotions across all three MFMs. Deactivation and steering produce opposite, emotion-specific effects, establishing a basis for cross-modal analyses.

  • A. Mono-Modal Causal Validation of ESNs: Across all three MFMs, deactivating FER-derived V-ESNs and SER-derived A-ESNs selectively reduces recognition of their matched emotion.The matched-emotion decrease exceeds effects on non-matched emotions, yielding negative Self-Cross Gaps.
  • A. Mono-Modal Causal Validation of ESNs: Steering V-ESNs and A-ESNs selectively enhances responses to their associated emotion relative to non-matched emotions.Steering produces positive Self-Cross Gaps, opposite to deactivation.
  • A. Mono-Modal Causal Validation of ESNs: Same-size random masks have only mild effects, supporting the specificity of the observed causal changes.The comparison distinguishes ESN interventions from generic masking effects.
  • A. Mono-Modal Causal Validation of ESNs: FER heatmaps show broadly matched-emotion structure, with deactivation often producing the largest or among the largest diagonal decreases.The diagonal pattern indicates that ESN removal most strongly affects the corresponding target emotion.
  • A. Mono-Modal Causal Validation of ESNs: Steering often increases the corresponding target-emotion response more than most non-target responses, especially for anger, fear, happiness, and sadness.Neutral exhibits a less uniformly targeted pattern, plausibly because it reflects an absence of overt affect.
  • A. Mono-Modal Causal Validation of ESNs: The facial-expression results confirm that ESNs can be identified from facial-expression activations, extending prior evidence for SER-derived A-ESNs in speech emotion recognition.Together, the approximately opposite and emotion-selective effects support ESNs as emotion-specific causal components.

B. Acoustic and Visual ESNs Show Cross-Modal Alignment

Across three multimodal foundation models, acoustic and visual emotion-sensitive neurons show sparse, emotion-matched overlap and broadly similar layer-wise distributions. Despite being identified from different modalities, tasks, and datasets, these patterns suggest partial convergence onto sparse decoder-level emotion units.

  • Emotion-level overlap: A-ESN and V-ESN overlaps are sparse but diagonal-dominant, with matched speech and facial emotions sharing more neurons than mismatched pairs.Mismatched emotion-pair similarities are often near zero across all three models.
  • Emotion-level overlap: The strongest matched-category overlaps vary by model: Gemma-4-12B-it aligns most clearly for anger, happiness, and sadness, while MiniCPM-o-4.5 aligns most clearly for sadness.MiniCPM-o-4.5 also shows additional alignment for anger, happiness, and neutral.
  • Layer-wise localization: Both A-ESNs and V-ESNs span multiple decoder layers, are relatively rare early, and become more frequent in middle and later layers across all models.The FER and SER profiles are broadly similar within each model, while late-layer concentration may partly reflect category-level decision or answer-selection processes.
  • Cross-modal interpretation: Emotion-matched overlap and comparable layer-wise profiles persist despite A-ESNs and V-ESNs being identified from different tasks, datasets, and input modalities.This makes it less likely that the alignment is explained solely by the selection procedure.
  • Cross-modal interpretation: The structural findings suggest that speech and facial emotion recognition may converge onto a sparse set of decoder MLP units associated with emotion-relevant processing.The interpretation is supported by emotion-specific causal effects, including under cross-modal transfer, and is not attributed solely to lexical association.

C. Cross-Modal ESN Interventions Reveal Bidirectional Transfer

Cross-modal interventions show that emotion-sensitive neurons identified from speech or faces produce structured, emotion-specific effects when transferred to the other modality. Bidirectional deactivation and steering effects, alongside near-baseline random controls, indicate partial rather than complete cross-modal sharing.

  • Cross-modal transfer: Cross-modal masks transfer emotion-specific effects across all three models, while random interventions generally remain close to the unmasked baseline.SER-derived A-ESNs were applied during FER, and FER-derived V-ESNs during SER.
  • Speech-to-face transfer: Applying A-ESNs to FER reduces matched self-emotion performance under deactivation and increases it under steering, producing positive Self-Cross Gaps.Transfer was strongest for Gemma-4-12B-it and more modest but consistent for MiniCPM-o-4.5 and Qwen2.5-Omni-7B.
  • Face-to-speech transfer: Applying V-ESNs to SER produces the same qualitative pattern: deactivation reduces corresponding self-emotion recognition, whereas steering improves it and yields positive Self-Cross Gaps.Random deactivation remains close to zero across models, supporting the specificity of the transferred effects.
  • Emotion-specific patterns: For MiniCPM-o-4.5, transferred interventions show clearer matched-emotion structure than random controls, but non-negligible off-diagonal effects remain.Deactivation preferentially reduces the corresponding self-emotion, while steering increases its response.

VI. CONCLUSION

Across three multimodal foundation models, sparse acoustic and visual emotion-sensitive neurons showed emotion-matched overlap, comparable layer-wise distributions, and partial bidirectional causal transfer between speech and facial emotion recognition. The work provides an activation-level mechanistic analysis of affective functional units and motivates controllable affective interfaces for future calibration, debiasing, and personalization.

  • Conclusion: Across Gemma-4-12B-it, MiniCPM-o-4.5, and Qwen2.5-Omni-7B, SER and FER identified sparse acoustic and visual ESNs from decoder MLP activations.The ESNs were investigated as potentially shared rather than exclusively modality-specific functional units.
  • Conclusion: A-ESNs and V-ESNs exhibited emotion-matched overlap and broadly comparable layer-wise distributions, tending to appear in middle and later decoder layers.These findings indicate partial structural alignment between affective representations across speech and faces.
  • Conclusion: Cross-modal interventions revealed partial bidirectional causal transfer: A-ESNs affected facial emotion recognition, while V-ESNs affected speech emotion recognition.Matched self-emotion effects were stronger than corresponding random-mask controls.
  • Conclusion: The study provides one of the first cross-modality activation-level analyses of affective functional units across speech and facial emotion recognition in MFMs.It offers a mechanistic view of how MFMs internally organize affective information across modalities.
  • Conclusion: Future work could develop controllable affective interfaces using shared ESNs to calibrate, debias, or personalize emotion perception across modalities.This direction moves beyond post-hoc localization.

APPENDIX · A. Datasets and Models

This appendix section reports dataset statistics and documents the sources and licenses of the three evaluated multimodal foundation models.

  • A. Datasets and Models: Table III reports utterance and image counts for each emotion in the datasets.The supplied passage identifies the table’s subject but does not provide the individual counts.
  • A. Datasets and Models: Table IV documents the sources and licenses associated with the three evaluated MFMs.The supplied passage does not list the individual sources or license terms.
Loading 2608.17102v1…