Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition

I. INTRODUCTION

• A 9 × 9 V–A head trained with a 2D-Gaussian target and KL + cross-entropy emits an affect distribution alongside the class decision.Its per-utterance entropy is weakly but consistently associated with human categorical ambiguity.Source: §I · p. 1
• The evaluation uses speaker-independent LOSO with rotating-session inner validation, keeping model selection on validation rather than the held-out test session.The contribution list argues this protocol should be the reporting standard for the task.Source: §I · p. 1
• State-space backbones are numerically on par with or slightly above the Transformer, with no latency or memory advantage at the studied utterance lengths.The paper also reports that the probabilistic V–A head adds its distributional output at no cost to classification accuracy.Source: §I · p. 2

A. Multimodal and Speech Emotion Recognition

• The authors distinguish their utterance-level text-and-speech setting from related Mamba systems for conversation-level recognition with video.They keep two-stream cross-attention fixed to isolate the temporal backbone.Source: A. Multimodal and Speech Emotion… · p. 2
• IEMOCAP results are difficult to compare across protocols: random utterance-level splits can leak speaker identity, unlike speaker-independent LOSO.The paper reports protocol, modalities, and metrics when tabulating prior results and limits comparisons to comparable settings.Source: A. Multimodal and Speech Emotion… · p. 2

B. Dimensional vs. Categorical Affect, and Soft Labels

• The paper’s joint V–A grid predicts a normalized two-dimensional distribution, unlike prior speech soft-label work on categories or independent per-attribute uncertainty.It is designed for possible use as a downstream language-model soft prompt, a design intention rather than a demonstrated result.Source: B. Dimensional vs. Categorical Affect,… · p. 2

C. State-Space Models

• Mamba variants serve as temporal backbones, but the paper makes no claim that they overcome an attention complexity wall at the studied utterance lengths.The paper reports their cost explicitly.Source: C. State-Space Models · p. 3

A. Overview

• The backbone mixes information within each modality over time, while fixed cross-attention exchanges information between modalities.The pipeline holds the other components fixed across experiments, although state-space parameters receive a ×3 learning-rate multiplier and long-sequence models use per-model batch sizes.Source: A. Overview · pp. 3–4

B. Acoustic Front-End

• The frozen Wav2Vec2 front-end produces about 550 frames for an 11-second utterance; the trainable raw-convolution option produces about 2,750.The latter is a 5× longer sequence and is the regime in which state-space backbones are expected to matter.Source: B. Acoustic Front-End · p. 4
• The raw-convolution encoder uses valid-frame masking and re-zeroing so its output is independent of batch padding.The Wav2Vec2 path instead pads each batch to a fixed 11-second cap to make per-utterance features independent of batch composition.Source: B. Acoustic Front-End · p. 4
• The paper avoids hand-crafted spectral or prosodic inputs to reduce feature-engineering confounds and enable end-to-end learned filters.A 12-dimensional prosody vector is implemented as an optional late-fused component, but no prosody-enabled configuration is evaluated.Source: B. Acoustic Front-End · pp. 4–5

C. Temporal Backbone

• A selective state-space model maps an input sequence through a latent state governed by continuous-time dynamics, then discretizes those dynamics for frame-level processing.The paper uses zero-order hold discretization to derive a linear recurrence and read-out.Source: C. Temporal Backbone · p. 5
• Mamba makes B_k, C_k, and the step size input-dependent, enabling per-token write and forget control; its recurrence uses O(T) time and memory.Mamba-2 rewrites the recurrence as structured matrix multiplication, and the paper compares v1 and v2 as realizations of the same block.Source: C. Temporal Backbone · p. 5
• The bidirectional block merges separate forward and backward scans with a linear map and residual connection.The backward scan reverses only the valid prefix, then restores time order so backward outputs align with forward outputs; a padding-invariance regression test checks this behavior.Source: C. Temporal Backbone · p. 5

D. Cross-Attention Fusion

• Each cross-attention block lets one modality query the other, and the text and audio streams are updated sequentially, text first.The audio update attends to the already-updated text stream.Source: D. Cross-Attention Fusion · p. 5
• Because text is capped at 64 tokens, cross-attention is linear in audio length; the state-space backbone replaces the potentially quadratic audio self-attention instead.The two cross-attention blocks are stacked, and their projection and feed-forward terms are also linear in sequence lengths.Source: D. Cross-Attention Fusion · pp. 5–6
• Mask-aware mean pooling excludes padded positions before the pooled text and audio streams are concatenated for the output heads.Validity masks are inverted before attention because the attention API treats True as ignore; pooling continues to use only valid positions.Source: D. Cross-Attention Fusion · p. 6

E. Valence–Arousal Probability Head

• The V–A head maps the shared fused representation to 81 logits, then reshapes their softmax into a 9 × 9 probability matrix over the affect grid.The matrix is an auxiliary output; the separate categorical head supplies the reported class decision and UA/WA/F1.Source: E. Valence–Arousal Probability Head · p. 6
• The distribution’s center of mass provides a continuous V–A read-out, while its Shannon entropy summarizes its sharpness.The ground-truth coordinates supervise the head through the soft target’s location rather than a point-regression loss.Source: E. Valence–Arousal Probability Head · p. 6
• A two-dimensional Gaussian centered on the ground-truth V–A annotation supplies a soft target, with σ = 0.5 as a researcher-chosen smoothing prior, not measured rater uncertainty.The paper reports mean within-utterance rater standard deviations of 0.31 for valence and 0.44 for arousal; the same σ is used across utterances.Source: E. Valence–Arousal Probability Head · p. 6
• The joint objective combines categorical cross-entropy on the four-way head with KL divergence from the Gaussian target to the V–A prediction, using λ = 0.3.The categorical term anchors the discrete label; the KL term is minimized when the predicted distribution matches the spread target.Source: E. Valence–Arousal Probability Head · p. 6
• The geometry-aware transport alternative was rejected: used alone it tends to collapse mass onto one cell and is seed-unstable, while adding it to KL does not sharpen distributions beyond σ alone.The stated limitation of KL is that it treats grid cells as unordered and does not encode their V–A proximity.Source: E. Valence–Arousal Probability Head · p. 7

A. Datasets

• IEMOCAP provides about 5.5k utterances across five sessions, with four emotion classes and released valence-arousal ratings averaged across raters.The four classes are angry, happy+excited, sad, and neutral; dimensional ratings use a [1, 5] scale.Source: A. Datasets · p. 7
• The 9 × 9 V–A head requires dimensional valence and arousal labels, restricting direct use to corpora that provide both; trained extensions to MSP-IMPROV and MSP-Podcast are future work.Sentiment-only corpora supply valence but not arousal; MSP-IMPROV is used here only as a zero-shot transfer probe.Source: A. Datasets · p. 7

B. Evaluation Protocol

• Five-fold leave-one-session-out evaluation makes each test fold speaker-independent, with early stopping and checkpoint selection using only the rotating validation session.Headline metrics are computed once on the untouched test session; the same five test sessions are reused across the paper’s ablations, a study-level caveat.Source: B. Evaluation Protocol · p. 7
• Unweighted accuracy is the primary metric because it averages per-class accuracies; weighted accuracy and Macro-F1 are also reported across the five test sessions.The protocol also reports continuous-read-out metrics, calibration error, and distributional entropy.Source: B. Evaluation Protocol · p. 7

C. Implementation Details

• The shared setup freezes BERT-base and Wav2Vec2, trains the remaining components, and uses a 9 × 9 V–A grid with σ = 0.5 and λ = 0.3.The raw-convolution front-end is pretrained on LibriSpeech and either frozen or fine-tuned; audio is capped at 11 seconds and text at 64 tokens.Source: C. Implementation Details · p. 7
• The paper bases comparisons on paired within-fold statistics because a separate three-seed rerun yielded 0.721 UA versus 0.730 in the headline campaign, an approximately 0.01-UA campaign difference.The reported significance tests use five session-level deltas; cluster-robust inference is described as directional support at this fold count.Source: C. Implementation Details · p. 8, (Data and Code Availability) · p. 8

A. Main Comparison

• The two backbone studies use Wav2Vec2 at T ≈550 and a trainable raw-convolution front-end at T ≈2750; within each, only the temporal backbone and V–A head presence vary.The panels are not a clean raw-convolution-versus-Wav2Vec2 comparison because both the encoder and sequence length differ.Source: A. Main Comparison · p. 8
• At long context, mamba_dual_head exceeded cross_attention by +0.030 UA (paired t p=.044; session-cluster bootstrap p<.001), but this system-level contrast includes the V–A head.The long-context single-head Mamba backbones lead the Transformer by 1.4–2.0 UA points, but that margin is not supported by both paired tests.Source: A. Main Comparison · p. 8
• At matched depth and width, short-context mamba_fusion scored 0.688 UA versus 0.679 for cross_attention; long-context single-head Mamba scored 0.714–0.719 versus 0.700.The short-context margin is within noise under both paired tests; the long-context single-head margin is supported by session-cluster bootstrap but not the paired t-test.Source: A. Main Comparison · p. 8
• Mamba-3 scored 0.715 UA, below Mamba-2, and had the highest long-sequence latency and memory, leaving no stated accuracy or efficiency reason to prefer it here.At equal depth and width, bidirectional Mamba blocks also carry about 2.6–3.2M more trainable parameters than the Transformer encoder.Source: A. Main Comparison · p. 8

B. Encoder Pre-Training Ablation

• The added Mamba-2 pre-training context module lowered the pre-training objective but did not improve downstream emotion recognition across the seven backbones.The two pre-training variants were within fold standard deviation on every model; the context module was discarded before fine-tuning.Source: B. Encoder Pre-Training Ablation · pp. 8–9, Table III · p. 9

C. Effect of the Probabilistic V–A Head

• The raw-convolution dual head lifted mamba_fusion from 0.714 to 0.730 UA, a paired gain of +0.016 UA (95% CI [+0.001, +0.030]).The gain is attributed to V–A auxiliary supervision generally: simpler V–A auxiliaries reproduced it within noise, while the distributional head’s specific value is its output format.Source: C. Effect of the Probabilistic V–A Head · p. 9

D. Analysis of the V–A Probability Matrix

• The distribution’s center of mass tracked valence and arousal at CCC 0.66 and 0.66, but within-class tracking was stronger for arousal (r = 0.52) than valence (r = 0.27).The direct regression head tracked ratings slightly better, reaching CCC 0.687/0.678.Source: D. Analysis of the V–A Probability… · p. 10
• Lowering soft-target width σ from 0.5 to 0.25 reduced mean entropy from 3.40 to 2.97 nats without significant change in classification UA.The entropy change was stable across frozen and fine-tuned encoders; the wider sweep left UA within per-fold standard deviation.Source: D. Analysis of the V–A Probability… · p. 10, Table IV · p. 11
• High-entropy outputs spread mass across multiple V–A regions and could straddle the boundary between the true and confused categories.The distributions’ entropy was only a weak indicator of correctness because correct and incorrect predictions overlapped almost entirely.Source: D. Analysis of the V–A Probability… · p. 10, Figure 8 · p. 13
• Predicted entropy was weakly but consistently associated with categorical rater ambiguity, while its association with dimensional V–A disagreement was marginally negative.The categorical association persisted after controlling for distance to the grid edge and was positive in all three seeds, but explained less than 1% of variance.Source: D. Analysis of the V–A Probability… · p. 11
• On IEMOCAP, the predicted joint distributions were nearly separable, with mean per-utterance KL divergence of 0.016 nats from the product of their marginals.The paper therefore locates the head’s value in its normalized distribution format, not measured V–A dependence.Source: D. Analysis of the V–A Probability… · pp. 11–12
• The categorical head was over-confident (ECE ≈0.20), while V–A entropy was a weaker error-ranking signal than softmax confidence and was not reliable for abstention.Abstaining on the 20% most diffuse predictions raised retained-utterance accuracy from 0.705 to 0.743, a selective accuracy rather than system-level UA.Source: D. Analysis of the V–A Probability… · p. 12, Figure 7 · p. 12

E. Pre-Specified Controls and Extensions

• Scalar regression (∆UA −0.007 [−0.019, +0.003]) and factorized marginals (−0.005 [−0.016, +0.006]) leave classification statistically unchanged.Both are equivalent within ±0.02 by TOST; the classification lift comes from V–A auxiliary supervision, not the joint grid.Source: E. Pre-Specified Controls and Extensions · p. 13
• For continuous read-out, direct regression (CCC 0.687/0.678) and marginals (0.673/0.675) exceed the joint head’s center of mass (0.661/0.661).The joint head’s case is its single normalized joint probability mass function, which supports coverage and calibration analyses.Source: E. Pre-Specified Controls and Extensions · p. 13
• Rater-matched anisotropic widths (σV = 0.31, σA = 0.44) change neither UA (−0.002, TOST-equivalent) nor read-out CCC (0.668/0.665).Within this recipe, the isotropic σ=0.5 default is not a liability.Source: E. Pre-Specified Controls and Extensions · p. 13
• Vote-mixture supervision matches UA (0.728 ± 0.007) but lowers arousal CCC (0.609 vs. 0.661), a negative result for plain mixture supervision at this scale.Distributing target mass across categories dilutes the location signal on the dimension acoustics carry best.Source: E. Pre-Specified Controls and Extensions · p. 13
• Audio-only and text-only baselines reach 0.583 and 0.652 UA, respectively; the bimodal system exceeds the stronger branch by 7.9 points.The replication campaign’s per-class recalls locate residual errors in neutral (0.584), with confusions toward happy/excited and sadness.Source: E. Pre-Specified Controls and Extensions · p. 13
• Frozen WavLM-Large features lift the identical downstream architecture to 0.766 ± 0.013 UA, +0.036 [+0.028, +0.043] over the featured arm.The encoder remains frozen; learnable softmax weights combine its 25 hidden states, and the read-out reaches CCC 0.743/0.701.Source: E. Pre-Specified Controls and Extensions · p. 13

F. Backbone Cost

• At every IEMOCAP-relevant length (T ≤ 2750), the Transformer is fastest, and its peak activation memory is lowest even at T = 8192.The measurements therefore show no latency or memory advantage for Mamba at these lengths.Source: F. Backbone Cost · pp. 13–14, Table VI · p. 14
• Mamba-2 latency stays near 29 ms through T ≤ 4096, then rises to 63.9 ms at T = 8192; Mamba-1 has a slow-kernel band around T ≈ 2750.Mamba-2’s chunked-matmul form amortizes fixed overhead.Source: F. Backbone Cost · p. 14
• Mamba-3 is costliest at long sequence lengths on both latency and memory; any latency benefit is deferred to much longer inputs.Latency was measured for isolated six-layer bidirectional encoders (d=256, batch 8, bf16) on an RTX 5090.Source: F. Backbone Cost · pp. 13–14

• The fixed Gaussian target uses no per-rater ratings, and predicted entropy is far too weak to serve as calibrated uncertainty.Entropy has a small, positive association with categorical rater ambiguity, but the authors identify per-rater category distributions as the direct route to strengthening that link.Source: §VI · p. 14
• Frozen WavLM-Large features lift the identical architecture by +3.6 UA points and the valence read-out by +0.08 CCC, making the front-end the binding constraint at this scale.The authors frame their contribution as the representation and protocol, not a leaderboard maximum.Source: §VI · p. 15
• The 9 × 9 grid resolution was not swept; finer grids would trade sharper localization for more sparsely supervised cells.Per-utterance distributions remain diffuse, while validation-fitted temperature scaling reduces classifier ECE from approximately 0.20 to approximately 0.02 without changing decisions.Source: §VI · p. 15
• The evidence base is acted, English IEMOCAP with ten speakers and reference transcripts, so it licenses no direct deployment inference.The V–A harmonization also assumes a lossy two-axis projection that may not hold equally across acted and naturalistic corpora.Source: §VI · p. 15
• At IEMOCAP utterance lengths, state-space backbones have neither lower latency nor lower peak memory than the Transformer.The Mamba-3 negative result applies to a reduced SISO variant because its MIMO backward kernel exceeds the available shared-memory limit.Source: §VI · p. 15
• Zero-shot evaluation on 7,798 MSP-IMPROV utterances yields 45.6% UA, and the entropy–ambiguity link vanishes out of domain.The authors frame this as distribution transfer under a known domain gap, not a tuned cross-corpus result.Source: §VI · p. 15
• A counseling deployment would fall under the EU AI Act’s high-risk regime and requires consent, data protection, accountability, transparency, and oversight.The intended role is decision support for a human counselor, never autonomous intervention.Source: §VI · pp. 15–16
• The proposed ECG stream, language-model soft prompt, empathetic response generation, and conformal coverage sets are future work, not implemented or evaluated here.The current study is Phase 1 and uses text and speech only.Source: §VI · p. 16

VII. CONCLUSION

• 0.730 ± 0.003 UA over three seeds is reported for the featured long-operating-point configuration, which exceeds the Transformer fusion baseline significantly under paired t-test and session-cluster bootstrap.The separate three-seed rerun reached 0.721 ± 0.002; with only five session clusters, the strictest cluster-robust tests cannot reach p < 0.05.Source: §VII · p. 16
• Mamba-1/2 are numerically on par with or slightly above the Transformer, but show no latency or memory advantage at the studied lengths; the distribution-to-language-model and physiological-stream integration remains future work.The conclusion also identifies soft-target width as a near-free sharpness knob and reports that geometry-aware transport loss did not improve on KL+CE.Source: §VII · p. 16

TECHNOLOGIES IN THE MANUSCRIPT PREPARATION

• Claude assisted with manuscript drafting, editing, programming, and code generation, with authors reporting that they reviewed and verified the content, code, and scientific claims.The authors state that code was checked through regression tests, automated table regeneration gates, and independent experiment reruns.Source: TECHNOLOGIES IN THE MANUSCRIPT… · p. 17