Source-linked AI summary
Dual-Scale State-Space Modeling with Speaker-Wise Dynamic CRF for Speech Emotion Recognition in Conversation
Guan-Hua Wen, Kuan-Yu Chen, Hou-Chiang Tseng
TL;DR
Conversational SER must reconcile multi-scale acoustic evidence with cross-speaker context and within-speaker emotion evolution. DSSM-CRF separates contextual encoding from speaker-wise dynamic CRF decoding, achieving strong results on IEMOCAP and MELD, with matched controls supporting complementary gains from factorization and CRF modeling.
Problem
Conversational SER must model acoustic evidence across temporal scales while capturing shared dialogue context and speaker-specific emotion transitions.
Method
DSSM-CRF uses self-supervised acoustic representations, dual-scale bidirectional state-space encoders, and independent speaker-wise dynamic CRF chains with contextual transition residuals.
Results
75.81 UA and 74.90 WA on IEMOCAP, and 54.72 WA and 49.31 WF1 on MELD, with matched controls supporting complementary gains from speaker-wise factorization and CRF modeling.
Takeaways & Limitations
Speaker-wise factorization preserves full-dialogue contextual information while restricting structured transitions to each speaker’s own trajectory.
Abstract
from arXiv · showhide
Conversational speech emotion recognition must reconcile acoustic evidence across temporal scales with two interaction processes: cross-speaker contextual influence and within-speaker emotion evolution. We propose DSSM-CRF, an audio-only architecture that explicitly separates these processes. Bidirectional state-space models encode fused self-supervised speech representations at frame and dialogue scales, so each utterance representation captures local prosody and context from all speakers. The decoder then orders each speaker's utterances into an independent dynamic conditional random field chain. Consecutive utterances in a speaker's chain form a transition pair whose score combines a corpus-level transition matrix with a residual predicted from the two contextualized utterances. An auxiliary objective supervises whether each pair changes emotion but does not participate in Viterbi inference. Thus, interlocutor turns affect contextual emotion scores without being treated as transitions in another speaker's emotion trajectory. DSSM-CRF achieves 75.81% UA and 74.90% WA on IEMOCAP, and 54.72% WA and 49.31% WF1 on MELD. Matched controls demonstrate complementary gains from speaker-wise factorization and CRF modeling.
1. INTRODUCTION
Conversational SER requires modeling brief acoustic events, long-range dialogue context, and speaker-specific emotional evolution. DSSM-CRF addresses these demands with dual-scale state-space encoding and speaker-wise dynamic CRF decoding.
- Conversational emotion depends on acoustic realization and surrounding context, requiring audio-only modeling of short events, long-range context, and speaker-specific evolution.
- Existing methods separately model temporal scales, speaker dependencies, interaction structure, pretrained representations, or structured sequence decoding.
- DSSM-CRF uses WavLM features with bidirectional state-space encoders at frame and dialogue scales.The dual-scale design targets local prosody and conversational context while separating contextual representation from structured decoding.
- The speaker-wise dynamic CRF combines a global transition prior with pair-conditioned residual transitions.Its independent speaker chains restrict transition semantics to each speaker’s emotion trajectory while retaining contextual information from the dialogue.
- The study evaluates audio-only DSSM-CRF on IEMOCAP and MELD using ablations, matched topology and decoder controls, backbone comparisons, and Overall, Shift, and Inertia analyses.
2. METHODOLOGY
DSSM-CRF models conversational emotion from fused acoustic representations at frame and utterance scales, then contextualizes them across the dialogue. Speaker-wise dynamic CRF chains structure transitions only between a speaker’s own turns while retaining cross-speaker information in contextual representations.
- Problem definition: DSSM-CRF infers utterance emotions from waveforms and speaker identities while modeling shared dialogue context and speaker-specific transitions.The task assigns an emotion label to every dialogue position.
- Acoustic feature extraction: Learned fusion combines the final twelve WavLM-Large layers into a frame sequence for subsequent acoustic modeling.The fused sequence H_t contains one D-dimensional hidden state per frame.
- Acoustic feature extraction: Multi-scale attentive statistics pooling uses one, two, and four contiguous partitions to preserve global and coarse temporal structure in a 256-dimensional utterance representation.Seven partition descriptors are concatenated before learned projection.
- Dual-scale context modeling: Bidirectional state-space encoders model frame dynamics and dialogue context, with frame recurrence reset per utterance and dialogue recurrence spanning ordered turns.Both encoders use two bidirectional state-space layers; cross-speaker turns affect contextual representations without creating speaker-chain edges.
- Speaker-wise dynamic CRF: The decoder forms an independent chain for each speaker, combining emotion emissions with pair-conditioned transition residuals and a global transition matrix.Viterbi decoding runs independently on each speaker chain, while contextual representations still encode the full dialogue.
- Speaker-wise dynamic CRF: An auxiliary shift objective supervises whether same-speaker edges change emotion, while inference uses the CRF chains without that auxiliary loss.The shift target distinguishes emotion changes from inertia on same-speaker edges.
3. EXPERIMENTAL SETUP
The experiments evaluate DSSM-CRF on IEMOCAP and MELD using audio-only inputs, staged training, held-out validation, and matched ablations. Evaluation reports overall and subset metrics, varies shift-loss weighting and backbone choice, and excludes ensembles.
- Datasets: IEMOCAP uses four emotion classes, 5,531 retained utterances, 151 dialogue boundaries, and five-fold session rotation.Each fold assigns three sessions to training, one to validation, and one to testing.
- Datasets: MELD uses the official seven-class split with 9,988 training, 1,108 validation, and 2,610 test utterances.Audio is truncated to 30 seconds, and neither dataset uses transcripts, sentiment annotations, or video.
- Training: Training first adapts the speech encoder and task heads with a class-weighted utterance objective, then freezes the selected encoder while training the state-space model and CRF.The staged schedule progressively unfreezes the encoder before reusing frozen representations for the structured model.
- Evaluation: UA, WA, and WF1 are reported, while Shift and Inertia subsets classify same-speaker predecessors by whether the gold emotion changes.WF1 and subset metrics do not affect model selection.
- Ablations: Ablations remove frame or dialogue state-space branches, contextual transition residuals, or shift supervision, with topology compared against a single-chain CRF.The single-chain control connects adjacent dialogue turns regardless of speaker, whereas the speaker-wise model preserves separate chains.
- Robustness: λ_shift is selected from six values, with 0.2 used for reported models; backbone robustness compares wav2vec 2.0 Large, HuBERT-Large, and WavLM-Large without ensembles.Backbones are adapted within the same fold and paired with otherwise unchanged DSSM-CRF models.
4. RESULTS AND ANALYSIS
DSSM-CRF performs strongly on IEMOCAP and MELD, while ablations and matched controls show complementary contributions from contextual transitions, speaker-wise chains, and CRF decoding. Shift-loss and backbone analyses further characterize robustness and the persistence–shift trade-off.
- 75.81 UA, 74.90 WA, and 74.67 WF1 are obtained on IEMOCAP with λshift = 0.2.The five-fold evaluation uses the reported full model.
- 54.72 WA and 49.31 WF1 are obtained on MELD, with γ = 0.4 selected by validation Overall WA.
- 1.72/2.11 UA/WA points are lost on IEMOCAP when contextual transitions are removed, the largest component-ablation effect.The same removal reduces I-UA by 2.44 points, while MELD ablation effects vary by metric.
- 59.17 Shift UA versus 79.06 Inertia UA yields a 19.89-point gap, showing abrupt within-speaker changes remain harder than persistence.Removing shift supervision lowers Shift UA by 0.98 points, whereas removing contextual transitions raises Shift UA by 0.57 but lowers Overall and Inertia UA.
- 0.99/0.91 UA/WA points are gained by speaker-wise chains over a single dialogue chain, although the single chain is 1.47 points higher on S-UA.Speaker-wise chains also improve I-UA by 1.49 points.
- 78.56% UA and 77.91% WA peak at λshift = 0.2, while the evaluated sweep spans only 0.44 UA and 0.40 WA points.Across adapted backbones, DSSM-CRF gains 15.22/18.61 UA/WA points for wav2vec 2.0, 10.90/13.10 for HuBERT, and 13.54/14.12 for WavLM.
5. CONCLUSION
The paper concludes that DSSM-CRF combines dual-scale state-space encoding with speaker-wise dynamic CRF decoding for audio-only conversational emotion recognition. Results support transfer across dyadic and multi-party conversations, while motivating improved transition calibration and causal dialogue modeling.
- DSSM-CRF combines self-supervised acoustic representations, dual-scale bidirectional state-space encoding, and speaker-wise dynamic CRF decoding.
- Results demonstrate transfer across dyadic and multi-party conversations.
- Ablations identify pair-conditioned transitions as the largest contributor, while matched controls support speaker-wise factorization and CRF training.
- The Shift–Inertia gap motivates improved transition calibration, and offline encoding motivates causal dialogue modeling for practical streaming SER.