Source-linked AI summary
Robust Speech Emotion Recognition under Tone-Word Conflict: A Benchmark and Framework
Xiaojiang Peng, Dawei Huang, Yongjie Lv, Ruijie Xiong, Chunxiang Jin, Bin Li, Xiaohui Wang, Zitong Yu
TL;DR
Tone-word conflict exposes a gap in SER systems that assume vocal tone and lexical semantics agree, motivating systematic robustness evaluation. The paper introduces TWIN-SER and DAS, which separates acoustic and semantic pathways and fuses selected representations with query-based attention. Across conflict-focused, in-domain, and zero-shot settings, DAS is reported to outperform existing methods, although TWIN-SER remains limited in language coverage and size.
Problem
Existing SER systems largely assume acoustic-semantic congruence, leaving tone-word conflict insufficiently evaluated and difficult for current models to handle.
Method
The paper introduces TWIN-SER and DAS, which separately represent acoustic and semantic information, select informative embeddings, and fuse them through query-based attention.
Results
DAS consistently outperforms existing methods under tone-word conflict while achieving competitive performance on in-domain and zero-shot benchmarks.
Takeaways & Limitations
Explicitly separating and adaptively bridging acoustic and semantic pathways supports robust SER evaluation and prediction under incongruent conditions.
Takeaways & Limitations
TWIN-SER currently covers only English and Chinese and contains 378 samples, limiting statistical power for fine-grained analysis, especially for rare emotions.
Abstract
from arXiv · showhide
Speech emotion recognition (SER) is a crucial component of human-computer interaction, attracting extensive attention from both industry and academia. However, existing SER systems typically assume alignment between vocal tone and lexical semantics, overlooking the real-world scenarios that involve tone-word conflict-where the emotion conveyed by speech contradicts the literal meaning of the words. To bridge this gap, we introduce TWIN-SER (Tone-Word Incongruent SER), a benchmark for systematic evaluation under acoustic-semantic incongruence, and show that state-of-the-art models degrade severely under such incongruence. To address this, we propose DAS (Disentangled Acoustic-Semantic fusion), a framework that mitigates tone-word conflict by explicitly disentangling acoustic and semantic pathways, selecting informative high-energy embeddings, and adaptively fusing them via a lightweight query-based attention mechanism. Specifically, DAS comprises three crucial modules: i) a heterogeneous feature extraction module that separately captures complementary acoustic and semantic representations from raw input; ii) a high-energy embedding selection module that identifies and retains the most discriminative embeddings; and iii) a Q-Former combination module that bridges the two pathways through cross-attention, enabling robust emotion prediction under incongruent conditions. Extensive experiments demonstrate that DAS consistently outperforms existing methods in tone-word conflict scenarios, as well as in standard in-domain and zero-shot settings. Our code and datasets are available at https://github.com/24DavidHuang/FAS
I. INTRODUCTION
Existing speech emotion recognition systems often assume vocal tone aligns with lexical meaning, leaving them vulnerable to tone-word conflict. The paper introduces TWIN-SER to evaluate this problem and DAS to disentangle and adaptively fuse acoustic and semantic information.
- Motivation: Tone-word conflict occurs when acoustic emotion contradicts an utterance’s literal meaning, as in sarcasm or a resentful congratulation.This setting is common in real-world communication but is overlooked by conventional SER systems.
- Limitations of existing methods: Speech-text pretrained methods can exhibit semantic bias, while self-supervised methods produce entangled representations that conflate affective and semantic information.These structural weaknesses make conflicting emotional cues difficult to resolve.
- Benchmark gap: Existing benchmarks lack a high-density, controlled setting for systematically evaluating SER robustness under acoustic-semantic conflict.MELD and IEMOCAP contain only sporadic and structurally uncontrolled conflict samples.
- DAS framework: DAS disentangles acoustic and semantic pathways, selects discriminative high-energy embeddings, and adaptively fuses them through lightweight query-based cross-attention.Its modules separately extract heterogeneous features, retain informative embeddings, and use a Q-Former to weight modalities under conflict.
- TWIN-SER: 378 high-quality conflict samples form TWIN-SER, a multilingual and speaker-diverse benchmark created with LLM-assisted utterance generation, text-to-speech synthesis, and manual verification.The benchmark is designed to evaluate robustness and study interactions between acoustics and semantics.
- Reported outcomes: DAS consistently outperforms state-of-the-art baselines in conflict-centric settings while remaining competitive on in-domain and zero-shot benchmarks.The claimed evaluation scope includes both tone-word conflict scenarios and standard SER settings.
II. RELATED WORK
SER has progressed from handcrafted acoustic features to large pretrained encoders, but unresolved feature variability and representation biases still complicate reliable emotion recognition. In particular, semantic bias and entangled prosodic representations remain central challenges.
- Traditional SER: Early SER systems used handcrafted features such as pitch, energy, and MFCCs with classifiers including SVMs and GMMs.OpenSMILE toolkits provided feature sets such as eGeMAPS and ComParE.
- Pretrained representations: Large self-supervised encoders such as WavLM, Whisper, and HuBERT extract high-level semantic-acoustic representations and help address SER data scarcity.Research has also studied transferring, distilling, and adapting these representations for emotion recognition.
- Open challenges: No universally optimal acoustic feature set reliably distinguishes emotions across speakers, linguistic content, and speaking rates.Speech variability remains a core unresolved challenge despite advances in representation learning.
- Open challenges: Whisper and CLAP exhibit semantic bias, whereas SSL encoders produce entangled representations that conflate affective prosody with phonetic content.These limitations make prosodic emotion cues difficult to separate from semantic or phonetic information.
B. SER in Acoustic-Semantic Conflict
Prior conflict-focused resources are limited, motivating TWIN-SER and DAS, which separately model acoustic and semantic information before selectively fusing emotionally informative representations.
- Existing acoustic–semantic conflict datasets are limited in scale and scope, leaving a need for controlled, high-density evaluation resources.The paper positions TWIN-SER as a more controlled and challenging testbed for this setting.
- DAS uses heterogeneous feature extraction to capture complementary semantic and paralinguistic streams from Whisper and MingTok.The streams differ in sequence length and feature dimension before fusion.
- Salient patch selection scores tokens by ℓ2 energy and retains the highest-scoring acoustic and semantic patches.The non-parametric top-k strategy reduces sequence length while preserving emotionally relevant temporal information.
- The Q-Former fusion module concatenates selected sequences and uses learnable queries with cross-attention to produce fused representations.Residual connections and a feed-forward network follow the cross-attention operation.
- A mean fused vector is passed to an MLP prediction head that outputs probabilities across seven emotion categories.
C. The TWIN-SER Benchmark
TWIN-SER is a dedicated benchmark for tone–word conflict, constructed through scenario generation, metadata annotation, audio synthesis, and human verification.
- TWIN-SER is designed as a four-stage benchmark for systematic robustness evaluation under acoustic–semantic conflict.Its construction emphasizes logical coherence, high conflict density, and plausible real-world scenarios.
- Scenario generation creates utterances whose semantic emotion differs from the underlying true emotion specified by the scenario.The paper illustrates this with birthday wishes delivered in a perfunctory tone toward someone disliked.
- Metadata annotation records ground-truth emotion, semantic emotion inferred from text, and conflict level for each utterance.
- The benchmark contains 378 high-quality audio samples synthesized with 21 speaker timbres.
- TWIN-SER compares ground-truth and semantic-only emotion labels across seven categories, with semantic labels predominantly neutral and ground-truth labels more evenly distributed.
A. Experiment Setup
The experiments train DAS on a heterogeneous corpus assembled from multiple open-source datasets and evaluate it using in-domain, conflict-focused, and zero-shot benchmarks.
- The training corpus aggregates multiple open-source datasets into more than 66 hours of heterogeneous emotional speech.It includes MER2024, IEMOCAP, and CMU-MOSEI among its component resources.
- All training samples from the aggregated datasets are used for model optimization.The corpus includes parallel utterances from English and Chinese native speakers.
- Weighted accuracy and weighted F1 are the primary evaluation metrics.
- General in-domain evaluation uses MELD, RAVDESS, and ESD test sets, while TWIN-SER measures robustness to tone–word conflict.
- Zero-shot generalization is evaluated on the Emo-Emilia and EmoDB test sets.
- DAS uses Whisper-large semantic features and MingTok-Audio acoustic features, then trains the fusion model from scratch on precomputed features.Experiments use a fixed random seed of 42 and eight NVIDIA A6000 GPUs.
B. Evidence of Degradation under Tone-Word Conflict
Existing SER methods are evaluated across multiple datasets and show dramatic accuracy degradation on the tone-word conflict TWIN-SER benchmark.
- The comparison covers self-supervised, speech-text, neural-tokenizer, emotion-specific, and multimodal audio models across MELD, RAVDESS, ESD, and TWIN-SER.
- TWIN-SER is presented as the benchmark for evaluating severe tone-word conflicts, with results reported alongside existing datasets.
- All models except neural audio tokenizers perform strongly on the three existing datasets.Neural audio tokenizers underperform because they rely exclusively on paralinguistic and prosodic features without lexical semantics.
- HuBERT drops from 80.1% accuracy on ESD to 32.9% on TWIN-SER.
C. Comparison with State-of-the-Art Methods
DAS consistently performs strongly across standard and conflict-rich benchmarks, with its selective patch strategy and Q-Former fusion providing particular advantages on TWIN-SER.
- DAS achieves superior or competitive accuracy and F1 across MELD, RAVDESS, ESD, and TWIN-SER, with its largest advantage on TWIN-SER.
- On TWIN-SER, Whisper reaches 47.26% accuracy and 44.97% F1, while DAS exceeds it by over 12% and 10% in the respective measures.
- Fusion and selection ablations: DAS reaches 59.38% accuracy versus 55.99% for full attention and 55.47% for random selection on TWIN-SER.
- Backbone generalizability: Whisper paired with alternative acoustic encoders consistently gains performance, while XCodec2+Whisper approaches the default DAS configuration on TWIN-SER.XCodec2+Whisper achieves 58.33% WA and 54.46% WF1, compared with 59.38%/55.08% for MingTok+Whisper.
- Patch selection: The default TWIN-SER patch configuration, kaco = 8 and ksem = 16, achieves 59.38% WA and 55.08% WF1.Increasing acoustic patches can degrade performance by introducing misleading prosodic cues.
- Learnable queries: Across learnable-query settings, MELD and RAVDESS remain robust to Nq, while TWIN-SER is evaluated through average accuracy across the three datasets.
E. Zero-Shot SER Evaluation
DAS is evaluated zero-shot on TWIN-SER and two unseen acted-emotion datasets, achieving its strongest reported advantage on TWIN-SER.
- The zero-shot evaluation includes TWIN-SER, Emo-Emilia, and EmoDB, with models trained without exposure to TWIN-SER.
- DAS outperforms Whisper by over 12% on TWIN-SER under zero-shot evaluation.
- On Emo-Emilia and EmoDB, DAS achieves 51.14%/42.92% and 68.10%/65.07%, respectively.
F. Visualization
Visualization analyses show that DAS forms more separated emotion representations and reduces confusion among major and high-arousal emotion classes on TWIN-SER.
- UMAP visualization: DAS produces reasonably separated UMAP clusters corresponding to distinct emotion categories under tone-word conflict.
- Confusion matrices: DAS achieves the highest accuracy on major classes, particularly anger, sadness, and neutral.
- Confusion matrices: DAS shows less confusion between high-arousal emotions such as anger and surprise than the compared models.
- Confusion matrices: No method correctly predicts any fear or disgust samples, partly because these classes contain only 33 and 28 samples and have overlapping acoustic expressions.
V. CONCLUSION
The paper addresses acoustic-semantic conflict in SER with DAS and TWIN-SER, reporting improved performance under in-domain and zero-shot conditions. The authors also identify dataset and integration limitations that motivate broader evaluation and end-to-end development.
- Contributions: DAS disentangles and bridges acoustic and semantic pathways through query-based fusion, while TWIN-SER evaluates robustness to their conflict.The framework targets semantic bias and entangled representations in conflicting speech conditions.
- Results: FAS outperforms state-of-the-art methods under both in-domain and zero-shot conditions.
- Limitations: TWIN-SER covers only English and Chinese and contains 378 samples, limiting statistical power for fine-grained analysis, especially for rare emotions.
- Limitations: DAS is a lightweight standalone module using pre-extracted features, leaving its integration into end-to-end audio-language models unexplored.Future work could investigate end-to-end fine-tuning with large multimodal backbones.