Source-linked AI summary
SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis
Shicheng Ma, Wenqian Cui, Irwin King
TL;DR
Existing speech sentiment systems often lose prosodic cues and use labels too coarse for interpersonal stances. SpeechSense introduces an 8-class, prosody-focused dataset with human validation, and its best audio model reaches 56.95% Acc versus 26.76% for the best text model.
Problem
Existing speech sentiment research relies on text-centric ASR pipelines that discard prosody and uses basic-emotion labels instead of nuanced interpersonal stances.
Method
SpeechSense defines an 8-class interpersonal-stance taxonomy and builds a curated prosody-focused dataset from synthesized speech with rigorous human validation.
Results
56.95% Acc for the best audio model versus 26.76% Acc for the best text model demonstrates stronger performance with acoustic access.
Takeaways & Limitations
Acoustic access is necessary for detecting fine-grained interpersonal stances, while semantic cues alone are insufficient across model scales.
Takeaways & Limitations
SpeechSense uses entirely synthesized English speech, creating a potential domain gap from real speech and limiting coverage to eight stances and 30 voice profiles.
Abstract
from arXiv · showhide
Recent advances in AI have revolutionized speech processing, yet effective speech understanding requires discerning not just what is said, but how it is said. Speech Sentiment Analysis plays a critical role in decoding these paralinguistic cues for diverse real-world applications such as recruitment and customer service. However, existing Speech Sentiment Analysis research faces two primary limitations. First, dominant approaches rely on text-centric pipelines that cascade Automatic Speech Recognition with text analysis. This process inevitably discards essential acoustic features like prosody and tone, failing to capture attitudinal meanings in acoustically ambiguous utterances. Second, current benchmarks suffer from a mismatch in label granularity, prioritizing basic emotions (e.g., happy, sad) over the nuanced interpersonal stances (e.g., confident, impatient) necessary for social sensitivity. To address these limitations, we propose a novel dataset, SpeechSense, for fine-grained speech sentiment analysis. Specifically, we define a specialized 8-class taxonomy of interpersonal stances detectable primarily through prosodic cues beyond lexical content alone. We then construct a curated dataset based on this taxonomy, built from high-fidelity speech synthesis and rigorous human validation. Comprehensive experiments across multi-modal LLMs, text-only LLMs, and speech encoders demonstrate that models with acoustic access consistently outperform text-only baselines. These results empirically validate the primacy of acoustic cues in detecting subtle speaker attitudes, highlighting the necessity of SpeechSense. Dataset and supplementary materials are available at https://github.com/Sher13cked/SpeechSense.
1 Introduction
SpeechSense addresses limitations in speech sentiment analysis caused by text-centric pipelines and coarse labels by introducing an 8-class, paralinguistic-focused dataset. It combines high-fidelity synthesized speech with human validation and evaluates acoustic access across multiple model types.
- Motivation: Effective speech understanding requires discerning paralinguistic information in addition to linguistic content.Paralinguistic information conveys meanings beyond what is said, including how it is said.
- Limitations: ASR-to-text sentiment pipelines discard prosody, tone, pauses, stress, and tempo, while transcription errors can further degrade sentiment analysis.Acoustic delivery can distinguish meanings such as passionate advocacy from impatient irritation even when the words are identical.
- Limitations: Existing datasets often use broad sentiment or basic emotions, creating a mismatch with nuanced interpersonal speaker states.Examples of coarse labels include positive versus negative and happy versus sad.
- SpeechSense: SpeechSense introduces a specialized 8-class taxonomy—Confident, Nervous, Warm, Apathetic, Passionate, Impatient, Sarcastic, and Neutral—designed for prosodic detection beyond lexical content.The dataset prioritizes prosodic features over semantic content and addresses the scarcity of human-recorded speech through high-fidelity synthesis and human validation.
- Evaluation: Experiments across multi-modal LLMs, text-only LLMs, and speech encoders demonstrate that acoustic access is necessary for fine-grained sentiment detection.The validation supports the dataset’s focus on paralinguistic cues and acoustic information.
2 Related Work
Prior work on speech sentiment analysis builds on emotion-recognition datasets and compares cascade pipelines with end-to-end acoustic approaches. Recent research also uses expressive synthetic speech to address scarce labels for paralinguistic states.
- Speech Emotion Recognition Datasets: Established SER datasets primarily label basic emotions, limiting their coverage of complex social attitudes.IEMOCAP contains approximately 12 hours of audiovisual data with categorical labels including anger, happiness, sadness, and neutrality; RAVDESS and CREMA-D provide acted speech.
- SSA Approaches: Cascade SSA systems use ASR followed by text sentiment analysis, but can lose prosodic information when acoustic and linguistic cues diverge.Prosodic cues such as pitch contours and rhythm may be independent of or contradictory to linguistic content, including in sarcasm.
- SSA Approaches: End-to-end systems process raw audio or continuous speech representations, preserving paralinguistic information through speech encoders and multimodal LLMs.Whisper, HuBERT, and Wav2Vec2 learn acoustic representations from large-scale audio data and perform strongly in speech emotion recognition; Qwen2.5-Omni and GPT-4o demonstrate direct speech understanding.
- Synthetic Data Generation: Expressive text-to-speech synthesis has become a viable way to generate training and augmentation data for low-resource paralinguistic states.Ma et al. report that SER models trained on high-quality synthesized emotional speech can achieve competitive performance, addressing data scarcity.
3 SpeechSense Dataset
SpeechSense targets nuanced interpersonal stances rather than basic emotions, using a three-stage pipeline that decouples lexical content from prosody, synthesizes role-played speech, and validates clips with human annotators. The resulting test set contains 669 curated clips with moderate inter-annotator agreement.
- Dataset Motivation: SpeechSense captures nuanced interpersonal stances and speaker attitudes rather than high-intensity basic emotions or internal affective states.The dataset addresses communicative stances directed toward an interlocutor, for which no standardized discrete category set exists.
- Construction Pipeline: The dataset construction pipeline has three stages: semantic-prosodic text design, role-play synthesis, and dual-stage human validation and filtering.The pipeline is designed to minimize artifacts and maximize attitudinal distinctiveness.
- Construction Pipeline: Stage 1 generates semantically neutral carrier sentences decoupled from target labels, while Stage 2 holds text constant and uses situational acting directives to produce organic prosodic variation.Qwen3-Max generates 120 carrier sentences per label; role-play synthesis maps labels to explicit acting scenarios rather than mechanically adjusting pitch.
- Human Validation: Each audio clip receives evaluations from at least three annotators, followed by Majority Vote retention and Reference Alignment for ambiguous cases.Clips are retained automatically with 3/3 consensus or 2/3 majority agreement, while ambiguous clips require at least one annotator to match the intended label.
- Dataset Statistics: 669 high-quality clips form the final curated test dataset, achieving Fleiss’ Kappa of 0.4437, interpreted as moderate agreement.The reported agreement is consistent with CREMA-D (κ= 0.42) and IEMOCAP (κ= 0.40), and exceeds EmoNet-Voice (Krippendorff’s α= 0.14).
4 Experiments and Results
SpeechSense audio is sufficiently clear for evaluation, while experiments show that acoustic access and prosodic training are essential for fine-grained sentiment classification. Audio-enabled models substantially outperform text-only systems, with speech encoders also capturing robust attitudinal signal.
- Audio Quality: 3.70% overall WER on the curated Test Set and 5.42% on the weakly-supervised Training Set verify clear, artifact-free audio.Sarcastic is harder to transcribe, with 7.98% Test WER and 8.57% Train WER, reflecting its acoustic complexity.
- Overall Classification Results: Zero-shot Macro-F1 remains near-random at 1.31%–6.63%, whereas SpeechSense fine-tuning substantially improves performance across architectures.This indicates that pretrained models do not inherently encode the dataset’s fine-grained attitudinal representations.
- Overall Classification Results: 56.95% Acc for the best audio model versus 26.76% Acc for the best text model shows that performance depends primarily on paralinguistic information.Supervised text models remain weak: Qwen2.5-Omni text models reach 20–22% F1, while Qwen2.5-Instruct reaches as low as 4.60% F1.
- Overall Classification Results: Speech encoders achieve 42–45% Macro-F1, while multi-modal LLM audio models reach 53–57% F1, demonstrating complementary acoustic and language-understanding benefits.Whisper, HuBERT, and Wav2Vec2 capture substantial acoustic signal even without language modeling capacity.
- Class-wise and Scaling Analysis: Nervous is the most reliably detected attitude at 68–80% F1, whereas Neutral and Confident remain below 45% F1 for most audio models.Text-only systems collapse toward architecture-dependent classes, while scaling audio models from 3B to 7B improves Macro-F1 from 53.38% to 56.76%.
5 Limitations
SpeechSense is limited by its reliance on synthesized English speech and its eight-class interpersonal-stance taxonomy. The authors identify potential synthetic-to-real domain gaps and call for broader attitudes, languages, and real-world adaptation.
- Synthetic speech: SpeechSense is built entirely from synthesized speech, creating a potential domain gap with real-world speech despite prosodic-fidelity validation.Validation uses human annotation and WER analysis; the dataset is positioned as a foundational cold-start resource for future real-world adaptation.
- Taxonomy scope: The current taxonomy covers only eight interpersonal stances in English.The authors propose extending the taxonomy to additional attitudes and languages.
6 Conclusion
SpeechSense addresses limitations in emotion-recognition benchmarks by focusing on fine-grained interpersonal stances and isolating paralinguistic attitude from semantic content. It defines eight sentiment states and uses high-fidelity synthesis to create a resource requiring acoustic generalization.
- Dataset contribution: SpeechSense shifts speech sentiment analysis from basic emotions to eight fine-grained interpersonal sentiment states.The taxonomy targets interpersonal stances while addressing limitations of existing emotion-recognition benchmarks.
- Dataset contribution: The dataset rigorously isolates paralinguistic attitude from semantic content, making acoustic generalization critical.High-fidelity synthesis is used to build the resource.
- Experiments: Experiments evaluate SpeechSense across multi-modal LLMs, text-only LLMs, and speech encoders.The supplied passage identifies these model categories but does not provide their quantitative results.