PaperlayerView exact revision on arXiv

Source-linked AI summary

EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoning

Dingdong Wang, Shujie Liu, Tianhua Zhang, Youjun Chen, Jinyu Li, Helen Meng

arXiv:2601.15668v2cs.SD

TL;DR

SpeechLLMs commonly treat emotion understanding as categorical classification, limiting interpretability and leaving reasoning capabilities underused. EmotionThinker addresses this with prosody-aware supervision, EmotionCoT-35K, and GRPO-PTR, achieving state-of-the-art emotion accuracy and reasoning quality across benchmarks.

  • Problem

    Current SpeechLLMs treat speech emotion understanding as categorical classification, limiting interpretability while underusing their expressive and reasoning capabilities.

  • Method

    EmotionThinker combines the EmotionCoT-35K reasoning dataset, a prosody-enhanced foundation model, and GRPO-PTR with progressive, trust-aware multidimensional reasoning rewards.

  • Results

    EmotionThinker achieves state-of-the-art performance in both emotion accuracy and reasoning quality across multiple benchmarks, reaching 68.89% average emotion-recognition accuracy and approximately 3% above BLSP-Emo.

  • Takeaways & Limitations

    The work advances speech emotion recognition from surface-level label prediction toward fine-grained, logically coherent explanations grounded in acoustic and semantic cues.

Abstract

from arXiv · show

Emotional information in speech plays a unique role in multimodal perception. However, current Speech Large Language Models (SpeechLLMs), similar to conventional speech emotion recognition (SER) systems, still treat emotion understanding as a simple classification problem. This provides limited interpretability of predictions, while leaving the LLMs' expressive and reasoning capabilities underutilized. In this work, we take the first step to reformulate SER as a deep reasoning problem through reinforcement learning (RL). We propose EmotionThinker, which is designed to generate accurate emotion predictions with interpretable explanations grounded in fine-grained acoustic cues. To achieve this, we first construct EmotionCoT-35K, an emotional reasoning dataset with Chain-of-Thought annotations and detailed captions. Second, we observe that current SpeechLLMs exhibit weak prosody perception, whereas prosodic cues constitute fundamental signals for interpreting emotions. To address this, we develop the prosody-enhanced foundation model EmotionThinker-Base, and demonstrate that prosody enhancement improves emotion understanding. Third, we introduce Group-Relative-Policy-Optimization with Progressive-Trust-aware-Reasoning-Reward (GRPO-PTR) for RL. Different from standard GRPO, which relies only on rule-based outcome rewards, GRPO-PTR progressively introduces reasoning reward, dynamically adjusts it with a trustworthiness weight reflecting the alignment between reasoning and outcome, and evaluates the overall reasoning quality with a reward model based on multi-dimensional criteria. EmotionThinker outperforms previous state-of-the-art evaluation models both in emotion accuracy and explanation quality, advancing SER toward interpretable multimodal reasoning. Project page: https://github.com/dingdongwang/EmotionThinker

1 INTRODUCTION

EmotionThinker reframes speech emotion recognition as explainable reasoning rather than categorical prediction. It combines prosody-aware modeling, fine-grained acoustic supervision, and reinforcement learning rewards designed to improve reasoning quality and trustworthiness.

  • Current SpeechLLMs largely treat emotion understanding as categorical classification, limiting interpretability and underusing multimodal language-model reasoning capabilities.
  • Existing explainability methods generate acoustic descriptions but often fail to connect observations with their emotional significance.
  • EmotionThinker addresses scarce reasoning supervision, weak prosody perception, and the inadequacy of outcome-only rule-based rewards.
  • EmotionCoT-35K supplies Chain-of-Thought training data with speaker traits, prosodic features, and transcription content extracted through an automated annotation pipeline.
  • GRPO-PTR progressively adds a multidimensional reasoning reward and uses trustworthiness weighting to penalize high reasoning scores paired with incorrect outcomes.

2 RELATED WORK

Related work spans general-purpose SpeechLLMs, empathetic dialogue systems, and descriptive emotion-captioning models. The paper also situates its approach within broader multimodal reasoning research using reinforcement learning.

  • General-purpose SpeechLLMs treat emotion recognition as one downstream evaluation task.
  • Empathetic dialogue systems use emotion recognition as a prerequisite for generating contextually appropriate responses.
  • Descriptive emotion-captioning models generate natural-language emotion descriptions alongside contextual information such as transcriptions.
  • Recent multimodal reasoning work highlights reinforcement learning as a route to stronger reasoning capabilities across domains.

3 METHOD

EmotionThinker uses a three-stage pipeline combining EmotionCoT-35K, prosody-centric supervised fine-tuning, and GRPO-PTR reinforcement learning to produce explainable speech emotion recognition. GRPO-PTR progressively adds a trust-aware reasoning reward to improve reasoning while preserving stable outcome optimization.

  • EmotionCoT-35K: EmotionCoT-35K contains 35,000 speech–reasoning pairs spanning about 200 hours, with nine emotion categories and prosody-aware Chain-of-Thought annotations.The dataset is sourced from five emotion corpora and is designed to support perceptually grounded explanations.
  • EmotionCoT-35K: EmotionCoT-35K is constructed through an automated pipeline that extracts speed, pitch, energy, stressed words, and other emotion-relevant acoustic cues.The pipeline prioritizes cues useful for affective inference while reducing annotation cost and human bias.
  • Prosody-centric supervised fine-tuning: EmotionThinker-Base is built with prosody-centric supervised fine-tuning before reinforcement learning because the Qwen2.5-Omni-7B backbone lacks fine-grained prosody perception.The SFT corpus integrates word-level stress perception, prosodic attribute classification, and comparative prosodic augmentation tasks.
  • GRPO-PTR reinforcement learning: GRPO-PTR combines format and outcome rewards with an open-ended reasoning reward evaluated across factual alignment, interpretative quality, caption completeness, and fluency.The reasoning reward model is trained on 101,400 scored reasoning tuples and produces four criterion-specific ratings on a 1–5 scale.
  • GRPO-PTR reinforcement learning: GRPO-PTR delays the reasoning reward until format and outcome rewards stabilize, then uses a trustworthiness weight to reduce its influence when reasoning quality does not align with correctness.This progressive schedule is intended to avoid unstable early optimization and misleading reasoning trajectories.

4 EXPERIMENT

The experiments evaluate EmotionThinker across SER benchmarks, reasoning dimensions, human judgments, prosody perception, ablations, and case studies. Results show strong accuracy and reasoning performance, with prosody enhancement and GRPO-PTR contributing to the reported gains.

  • Experimental setup: Evaluation uses IEMOCAP and MELD test sets plus zero-shot RAVDESS and SAVEE, benchmarking against 13 general-purpose SpeechLLMs and OmniLLMs.Emotion recognition is measured by unweighted accuracy, while reasoning quality is rated across four dimensions.
  • Main results: EmotionThinker achieves 68.89% average emotion-recognition accuracy, approximately 3% above the second-best BLSP-Emo, and a 3.98 average reasoning score.The reported reasoning result is the highest average score across the evaluated dimensions.
  • Main results: Human evaluation confirms superiority across all reasoning dimensions, with EmotionThinker scoring 3.7 in Factual Alignment and 4.7 in Caption Completeness.Four independent reviewers rated 100 anonymized outputs from five representative models.
  • Analysis: EmotionThinker-Base shows improved sensitivity to pitch, energy, speed, and stress over the original backbone after prosody-centric supervised fine-tuning.The comparison uses a prosody perception test set containing 1,000 unseen utterances.
  • Analysis: SFT reaches 53.91% accuracy and 3.78 reasoning score, standard GRPO improves both, and GRPO-PTR further enhances emotion accuracy and reasoning quality.The comparison uses the same EmotionCoT-35K training data across training strategies.
  • Analysis: Ablations show that removing the trained reward model reduces accuracy to 66.67% and reasoning score to 3.36, while case studies show more accurate, comprehensive, and prosody-grounded reasoning for GRPO-PTR.The case study contrasts GRPO-PTR with Qwen2.5-Omni-7B and EmotionThinker-Base + GRPO; the latter includes hallucinated interpretations inconsistent with the input.

5 CONCLUSION

The paper concludes that EmotionThinker reframes SER as explainable emotion reasoning through reinforcement learning. Its experiments report state-of-the-art emotion accuracy and reasoning quality, with explanations grounded in acoustic and semantic cues.

  • Conclusion: EmotionThinker extends speech emotion recognition from simple classification to explainable emotion reasoning through reinforcement learning.The framework includes the GRPO-PTR strategy for refining reasoning coherently and trustworthily.
  • Conclusion: The framework produces fine-grained, logically coherent explanations grounded in relevant acoustic and semantic cues.This conclusion describes the form of the generated explanations rather than only their emotion labels.
  • Conclusion: Comprehensive benchmark experiments report state-of-the-art performance in both emotion accuracy and reasoning quality.The conclusion states this result across various benchmarks.

Supplementary Material

The supplementary material includes a GPT prompt entry in its contents.

  • Supplementary Material: The supplementary material lists the GPT Prompt in section D.2 on page 19.The passage is a table-of-contents entry rather than a description of the prompt itself.

A THE USE OF LARGE LANGUAGE MODELS

The paper uses large language models as assistive tools for data construction, evaluation, and copyediting. Their evaluation use follows a fixed rubric for emotion reasoning quality.

  • Data construction: GPT-4o synthesizes EmotionCoT-35K samples and creates training pairs for the reward model.These uses support data construction rather than the paper’s final prose generation.
  • Evaluation: GPT-4o evaluates emotion reasoning quality under a fixed rubric.The evaluation is described as occurring under the rubric in Section E.
  • Writing: LLMs are used for copyediting and phrasing to improve clarity and are not described as replacing the paper’s research process.The passage characterizes writing assistance as copyediting and phrasing support.

B.1 DATA SOURCE

The data sources span acted, conversational, expressive, and controlled emotional speech, with multimodal recordings and varied prosodic styles. Together, they provide diverse speakers, settings, emotions, and recording conditions for emotion reasoning.

  • IEMOCAP: IEMOCAP provides approximately 12 hours of acted, multimodal, multispeaker audiovisual data from dyadic emotional interactions.Its sessions use improvisations and scripted scenarios designed to elicit emotional expressions.
  • MELD: MELD contains more than 1,400 dialogues and 13,000 utterances from Friends, with audio, visual, and text modalities.Its utterances are labeled with seven emotions and involve multiple speakers.
  • Expresso: Expresso contributes 40 hours of 48kHz expressive speech across read styles and improvised dialogues.The dataset includes eight read-speech styles, 26 improvised-dialogue styles, and four speakers.
  • MEAD: MEAD contains recordings from 60 actors under eight distinct emotions and three intensity levels except neutral.The data were collected in a strictly controlled environment for emotional audiovisual modeling.
  • EARS: EARS provides approximately 100 hours from 107 demographically diverse speakers across expressive styles, vocalizations, and 22 emotional styles.Recordings were made at 48 kHz in anechoic conditions.

B.2 AUTOMATIC ANNOTATION PIPELINE

The automatic annotation pipeline extracts prosodic, stress, speaker, and semantic-supporting information, then uses these annotations and GPT-4o prompting to create emotion reasoning supervision. It also adds explicit stress and controlled prosodic-contrast data for prosody learning.

  • Attribute extraction: The pipeline automatically extracts pitch, energy, speaking rate, word-level stress, gender, and age for fine-grained expressive speech annotation.Pitch and energy come from short-time acoustic frames, while speaking rate combines phoneme alignments with utterance duration.
  • Stress annotation: WhiStress supplies token-level stress labels through an alignment-free stress-detection model built on a frozen Whisper backbone.Its architecture includes a transformer decoder block and feedforward classifier.
  • Reasoning construction: GPT-4o generates structured emotion explanations grounded in prosodic and semantic cues to provide Chain-of-Thought-style supervision.The prompt template shown in Figure 5 guides the construction of the reasoning process.
  • Stress data: Stress-17K adds paired audio and explicit stress annotations across contrastive, emphatic, new-information, and focus categories.It provides both stress detection and reasoning supervision.
  • Prosodic classification: GigaSpeech utterances are converted into supervised tasks for pitch, energy, speaking rate, and intonation classification.The annotations are produced with the EmotionCoT pipeline and used to query the model about prosodic categories.
  • Prosodic augmentation: Controlled augmentations vary pitch, energy, and speaking rate before concatenating samples into sequences with systematic prosodic shifts.These comparative examples are constructed from GigaSpeech utterances to strengthen robustness.

C.2 TRAINING DETAILS OF EMOTIONTHINKER-BASE

EmotionThinker-Base uses Qwen2.5-Omni-7B and a two-stage training process that first builds prosody capabilities, then adds cold-start emotion reasoning. Text-only and ASR data help preserve instruction-following and ASR abilities.

  • Stage I: EmotionThinker-Base is built on Qwen2.5-Omni-7B and trains its audio encoder, adapter, and language model jointly in Stage I.Stage I uses the constructed Prosody-Centric corpus for full-parameter training.
  • Training schedule: The training setup uses one epoch at a learning rate of 1 × 10^-5 in Stage I and proceeds to a separate Stage II adaptation.Stage I updates the full model components, whereas Stage II applies LoRA only to the language-model layers.
  • Capability preservation: Stage I additionally incorporates 20% text-only data and 20% ASR data from LibriSpeech and GigaSpeech.These data are included to preserve the backbone’s instruction-following and basic ASR capabilities.
  • Stage II: Stage II samples 5K EmotionCoT examples for cold-start reasoning supervision and trains only the language-model layers with LoRA.The audio encoder and adapter remain fixed during this stage.

D.1 DATA CONSTRUCTION DETAILS

The thinking reward model is trained from EmotionCoT examples paired with controlled reasoning-quality variations and four criterion-wise ratings. This construction yields diverse supervision for evaluating emotion reasoning quality.

  • Gold traces: The reward-model data start from 20K high-quality EmotionCoT instances treated as gold reasoning traces.Each gold trace receives a perfect score of 5 across four evaluation dimensions.
  • Quality criteria: The four evaluation dimensions are Factual Alignment, Interpretative Quality, Caption Completeness, and Fluency and Structural Clarity.GPT-4o is used to obtain 1–5 ratings for these dimensions.
  • Quality variation: Controlled quality variation produces 101,400 (q, r, g) tuples by assigning criterion-wise scores from 1 to 5 and generating matching reasoning traces.Lower scores produce targeted degradation, such as factual inconsistency or poor fluency and structure.
  • Reward-model supervision: The resulting supervision provides balanced coverage across quality levels and diverse signals for training the thinking reward model.The generated traces are conditioned on target score configurations across the four dimensions.

F CASE STUDY

The case study compares EmotionThinker with 12 representative SpeechLLMs on the same speech sample, whose ground-truth emotion is sad. EmotionThinker provides more accurate acoustic analysis and more logically consistent reasoning than the compared systems.

  • Case study setup: The case study evaluates EmotionThinker against 12 representative SpeechLLMs using the same audio sample, labeled sad in the ground truth.The comparison focuses on prosodic cue recognition, semantic integration, and logical coherence.
  • Case study conclusion: EmotionThinker is reported to capture acoustic information more accurately and comprehensively while maintaining stronger logical consistency in its reasoning.
  • Model outputs: Qwen2.5-Omni-7B predicts neutral and supports it with even prosody and matter-of-fact semantic content.
  • Model outputs: Phi-4-Multimodal predicts anger by associating high pitch, fast speech, high energy, and stressed words with frustration.
  • Model outputs: DIVA predicts frustration using flat, monotone prosody, a slightly rising pitch, deliberate pacing, and dissatisfaction about waiting.
  • Model outputs: Another model predicts sadness from flat, monotonous intonation and the speaker’s disappointing experience of waiting in line.
Loading 2601.15668v2…