Source-linked AI summary
E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models
Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Mengzhe Chen, Qian Chen, Lei Xie
TL;DR
Existing speech-capable models do not generate appropriate responses based on emotional speech. E-chat combines speech-derived emotion embeddings with an LLM and introduces the E-chat200 dataset; it reports stronger results than baseline models across objective and subjective evaluations.
Problem
Existing models can perceive speech content and emotion separately but cannot generate appropriate responses based on emotional speech.
Source: §1 · p. 1Method
E-chat uses a speech encoder to extract emotion embeddings and combines them with LLM decoder inputs, supported by the E-chat200 emotion-sensitive dialogue dataset.
Source: §1 · p. 1Results
E-chat outperforms baseline models across objective metrics; against ParalinGPT, it scores 74.83 vs 65.84 SIM score and 3.56 vs. 3.37 MOS score.
Source: §1 · p. 1, §5.3 · p. 4Takeaways & Limitations
Evaluations demonstrate E-chat’s capability to discern and respond to various emotional cues in speech.
Source: §6 · p. 4E-chat currently produces text only; the paper identifies integration with a text-to-speech model as a possible way to address this limitation.
Source: §6 · p. 4
Abstract
from arXiv · showhide
This study focuses on emotion-sensitive spoken dialogue in human-machine speech interaction. With the advancement of Large Language Models (LLMs), dialogue systems can handle multimodal data, including audio. Recent models have enhanced the understanding of complex audio signals through the integration of various audio events. However, they are unable to generate appropriate responses based on emotional speech. To address this, we introduce the Emotional chat Model (E-chat), a novel spoken dialogue system capable of comprehending and responding to emotions conveyed from speech. This model leverages an emotion embedding extracted by a speech encoder, combined with LLMs, enabling it to respond according to different emotional contexts. Additionally, we introduce the E-chat200 dataset, designed explicitly for emotion-sensitive spoken dialogue. In various evaluation metrics, E-chat consistently outperforms baseline model, demonstrating its potential in emotional comprehension and human-machine interaction.
1. Introduction
Speech conveys emotional cues as well as words, but existing speech-capable models do not generate appropriate responses based on emotion. E-chat pairs a speech-derived emotion embedding with an LLM, alongside the E-chat200 dataset, and reports stronger objective and subjective results than baseline models.
1. Introduction
- Speech carries emotional undertones as well as textual content, and recognizing them is presented as essential to natural, effective human-machine interaction.The paper illustrates this with a cheerful tone changing how a system might answer a question about a phone that will not turn on.Source: §1 · p. 1
- Existing models can perceive speech content and emotion separately but cannot generate appropriate responses based on emotion.This limits their practicality and interactivity for emotion-sensitive spoken dialogue.Source: §1 · p. 1
- E-chat extracts emotion embeddings with a speech encoder and combines them with LLM decoder inputs to respond according to different emotions.The paper also introduces E-chat200, a dataset designed for emotion-sensitive spoken dialogue.Source: §1 · p. 1
- E-chat outperforms baseline models across objective metrics and earns a higher Mean Opinion Score for emotionally nuanced responses.The introduction reports superior subjective evaluation as well as objective-metric performance.Source: §1 · p. 1
2. Related work
Prior work connects audio features to LLMs and explores emotional responses in spoken dialogue. E-chat is positioned as an end-to-end approach that directly uses speech’s textual and emotional features to generate responses.
2. Related work
- E-chat connects a speech encoder and LLM decoder with Transformer blocks and reports better results than Q-former while targeting emotion-sensitive responses.The paper contrasts its focus with audio-language systems such as SpeechGPT, LTU, SALMMON, and Qwen-Audio.Source: §2 · p. 2
- Prior emotion-sensitive dialogue work predicts emotion labels or uses them to guide text generation, while ParalinGPT combines ASR text with speech-derived emotion embeddings.E-chat instead generates responses directly from speech’s textual and emotional features without prior prediction of emotion or response labels.Source: §2 · p. 2
3. Method
E-chat maps speech and emotion features into an LLM decoder through a speech encoder and connection module. Training proceeds in two stages: first mapping speech features to the decoder’s text space, then fine-tuning for emotion-based responses.
3.1. E-chat Architecture
- A Chinese HuBERT speech encoder supplies weighted sums of its 24 Transformer-layer outputs as speech embeddings and separate weighted outputs as emotion embeddings.The encoder is trained on WenetSpeech and has seven convolutional subsampling layers and 317M parameters.Source: §3.1 · p. 2
- A trainable four-layer Transformer connection module maps speech features toward the LLM’s textual space, which the authors preferred over Q-former based on their tests.The connection module contains 52M parameters; a projection layer is also part of the module.Source: §3.1 · p. 2
- The decoder is Baichuan2-7B-Chat, a 7.7B-parameter Chinese LLM whose parameters remain frozen during training.It has a 32-layer Transformer decoder and is based on the LLaMA structure.Source: §3.1 · p. 2
3.2. Two-Stage Training
- Stage 1 trains the speech encoder and connection module on ASR data while freezing the LLM, using speech embeddings and a fixed prompt with cross-entropy loss.The emotion component is not trained in this stage.Source: §3.2 · p. 2
- Stage 2 fine-tunes on E-chat200 with the LLM and encoder frozen, training the connection module to generate emotion-based responses and classify emotion.Emotion classification is auxiliary; predicted emotion labels are not fed to the decoder.Source: §3.2 · p. 2
- The total loss combines decoder and emotion losses, with α set to 0 in stage 1 and 0.1 in stage 2.The stage-1 setup trains with cross-entropy for ASR, while stage 2 uses losses for spoken dialogue and emotion classification.Source: §3.2 · p. 2
4. Echat-200
E-chat200 is designed for emotion-sensitive spoken dialogue and pairs spoken questions with response and emotion labels. It contains 178k tuples spanning five emotion categories and ten speakers.
4. Echat-200
- Each E-chat200 entry contains question text, response, emotion, and question speech; the speech is input, while emotion and response are labels.The dataset was developed because existing datasets were inadequate for E-chat’s requirements.Source: §4 · p. 2
- The dataset construction uses emotion-labeled texts, GPT-3.5-Turbo-generated responses, and emotional speech produced with Microsoft’s TTS API using randomly selected speakers.The paper notes the absence of high-quality emotional question-audio datasets alongside available high-quality emotional text datasets.Source: §4 · p. 3
- E-chat200 contains 178k tuples across cheerful, fearful, angry, sad, and neutral categories, with ten speakers.The split is 143k training, 12k validation, and 23k test entries.Source: §4 · p. 3
5. Experiments
E-chat is evaluated with objective and subjective response measures, recorded-speech examples, and analyses of stage-1 and emotion recognition. It outperforms ParalinGPT on reported SIM and MOS comparisons, while stage-1 outperforms Whisper-large-v2 on the listed test sets.
5.1. Setup
- The setup trains the connection module on 10,000 hours of speech data in stage-1, then fine-tunes on E-chat200 in stage-2 with a comforting-response prompt.The listed stage-1 corpora are WenetSpeech, AISHELL-1, and AISHELL-2.Source: §5.1 · p. 3
5.2. Evaluation Metric
- Evaluation combines sentence-level SIM, BLEU1 and BLEU4, and human MOS ratings on a 1–5 scale.GPT-3.5 responses with correct question text and emotion labels provide an objective topline.Source: §5.2 · p. 3
5.3. Results
- E-chat scores 74.83 SIM versus ParalinGPT’s 65.84 and 3.56 MOS versus ParalinGPT’s 3.37, while its 3.56 MOS is comparable to GPT-3.5 annotations at 3.71.The objective and subjective comparisons are reported in Table 2.Source: §5.3 · p. 4, Table 2 · p. 4
- In the recorded-speech examples, E-chat understands the questions and responds appropriately when each question is recorded in two emotional tones.The examples use sentences manually recorded on an Android phone.Source: §5.3 · p. 4, Figure 3 · p. 4
5.4. Analysis and Discussion
- Stage-1 outperforms Whisper-large-v2 on WenetSpeech and AISHELL-1 and AISHELL-2 test sets, supporting its role in mapping speech embeddings to an LLM-suitable feature space.Training solely with stage-2 learned emotional and subject information within sentences, motivating the stage-1 analysis.Source: §5.4 · p. 4, Table 3 · p. 4
- E-chat achieves 73.6% emotion-recognition accuracy on the E-chat200 test set and is similar to other speech emotion recognition systems on relevant metrics.The passage also reports that soft emotion embeddings improve the accuracy of appropriate responses.Source: §5.4 · p. 4
6. Conclusion
The conclusion presents E-chat and E-chat200 for emotion-sensitive spoken dialogue, with evaluations indicating that E-chat can discern and respond to emotional cues in speech. Its current output is text only; text-to-speech integration and end-to-end emotion-sensitive speech-to-speech models are future directions.
6. Conclusion
- E-chat, paired with the E-chat200 dataset, is designed to discern and respond to emotional cues in speech, and evaluations demonstrate this capability.The dataset is designed specifically for emotion-sensitive spoken dialogue.Source: §6 · p. 4
- The current model produces text only; integrating text-to-speech and exploring end-to-end emotion-sensitive speech-to-speech models are future directions.Source: §6 · p. 4