PaperlayerView exact revision on arXiv

Source-linked AI summary

SECap: Speech Emotion Captioning with Large Language Model

Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shixiong Zhang, Guangzhi Li, Yi Luo, Rongzhi Gu

arXiv:2312.10381v3cs.SDeess.AS

TL;DR

Speech emotion research has often used fixed categories, although speech can express complex or mixed emotions that labels may not capture. SECap combines HuBERT, Q-Former, and LLaMA to generate natural-language emotion captions, outperforming HTSAT-BART on objective metrics and reaching subjective MOS on par with human annotations.

  • Problem

    Fixed emotion categories can miss nuanced or mixed speech emotions, and natural-language approaches to describing them remain limited.

    Source: Introduction · p. 1

  • Method

    SECap uses HuBERT speech features, a Q-Former Bridge-Net trained to separate content from emotion-related features, and LLaMA to generate captions.

    Source: Introduction · p. 1, Obtain Emotion-Related Representations · p. 3

  • Results

    3.77 vs. 3.85 MOS against human annotations, while SECap outperforms HTSAT-BART across all objective metrics.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The paper presents natural-language captioning as an alternative to fixed categories for characterizing speech emotions.

    Source: Introduction · p. 1

Abstract

from arXiv · show

Speech emotions are crucial in human communication and are extensively used in fields like speech synthesis and natural language understanding. Most prior studies, such as speech emotion recognition, have categorized speech emotions into a fixed set of classes. Yet, emotions expressed in human speech are often complex, and categorizing them into predefined groups can be insufficient to adequately represent speech emotions. On the contrary, describing speech emotions directly by means of natural language may be a more effective approach. Regrettably, there are not many studies available that have focused on this direction. Therefore, this paper proposes a speech emotion captioning framework named SECap, aiming at effectively describing speech emotions using natural language. Owing to the impressive capabilities of large language models in language comprehension and text generation, SECap employs LLaMA as the text decoder to allow the production of coherent speech emotion captions. In addition, SECap leverages HuBERT as the audio encoder to extract general speech features and Q-Former as the Bridge-Net to provide LLaMA with emotion-related speech features. To accomplish this, Q-Former utilizes mutual information learning to disentangle emotion-related speech features and speech contents, while implementing contrastive learning to extract more emotion-related speech features. The results of objective and subjective evaluations demonstrate that: 1) the SECap framework outperforms the HTSAT-BART baseline in all objective evaluations; 2) SECap can generate high-quality speech emotion captions that attain performance on par with human annotators in subjective mean opinion score tests.

Introduction

The paper proposes speech emotion captioning as a natural-language alternative to fixed emotion labels, which can miss nuance and mixed affect. SECap combines HuBERT, Q-Former, and LLaMA, and its captions outperform HTSAT-BART on objective metrics while reaching human-annotation-level subjective scores.

Introduction

  • Speech emotion captioning describes emotions in natural language, addressing the limited nuance of single labels for complex or mixed affect.The paper introduces SEC as a task and SECap as its framework.Source: Introduction · p. 1, Figure 1 · p. 2
  • The authors frame SEC as a pioneering effort to characterize human speech emotions with natural-language descriptions rather than predefined categories.They describe this as among the pioneering works in the direction.Source: Introduction · pp. 1–2
  • SECap uses HuBERT to extract speech features, Q-Former as Bridge-Net, and LLaMA to generate coherent emotion captions.Q-Former compresses frame-level HuBERT features before they reach the text decoder.Source: Introduction · pp. 1–2
  • Speech-Caption Contrastive Learning extracts emotion-related acoustic features, while Speech-Transcription Mutual Information Learning separates them from speech content.The paper treats acoustic information as more directly related to emotion and content as recoverable through transcription.Source: Introduction · p. 2
  • 3.77 vs. 3.39 MOS against human labels and 3.77 vs. 3.85 MOS against human annotations; SECap also outperforms HTSAT-BART across all objective metrics.The subjective results place SECap above human labels and on par with human annotations.Source: Introduction · p. 2

Related Work

The related work reviews categorical speech emotion recognition, language models including multimodal approaches, and natural-language captioning for ambient audio. These areas provide context for describing speech emotions with natural sentences.

Speech Emotion Recognition

  • SER research extracts and classifies speech features into emotion categories, while this paper describes emotions with natural sentences instead.Prior classifiers have progressed from methods such as LDA to CNN and Transformer architectures.Source: Speech Emotion Recognition · p. 2

Large Language Model

  • LLM multimodal approaches either connect specialized downstream models or map modalities into a shared space, as in BLIP-2's Q-Former.AudioGPT and HuggingGPT illustrate the task-orchestration approach; BLIP-2 and Video-LLaMA illustrate modality mapping.Source: Large Language Model · p. 2

Automated Audio Captioning

  • AAC describes ambient sounds in natural language, commonly using encoder-decoder models and contrastive methods that connect audio and text.AudioClip and CLAP are cited as examples of contrastive learning approaches.Source: Automated Audio Captioning · p. 2

Method

SECap uses HuBERT, a Q-Former Bridge-Net, and LLaMA to produce speech emotion captions. The Bridge-Net compresses speech features and is trained to emphasize emotion-related information while reducing content information.

Method

  • The overall architecture sends extracted speech features through a Bridge-Net into a text decoder that generates speech emotion captions.Q-Former compresses HuBERT features while aiming to retain emotion-related acoustic information and eliminate content information.Source: Method · p. 2, Model Architecture · p. 3

Model Architecture

  • SECap combines a HuBERT audio encoder, Q-Former Bridge-Net, and LLaMA text decoder to generate speech emotion captions from extracted features.The Bridge-Net extracts emotion-related features and transforms them into the text decoder’s feature space.Source: Method · p. 2, Model Architecture · p. 3

Q-Former

  • Q-Former compresses redundant HuBERT features into a fixed-length Q-Embedding whose length is independent of input speech duration.Learnable Q-queries retrieve speech features through self-attention and cross-attention.Source: Q-Former · p. 3

Obtain Emotion-Related Representations

  • The representation-learning design uses transcription and human-labeled emotion captions alongside speech features to provide LLaMA with content-unrelated, emotion-related features.STMIL targets speech-content disentanglement, while SCCL extracts features more related to emotion captions.Source: Obtain Emotion-Related Representations · p. 3, Figure 3 · p. 4

Speech-Transcription Mutual Information Learning (STMIL)

  • STMIL compares speech and transcription embeddings in a shared representation space and uses an estimated mutual-information upper bound to reduce their correlation.This is intended to mitigate speech content’s impact on LLaMA’s emotion-caption generation.Source: Speech-Transcription Mutual… · pp. 3–4

Speech-Caption Contrastive Learning (SCCL)

  • SCCL brings speech embeddings closer to their matching emotion-caption embeddings and separates irrelevant caption features using a margin-based contrastive objective.Training samples include captions with similar emotions and captions with dissimilar emotions.Source: Speech-Caption Contrastive Learning… · p. 4

Training Process

  • The two-stage training first freezes HuBERT while jointly training Q-Former with STMIL and SCCL, then aligns Q-Former features to LLaMA while freezing HuBERT and LLaMA.The second stage also fine-tunes the projection layer and uses prompted, teacher-forced caption generation with cross-entropy loss.Source: Training Process · pp. 4–5

Dataset

EMOSpeech is an internally assembled Chinese speech-emotion-caption dataset with multiple human captions, emotion labels, and transcriptions per utterance. The paper describes its annotation procedure and train, validation, and test split.

Dataset

  • EMOSpeech contains 41.6 hours of 24 kHz speech from 5 female and 2 male speakers, covering 30,526 sentences.Each sentence has three to five human-labeled emotion captions, emotion labels, and a corresponding transcription.Source: Dataset · p. 5

Annotation

  • Annotators used guidelines covering overall emotion, intensity, and a comprehensive description of emotion, volume, and speech rate.They discussed 50 sample clips to standardize the rules, and other annotators reviewed 5 of every 100 clips for consistency.Source: Dataset · p. 5

Data Split

  • The dataset was split into 29,326 training sentences, 600 validation sentences, and 600 test sentences.The sentences were assigned to the splits randomly.Source: Dataset · p. 5

Evaluation Metric

Because no existing method evaluates speech emotion captions, the paper combines objective text-similarity measures with subjective ratings. The subjective procedure checks emotional content, match to the speech, and intensity.

Evaluation Metric

Objective Evaluation

  • Objective evaluation combines BLEU1, BLEU4, METEOR, ROUGE-L, CIDEr, and SPICE with sentence-similarity indicators SIM1 and SIM2.The sentence-level measures complement metrics that primarily focus on word-level matching.Source: Objective Evaluation · p. 5

Subjective Evaluation

  • Subjective evaluation scores captions from 1 to 5 after assessing emotion description, match between summarized emotion and speech, and emotion intensity.The scoring method is described as similar to Mean Opinion Score ratings used in text-to-speech systems.Source: Subjective Evaluation · p. 5

Results and Analysis

Experiments use EMOSpeech and compare SECap configurations against HTSAT-BART, human labels, and human captions. SECap’s objective and subjective results support speech emotion captioning, with outcomes depending on how speech and text features are combined.

Experiment Setup

  • Experiments use only EMOSpeech, with HuBERT-large pretrained on the 10k-hour WenetSpeech L subset and a Chinese-finetuned LLaMA decoder.The analysis also compares transcription inputs, including raw text and T-Embedding, alongside speech-feature configurations.Source: Experiment Setup · p. 5, Performance Analysis · p. 5

Performance Analysis

  • SECap using speech features surpasses HTSAT-BART across all objective metrics, while its best model matches human-labeled speech emotion captions in subjective evaluation.The objective comparison uses SECap configuration #6; the subjective finding compares the best SECap model with human-labeled captions.Source: Table 2 · p. 6, Performance Analysis · p. 6, Figure 4 · p. 6
  • Using T-Embedding instead of raw transcription improves SIM values by 16.66% and 178.11%.The authors attribute this to T-Embedding constraining LLaMA’s output and extracting more emotion-related features.Source: Performance Analysis · p. 6, Table 2 · p. 6
  • Q-Embedding alone increases SIM values by 9.18% and 13.29% over T-Embedding alone; adding T-Embedding to Q-Embedding improves them by 0.61% and 3.73%, but remains below Q-Embedding alone.Adding raw transcription to Q-Embedding instead decreases SIM values by 3.77% and 4.27%.Source: Performance Analysis · p. 6, Table 2 · p. 6
  • Human-labeled captions outperform human emotion labels and SER-model labels, while SECap with raw transcription alone performs suboptimally in subjective evaluation.Combining Q-Embedding and T-Embedding produces better subjective outcomes than using Q-Embedding alone.Source: Figure 4 · p. 6, Performance Analysis · p. 6
  • Evaluators may favor content-relevant captions in content-emotion conflicts, while objective metrics may miss naturalness and contextual details that shape subjective judgments.The authors illustrate this with a flat-toned utterance: text embedding can yield a caption combining sadness and flatness, unlike speech embedding alone.Source: Performance Analysis · pp. 6–7

Ablation Study on Different Model Components

  • Replacing HTSAT-BART’s decoder with LLaMA raises SIM by 8.92%, replacing its encoder with HuBERT raises SIM by 7.26%, and replacing both raises SIM by 15.10%.Replacing the linear layer with Q-Former adds a 4.85% SIM increase.Source: Table 3 · p. 7, Ablation Study on Different Model… · p. 7

Comparison of Training Methods

  • Using STMIL and SCCL together raises SIM by 6.92% over omitting first-stage training and exceeds either method alone by 4.65% and 3.67%, respectively.Using STMIL alone and SCCL alone raises SIM by 2.17% and 3.14%, respectively, compared with omitting the stage.Source: Table 4 · p. 7, Comparison of Training Methods · p. 7

Conclusion

SECap introduces speech emotion captioning and combines HuBERT, Q-Former, and LLaMA to generate natural-language descriptions of speech emotions. Its captions perform on par with human annotators.

Conclusion

  • SECap generates high-quality speech emotion captions with performance on par with human annotators.Source: Conclusion · p. 7
  • SECap combines a HuBERT-based audio encoder, a Q-Former-based Bridge-Net, and a LLaMA-based text decoder.Source: Conclusion · p. 7
  • Speech emotion captioning characterizes speech emotions with natural-language descriptions rather than singular labels.Source: Conclusion · p. 7
Loading 2312.10381v3…