Can Large Language Models Aid in Annotating Speech Emotional Data? Uncovering New Frontiers

I. INTRODUCTION

• The study targets an unexamined setting—LLM annotation of speech emotion from transcripts—and adds audio context to address text-only annotations’ limited generalisation to speech.The evaluation compares LLM-based and human annotations on IEMOCAP and MSP-IMPROV.Source: §I · pp. 1–2, §II · p. 3
• Speech-emotion annotation depends on context, while crowd- and expert-based labeling can involve bias, verification demands, and dependence on a large annotator pool.Unsupervised annotation methods also require large samples and may omit context.Source: §I · p. 1

II. RELATED WORK

• Prior annotation approaches trade off quality, speed, cost, complexity, and scalability; human annotation can be high quality but costly and vulnerable to bias and inconsistency.Bulk annotation is faster and less expensive but may lower quality; semi-supervised approaches can face generalisability and robustness difficulties.Source: §II · p. 2
• ChatGPT re-annotation performance varies by task: Zhu et al. [13] report 60.9% average across five datasets, while fact-checking re-annotation reached 72.0%.Prior studies also report variation with ChatGPT version and prompts [13].Source: §II · p. 2
• The literature also frames GPT-3 annotation as a lower-cost option, while noting reliability concerns for high-stakes cases.GPT-3 annotation saved 50.0% to 96.0% in annotation costs across the reported tasks.Source: §II · p. 3

III. METHODOLOGY

• The approach represents audio context using utterance-level average energy and pitch, gender information, or a 64-dimensional discrete VQ-VAE representation supplied alongside text to ChatGPT.The study first annotates samples using text alone, then incorporates audio features and gender information.Source: §III · p. 3
• A BLSTM-based classifier is trained for speech-emotion classification using the annotations produced in the experiments.The paper introduces this classifier as the evaluation approach for the annotations.Source: §III · p. 3
• The audio-context experiments use average utterance energy and pitch as features and also test a discrete VQ-VAE audio representation.The representation is provided to ChatGPT as audio context.Source: §III · p. 3

A. VQ-VAE for Speech Code Generation

• A VQ-VAE encodes Mel-spectrograms, quantizes encoder outputs to nearest vectors in a learned embedding space, and decodes the selected vectors to reconstruct the input.The vector-selection step is nondifferentiable, so gradients are copied from the quantized vectors to the encoder outputs.Source: A. VQ-VAE for Speech Code Generation · p. 3
• The VQ-VAE objective includes reconstruction, codebook, and commitment losses; the codebook loss moves vector embeddings closer to encoder outputs.The commitment loss encourages the encoder to commit to an embedding, with β controlling its weight.Source: A. VQ-VAE for Speech Code Generation · p. 3
• The model represents speech as a discrete list of integer codes, which can be supplied to ChatGPT alongside transcriptions for annotation.The architecture consists of an encoder, vector quantiser, and decoder.Source: A. VQ-VAE for Speech Code Generation · p. 3

B. Emotion Label Annotation using LLMs

• The experiments annotate IEMOCAP training samples with transcripts in zero-shot and few-shot settings, using 10 randomly selected training samples as few-shot context.A BLSTM classifier is trained on ChatGPT-annotated samples and evaluated using UAR.Source: B. Emotion Label Annotation using LLMs · p. 4
• The study repeats zero-shot and few-shot annotation with average pitch and energy, then also provides gender information to ChatGPT.Classification UAR is measured with a BLSTM-based classifier.Source: B. Emotion Label Annotation using LLMs · p. 4
• The final audio-context condition supplies a VQ-VAE representation to ChatGPT and uses the resulting annotations to benchmark classification performance.The authors curated multiple prompts and also explored prompt variants to improve on the benchmark.Source: B. Emotion Label Annotation using LLMs · p. 4

C. Speech Emotion Classifier

• CNN layers extract emotionally salient features, and the BLSTM models temporal relationships in speech.The CNN acts as data-driven filter banks before passing features to the BLSTM.Source: C. Speech Emotion Classifier · p. 4
• Attention weights aggregate salient attributes across the BLSTM output sequence into an utterance-level representation.The attention weights are computed using a trainable parameter W.Source: C. Speech Emotion Classifier · p. 4
• The attentive representation feeds a fully connected layer for emotion classification, combining abstract features, context, and salient attributes.The classifier uses CNN layers, a BLSTM, attention, and a fully connected layer.Source: C. Speech Emotion Classifier · p. 4

A. Datasets

• The evaluation uses IEMOCAP, MSP-IMPROV, and MELD, which include acted dyadic interactions or dialogue from the Friends television series.IEMOCAP and MSP-IMPROV have similar labelling schemes; MELD contains Friends utterances.Source: A. Datasets · p. 4

1) IEMOCAP:

• IEMOCAP contributes four emotion categories: sad (1084), happy (1636), angry (1103), and neutral (1708).The database contains 12 hours of recordings from five male and five female speakers in unrehearsed dyadic interactions.Source: 1) IEMOCAP · p. 4

2) MSP-IMPROV:

• MSP-IMPROV contributes four categories—anger (792), sad (885), neutral (3477), and happy (2644)—from six sessions of actor recordings.Its emotional labels were collected through crowdsourced perceptual evaluations.Source: 2) MSP-IMPROV · p. 4

3) MELD:

• MELD contributes 11,353 utterances across the selected sadness, neutral, joy, and anger categories.The dataset contains over 1,400 dialogues and 13,000 utterances with seven emotion labels before the study's selection.Source: 3) MELD · p. 5

B. Speech Features

• 16 kHz audio is converted to normalized 80x256 Mel spectrograms, using zero-padding for shorter samples.The spectrograms use an STFT size and window size of 1024, hop size 256, 80 Mel bands, and an 8 kHz cutoff.Source: B. Speech Features · p. 5

C. Hyperparameters

• The VQ-VAE produces 64 discrete codes per utterance, which are passed to ChatGPT with textual data for annotation.The reported configuration uses 8,192 token embeddings of dimension 512 and trains for 1,000 epochs.Source: C. Hyperparameters · p. 5
• The classifier uses two CNN layers, a 128-unit BLSTM, attention, and a dense layer with 128 hidden units before softmax classification.Cross-entropy loss is used to calculate posterior class probabilities.Source: C. Hyperparameters · p. 5

V. EXPERIMENTS AND RESULTS

• Evaluation uses speaker-independent leave-one-speaker-out cross-validation, ten repetitions, and mean and standard deviation of UAR.For cross-corpus SER, IEMOCAP is used for training and MSP-IMPROV for validation and testing; the learning rate is halved after five epochs without validation-accuracy improvement and training stops below 0.00001.Source: §V · p. 5, C. Hyperparameters · p. 5

A. Within Corpus Experiments

• Few-shot ChatGPT annotation improves on zero-shot, while text-only annotation performs poorly against the original-label baseline; adding audio context, gender, and VQ-VAE codes improves results.Energy and pitch alone yield limited improvement, attributed primarily to confounding by gender; the reported substantial gains with discrete codes combine them with energy, pitch, and gender information.Source: Figure 2 · p. 5, A. Within Corpus Experiments · pp. 5–6

B. Cross-Corpus Evaluations

• ChatGPT-annotated IEMOCAP data generalizes to MSP-IMPROV, but the cross-corpus results do not surpass previous studies.The cross-corpus evaluation trains on IEMOCAP and tests on MSP-IMPROV.Source: Table I · p. 6, B. Cross-Corpus Evaluations · p. 6

C. Augmentating the Training Data

• ChatGPT-annotated MELD samples are used to augment training, with results compared against a CNN-BLSTM classifier trained using actual IEMOCAP labels.MELD is annotated in four emotions using IEMOCAP few-shot examples; Figure 3 compares classifier UAR with data augmentation.Source: C. Augmentating the Training Data · p. 6, Figure 3 · p. 6
• The authors attribute their gains over prior augmentation work to adding newly annotated data rather than perturbed versions of existing samples.They note that perturbing the same data can potentially lead to overfitting and also report stronger cross-corpus results than prior synthetic-data or audio-perturbation approaches,.Source: C. Augmentating the Training Data · pp. 6–7
• The approach achieves superior performance to previous studies in both within-corpus and cross-corpus settings.The authors report considerably improved results over prior augmentation studies, including, and suggest that adding new annotated data may explain the difference from perturbing existing data.Source: C. Augmentating the Training Data · p. 7

D. Limitations

• Augmentation was ineffective on LibriSpeech, whose recordings lack emotional variation, unlike the more promising MELD results.The authors report that LibriSpeech results were not shown.Source: D. Limitations · p. 7
• Small prompt changes can produce ambiguous or erroneous annotations, while adding context is challenging for short-time emotion variability.The authors suggest testing different prompts and report that more prompt context has improved result quality.Source: D. Limitations · p. 7
• ChatGPT was not specifically trained for speech-emotion annotation, and its labels are insufficient alone, though multi-shot annotations improved performance over zero-shot.The paper reports enhanced classification when ChatGPT annotations are incorporated alongside training data.Source: D. Limitations · p. 7
• ChatGPT annotation cost about 30 USD for IEMOCAP examples, but its accuracy was lower than human annotation, creating a cost–accuracy trade-off.The authors describe using ChatGPT annotations to augment human-annotated data as having improved SER performance.Source: D. Limitations · p. 7

• To the authors’ knowledge, this study is the first to explore ChatGPT’s capabilities in speech emotion recognition.Source: §VI · p. 7
• Text-based emotional annotations did not generalize effectively to speech, while incorporating audio context improved SER performance over the text-based approach.Source: §VI · p. 7
• Few-shot ChatGPT annotations considerably improved annotation quality compared with zero-shot, with a small number of annotated samples yielding improved evaluation results.Source: §VI · p. 7
• ChatGPT-annotated data augmentation improved SER results over state-of-the-art systems using conventional augmentation techniques.Source: §VI · pp. 7–8
• Future work will test the approach on new datasets and diverse contexts, compare different LLMs’ annotation abilities, and use LLMs in SER training.Source: §VI · p. 8