PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue

Yutong Hu, Jinho Choi

arXiv:2609.39072v1cs.CLcs.AIcs.LGcs.MM

TL;DR

LLMs have rarely been applied to continuous VAD evaluation in multimodal dialogue, where emotion includes acoustic cues unavailable from text alone. This work evaluates a framework for discrete and dimensional emotion tasks using textual audio descriptions. LoRA-fine-tuned LLaMA models outperform prompted GPT models on both tasks, with a Valence CCC of 0.7822.

  • Problem

    LLMs have rarely been systematically evaluated for continuous VAD prediction in multimodal dialogue, where acoustic cues such as tone and pitch carry affective information beyond transcription.

    Source: §1 · p. 1, §2.2 · p. 2

  • Method

    The framework evaluates discrete emotion recognition and continuous VAD using conversation context, target utterances, and acoustic cues rendered as natural-language descriptions.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    LoRA-fine-tuned LLaMA models outperform prompt-engineered GPT models on both tasks; LLaMA-3.3-70B achieves Valence CCC 0.7822.

    Source: §1 · p. 2, §12 · p. 9

  • Takeaways & Limitations

    The reported comparison indicates domain adaptation through parameter-efficient fine-tuning is more impactful than general instruction-following capacity for these emotion tasks.

    Source: §12 · p. 9

    Evaluation is limited to IEMOCAP, and converting raw audio to text descriptions inevitably loses some acoustic information.

    Source: §13 · p. 9

Abstract

from arXiv · show

Emotion recognition in conversation has been widely studied, but applying Large Language Models (LLMs) to continuous dimensional emotion evaluation in multimodal dialogue remains largely unexplored. We propose an LLM-based framework that performs discrete emotion recognition and Valence-Arousal-Dominance (VAD) dimensional evaluation on IEMOCAP, incorporating acoustic cues as natural language descriptions following the SpeechCueLLM approach. We evaluate six models spanning the LLaMA, GPT, and Qwen families under zero-shot prompting, few-shot prompting, and LoRA fine-tuning. LoRA fine-tuned LLaMA models substantially outperform prompt-engineered GPT models on both tasks despite GPT's larger scale, a gap we attribute to domain adaptation rather than model capacity. Our best model achieves a Valence CCC of 0.7822, a new state-of-the-art on IEMOCAP. Ablation studies confirm that textual audio descriptions meaningfully improve smaller models (+3.5 to 3.6 weighted F1) while contributing little for the largest model, suggesting audio cues are most valuable when linguistic capacity is limited. The performance asymmetry across VAD dimensions closely mirrors the annotator agreement hierarchy in IEMOCAP's own annotations.

1 Introduction

Emotion recognition in dialogue requires context across turns, while continuous VAD evaluation in multimodal dialogue has received little LLM study. This work adapts an LLM framework for discrete and dimensional tasks by representing acoustic cues as text and comparing prompting with LoRA fine-tuning.

1 Introduction

  • The proposed framework is the first systematic LLM-based approach to continuous VAD evaluation in multimodal dialogue, extending acoustic descriptions from discrete emotion recognition.The work compares prompting and LoRA fine-tuning across model families.Source: §1 · p. 1
  • LoRA fine-tuning substantially outperforms prompting for both discrete and dimensional emotion tasks, even against larger prompted models.The paper also examines error sources, acoustic-description contributions, and past VAD context.Source: §1 · p. 2

2.1 Discrete Emotion Recognition in Dialogue

  • The framework extends instruction-following emotion recognition approaches, including SpeechCueLLM’s natural-language descriptions of vocal information.SpeechCueLLM is cited as the source of the acoustic-information approach.Source: §2.1 · p. 2

2.2 Dimensional Emotion Evaluation

  • LLMs remain largely untested for continuous VAD prediction in multimodal dialogue, despite prior multimodal work using LSTM and CNN-1D models.Tone, pitch, rhythm, and speaking rate carry affective signals that transcription discards.Source: §1 · p. 1, §2.2 · p. 2

3.1 Framework Overview

  • The framework uses conversation history, a target utterance, and textual audio descriptions to output either a discrete emotion label or VAD scores.It evaluates zero-shot and few-shot prompting alongside LoRA fine-tuning.Source: §3.1 · pp. 2–3

3.2 Audio Feature Extraction and Textual Description

  • Volume, pitch, and speaking rate are each summarized by central tendency and variation, then mapped to five quantile-based verbal levels.These descriptions provide non-lexical cues unavailable from transcribed text alone.Source: §3.2 · p. 3

3.3 Fine-Tuning Strategy (LoRA)

  • LoRA updates a small set of trainable parameters while freezing original model weights, reducing overfitting risk with IEMOCAP’s limited training data.The paper adopts it because full fine-tuning is computationally expensive for large models.Source: §3.3 · p. 3

4 Experiment Setup

Experiments use IEMOCAP, selected for its multimodal, conversational data and both discrete and continuous emotion annotations. The setup holds out one session for unseen-speaker testing and compares model families and training strategies.

4.1 Dataset

  • IEMOCAP was selected after reviewing 19 datasets because it meets the multimodal, conversational, and dual-annotation requirements.It contains 151 dyadic dialogues and 10,086 utterances.Source: §4.1 · p. 4, §4.1.1 · p. 4

4.1.1 IEMOCAP Dataset

  • The experiments use IEMOCAP audio and text, with majority-vote discrete labels and aggregated VAD ratings on a 1-to-5 scale.Although the corpus includes video and motion capture, this work uses only audio recordings and text transcriptions.Source: §4.1.1 · p. 4

4.1.2 Data Splitting

  • The fifth session is held out for testing unseen speakers, while the other four sessions are split 90/10 for training and validation.Source: §4.1.2 · p. 4

4.2 Models

  • Six language models span LLaMA, GPT, and Qwen; Qwen is evaluated only for discrete emotion recognition.LLaMA models use zero-shot, few-shot, and LoRA conditions for discrete recognition, but only LoRA for VAD; GPT models use prompting only.Source: §4.2 · p. 4

4.3 Implementation Details

  • LoRA training uses AdamW at learning rate 0.0003, rank r = 16, and 15 epochs.The 70-billion-parameter model uses DeepSpeed across two GPUs; smaller models use one GPU.Source: §4.3 · p. 4

5 Evaluation Metrics

Discrete recognition is evaluated with weighted F1, while continuous VAD prediction is evaluated with CCC. These metrics account for class imbalance and agreement with ground-truth scores, respectively.

5.1 Discrete ER

  • Weighted F1 evaluates discrete emotion recognition and weights each class’s F1 by its share of samples to account for class imbalance.Source: §5.1 · p. 5

5.2 VAD Evaluation

  • CCC evaluates VAD predictions by measuring correlation and agreement while penalizing mean offset and variance differences.CCC ranges from −1 to 1, with 1 indicating perfect agreement.Source: §5.2 · p. 5

6 Emotion Recognition Results

LoRA fine-tuned LLaMA models outperform prompt-engineered approaches in discrete emotion recognition and VAD evaluation. The best model establishes a new IEMOCAP Valence result, while Arousal and Dominance remain weaker and below prior best results.

7 VAD Evaluation Results

  • Across GPT prompting conditions, Valence CCC is 0.60–0.67, exceeding Arousal and Dominance at 0.13–0.39; GPT-5-mini outperforms GPT-4o-mini across most conditions.The largest stated Dominance difference is zero-shot CCC 0.299 for GPT-5-mini versus 0.146 for GPT-4o-mini.Source: §7 · p. 5
  • LLaMA-3.3-70B reaches a Valence CCC of 0.7822, establishing state-of-the-art IEMOCAP performance, while Arousal and Dominance remain competitive but below prior best results.Awatef et al. (2025) report higher Arousal (0.736 vs. 0.465) and Dominance (0.604 vs. 0.441) than this approach.Source: §7 · pp. 5–6

6 Emotion Recognition Results

  • Three LoRA fine-tuned LLaMA models achieved weighted F1 scores of 71.8–73.2, consistently outperforming prompt-engineered open-source models; fine-tuned models also surpassed all prompting-based approaches, including GPT-5-mini few-shot (59.6).LLaMA-3.3-70B zero-shot scored 60.299, performing comparably to GPT-4o-mini few-shot (56.8).Source: §6 · p. 5, Table 1 · p. 6

7 VAD Evaluation Results

  • On Valence, LLaMA-3.3-70B achieves a CCC of 0.7822 versus the best GPT-5-mini prompting result of 0.6697; smaller fine-tuned LLaMA models also score 0.7672 and 0.7433.LoRA also improves Arousal (CCC 0.44–0.48 vs. 0.34–0.39) and Dominance (around 0.44 vs. 0.12–0.30), though both remain below Valence.Source: §7 · p. 5, Table 2 · p. 6

8 Error Analysis for Discrete ER

Error analysis finds that GPT prompting struggles with anger–frustration confusions, while smaller prompted LLaMA models often collapse emotion labels. Across models, VAD prediction follows the same hierarchy as annotator agreement.

8 Error Analysis for Discrete ER

  • 46.5–75.3 confusion rates for GPT models on anger misclassified as frustration indicate reduced sensitivity to aggressiveness; GPT-5-mini also confuses Happy→Excite more than Excite→Happy.GPT-5-mini rates are 29.2–30.6 versus 18.4–22.1; GPT-4o-mini shows the reverse pattern, at 7.6–12.5 versus 22.7–29.8.Source: §8 · p. 6
  • Prompt-engineered LLaMA-2-7B and LLaMA-3.1-8B tend to collapse most emotions into one label, with excitement often misclassified as happiness.The analysis describes high excitement-to-happiness errors alongside low confusion rates for other emotion-label pairs.Source: §8 · p. 6

9 VAD Evaluation Performance Analysis

  • Across model families and training conditions, Valence is predicted better than Arousal, which is roughly level with Dominance, mirroring the annotator-agreement hierarchy.Valence has Krippendorff’s α = 0.680, compared with 0.304 for Arousal and 0.271 for Dominance; annotator disagreement makes the ground truth noisy.Source: §9 · pp. 6–7

8 Error Analysis for Discrete ER

  • All models except few-shot LLaMA-2-7B and LLaMA-3.1-8B perform best on Sad; GPT models have their lowest F1 scores on Happy and Angry.GPT models also generally improve slightly with few-shot prompting, and GPT-4o-mini performs similarly to GPT-5-mini on Frustrated and Sad.Source: §8 · p. 6, Table 4 · p. 7

10 Annotator Reliability

The section contextualizes model behavior with IEMOCAP annotator agreement and tests how audio descriptions and past VAD context affect predictions. Audio descriptions help smaller fine-tuned models, while past estimated VAD context has different effects for prompted and fine-tuned models.

10 Annotator Reliability

  • Valence has the highest inter-annotator agreement (α = 0.680), while Dominance is lowest (α = 0.271, κ = 0.014); discrete labels show α = 0.279 and κ = 0.273.Arousal agreement is intermediate (α = 0.304, κ = 0.085).Source: §10 · p. 7, Table 6 · p. 8

11.1 The Impact of Acoustic Feature Descriptions

  • Removing audio descriptions lowers weighted F1 from 73.2 to 69.7 for LLaMA-2-7B and 71.8 to 68.2 for LLaMA-3.1-8B, but leaves LLaMA-3.3-70B at 72.1.The authors suggest smaller models benefit from audio information, while the largest can partially recover prosodic context from linguistic patterns.Source: §11.1 · p. 7

11.2.1 GPT Models with Prompting

  • For GPT-5-mini, a 12-utterance past-VAD window raises Valence CCC from 0.663 to 0.689, with minimal or negative effects on Arousal and Dominance.Window size makes little difference overall.Source: §11.2.1 · p. 7

11.2.2 LoRA Fine-tuned LLaMA Models

  • Estimated past VAD generally degrades fine-tuned LLaMA performance; for LLaMA-3.3-70B, 12 prior utterances lower Valence CCC from 0.7822 to 0.5368.Ground-truth past VAD instead yields substantially higher performance than using no past scores; the authors attribute estimated-context degradation to propagated prediction errors.Source: §11.2.2 · pp. 7–8

12 Conclusion

The study applies LLMs to discrete emotion recognition and continuous VAD evaluation on IEMOCAP using conversation history, target utterances, and textual audio descriptions. It reports stronger results from LoRA fine-tuned LLaMA models than prompted GPT models, alongside findings on audio cues, annotator agreement, and past-VAD context.

12 Conclusion

  • The shared input construction combines conversation history, the target utterance, and textual audio feature descriptions, while treating discrete ER and VAD evaluation as independent tasks.LoRA fine-tuning uses a separately trained model instance for each task.Source: §12 · p. 8
  • Fine-tuned LLaMA models score 71.8–73.2 weighted F1 versus 54.7–59.6 for the best GPT prompting conditions, while LLaMA-3.3-70B reaches 0.7822 Valence CCC, a new IEMOCAP state-of-the-art.The authors attribute the gains to domain adaptation through parameter-efficient fine-tuning rather than general instruction-following capacity.Source: §12 · pp. 8–9
  • Removing audio descriptions reduces weighted F1 by 3.5–3.6 for LLaMA-2-7B and LLaMA-3.1-8B but has negligible effect on LLaMA-3.3-70B.The authors suggest larger models can partially recover prosodic context from linguistic patterns, while audio remains a cost-effective signal for smaller architectures.Source: §12 · p. 9
  • Valence's higher model performance and Arousal and Dominance's lower performance mirror the dimensions' annotator-agreement hierarchy.The authors link annotation noise to model-performance ceilings.Source: §12 · p. 9
  • Estimated past VAD context marginally helps prompt-engineered models but degrades LoRA fine-tuned models, which the authors attribute to error accumulation.The conclusion distinguishes the effect of model-estimated past scores from the informational value of past VAD itself.Source: §12 · p. 9

13 Limitation

  • The evaluation is restricted to IEMOCAP, whose discrete emotion, Arousal, and Dominance annotations have low inter-annotator agreement; converting raw audio to descriptions also loses some acoustic information.The authors identify both additional reliable multimodal dialogue datasets and direct processing of raw audio as future directions.Source: §13 · p. 9

A Methodology Pipeline Algorithm, Details, and Examples

The pipeline combines dialogue context and target-utterance audio descriptions for independent discrete-emotion and VAD inference. The accompanying material situates IEMOCAP among multimodal datasets and illustrates the pipeline and audio-description generation.

A Methodology Pipeline Algorithm, Details, and Examples

  • Figure 4 illustrates the proposed emotion-data pipeline, while Figure 5 visualizes conversion of IEMOCAP audio information into textual feature descriptions.The algorithm provides additional pipeline detail.Source: A Methodology Pipeline Algorithm,… · p. 11, Figure 4 · p. 12
  • Given a target utterance, its audio recording, and preceding conversation history, the pipeline outputs either an emotion label or three VAD scores, each on a 1–5 scale.Source: §B.1 · p. 12

B.1 Dataset Coverage by Modality

  • The algorithm separates task inference from model adaptation: it outputs a discrete emotion label or VAD scores and, in LoRA mode, fine-tunes separate models per task.The listed adaptation settings are r=16 and α=16.Source: §B.1 · p. 12

B Multimodal Emotion Dataset

  • IEMOCAP and CMU-MOSEI combine text, audio, and visual modalities with categorical and dimensional annotations, unlike text-only or primarily acted audio-visual datasets.IEMOCAP is described as 12 hours of audiovisual data; CMU-MOSEI contains 23.5k utterances.Source: §B.1 · pp. 11–12

B.2 Emotion Continuous Dimensional Annotations

Continuous dimensional emotion annotations remain less common than categorical labels across the described datasets. IEMOCAP and MEISD are identified as suitable resources for multimodal dimensional evaluation, but the authors select IEMOCAP because MEISD is unavailable to them.

B.2 Emotion Continuous Dimensional Annotations

  • Few datasets provide continuous emotion dimensions: IEMOCAP includes VAD ratings, while most datasets including MELD, EmotionLines, and GoEmotions provide categorical labels only.MEAD uses four intensity categories and MEISD provides intensity levels from 1–3 across eight emotion categories.Source: §B.2 · p. 12

B.3 Conversational Versus Non-Conversational Context

  • IEMOCAP and MEISD are identified as suitable for multimodal dimensional evaluation, but IEMOCAP is the final and only available choice because MEISD is unavailable.The passage contrasts conversational datasets, which preserve turn-taking and contextual emotion flow, with non-conversational datasets focused on isolated expressions.Source: §B.3 · pp. 12, 14

C Prompt Templates

The prompts use conversation context, the target utterance, and audio-feature descriptions for emotion analysis. Discrete emotion recognition selects one dominant label, while few-shot prompts add annotated examples covering all six categories.

C Prompt Templates

  • Reasoning was required in prompts for both discrete emotion recognition and VAD evaluation to boost performance.Source: C Prompt Templates · p. 14

C.1 Discrete ER Prompt

  • The discrete emotion prompt asks for one dominant label from six choices: happy, sad, neutral, angry, excited, or frustrated.Source: §C.1 · p. 14
  • Its inputs include conversation history, the target utterance, and audio features.Source: §C.1 · p. 14
  • The requested JSON output contains an emotion label and a reasoning field.Source: §C.1 · p. 14
  • Few-shot prompts prepend annotated input–output examples covering all six categories, including cases where audio features are diagnostic.Source: §C.1 · p. 14

C.2 VAD Evaluation Prompt

The VAD prompt rates the target utterance on three dimensions using conversation history and audio-feature descriptions. The section also identifies tables detailing past-VAD ablations and IEMOCAP summary statistics.

C.2 VAD Evaluation Prompt

  • VAD evaluation rates Valence, Arousal, and Dominance from 1 to 5, with higher scores indicating more positive, activated, and dominant affect, respectively.Source: §C.2 · p. 14
  • The VAD prompt inputs are conversation history, the target utterance, and audio features.Source: §C.2 · p. 14
  • The requested JSON output contains integer valence, arousal, and dominance scores, plus a reasoning field.Source: §C.2 · p. 14

C.3 Detailed Results for the Impact of Past VAD Values on VAD Evaluation Performance

Loading 2609.39072v1…