PaperlayerView exact revision on arXiv

Source-linked AI summary

EmoLASP: Emotion Recognition with Language Models and Answer Set Programming

Thao Le, Michael Thielscher

arXiv:2608.29035v1cs.AIcs.CL

TL;DR

Emotion-recognition language models can be unstable and costly when relying on fine-tuning or long dialogue histories. EmoLASP adds Answer Set Programming to generate and select context-consistent VAD hypotheses, improving MAE for most prompt-only LLMs and beating a history-fed baseline for 5 of 6 LLMs without fine-tuning. The study also identifies scope limitations from evaluating on one dataset and using fixed framework settings.

  • Problem

    Language-model approaches to conversational emotion recognition can be unstable and costly because they may require fine-tuning or long dialogue-history prompts.

  • Method

    EmoLASP combines a language model with an ASP reasoner that aggregates and optimizes VAD hypotheses using conversational history.

  • Results

    EmoLASP improves MAE for most prompt-only LLMs and beats a history-fed baseline for 5 of 6 LLMs while seeing only the current utterance.

  • Takeaways & Limitations

    A small reasoner can replace long prompt histories for LLM-based emotion recognition at lower input-token cost and without fine-tuning.

  • Takeaways & Limitations

    The evaluation uses a single dataset, while the framework’s main experiments fix the aggregation window, hypothesis count, and confidence threshold.

Abstract

from arXiv · show

Emotion recognition in conversations is increasingly tackled with language models, but these models can be unstable and expensive to fine-tune or to prompt with long dialogue histories. We propose EmoLASP, a framework that combines a language model with declarative reasoning via Answer Set Programming (ASP) to predict VAD scores (Valence-Arousal-Dominance) in conversations. Experiments on a widely used benchmark dataset (IEMOCAP) across six open-source LLMs (3B-120B) and two PLMs (BERT, RoBERTa) show that EmoLASP improves prediction performance compared to using the language model alone, even when the LLMs/PLMs are given no dialogue history in their prompts or input vectors. The gains are largest for prompt-only LLMs, which EmoLASP uses without any fine-tuning. However, for fine-tuned PLMs, the reasoner adds little once dialogue history is available. EmoLASP's LLM pipeline demonstrates the potential advantages of using a reasoning approach to ensure emotion prediction consistency and to reduce both the cost of fine-tuning and the cost of prompting with long dialogue histories.

1 Introduction

Emotion recognition in conversations benefits from dimensional VAD representations, but language-model approaches remain unstable, costly, and limited in their use of dialogue continuity. EmoLASP addresses these issues by combining language models with declarative Answer Set Programming reasoning.

  • ERC identifies the emotional state expressed in each utterance and supports applications including empathetic dialogue systems and mental health support.
  • VAD represents emotion dimensionally through valence, arousal, and dominance rather than categorical labels.
  • Language models can be unstable, biased, and unable to ensure logical continuity of emotional states across conversations.
  • Using language models alone can be costly because of domain-specific fine-tuning and long dialogue-history prompts.
  • EmoLASP combines language models with declarative reasoning through Answer Set Programming for emotion recognition.
  • ASP considers lower-confidence alternatives and uses weak constraints to prefer VAD hypotheses consistent with prior conversational turns.
  • The framework is presented as the first work combining language models and declarative reasoning for ERC, evaluated across LLMs and PLMs on a benchmark dataset.

2 Method

EmoLASP uses a two-layer pipeline in which a language model proposes VAD hypotheses and an ASP reasoner aggregates and optimizes them using conversational history. Confidence thresholds determine when reasoning is invoked, while weak constraints favor context-consistent and highly ranked hypotheses.

  • The pipeline combines an LLM or PLM with an ASP program implemented in Clingo.
  • For each utterance, the language model generates ranked VAD hypotheses that the ASP reasoner aggregates and optimizes into one prediction.
  • LLM prompts use either the current utterance alone or the full preceding dialogue history, with nohist reducing input tokens.
  • PLMs BERT and RoBERTa are fine-tuned with separate classification heads for the three VAD dimensions and Likert scores from 1 to 5.
  • Reasoning with Answer Set Programming: ASP is invoked only below a predefined confidence threshold; otherwise, the model’s top-ranked hypothesis is used directly.
  • Reasoning with Answer Set Programming: The confidence score is based on normalized Shannon entropy over the model’s hypothesis probabilities and lies in [0, 1].
  • Reasoning with Answer Set Programming: Aggregation averages predicted VAD scores from the most recent k turns without using gold labels during inference.
  • Reasoning with Answer Set Programming: Weak constraints first minimize distance to the history aggregate, then select the higher-ranked language-model hypothesis when distances tie.

3 Experiment

The experiments evaluate EmoLASP on IEMOCAP using MAE and Krippendorff’s α across open-source LLMs and fine-tuned PLMs under history and no-history conditions. Paired tests compare the reasoner against language-model-only baselines.

  • Dataset: IEMOCAP contains 115 dialogues across 5 dyadic sessions and 10,039 utterances with 5-point ordinal VAD annotations.
  • Dataset: Final VAD labels average ratings from multiple human annotators, with most utterances receiving two annotations.
  • Metrics: MAE measures average absolute error against averaged human ratings, while Krippendorff’s α measures agreement with ordinal human annotations.
  • Metrics: Human-only Krippendorff’s α is reported as a reference for inter-annotator agreement.
  • Baselines: The comparison includes open-source LLMs, BERT, and RoBERTa under nohist and hist context settings.

4 Results

EmoLASP generally improves emotion prediction, especially for prompt-only LLMs, while its benefits are limited for fine-tuned PLMs with dialogue history. It reduces history-related token costs but remains limited on extreme emotions expressed through brief or metaphorical language.

  • 16 of 24 MAE comparisons show significant improvement with EmoLASP, while 7 show no significant difference and 1 shows a decrease.The improvements are concentrated in the prompt-only LLM pipeline.
  • 14 of 18 prompt-only LLM comparisons improve significantly, whereas only 2 of 6 fine-tuned PLM comparisons do.The two significant PLM improvements occur in no-history settings.
  • 5 of 6 LLMs significantly outperform the full-history baseline when EmoLASP uses only the current utterance.For llama3.3:70b, MAE drops from 0.91 to 0.81 (p < 0.001), with fewer input tokens and no fine-tuning.
  • Fine-tuned PLMs gain little under the handicap comparison: bert shows no benefit, while roberta-large worsens from 0.72 to 0.77.These results motivate restricting the cost claim to prompt-only LLMs.
  • Krippendorff’s α is unchanged in 13 of 24 comparisons, improves in 3, and decreases in 8; the best model α of 0.32 remains below human agreement of 0.39.Excluding the two gemma3 models, matched-context comparisons are unchanged or improved.
  • All tested models predict mid-range VAD values around 2 to 4 for some utterances annotated at the extremes of 1 or 5.ASP cannot select extreme values absent from the language model’s hypotheses.
  • Brief or surface-neutral utterances and figurative language produce persistent errors on extreme emotions.Examples include “Hmm,” caring questions with low dominance, and metaphors interpreted with overly positive valence.

5 Conclusions

EmoLASP adds an ASP reasoner to a language model and improves MAE for most prompt-only LLMs while requiring only the current utterance. The approach preserves annotator agreement in most comparisons but remains vulnerable to extreme emotions in brief or metaphorical language.

  • EmoLASP improves MAE for most prompt-only LLMs, while fine-tuned PLMs gain little because they already use dialogue history well.
  • EmoLASP beats a history-fed baseline for 5 of 6 LLMs using only the current utterance, no fine-tuning, and fewer input tokens.
  • Agreement with human annotators mostly remains unchanged, so the MAE gains do not come at the cost of agreement.
  • LLM-based approaches struggle with extreme emotions expressed in brief utterances or metaphorical language.

Limitations

The study’s limitations concern its data coverage, framework design, and evaluation scope. These constraints affect generalisation, hypothesis generation, and comparability across models.

  • Data: Evaluation uses only IEMOCAP, the only publicly available dataset identified with dialogue-level human VAD annotations.IEMOCAP contains acted, multimodal dialogue data, while the models use text only.
  • Data: Text-only modelling does not use IEMOCAP’s audio and visual cues, which may contribute to disagreement with human annotations.Most utterances have only two annotators, and annotator counts vary across utterances.
  • Framework: The ASP implementation is modest, and its fixed hyperparameters were swept one at a time on a single dataset.Optimal settings may differ on other datasets or under joint hyperparameter search.
  • Framework: EmoLASP still struggles with very short utterances and figurative language, limitations linked to hypothesis generation rather than only ASP selection.Addressing these cases may require changes to the language models’ generation process.
  • Evaluation: The evaluation excludes directly comparable categorical-emotion ERC baselines and tests only open-source LLMs.The PLM setup also assumes independence among the three VAD dimensions, and conclusions may not generalise to closed-source models.

Ethics Considerations

The paper discusses ethical risks, dataset limitations, and prior uses of ASP for affective computing. It also situates EmoLASP among LM-ASP pipelines for reasoning and dialogue systems.

  • Risks: VAD predictors could be repurposed for affect-based advertising or surveillance without user consent.The paper frames empathetic dialogue and mental health support as its motivating applications.
  • Risks: Most evaluated LLMs compress VAD predictions toward the middle of the scale and miss extreme states in short or figurative utterances.The paper warns that this could underestimate distress in safety-critical mental-health settings, a limitation EmoLASP does not overcome.
  • Risks: IEMOCAP’s acted English dyadic dialogues from one cultural setting limit generalisation to other populations and languages.The corpus was recorded in a single US university drama-department setting.
  • Related work: Prior affective-computing systems use ASP to model user emotion, reason about emotional dynamics, or support agent planning.Examples include an E-friend chatbot and emotion-aware agent planning.
  • Related work: LM-ASP pipelines have been applied to NLU and task-oriented dialogue, but the cited systems had not applied the combination to emotion understanding.Those dialogue systems used LLMs to extract knowledge and ASP to choose actions such as recommendations or answers.

C Experiment Details

The experiments evaluate EmoLASP across multiple models, contexts, and reasoner settings. Prompt templates vary by whether dialogue history is included, while ablations examine reasoner hyperparameters and speaker weighting.

  • Compute: 80 LLM jobs cover 8 models, 2 context conditions, and 5 dialogue sessions.The experiments use shared NVIDIA V100, A100, and H200 infrastructure with one GPU per job.
  • PLM settings: Two PLMs, BERT and RoBERTa-large, are fine-tuned for 20 epochs with learning rate 2e-5, batch size 16, and maximum context length 512.BERT has approximately 110 million parameters and RoBERTa-large approximately 355 million.
  • LLM settings: Six LLMs range from 3 billion to 120 billion parameters, using temperature 0, m = 3, τ = 0.25, and k = 2.The six-LLM runs are estimated to require approximately 170 hours.
  • Reasoner settings: The confidence threshold τ = 0.25 applies ASP when the best of three hypotheses has probability below 0.7.The threshold follows from normalized entropy under the stated probability example.
  • C.3 Prompt Templates: The hist prompt includes prior same-dialogue turns with speaker labels, whereas the nohist prompt excludes dialogue context.The prompt templates use a 5-point Likert VAD scale based on the Self-Assessment Manikin.
  • C.3 Prompt Templates: Table 3 sweeps one of k, m, or τ while holding the other two at k = 2, m = 3, and τ = 0.25.Results aggregate across six LLMs and two PLMs after averaging over five cross-session splits.
  • C.3 Prompt Templates: Table 4 compares speaker-weighting ratios while holding the remaining reasoner hyperparameters at k = 2, m = 3, and τ = 0.25.The 1:1 ratio is the speaker-blind default.

D Results

The speaker-weighted ablation replaces equal weighting of recent history with speaker-dependent weights, while leaving optimization unchanged. The weighting ratio controls whether the aggregate emphasizes the target speaker or the interlocutor.

  • Speaker-weighted aggregation assigns wself to history turns from the target speaker and wother to turns from the interlocutor.
  • The weighted aggregate replaces the equal-weight aggregate used in the main method.
  • When wself > wother, the aggregate shifts toward the speaker’s previous turns; the reverse ratio favors the interlocutor.
  • The optimization step remains unchanged except that it uses Asw_i instead of Ai, with ties broken by the higher-ranked hypothesis.

D.2 Ablation Studies

Ablation studies vary history size, hypothesis count, confidence threshold, and speaker weighting. EmoLASP remains stable across settings, so the authors retain default hyperparameters and equal speaker weights.

  • Selected configuration: The authors retain k = 2, m = 3, τ = 0.25 with equal speaker weights because tuning has little effect and the reasoner has no trained parameters.
  • Hyperparameter ablations: Across twelve hyperparameter settings, MAE remains 0.75–0.79 and Krippendorff’s α remains 0.25–0.28.These variations are smaller than model-level standard deviations of up to ±0.09.
  • Hyperparameter ablations: Longer history or more hypotheses lowers MAE by about 0.02, but does not improve α and sometimes slightly decreases it.
  • Hyperparameter ablations: Raising τ from 0.1 to 0.6 causes ASP to run on more turns while barely changing scores.
  • Speaker-weighted aggregation: Speaker-weighted variants produce nearly identical results: MAE is 0.78 without history and 0.77–0.78 with history, while α is 0.26–0.28.

D.3 Qualitative Analysis

The qualitative analysis identifies turns where both the history-based No Reasoner and EmoLASP predictions have high error. These cases remain difficult for the evaluated systems even after ASP reasoning.

  • The analysis focuses on the five highest-error cases by MAE where all LLMs fail to predict accurate VAD scores after ASP.
  • Table 5 lists turns where both No Reasoner (hist) and EmoLASP (hist) predictions have high error.
Loading 2608.29035v1…