PaperlayerView exact revision on arXiv

Source-linked AI summary

Is ChatGPT Equipped with Emotional Dialogue Capabilities?

Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang, Yanpeng Tong, Bing Qin

arXiv:2304.09582v1cs.CL

TL;DR

This report examines whether ChatGPT exhibits emotional intelligence in dialogue by evaluating it across emotional dialogue understanding and generation tasks. ChatGPT shows useful generation capabilities, but its performance is constrained by dataset-label mismatches and advice-focused responses, while further prompt engineering and evaluation work remain necessary.

  • Problem

    Whether ChatGPT exhibits emotional intelligence in dialogue has not yet been explored, despite its advances in conversational technology.

    Source: §1 · p. 1

  • Method

    The report evaluates ChatGPT on multiple emotional dialogue tasks, including emotion-cause identification and empathetic response generation, using benchmark datasets and evaluations.

    Source: §1 · p. 1, §4.2.1 · p. 4, §5.1 · p. 7

  • Results

    ChatGPT generates informative and diverse emotional responses, but dataset-label inconsistencies and excessive advice can reduce alignment with expected emotions and empathy judgments.

    Source: §4.1.5 · p. 3, §5.1.4 · p. 8, §5.2.4 · p. 10

  • Takeaways & Limitations

    Future work should improve prompt engineering, align ChatGPT with labeling standards, and develop evaluation metrics that better capture emotional dialogue quality.

    Source: §4.1.5 · p. 3, §6 · p. 10

    The evaluation used simple prompting with no more than 3-shot examples, which may not reflect ChatGPT’s optimal downstream-task performance.

    Source: Limitations · p. 11

Abstract

from arXiv · show

This report presents a study on the emotional dialogue capability of ChatGPT, an advanced language model developed by OpenAI. The study evaluates the performance of ChatGPT on emotional dialogue understanding and generation through a series of experiments on several downstream tasks. Our findings indicate that while ChatGPT's performance on emotional dialogue understanding may still lag behind that of supervised models, it exhibits promising results in generating emotional responses. Furthermore, the study suggests potential avenues for future research directions.

1 Introduction

The report examines whether ChatGPT exhibits emotional intelligence in dialogue, focusing on emotional understanding and generation across downstream tasks. It frames this question against ChatGPT’s broader conversational advances and the applications of emotional dialogue technology.

1 Introduction

  • Emotional dialogue technology aims to help chatbots recognize, understand, and express emotions, supporting more engaging and human-like interactions.The report links this capability to open-domain chatbots, intelligent customer service, and voice assistants.Source: §1 · p. 1
  • ChatGPT has substantially improved semantic understanding and response generation in conversational robots.Source: §1 · p. 1
  • Figure 1 organizes emotional dialogue capability into understanding and generation dimensions evaluated through several downstream tasks.Source: Figure 1 · p. 1
  • The report investigates whether ChatGPT demonstrates emotional intelligence and analyzes its strengths, weaknesses, and future research directions.Source: §1 · p. 1

2 Task Settings

The task settings divide emotional dialogue capability into understanding and generation. The downstream evaluations ask whether ChatGPT interprets users’ emotions accurately and responds with empathy or support.

2 Task Settings

  • The evaluation framework compares ChatGPT across two dimensions: emotional understanding and emotional response generation.Source: §2 · pp. 1–2
  • Understanding tasks test whether ChatGPT can accurately understand and interpret the user’s emotions.Source: §2 · p. 1
  • Generation tasks test whether ChatGPT can elicit empathy or support toward the user’s emotional state.Source: §2 · pp. 1–2
  • The report provides detailed task definitions and evaluations in subsequent sections.Source: §2 · p. 2

3 Evaluation Methods

The evaluation uses state-of-the-art results from original task papers as comparisons and tests ChatGPT with the gpt-3.5-turbo API in zero-shot and few-shot settings.

3 Evaluation Methods

  • State-of-the-art comparison results are taken directly from the original papers for each task.Source: §3 · p. 2
  • ChatGPT is evaluated with the gpt-3.5-turbo model using both zero-shot and few-shot prompting settings.The tested API version was current through March 8.Source: §3 · p. 2

4.1 Emotion Recognition

This section defines Emotion Recognition in Conversations, evaluates ChatGPT against supervised baselines across four datasets, and analyzes prediction discrepancies and prompting limitations.

4.1.1 Task Definition

  • ERC maps a continuous dialogue to emotion labels for all utterances, requiring conversation background, context, and speaker information.The task is distinct from single-sentence emotion recognition because it evaluates utterances within their conversational setting.Source: §4.1 · p. 2, §4.1.1 · p. 2

4.1.2 Dataset and Evaluation Metrics

  • The evaluation covers IEMOCAP, MELD, EmoryNLP, and DailyDialog, using Weighted-F1 for the first three and Micro-F1 excluding neutral for DailyDialog.These datasets provide varied conversational sources and emotion categories.Source: §4.1.2 · pp. 2–3

4.1.3 Main Results

  • The study compares ChatGPT with pretrained-model and commonsense-based methods, alongside established ERC baselines and current state-of-the-art models.ChatGPT is evaluated in zero-shot and few-shot settings using the gpt-3.5-turbo public API.Source: §4.1.3 · p. 3
  • ChatGPT generally trails advanced fine-tuned models by 3–18 percentage points on ERC.The comparison uses Table 1 results across the evaluated ERC datasets.Source: §4.1.3 · p. 3, Table 1 · p. 3

4.1.4 Case Study

  • A doctor–patient example shows ChatGPT and dataset annotations assigning different emotions to the same utterance.The dataset labels the patient’s headache description neutral, whereas ChatGPT labels it sadness.Source: §4.1.4 · p. 3, §4.1.5 · p. 3, Table 2 · p. 4

4.1.5 Analysis and Discussion

  • The analysis attributes these disagreements to differing interpretive standards and notes that few-shot prompting cannot fully encode intricate annotation guidelines.The authors suggest prompting improvements when strict guideline conformity is not required, but supervised fine-tuning may remain preferable when it is.Source: §4.1.5 · pp. 3–4

4.2 Emotion Cause Recognition

The CEE task tests whether ChatGPT can identify conversation-history utterances that cause a target utterance’s non-neutral emotion. On RECCON-DD, ChatGPT performs comparably on negative causal pairs but remains below the SOTA supervised model overall, with errors linked to annotation-standard mismatches.

4.2.1 Task Definition

  • CEE predicts which conversation-history utterances trigger the target utterance’s non-neutral emotion, labeling each candidate as positive or negative.Source: §4.2.1 · p. 4, §4.2 · p. 4

4.2.2 Dataset and Evaluation Metrics

  • The evaluation uses RECCON-DD causal pairs and reports F1 for negative pairs, positive pairs, and their macro average.Source: §4.2.2 · p. 4

4.2.3 Main Results

  • ChatGPT is evaluated in zero-shot and one-shot settings against pretrained-model and commonsense-based supervised baselines.Source: §4.2.3 · p. 5
  • ChatGPT has a 11.95% macro F1 gap compared to the SOTA supervised method, despite comparable performance on negative causal pairs.Source: §4.2.3 · p. 5

4.2.4 Analysis and Discussion

  • ChatGPT often overlooks emotionally charged content inside the target sentence, while differing from the dataset’s annotation standard.Source: §4.2.4 · p. 5

4.3 Dialog Act Classification

Dialogue Act Classification maps each utterance in a contextual conversation to a dialogue-act label, evaluated on DailyDialog with weighted-F1 because ChatGPT produced labels outside the four categories. ChatGPT remained 11–17 percentage points behind fine-tuned baselines, while few-shot prompting improved performance but label-system alignment remained a concern.

4.3.1 Task Definition

  • Dialogue Act Classification labels the dialogue act of every utterance in a multi-utterance conversation, requiring conversational context.Given C = {u1, u2, ..., uN}, the task outputs a corresponding label sequence Z = {z1, z2, ..., zN}.Source: §4.3 · p. 5, §4.3.1 · p. 5

4.3.2 Dataset and Evaluation Metrics

  • The study evaluates DAC on DailyDialog using weighted-F1 because ChatGPT generated meaningless labels beyond the dataset’s four categories.Both weighted-F1 and macro-F1 were considered, but weighted-F1 was used for evaluation.Source: §4.3.2 · p. 5, Table 4 · p. 6

4.3.3 Main Results

  • ChatGPT generally trails fine-tuned baselines by 11–17 percentage points on Dialogue Act Classification.The comparison uses ChatGPT and baseline models including WEAKDAP, CASA, DCR-Net, and Co-GAT.Source: §4.3.3 · p. 6, Table 5 · p. 6

4.3.4 Case Study

  • A DailyDialog case study shows ChatGPT conflating questions with directives and informs with commissives when label definitions semantically overlap.Examples include “Can you come today?” as a directive question and a promise as a commissive inform.Source: §4.3.5 · p. 6, §4.3.4 · p. 6

4.3.5 Analysis and Discussion

  • The analysis attributes errors partly to mismatch between ChatGPT’s labeling standards and dataset guidelines rather than solely to model capability.The authors also note that the prompts were not intricate enough to test ChatGPT’s performance upper limit.Source: §4.3.5 · p. 6
  • Few-shot prompting was the most effective enhancement in this DAC experiment, using three examples without complex prompt engineering.The authors suggest that selecting more examples and refining prompts could reduce the difference between ChatGPT’s labels and the dataset’s labels.Source: §4.3.5 · p. 6

5 Generation Capability

The report evaluates ChatGPT’s emotional response generation through empathetic response generation and emotional support conversation tasks. ChatGPT produces longer, more diverse, and informative responses, but its advice-oriented style can reduce empathy alignment and lexical accuracy.

5.1.1 Task Definition

  • Empathetic response generation uses dialogue history and one of 32 emotion labels to produce a coherent response demonstrating empathy toward the user’s situation.Experiments use the 25k-conversation EMPATHETICDIALOGUES dataset and BLEU, ROUGE-L, and Distinct-n metrics, with human ratings of coherence, empathy, and informativeness.Source: §5.1.1 · p. 7, §5.1.2 · pp. 7–8

5.1.3 Main Results

  • ChatGPT generates longer and more diverse empathetic responses than baselines, averaging 21.77 tokens versus 14.89 tokens.This increased diversity may increase mismatch with golden responses.Source: §5.1.3 · p. 8

5.1.4 Analysis and Discussion

  • In empathetic response generation, ChatGPT provides context-specific information and expands on users’ situations, while baseline coherence and empathy remain comparable.Human evaluation found a significant difference in the amount of information provided.Source: §5.1.4 · p. 8
  • ChatGPT often suggests solutions for users in negative emotional states, which can deviate from reference responses and worsen word-overlap metrics.Its empathy expression also frequently restates the user’s emotion before adding information.Source: §5.1.4 · p. 8
  • Future work should improve personalized empathy, adaptively control the rhythm of emotional support, and develop automatic metrics that better align with human evaluation.The report notes that template-like empathy and differences between automatic and human evaluations remain unresolved.Source: §5.1.4 · pp. 8–9, §5.2.4 · p. 10

5.2.1 Task Definition

  • Emotional support conversation generates a supportive response that helps alleviate a seeker’s distress, using dialogue history and annotated support strategies.The task includes exploring the problem, offering comfort, and providing advice or information.Source: §5.2 · p. 9, §5.2.1 · p. 9

5.2.4 Analysis and Discussion

  • On the ESC task, ChatGPT generates longer and more diverse responses that outperform state-of-the-art methods on Distinct-n, although this diversity can also deviate from golden responses.The dataset contains 1,300 long dialogues and 38,350 utterances.Source: §5.2.2 · p. 9, §5.2.4 · p. 10
  • Human evaluation finds ChatGPT effective at providing information and suggestions in emotional support, but less empathetic than SOTA methods because it gives advice too quickly.The authors suggest prompt engineering could help ChatGPT slow down and provide emotional guidance before advice.Source: §5.2.4 · p. 10

6 Conclusion

The report presents a preliminary exploration of ChatGPT’s emotional conversation capability. It concludes that stronger prompting may improve results, while emotional understanding and generation still require better alignment and evaluation.

6 Conclusion

  • The report is a preliminary exploration whose results may not reflect ChatGPT’s optimal performance.The authors believe refined prompt engineering and context example selection could further improve performance.Source: §6 · p. 10
  • Future emotional dialogue understanding should examine alignment between ChatGPT and dataset labeling standards.For generation, the report calls for reasonable automatic metrics because automatic and human evaluations may differ.Source: §6 · p. 10

Limitations

The evaluation is limited by its model selection, low-resource prompting, and small human-evaluation sample. These constraints may prevent the results from representing ChatGPT’s best or broadly generalizable performance.

Limitations

  • The study evaluates only the gpt-3.5-turbo variant, so it does not comprehensively represent the abilities of other LLMs.The authors identify broader evaluation across models, including GPT-4, as necessary future work.Source: Limitations · pp. 10–11
  • Simple prompt engineering and at most 3-shot prompting may not reflect ChatGPT’s optimal downstream-task performance.This limitation arises from the study’s low-resource setting.Source: Limitations · p. 11
  • Human evaluation relies on a small group whose preferences may not represent a broader audience.The authors therefore call for a more objective and universal evaluation method.Source: Limitations · p. 11

A Definitions of Strategies in ESC

The ESCONV dataset annotates eight support strategies for emotional support dialogue. They range from understanding and validating the seeker’s experience to offering suggestions and information.

A Definitions of Strategies in ESC

Related papers

Loading 2304.09582v1…