PaperlayerView exact revision on arXiv

Source-linked AI summary

EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidance

Xiaolin Chen, Xuemeng Song, Jinlan Fu, Weili Guan, Mong-Li Lee, Wynne Hsu

arXiv:2609.38182v1cs.HCcs.CVcs.MM

TL;DR

Avatar-based empathetic generation must infer user emotions and produce synchronized text, speech, and talking-face video, but existing methods overlook cross-modal conflicts, expressive guidance, and text-stage error propagation. EmAvatar uses deliberative evidence gathering and iteratively refined scripts; experiments report gains over state-of-the-art methods across four tasks.

Abstract

from arXiv · show

Avatar-based multimodal empathetic response generation has emerged as a pivotal capability in human-centric systems, aiming to recognize user emotions and synthesize responses with synchronized text, audio, and talking-face video. Despite recent progress, existing methods still suffer from three critical limitations: (1) overlooking conflicting emotions across modalities, (2) lacking explicit multimodal synthesis guidance, and (3) neglecting inherent error propagation of multimodal response generation. To address these limitations, we propose EmAvatar, a novel framework for precise emotion perception and expressive response generation. It first performs deliberative multimodal emotion recognition by exposing inter-modal prediction conflicts and then initiates a multi-round QA process between a Conflict Inspector and an Evidence Collector to gather evidence for conflict resolution, leading to a robust, evidence-aware prediction. Regarding response generation, EmAvatar first synthesizes a composite script that couples the textual response with an expressive instruction. Moreover, to ensure high-quality synthesis, an iterative refinement mechanism evaluates and revises the script until it aligns with predefined criteria, serving as reliable guidance for subsequent audio and video synthesis. Extensive experiments across four tasks demonstrate that EmAvatar outperforms state-of-the-art methods. Our code will be publicly released.

Introduction

Avatar-based empathetic response generation combines emotion perception from multimodal input with synchronized text, expressive speech, and talking-face video. EmAvatar addresses cross-modal emotion conflicts and response-generation gaps with deliberative reasoning and refined expressive scripts.

Introduction

  • Avatar-based systems infer emotions from multimodal signals and generate synchronized textual, spoken, and talking-face responses.The motivation includes emotionally sensitive settings such as mental health support, education, and companionship.Source: Introduction · p. 1
  • Existing methods can miss conflicting emotional cues across modalities, such as positive wording paired with an irritated voice and facial expression.The cited example is “Great, another meeting” said in a superficially positive tone while voice and facial expression convey irritation.Source: Introduction · p. 1

Composite Script-Driven Empathetic Response Generation

  • Existing response-generation methods lack explicit expressive guidance and quality checks, while text-stage errors can propagate into audio and video.Without explicit guidance, outputs can have emotionally mismatched tone or facial expressions.Source: Composite Script-Driven Empathetic… · p. 2
  • EmAvatar exposes emotion-prediction conflicts for evidence-based resolution and uses iteratively refined composite scripts to guide expressive synthesis.The framework couples a textual response with an expressive instruction; experiments across four tasks demonstrate its effectiveness.Source: Composite Script-Driven Empathetic… · p. 2

Related Work

Empathetic response research has moved from text-only systems toward multimodal understanding and response generation. The latter still faces gaps in conflict handling, expressive guidance, and text quality.

Related Work

Methodology

The method targets training-free emotion recognition and multimodal empathetic response generation from dialog history, user input, and an avatar image. It combines parallel emotion prediction, evidence-based deliberation, and guided response synthesis.

Problem Formulation

  • The task is to recognize a user’s emotion and generate a corresponding multimodal response from dialog history, current input, and a target avatar image.The input and target response each include text, speech audio, and talking-face video.Source: Problem Formulation · p. 3

Deliberative Multimodal Emotion Recognition

  • Deliberative emotion recognition uses parallel predictions followed by evidence-enhanced reasoning to resolve inter-modal conflicts.The method treats accurate emotion prediction as a prerequisite for precise empathetic responses.Source: Deliberative Multimodal Emotion… · p. 3

Multi-Perspective Parallel Perception

  • Four emotion predictions come from text, audio, video, and a holistic multimodal view, forming the initial candidate set.The described predictors are Qwen3, emotion2vec, DeepFace, and Qwen3-Omni, respectively.Source: Multi-Perspective Parallel Perception · p. 3

Evidence-Enhanced Deliberative Reasoning

  • When predictions disagree, a Conflict Inspector asks discriminative questions, an Evidence Collector answers from the multimodal input, and an Arbiter integrates the evidence and candidates.The Inspector’s questions use the initial prediction set and prior QA trajectory; after K QA rounds, the Arbiter produces the final emotion.Source: Evidence-Enhanced Deliberative Reasoning · pp. 3–4

Composite Script-Driven Empathetic Response Generation

  • Conditioned on the recognized emotion, the generation module creates a content-synchronized response across text, speech audio, and talking-face video.The module has two parts intended to guide emotional expression in audio and video.Source: Composite Script-Driven Empathetic… · p. 4

1) High-Quality Composite Script Response Generation,

A composite script pairs the textual response with an instruction for expressing it in audio and video.

1) High-Quality Composite Script Response Generation,

  • The script combines a textual response with an expressive instruction specifying how audio and video should realize it.One example instruction is to speak with a soothing, slow-paced tone.Source: 1) High-Quality Composite Script… · p. 4

2) Composite Script-Driven Multimodal Synthesis,

EmAvatar turns summarized dialogue history into a composite script pairing a textual response with expressive guidance, then refines that script before rendering audio and video. The refiner checks coherence, empathy, feasibility, and alignment, while synthesis conditions speech and the talking-face video on the script and audio, respectively.

2) Composite Script-Driven Multimodal Synthesis,

  • The Dialog History Summarizer filters irrelevant material from verbose multimodal history into a salient context summary for response generation.This differs from feeding the full raw history directly into generation.Source: 2) Composite Script-Driven Multimodal… · p. 4
  • The Composite Script Generator combines the current input, predicted emotion, and history summary into a textual response plus an expressive instruction for audio and video.The instruction specifies how the textual response should be expressed in both modalities.Source: 2) Composite Script-Driven Multimodal… · p. 4
  • The Iterative Quality Refiner assesses coherence, empathy, feasibility, and alignment to keep the script suitable for downstream synthesis.These dimensions respectively cover history consistency, response appropriateness, instruction executability, and emotional consistency between response and instruction.Source: 2) Composite Script-Driven Multimodal… · p. 4
  • At each iteration, the refiner scores the candidate script on a five-level scale and uses identified deficiencies to guide revisions until the predefined threshold is met.The resulting script contains the refined textual response and expressive instruction.Source: 2) Composite Script-Driven Multimodal… · p. 4
  • The audio synthesizer renders speech from the refined text and expressive instruction, conditioned on retrieved acoustic references associated with the specified avatar.The method uses CosyVoice2 for speech synthesis.Source: 2) Composite Script-Driven Multimodal… · p. 4
  • The video synthesizer animates the avatar image from synthesized audio, with predicted emotion and avatar identity as additional constraints.The audio-driven video model receives no separate expressive instruction because it is already embedded in the synthesized audio.Source: 2) Composite Script-Driven Multimodal… · p. 4

Experiment

Across four tasks, EmAvatar is evaluated against competitive baselines, with results favoring its conflict-resolution and expressive-guidance framework and tests examining ablations, generalization, sensitivity, efficiency, and example cases.

Experimental Setup

  • Experiments use AvaMERG, which provides synchronized text, speech, and talking-face video across 33,048 dialogues, and compare EmAvatar with task-specific competitive baselines.The evaluation covers emotion recognition and textual, speech, and talking-face video generation.Source: Experimental Setup · p. 5

Comparison with State-of-the-Art Methods

Ablation Study

  • EmAvatar outperforms w/o-Conflict, Major-Vote, and Only-QA in emotion recognition, while the full model also beats w/o-Refiner on text and w/o-Instruction on speech.The ablations compare conflict resolution with direct prediction, voting, or QA alone, and test refinement and expressive guidance.Source: Ablation Study · p. 6, Table 5 · p. 6, Table 7 · p. 6

Generalization Study

  • The deliberative framework gains consistently across five backbones, while expressive instructions improve both VoxCPM and CosyVoice3 across all reported metrics.These tests examine backbone generalization and instruction transfer to other speech models.Source: Table 6 · p. 6, Table 7 · p. 6, Generalization Study · p. 6

Sensitivity Analysis

  • With 1, 2, and 3 QA rounds, accuracy is 70.75%, 70.87%, and 70.78%; more than 2 refinement iterations brings no consistent gain.The study identifies 2 rounds and 2 iterations as a favorable quality-cost balance.Source: Sensitivity Analysis · p. 6, Table 8 · p. 7

Inference Efficiency Analysis

  • The full pipeline takes 106.93 seconds, only 9.92 seconds above baseline; response generation takes 94.57 seconds versus the 95.32-second baseline.Video synthesis is the largest component at 61.81 seconds; emotion recognition takes 12.36 seconds, including 8.03 seconds for the Evidence Collector.Source: Inference Efficiency Analysis · pp. 6–7, Table 9 · p. 7

Case Study

  • In the examples, deliberative reasoning corrects a video-branch emotion conflict, while script refinement replaces a dismissive response and generic tone with validating text and an empathetic instruction.The refined script satisfies all four criteria; Figure 3 illustrates conflict resolution and Figure 4 marks satisfied and unsatisfied refinement criteria.Source: Figure 3 · p. 7, Figure 4 · p. 7, Case Study · p. 7

Conclusion

EmAvatar combines deliberative multimodal emotion recognition with composite-script-driven response generation. Experiments across four tasks, case studies, and refinement experiments support its effectiveness, while inference efficiency remains a future focus.

  • EmAvatar pairs deliberative multimodal emotion recognition with composite-script-driven response generation for explicit expressive guidance.Source: Conclusion · p. 7
  • Experiments across four tasks quantitatively and qualitatively verify EmAvatar’s effectiveness.Source: Conclusion · p. 7
  • Case studies confirm cross-modal emotion conflicts and show that deliberative reasoning mitigates them.Source: Conclusion · p. 7
  • Experiments show iterative quality refinement is crucial for reliable composite scripts and precise guidance in expressive multimodal synthesis.Source: Conclusion · p. 7
  • The authors identify inference efficiency for real-time applications and practical deployment as future work.Source: Conclusion · p. 7
Loading 2609.38182v1…