PaperlayerView exact revision on arXiv

Source-linked AI summary

Closing the Affective Loop: Multimodal Speaker-Listener Emotion-Dynamics-Aware Empathetic Social Robots

Zi Haur Pang, Casey Kennington, Tatsuya Kawahara

arXiv:2608.16686v1cs.HCcs.CLcs.RO

TL;DR

Empathetic social robots rarely model the dynamic affective exchange between human speakers and robot listeners across verbal and nonverbal signals. AffectLoop conditions spoken responses and embodied behavior on both speaker emotional dynamics and the robot listener’s affective state, receiving higher ratings for empathetic response and user satisfaction and showing stronger affective alignment and distress recovery than its baseline.

  • Problem

    Existing empathetic robot systems rarely model dynamic affective exchange between human speakers and robot listeners across multimodal signals and the listener’s own affective state.

  • Method

    AffectLoop conditions LLM-based spoken response generation and emotionally congruent robot behavior on coupled speaker verbal/facial dynamics and listener verbal/behavioral affective state.

  • Results

    Overall impression ratings increased from 4.75 to 5.10, with empathetic response rising from 4.80 to 5.35 and user satisfaction from 4.20 to 4.68 versus baseline.

  • Takeaways & Limitations

    The pilot provides preliminary evidence that incorporating speaker emotional dynamics and listener affective state can make social-robot responses feel more supportive and improve the interaction’s affective process.

Abstract

from arXiv · show

Empathetic social robots should respond not only to what users say, but also to how their emotions dynamically evolve during interaction. However, existing empathetic dialogue systems are often text-centered and primarily model empathy as a one-way mapping from the user's emotion to the system response, limiting their ability to capture embodied speaker--listener affective exchange. We present AffectLoop, a multimodal speaker-listener emotion-dynamics-aware spoken dialogue system implemented on the Misty II robot. The system tracks the speaker's verbal and facial affective dynamics, estimates the robot listener's own verbal and behavioral affective state, and conditions LLM-based response generation on both affective streams. The robot then generates a short spoken empathetic response together with emotionally congruent embodied behavior, forming a closed speaker--listener affective loop. We evaluate the system in a pilot within-subject study with five participants, comparing it with an otherwise identical utterance-conditioned baseline that omits the speaker- and listener-affective-state inputs. The proposed system received higher overall impression ratings, especially for empathetic response and user satisfaction. Post-hoc log analysis further showed higher speaker-listener affective alignment and stronger valence-based distress recovery. These preliminary results suggest that explicitly modeling both speaker emotional dynamics and listener affective state can improve embodied empathetic interaction.

I. INTRODUCTION

AffectLoop addresses the limits of text-centered, one-way empathy modeling by conditioning multimodal robot responses on coupled speaker and listener affective dynamics. A pilot study and interaction-log analysis suggest improved perceived empathy, satisfaction, affective alignment, and distress recovery.

  • Motivation: Existing empathetic dialogue systems are largely text-centered and model empathy as a one-way mapping from speaker emotion to system response.
  • Motivation: Social robots require multimodal empathy because emotions evolve over time, are expressed through verbal and facial signals, and shape speaker–listener alignment.
  • AffectLoop: AffectLoop conditions LLM response generation on the speaker’s verbal and facial emotional dynamics and the robot listener’s verbal and behavioral affective state.The robot produces a short spoken empathetic response with emotionally congruent embodied behavior.
  • Evaluation: A pilot within-subject study found improvements in perceived naturalness, empathetic listening, empathetic response, and user satisfaction versus a text-only LLM baseline.
  • Evaluation: Post-hoc interaction-log analysis showed higher speaker–listener affective alignment and greater distress recovery for AffectLoop.The evidence is preliminary and supports explicitly closing the affective loop for embodied empathetic interaction.

II. RELATED WORK · A. Emotional Dynamics and Speaker–Listener Alignment in Dialogue Systems · B. Affective and Multimodal Social Robots

Related work increasingly models emotion as evolving over time and recognizes affective alignment between conversational partners, while social-robot research emphasizes multimodal empathy. AffectLoop extends these directions by coupling speaker emotional dynamics with the robot’s own affective state in a speaker–listener affective loop.

  • A. Emotional Dynamics and Speaker–Listener Alignment in Dialogue Systems: Emotion is dynamic, changing over time, within and across utterances, and through interaction with the conversational partner.
  • A. Emotional Dynamics and Speaker–Listener Alignment in Dialogue Systems: Recent emotional-support dialogue research evaluates and models affective trajectories across time and multi-turn conversations rather than relying only on snapshots.ETrajEval examines emotional-trajectory improvement and stabilization, while AFlow models continuous affective flow.
  • A. Emotional Dynamics and Speaker–Listener Alignment in Dialogue Systems: Empathy also involves alignment between conversational partners, with psychological work linking emotional communication to amplified speaker–listener neural alignment.
  • B. Affective and Multimodal Social Robots: Social robots express empathy through verbal and nonverbal behaviors, including facial expressions, gestures, gaze, body motion, and speech style.Prior studies identify nonverbal affective cues as important in social and emotionally supportive human–robot interaction.
  • B. Affective and Multimodal Social Robots: LLM-based social-robot research has explored generating responses from multimodal emotional-support context, including visual and nonverbal information.
  • B. Affective and Multimodal Social Robots: Most prior systems focus separately on recognizing user emotion or generating robot expressions rather than connecting both sides of the interaction.
  • B. Affective and Multimodal Social Robots: AffectLoop connects these sides through a speaker–listener affective loop that conditions responses on the speaker’s verbal and facial dynamics and the robot’s verbal and behavioral affective state.

III. PROPOSED SYSTEM · A. Incremental Dialogue Framework · B. Multimodal Input Module

The proposed system is an incremental multimodal spoken-dialogue pipeline implemented on Misty II and Retico. It integrates speech and facial-behavior inputs to estimate affective states and guide LLM-based response generation.

  • III. PROPOSED SYSTEM: The system is implemented on Misty II, an open programmable robotics platform, as an incremental multimodal spoken-dialogue pipeline built on Retico.Retico is a Python framework for incremental spoken-dialogue systems.
  • A. Incremental Dialogue Framework: Retico uses an Incremental Unit model in which modules process and pass small units such as audio frames, ASR hypotheses, images, and dialogue-state updates.This supports incremental processing across the dialogue pipeline.
  • A. Incremental Dialogue Framework: The modular pipeline connects perception, language understanding, response generation, and robot behavior execution.The architecture links these components through modular information flow.
  • B. Multimodal Input Module: The system receives user input through speech and facial behavior, enabling multimodal processing of the speaker’s interaction signals.Speech is captured through a hand microphone, while facial behavior is captured by the robot camera.
  • B. Multimodal Input Module: Speech processing uses an on-device Whisper-Tiny ASR module with buffered audio frames, voice activity detection, and periodically generated partial recognition hypotheses.The design supports local incremental speech recognition during user speaking turns.
  • B. Multimodal Input Module: Visual processing uses a Retico image module that creates image IUs from webcam, IP-camera, or video sources, independently of the speech stream.Each image IU contains the current frame and frame-rate information.

C. Affect Modeling Module

The Affect Modeling Module represents affective dynamics for both the human speaker and robot listener across verbal and nonverbal channels. Speaker affect is tracked through token-level VAD trajectories and temporally aggregated facial estimates, while listener affect is derived from response speech and robot actions.

  • Speaker affect modeling: Speaker verbal affect is incrementally represented as token-level valence, arousal, and dominance (VAD) vectors rather than a single utterance-level emotion score.The trajectories are computed from ASR output using the NRC-VAD lexicon, which covers over 20,000 English words.
  • Speaker affect modeling: Speaker nonverbal affect is estimated from camera frames with EmoNet, jointly capturing categorical emotion, valence, and arousal under naturalistic conditions.Frame-level estimates are aggregated over a short temporal window to represent nonverbal emotional dynamics.
  • Listener affect modeling: Robot listener verbal affect is tracked from its generated response using the same VAD-based module, producing a listener verbal VAD trajectory.This extends affect tracking to the robot’s generated speech rather than limiting modeling to the human speaker.
  • Listener affect modeling: Robot listener nonverbal affect is inferred from generated actions with EMRO, mapping behaviors to six affective categories spanning negative, positive, and understanding-related states.The categories include anger/frustration; confusion/sorrow/boredom; disgust/surprise/alarm/fear; interest/desire; joy/hope; and understanding/gratitude/relief.
  • Prompt conditioning: The baseline prompt excluded all speaker- and listener-side affective-state information and the corresponding instruction to consider these states.The proposed framework’s prompt template is therefore distinguished by explicitly including both affective-state streams.

D. Response and Behavior Generation Module

The response and behavior generation module uses GPT-4.1-nano2 to condition short spoken responses on speaker–listener affective context. It also predicts the robot’s response emotion to generate synchronized affect-conditioned behaviors and speech, followed by listener-affect analysis.

  • Affective response generation: GPT-4.1-nano2 generates short spoken responses from dialogue context and speaker and listener verbal and affective trajectories.These inputs are serialized into a structured prompt for interactive deployment.
  • Affective behavior generation: The LLM predicts the robot’s next-response emotion, which conditions GRED-generated behavior sequences executed alongside onboard text-to-speech.The spoken response and action sequence are analyzed again by listener affect modules after execution.

IV. EXPERIMENTAL SETUP · A. Study Design

A pilot within-subject study compared a speaker-utterance-only baseline with a proposed system incorporating speaker- and listener-side dynamic affective states. Five participants interacted with both Misty II systems in randomized five-minute conditions and rated four dimensions on a 7-point Likert scale.

  • A. Study Design: The pilot within-subject study compared a baseline system with the proposed speaker–listener emotion-dynamics-aware system.Each participant interacted with both systems.
  • A. Study Design: The baseline generated empathetic responses only from the speaker’s utterance.This provided the utterance-conditioned comparison condition.
  • A. Study Design: The proposed condition incorporated both speaker-side and listener-side dynamic affective states into the response-generation context.These two affective streams distinguished it from the baseline.
  • A. Study Design: Five participants provided informed consent and interacted with both systems for five minutes each.The study involved a small pilot sample.
  • A. Study Design: The order of the two conditions was randomized to reduce order effects.Participants talked with the Misty II robot in an open-ended attentive-listening setting.
  • A. Study Design: Participants rated each system on a 7-point Likert scale across Naturalness, Empathetic Listening, Empathetic Response, and User Satisfaction.The questionnaire followed prior-work evaluation metrics and its detailed items were listed in Table I.

B. Exploratory Interaction-Process Analysis

An exploratory post-hoc interaction-log analysis compared affective processes across conditions using dyadic affective alignment and speaker affect shift, including valence-based distress recovery. Alignment measured how the robot listener’s affective movement followed the speaker’s affective state, while affect shift measured changes across adjacent speaker turns.

  • Process-level metrics: The analysis examined two process-level metrics: dyadic affective alignment and speaker affect shift.These metrics were computed from interaction logs to compare affective processes between the two conditions.
  • Dyadic affective alignment: Dyadic alignment was measured as the cosine similarity between the speaker’s VAD state and the robot listener’s affective change.Turns with a zero-norm vector were excluded, and a normalized score mapped cosine similarity from [−1, 1] to [0, 1].
  • Dyadic affective alignment: Dimension-level alignment used sign-match rates to assess whether listener affective movement followed the speaker’s directional tendency in valence, arousal, and dominance.Near-zero values in either term were excluded when computing each dimension’s mean sign-match rate.
  • Speaker affect shift: Speaker affect shift was computed as the change between consecutive speaker VAD states, with mean changes reported for valence, arousal, and dominance.The analysis used valid adjacent speaker-turn transitions.
  • Speaker affect shift: Valence-based distress recovery was evaluated on turns with negative current valence, where positive scores indicated reduced negative-valence magnitude in the next turn.Negative scores indicated increased negative valence; the analysis reported mean distress recovery and positive recovery rate.

V. RESULTS AND DISCUSSION

The proposed system outperformed the baseline in subjective impression ratings and showed stronger speaker–listener affective alignment and valence-based distress recovery. Together, these findings suggest that conditioning on both affective streams may help close the affective loop.

  • Subjective evaluation: Overall impression increased from 4.75 to 5.10, with the largest gain in empathetic response, which rose from 4.80 to 5.35.The proposed system was also rated higher for encouraging, praising, comforting, and helping the user.
  • Affective trajectory analysis: Valence sign match Mv increased from 46.4% to 59.7%, whereas arousal sign match Ma remained similar.The clearest dimensional change concerned following the user’s positive–negative affective direction rather than uniformly increasing synchrony.
  • Affective trajectory analysis: Mean recovery ¯R increased from 0.151 to 0.339, and positive recovery rate Prec increased from 72.2% to 100.0%.After negative-valence user turns, the next user turn was more often less negative with the proposed system.
  • Overall interpretation: Subjective ratings and post-hoc process analysis consistently suggest that modeling speaker emotional dynamics and listener affective state may help close the affective loop.The conclusion concerns embodied empathetic interaction.

VI. CONCLUSION

AffectLoop is a multimodal speaker–listener emotion-dynamics-aware spoken dialogue system that conditions response generation on both participants’ affective states. In a pilot study, it outperformed a text-only LLM baseline in overall impression, with gains in empathy and satisfaction, alongside stronger affective alignment and distress recovery.

  • System contribution: AffectLoop conditions LLM-based response generation on the speaker’s verbal and facial emotional dynamics and the robot listener’s verbal and behavioral affective state.This multimodal conditioning supports spoken empathetic responses and emotionally congruent embodied behavior.
  • System contribution: The system generates spoken empathetic responses together with emotionally congruent embodied behavior, extending beyond text-only empathetic dialogue systems.The design explicitly represents affective exchange between speaker and listener.
  • Evaluation findings: The pilot within-subject study rated the proposed system higher than a text-only LLM baseline in overall impression, especially for empathetic response and user satisfaction.These were the clearest reported gains in the evaluation.
  • Evaluation findings: Post-hoc interaction-log analysis suggested stronger speaker–listener affective alignment and greater valence-based distress recovery with the proposed system.The findings provide preliminary evidence for incorporating both speaker emotional dynamics and listener affective state.
Loading 2608.16686v1…