Source-linked AI summary
Listening and Mirroring: The Effects of Verbal Attunement and Behavioral Mimicry on Social and Empathic Perceptions of Embodied AI Agents in VR
Nathalia Gomez, Haig Shamlian, Omar Khan, Tiffany D. Do
TL;DR
The paper addresses limited evidence on how verbal attunement and behavioral mimicry jointly shape perceptions during real-time conversations with embodied AI agents in VR. It tests an AI counselor with these cues varied in a within-subjects study, finding that verbal attunement was clearest for perceived empathy while mimicry showed different associations. These findings caution against treating multimodal synchrony as a simple additive strategy for empathic VR agents.
Problem
Prior mimicry research largely predates conversational AI with contextually responsive dialogue, leaving limited evidence about how verbal attunement and mimicry jointly shape perceptions in real-time embodied VR conversation.
Source: §1 · p. 2, §2.2 · p. 3Method
The authors tested an embodied AI counselor that varied verbal attunement and behavioral mimicry in a 2 x 2 within-subjects study with 20 participants.
Source: §1 · p. 2Results
Verbal attunement was the clearest driver of perceived empathy, while mimicry showed a marginal effect on humanness and greater exposure was positively associated with empathy, positivity, and humanness.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
The findings suggest that multimodal synchrony is not a simple additive strategy for designing empathic conversational agents in VR.
Source: §1 · p. 2Responses were not matched for length or speaking duration, so these differences may have contributed alongside verbal attunement to empathy ratings.
Source: §6.5 · p. 9
Abstract
from arXiv · showhide
As embodied agents take on increasingly social and relational roles in VR, visual realism and embodiment alone may be insufficient; users must also perceive these agents as emotionally attuned, supportive, and humanlike. Prior work suggests that verbal attunement and nonverbal mimicry can each improve users' social evaluations of embodied agents. However, behavioral mimicry has largely been studied outside of real-time, conversational AI interactions, leaving limited understanding of how users respond when an agent simultaneously generates contextually responsive dialogue and adapts its nonverbal behavior during an immersive conversation. To address this gap, we developed an embodied AI counselor that combines conversational AI with real-time facial-expression and posture mimicry, while producing either verbally attuned or neutral responses. We evaluated the system in a 2 X 2 within-subjects study with 20 participants, manipulating verbal attunement and behavioral mimicry. Results showed that verbal attunement was the most reliable driver of perceived empathy. Behavioral mimicry showed a marginal relationship with perceived humanness, while greater mimicry exposure showed preliminary, exploratory positive associations with empathy, positivity, and humanness, particularly among female participants. Together, these findings show that multimodal synchrony is not a simple additive strategy for designing empathic conversational agents in VR and underscore the need to consider how verbal and nonverbal behaviors are combined during real-time interaction.
1 Introduction
This study asks how verbal attunement and behavioral mimicry shape perceptions of an embodied AI counselor during real-time VR conversation. Its findings distinguish the cues’ roles, with verbal attunement clearest for empathy and mimicry contributing more to humanness and exposure-based patterns.
1 Introduction
- Earlier mimicry research largely predates conversational AI capable of contextually responsive dialogue, leaving its joint effects with verbal attunement unclear in real-time VR conversation.Source: §1 · p. 2
- The authors evaluated an embodied AI counselor in a 2 x 2 within-subjects study with 20 participants, varying verbal attunement and behavioral mimicry.Participants also provided open-ended feedback after each interaction.Source: §1 · p. 2
- The study examines how verbal attunement and behavioral mimicry affect empathy, positivity, humanness, and eeriness, including how the cues operate together.Source: §1 · p. 2
- Verbal attunement was the clearest driver of perceived empathy, while mimicry played a different role in perceived humanness and exposure-based patterns.The authors conclude that multimodal synchrony should not be treated as a simple additive strategy for making VR agents empathic.Source: §1 · p. 2
2 Related Work
Prior work positions embodied agents as social partners in VR and identifies empathy and nonverbal mimicry as relevant to their evaluation. This paper examines how verbal attunement and mimicry operate together in real-time embodied conversation.
2.1 Embodied AI in VR for Healthcare and Collaboration
- Embodied AI has been applied in VR healthcare and collaboration, where language, movement, appearance, and other dimensions shape user experience.Examples include mental-health treatment, telecare, aphasia rehabilitation, and collaborative decision-making.Source: §2.1 · p. 2
2.2 Importance of Empathy in Human-AI Interaction
- Empathic, socially present agents matter in counseling and other relational roles, and embodied VR adds expressive cues beyond text-based interaction.Prior work links therapist empathy to therapeutic outcome and client experience, and describes embodied agents’ shared social context.Source: §2.2 · p. 3
2.3 Behavioral Mimicry in Embodied Agents
- Prior mimicry studies report effects on persuasion, rapport, attachment, and social presence, but its role in empathy during reflective spoken conversation remains unclear.This study examines mimicry of facial expressions, head tilts, and posture during real-time VR conversation.Source: §2.3 · p. 3
2.4 LLMs and Verbal Attunement
- Research associates empathic language with positive evaluations, motivating a test of verbal attunement in spoken VR interaction with voice, timing, spatial presence, and embodied cues.Described verbal features include validation, personalized active listening, nonjudgmental language, and emotional reflection.Source: §2.4 · p. 3
3 System Design: An Empathic Virtual Counselor
The system combines a controlled spoken-conversation pipeline with prompts that vary verbal attunement and contingent facial, head, and posture mimicry. The counselor is presented in a seated, face-to-face VR office setting.
3.1 Conversational Architecture and Interaction Flow
- The conversation pipeline streamed microphone audio to an AI agent, generated counselor turns from transcripts and study context, and returned speech through the VR counselor.The program controlled the interview sequence, follow-ups, time limit, and closing sequence.Source: §3.1 · p. 4
3.2 Verbal Attunement
- The verbally attuned prompt encouraged validation, personalized listening, emotional reflection, and relevant follow-up, while the neutral prompt used professional, task-focused language without explicit emotional validation.The conditions shared the same technical pipeline and interview structure.Source: §3.2 · p. 4
3.3 Behavioral Mimicry
- Mimicry conditions mirrored selected participant facial expressions, head tilts, and posture changes, with separate mappings for emotional expressions and speech-related mouth movements.Facial and posture changes required 30 stable frames; head-tilt responses used a one-second delay, and tested onset delays ranged from approximately 2 to 4 seconds.Source: §3.3 · pp. 4–5
- Facial-expression mimicry classified expressions as smiling, brow lowering, or neutral, while head-tilt and posture responses used thresholds and stable changes to trigger counselor movements.Source: §3.3 · p. 5
3.4 Virtual Environment and Embodiment
- The counselor used a facial-blendshape avatar seated across a table in a Unity office, with neutral idle behavior by default and responsive movement in mimicry conditions.Source: §3.4 · p. 5
4.1 Participants
The study recruited 20 adults in the United States who were fluent in English.
4.1 Participants
- The sample included 20 participants: 10 identified as male, eight as female, one as non-binary, and one preferred not to answer.Participants were recruited via listservs and flyers and screened for adulthood and English fluency.Source: §4.1 · p. 5
4.2 Experimental Design
The study crossed verbal attunement and behavioral mimicry in a four-condition, within-subjects design. Participants answered moderately personal reflection questions selected from the Fast Friends procedure, and condition order was counterbalanced.
4.2 Experimental Design
- The 2 x 2 design varied verbal attunement and behavioral mimicry, producing four counselor conditions: VA+BM, VA+NBM, NVA+BM, and NVA+NBM.Each participant experienced all four conditions.Source: §4.2 · p. 5
- A Latin square assigned participants to counterbalanced condition orders, with five participants in each of four groups.This scheme was used to reduce order effects.Source: §4.2 · p. 5
- Table 1 summarizes main and interaction effects from the 2 x 2 repeated-measures ANOVAs.Its caption says bold values mark statistically significant effects (p < .05) or marginal trends (p < .10).Source: Table 1 · p. 6
- Two questions from a randomly selected set of eight were assigned to each condition, using Fast Friends Set II prompts designed for moderately personal reflection.The questions kept topic structure consistent while allowing opportunities for empathic responses.Source: §4.2 · p. 6
4.3 Procedure
Participants completed four short VR conversations with a virtual counselor, one per experimental condition. Each conversation used two selected Fast Friends questions, followed by a questionnaire about that interaction.
4.3 Procedure
- Participants completed four short conversations with the counselor, one for each experimental condition, and the counselor used the behavior assigned to that condition.The counselor asked two selected Fast Friends questions in each conversation.Source: §4.3 · p. 6
- After each conversation, participants removed the headset and completed a post-condition questionnaire before proceeding to the next condition.They followed their assigned counterbalanced order.Source: §4.3 · p. 6
- Before the conversations, participants completed prescreening and demographics, then received headset setup instructions and a brief task overview at an in-person session.Source: §4.3 · p. 6
4.4 Measures
The study gathered post-interaction self-reports, logged mimicry exposure, and collected open-ended feedback. Measures covered empathy, positivity, humanness, eeriness, and perceived verbal attunement.
4.4 Measures
- After each interaction, participants rated the counselor and answered open-ended questions; the system also logged mimicry exposure.Complete surveys, including scale items and open-ended questions, were provided in supplementary materials.Source: §4.4 · p. 6, §4.4.5 · p. 6
4.4.1 Empathy and General Positivity.
- Empathy was measured with a 12-trait Counselor Empathy Scale, while general positivity was measured with the General Positivity Scale.Both scales were adapted from Guadagno et al..Source: §4.4.1 · p. 6
4.4.2 Humanness and Eeriness.
- Humanness and eeriness were measured using adapted Uncanny Valley Index items on 7-point semantic differential scales.Three items measured humanness and two measured eeriness; responses were averaged separately.Source: §4.4.2 · p. 6
4.4.3 Verbal Attunement Manipulation Check.
- A three-item, 7-point Likert manipulation check assessed perceived verbal attunement and had high internal consistency (Cronbach’s α = .914).Source: §4.4.3 · p. 6
4.4.4 Mimicry Exposure.
- Mimicry exposure was quantified by summing the durations of behaviors mimicked during each session.The system recorded duration and frequency of mimicked behaviors.Source: §4.4.4 · p. 6
4.4.5 Open-Ended Feedback.
- Open-ended responses about what participants liked and disliked were used to contextualize ratings and identify patterns or design issues the scales might miss.Source: §4.4.5 · p. 6
5 Results
Verbal attunement had a significant main effect on empathy, while mimicry showed a marginal main effect on humanness and no interaction effects were found. Exploratory correlations and open-ended responses showed additional patterns involving mimicry exposure, conversational breakdowns, and embodied behavior.
5.1.1 Data Analysis Approach.
- Verbal attunement had a significant main effect on empathy, while mimicry’s humanness effect was marginal; no interaction effects were significant.The reported tests were F(1, 19) = 7.425, p = .013 for attunement on empathy and F(1, 19) = 4.314, p = .052 for mimicry on humanness.Source: §5.1.1 · p. 7
- Verbal attunement showed a nonsignificant trend on positivity, neither factor significantly affected eeriness, and attunement significantly affected its manipulation check.The reported p-values were .078 for attunement on positivity, .823 and .86 for the eeriness effects, and .028 for attunement on the manipulation check.Source: §5.1.1 · p. 7
5.1.2 Exploratory Analysis of Mimicry Exposure by Gender.
- Across participants, mimicry duration correlated positively but not significantly with empathy (r(38) = .241, p = .135), positivity (r(38) = .269, p = .093), and humanness (r(38) = .294, p = .065).Mimicry exposure varied because it depended on behaviors participants displayed.Source: §5.1.2 · p. 7
- Among female participants, mimicry duration correlated significantly with empathy, positivity, and humanness; among male participants, none of these correlations was significant.The subgroup analyses were exploratory and had small sample sizes, so the patterns warrant caution.Source: §5.1.2 · p. 7
5.2 Open-Ended Observations
- Participants described more coldness or emotional distance in neutral-attunement conditions (10 comments) than in attuned conditions (4), and more responsiveness or involvement in attuned (14) than neutral conditions (5).Source: §5.2.1 · p. 7
- Interruptions, abrupt topic changes, long pauses, and responses that missed or cut off answers reduced perceived responsiveness, including in verbally attuned conditions.Participants described feeling rushed or not being understood.Source: §5.2.2 · pp. 7–8
- Participants found embodied behavior helpful when it signaled attentiveness, but distracting or uncanny when it appeared unnatural, mechanical, or poorly synchronized.Comments described inviting posture alongside contrived head movement, unnatural mouth movement, and rigid body language.Source: §5.2.3 · p. 8
6 Discussion
The discussion distinguishes verbal attunement’s strong role in perceived empathy from mimicry’s more limited humanness and exposure patterns. It also emphasizes conversational timing and study limitations.
6.1 Verbal Attunement Was the Primary Driver of Perceived Empathy
- Verbal attunement was the clearest driver of perceived empathy, with participants responding to specific, relevant reflections of their disclosures rather than general friendliness alone.Open-ended comments described the counselor as responsive and emotionally aware when it reflected participants’ feelings and addressed what they had said.Source: §6.1 · p. 8
6.2 Behavioral Mimicry Supported the Counselor’s Humanlike Presence
- Mimicry showed a marginal effect on perceived humanness, and participant comments suggest contingent movement could make the counselor seem attentive and socially present.The authors caution against treating humanness as an unconditional goal; coherent humanlike cues are framed as supporting comfort, attention, and social engagement.Source: §6.2 · p. 8
6.3 Mimicry Exposure Showed Exploratory Gender-Stratified Patterns
- The restricted mimicry cues had no significant main effect on empathy or positivity, while female participants’ mirroring duration was positively correlated with empathy, positivity, and humanness.The gender-stratified pattern is exploratory because subgroup samples were small and gender was not a primary study factor.Source: §6.3 · pp. 8–9
6.4 Turn-Taking and Conversational Flow Shaped Perceived Responsiveness
- Turn-taking errors, including interruptions during reflective pauses, can undermine perceived responsiveness even when the agent’s language is designed to be empathic.Pauses may signal that users are thinking or deciding what to disclose, so turn-taking and end-of-utterance detection belong in empathy design.Source: §6.4 · p. 9
6.5 Limitations and Future Work
- Attuned and non-attuned responses were not matched for length or speaking duration, so response elaboration may have contributed to empathy ratings.The authors suggest length-matched responses or accounting for word count and speaking duration in future studies.Source: §6.5 · p. 9
- Mimicry exposure depended on detectable user behaviors, the cues were restricted and categorical, and the study focused on an emotionally reflective AI counselor.The authors call for testing richer, context-sensitive mirroring and whether findings extend to other agent roles; they also identify race and identity as future questions.Source: §6.5 · p. 10
7 Conclusion
In VR, verbal attunement most clearly shaped perceived empathy, while behavioral mimicry showed a different pattern involving humanness and exposure. The conclusion emphasizes that these cues and conversational timing must work together.
7 Conclusion
- Verbal attunement had the clearest effect on perceived empathy, with users judging empathy through supportive responses that reflected emotional meaning and specific disclosures.Source: §7 · p. 10
- Implemented mimicry cues showed a marginal effect on perceived humanness, while greater mimicry exposure was positively associated with empathy, general positivity, and humanness.Source: §7 · p. 10
- The findings suggest that empathic VR agents depend on verbal attunement, embodied behavior, and conversational timing working together rather than on simple multimodal synchrony.Source: §7 · p. 10