Source-linked AI summary
AffAdapt: AFFect-driven ADAPTive AI Personas for Seamless Conversations
Nishanth Chidambaram, Kaustubh Paliwal, Kayla Hom, Shaoze Zhou, Chen Chen, Manas Satish Bedmutha, Nadir Weibel
TL;DR
AI personas need coordinated timing, identity, affect, and embodiment to support naturalistic conversation, but existing interaction layers must decide when to listen or respond while preserving character. AffAdapt addresses this gap with a unified loop combining streaming recognition, proactive turn management, persistent emotional state, persona-grounded generation, and synchronized embodied output. The prototype demonstrates this approach for sensitive, high-stakes conversations while identifying open challenges in immersion, animation fidelity, and richer emotion modeling.
Problem
AI personas must coordinate turn timing, emotional state, identity, and multimodal behavior rather than merely generate plausible responses.
Method
AffAdapt integrates proactive turn management, a persistent trainee-driven persona state, persona-grounded dialogue, and synchronized embodied output into one interaction loop.
Results
AffAdapt provides a voice-driven architecture for natural, two-way interaction with an embodied virtual agent in sensitive, high-stakes conversations.
Takeaways & Limitations
The interaction layer is designed to coordinate timing, persona identity, emotional state, and embodiment independently of a single application domain.
Takeaways & Limitations
The current direction remains bounded by screen-based deployment, occasional verbal–expressive mismatches, and emotion modeling that the authors plan to deepen.
Abstract
from arXiv · showhide
AI-generated personas are being increasingly used for support, training and simulations. While generative AI models possess abilities to generate affect-aware responses, their embodiment into visual personas is an active area of investigation. Naturalistic exchanges require understanding of the conversational partners' turn completions, whether the agent should respond or keep listening and rely on non-verbal cues aligned with one's emotional states. Seamless human-AI conversation in a multimodal setting requires all modalities being generated to act in coordination. We present AffAdapt, a seamless interaction design framework for AI-personas, which coordinates streaming speech recognition, proactive turn-management, persona-grounded response generation, a persistent emotional state, and synchronized embodied output into a single interaction loop. We demonstrate the architecture in the context of practicing sensitive, high-stakes conversations, and report an initial case study showing fluid turn management and adaptive, persona-consistent behavior, alongside open challenges in interruption handling, open-ended dialogue, and multimodal affective alignment. AffAdapt's interaction loop is a generalizable pattern for coordinating timing, identity, and affect in real-time AI personas - applicable to training, coaching, education, and simulation contexts wherever believable, responsive interaction matters.
1 Introduction
AI personas must coordinate timing and control decisions while maintaining identity, emotional state, and relationship trajectory. AffAdapt addresses this challenge by integrating proactive interaction management with persona-consistent, embodied behavior for sensitive conversations.
- AI personas must decide when to listen, yield, respond, and recover from interruptions, because mistimed responses can feel unnatural despite strong generation.
- Persona-based agents must maintain consistent identity, emotional state, social stance, and relationship trajectory while managing interaction timing.
- AffAdapt instantiates a coordinated interaction layer for practicing sensitive, high-stakes conversations.
- The framework combines proactive turn management, persona-grounded response generation, and embodied output in one loop.
2 System Design
AffAdapt coordinates turn detection, persistent emotion modeling, persona-grounded dialogue, and synchronized embodiment in a streaming interaction loop. The persona state links how the trainee communicates to both the agent’s responses and its presentation.
- Proactive Turn Management: Streaming recognition and voice-activity detection segment speech without explicit turn signals, while token and sentence streaming produce replies about 1.5 s after speech stops.Recognition takes about 0.3 s and first-sentence generation about 1.1 s before synthesis.
- Persistent Persona State: A persistent fourteen-dimensional persona state updates after each turn from detected trainee emotions and conditions both dialogue and embodied behavior.
- Persistent Persona State: The state accumulates across the conversation rather than resetting each turn, allowing behavior labels such as withdrawn, open, anxious, or engaged.
- Persona-Grounded Dialogue: Persona-grounded dialogue combines a structured profile, emotional state, disclosure policy, and recent history in the prompt to preserve character consistency.
- Embodied Response Timing: Affect scores synchronize voice, facial expression, posture, gesture, and lip synchronization as the reply begins, although discrete gestures and postures can mismatch verbal content.
- Emotion Inferencing and State Dynamics: Emotion inference currently uses transcript text alone, with prosodic and acoustic affect features left for future work.
3 Conclusion and Future Work
AffAdapt is presented as a voice-driven architecture for natural, two-way interaction with embodied personas in sensitive conversations. The authors identify immersion, more realistic animation, and richer emotion modeling as future directions.
- AffAdapt supports natural, two-way spoken interaction with an embodied virtual agent for sensitive, high-stakes conversations.
- Immersive VR deployment: VR deployment would add posture, gaze, and other non-verbal cues to the interaction beyond the current screen-based setting.
- More realistic embodied animation: The authors aim for continuous facial and bodily animation to reduce occasional mismatches between verbal content and expression observed in the case study.
- Richer emotion modeling and session scoring: Future emotion modeling will deepen the persona’s evolving disposition and support session feedback on rapport, openness, and attunement.