Aura: Dynamic Intra-Turn Emotion-Aware Adaptation of Large Language Model Responses

Rachel Schuchert; Christian Holz

Source-linked AI summary

Aura: Dynamic Intra-Turn Emotion-Aware Adaptation of Large Language Model Responses

Rachel Schuchert, Christian Holz

arXiv:2608.24224v1Aug 25, 2026cs.HC

TL;DR

LLM responses are typically generated in static, open-loop turns that do not sense users’ immediate reactions, leaving limited support for real-time clarification and adaptation. Aura combines facial-expression perception, probabilistic policy selection, and mid-turn LoRA activation to modulate response strategies. In a within-subjects study, it achieved higher normalized perceived learning gains than Llama-3, reduced interaction time by 21% against GPT-4o and Llama-3, and showed no observable degradation in factual accuracy.

Problem
Static LLM generation remains largely insensitive to immediate user feedback during a response, limiting intra-turn communicative synchrony.
Method
Aura estimates facially expressed emotion, tracks uncertain interaction states with a POMDP, and executes selected interventions through dynamic LoRA adapters.
Results
Aura achieved statistically significantly higher normalized perceived learning gains than Llama-3 and reduced interaction time by 21% relative to GPT-4o and Llama-3 without observable factual-accuracy degradation.
Takeaways & Limitations
Aura’s results support real-time communicative alignment as a potential route to more efficient and satisfying human-AI interaction.
Takeaways & Limitations
Webcam facial cues may miss subtle or nonfacial states, and facial-emotion mappings can be culturally biased and imperfect proxies for internal affective or cognitive states.

Abstract

from arXiv · show

Effective human-AI interaction requires systems that dynamically adapt to a user's behavior and evolving understanding. When users interact with Large Language Models (LLMs), these models typically respond to prompts without sensing the user's immediate reactions. This lack of communicative synchrony can lead to information overload or leave confusion unresolved in real time. In this paper, we introduce Aura, a framework that enables LLM systems to dynamically modulate output based on a user's evolving emotions. Aura's Perception Module continuously estimates the user's emotional state from facial expressions. Our Policy Module then selects interventions through a probabilistic belief model. Finally, Aura's Generation Module uses parameter-efficient Low-Rank Adaptation (LoRA) adapters to produce contextually tailored responses mid-turn during response generation. We evaluated Aura in a within-subjects user study (N=20) on information-seeking tasks, where it achieved statistically significantly higher normalized perceived learning gains than a Llama-3 baseline and reduced interaction time by 21% relative to existing LLM baselines (GPT-4o, Llama-3). Our results indicate that real-time, context-sensitive interventions can improve learning efficiency and user satisfaction without observable degradation in factual accuracy. Aura thus supports the potential for more responsive and effective human-AI interaction.

1 Introduction

Aura addresses the mismatch between static LLM generation and the immediate, evolving feedback available in human communication. It uses facial-expression-based emotion estimates, probabilistic intervention selection, and mid-turn LoRA adaptation to support synchronized responses.

Human speakers use continuous facial, gaze, and posture cues to adjust explanations in real time.
Static LLM exchanges remain blind to users during generation, often requiring later turns for clarification or reaction.
Aura adapts output during response generation using spontaneously expressed facial expressions rather than only between turns.
Aura supports sustained engagement, interaction recovery, and efficiency optimization through base, clarification, simplification, and pacing strategies.
Aura combines POMDP-based state tracking with LoRA adapters that modulate detail, examples, and response termination for a frozen base LLM.
The framework integrates user-state modeling into the model’s parameter space for fine-grained intra-turn adaptation while preserving conversational continuity.

2 Related Work

Prior work adapts language generation using user context, emotion, and dialogue policy, but most systems act at response or turn boundaries. Aura’s framework instead connects probabilistic user-state modeling to language generation during an ongoing response.

Research has explored adapting language generation to latent background cues, affective signals, and unstated preferences to improve interaction quality.
Emotion-aware generation commonly conditions responses on explicit or inferred emotion representations to improve perceived empathy and conversational quality.
Aura’s framework combines facial-expression perception, POMDP policy selection, and LoRA-based generation interventions.
Prior systems often use emotion as a fixed control signal for controllable generation or style conditioning.
Emotion-aware dialogue systems can use inferred emotions to guide response selection or dialogue policy, often with probabilistic emotional dynamics.
Most prior approaches adapt externally through templates, stylistic labels, or next-action selection at turn boundaries rather than changing language generation mid-response.

3 Method

Aura estimates interaction-centered user states from facial signals, tracks uncertainty with a POMDP, and selects LoRA interventions during generation. Its modular training and switching design targets low-latency, context-preserving adaptation.

Aura’s three modules infer facially expressed emotion distributions, select interventions through a POMDP, and execute them with LoRA adapters.
Perception Module: A ResNet-50 and LSTM process webcam facial features over N = 10 frames to estimate probabilities across seven basic emotions.
Perception Module: The system maps seven-emotion probabilities into four actionable states: Engaged, Confused, Frustrated, and Bored.
Perception Module: Exponential moving-average smoothing and spike detection stabilize observations against transient artifacts before policy decisions.
Policy Module: The POMDP tracks uncertain user states and selects among communicative interventions including Reframe, Clarify, De-escalate, Pace, Simplify, and Base.
Policy Module: A synthetic corpus of 24,929 sessions and 131,283 turns parameterized the POMDP because longitudinal data were scarce.
Policy Module: Offline tabular Q-learning computes the state–action value function, while hysteresis, competitive counters, and a 3.0-second cooldown reduce strategy flickering.
Generation Module: LoRA adapters are trained independently from 2,500 strategy-specific variants generated from a 500-interaction neutral seed corpus using QLoRA.

4 Experimental Setup

Aura was evaluated in a within-subject laboratory study comparing adaptive responses with non-adaptive Llama-3-8B and GPT-4o baselines across conceptual classification tasks. The protocol measured learning, interaction dynamics, subjective experience, and output quality.

Study design: The within-subject design compared Aura with non-adaptive Llama-3-8B and GPT-4o while counterbalancing condition order and task blocks.The design aimed to reduce inter-participant variability and control learning and fatigue effects.
Apparatus: Sessions were conducted in a standardized lab setting using a webcam, an RTX 4090, and a shared web interface for equivalent end-to-end latency.All systems used temperature 0.7 and a per-character streaming delay of 0.05.
Tasks: Participants completed 18 binary classification tasks involving frequently conflated concepts across statistics, moral philosophy, typography, and other domains.Tasks were difficulty-normalized and distributed across three balanced blocks.
Procedure: Each task used pre-task understanding ratings, an information phase, and application to four novel instances with post-task understanding ratings.This structure focused evaluation on conceptual understanding rather than example recall.
Measures: Learning was assessed with binary task accuracy and normalized learning gain, while interaction dynamics included turns, duration, completion time, and content consumed.Subjective measures covered self-efficacy, confidence, satisfaction, usefulness, and perceived quality.
Participants: The study recruited twenty participants who reported high English proficiency, comfort with new technologies, and regular chatbot or LLM use.Participants had mean LLM familiarity of 4.50/5.0.

5 Results

Aura improved normalized learning gain and interaction efficiency relative to the evaluated baselines while maintaining comparable accuracy. Participants also rated it favorably for effectiveness, satisfaction, relevance, confidence, engagement, and cognitive load.

Learning effectiveness: Aura achieved the highest normalized learning gain (g = 0.59 ± 0.22), significantly exceeding Llama-3 (0.41±0.38; p = .028).GPT-4o reached 0.54±0.33; the global Friedman test was p = .167.
Interaction dynamics: Aura reduced interaction time to 73.18s±27.33 versus GPT-4o (92.89s) and Llama-3 (99.88s), with a significant reduction against Llama-3 (p < .001).The global Friedman effect was χ2 = 12.90, p = .002, W = 0.323.
Output quality: Accuracy remained high and comparable across conditions: GPT-4o 90.4%, Llama-3 89.8%, and Aura 89.3%.The reported accuracy pattern did not indicate an observable accuracy advantage for Aura.
Subjective quality: Aura led retrospective Understanding at 4.35, tied GPT-4o on perceived Accuracy at 4.05 and Predictability at 2.85, and produced no reported abrupt adaptations.Per-task ratings also favored Aura for Satisfaction (3.78), Relevance (4.14), and Confidence (3.76).
Affective state and effort: Aura matched GPT-4o for highest Engagement (3.45) and had the lowest Arousal (2.20), Demand (2.15), and Effort (2.10).Boredom and frustration reductions were directionally favorable but not statistically significant.
Participant preference: Aura was ranked first by 11 of 20 participants, compared with 4 for GPT-4o and 5 for Llama-3.Participants commonly described Aura as concise, effective, and producing the least overload.
Qualitative feedback: Llama-3 feedback varied from good structure and helpful summaries to inconsistent depth, confusing length, missing information, and attention-reducing bullet use.These comments indicate heterogeneous perceptions of response depth and organization.

6 Discussion

The discussion attributes Aura’s advantage to communicative tailoring: adapting how information is delivered to the user’s interaction state rather than relying only on static response generation. The reported pattern combines learning and efficiency gains with lower perceived cognitive load and no comparable accuracy loss.

Interpretation: Aura balanced depth and brevity, producing the highest normalized learning gains alongside the lowest perceived mental demand and effort.The comparison covered all three evaluated conditions.
Interpretation: Comparable perceived and quiz accuracy across systems suggests Aura’s advantage was not primarily attributable to differences in the underlying LLM knowledge base.This interpretation is paired with higher satisfaction and retrospective engagement relative to Llama-3.
Efficiency: Aura reduced mean interaction time by ≈21% relative to GPT-4o and Llama-3 while maintaining learning outcomes.The discussion connects this efficiency to mid-turn modulation and communicative tailoring.
Affective implications: Directional reductions in boredom and frustration, together with increased engagement, suggest that delivery modulation may mitigate interaction fatigue.The discussion describes these affective changes as directional rather than uniformly statistically significant.
Mechanism: The POMDP is described as automating repair before users recognize confusion and formulate clarifying prompts, helping shorten interactions and sustain engagement.This frames intervention timing as a mechanism for reducing conversational friction.

7 Conclusion

Aura is a real-time adaptive LLM framework that uses inferred facially expressed user states to modulate response strategies. In a user study, it reduced interaction time, achieved the highest learning gains, and was preferred for conciseness and reduced overload.

Aura combines POMDP-based policy selection with dynamic LoRA activation to adapt response strategies from inferred facial signals.The framework moves beyond static prompt–response exchanges toward communicative alignment.
Aura reduced interaction time compared to GPT-4o and Llama-3 while achieving the highest learning gains.
Participants preferred Aura overall, citing its conciseness and reduced overload.
The results frame conversational AI effectiveness as depending on both information quality and synchronization with the user’s immediate context.

8 Limitations

Aura’s findings are bounded by limitations in sensing, modeling, task design, and user agency. These constraints limit generalizability beyond the evaluated setting and interaction paradigm.

Sensing and generalizability: Webcam-based facial cues may miss subtle or non-facial user states, while facial emotion mappings can be affected by cultural bias.The paper also describes facial emotion signals as imperfect proxies for internal affective or cognitive states.
Modeling constraints: Synthetic training data for POMDP transition and reward models may not cover the more complex dynamics encountered in real-world interactions.
Modeling constraints: Aura’s discrete LoRA-based rhetorical strategies simplify the continuous and nuanced stylistic shifts characteristic of human communication.
Evaluation scope: The evaluation used binary classification tasks designed to induce controlled cognitive friction, leaving open-ended and multi-step tasks for future study.
User agency: Proactive interventions based on implicit cues can feel weird to users because they bypass explicit agency.

9 Ethical Considerations

Aura raises privacy and fairness considerations because it processes facial signals to infer interaction states. The paper limits privacy exposure through local, transient processing but acknowledges bias and interpretive risks.

Privacy: Facial landmark extraction occurs locally and in real time, with no raw video or identifiable biometric images stored or transmitted externally.
Fairness and interpretation: Facial emotion models may fail to generalize across ethnicities, genders, ages, and cultural backgrounds.
Fairness and interpretation: Non-universal expressions can cause misinterpretation of users with different expressive norms, diminished facial expressivity, or facial paralysis.
Study ethics: The study used informed consent, transparency about sensing technology, privacy protections, and voluntary participation.

A.3 Robustness via Nonlinear Smoothing

Aura’s robustness pipeline smooths perceptual state estimates, learns smoothed transition dynamics, tracks beliefs from observations, and selects actions using QMDP values. Strategy-specific training data is designed to preserve base-model factual accuracy while modulating style.

Nonlinear smoothing: A dual-mode filter uses β = 1.4 for spike detection and γ = 0.8 for smoothing to mitigate sensor artifacts while preserving state persistence.
Transition modeling: The transition model is learned from synthetic-corpus transition frequencies with Laplace smoothing κ = 1 to address data sparsity.
Recursive belief tracking: Aura updates its belief state by combining the current Perception Module observation with transition dynamics and the previous action.The observation model and normalization constant are used in the belief update.
Policy selection: Tabular Q-learning optimizes the state-action value function, with α = 0.1 and γ = 0.95 producing Q∗(s, a) values for real-time QMDP action selection.
Strategy training: Synthesized strategy-specific training data uses five meta-prompts to modulate style while preserving the frozen base model’s factual accuracy.

C.2 Adapter Evaluation Details

Aura's adapter evaluation examines specialized capabilities, synthetic emotion-state trajectories, strategy-distillation prompts, study tasks, and a controlled chat interface. The evaluation also describes how adapter specialization was checked against core capabilities and how participants encountered the classification tasks.

Adapter evaluation: Adapters were evaluated zero-shot with Prometheus-eval on a 5-point Likert scale across nine functional dimensions.The dimensions included grounding, instruction following, planning, reasoning, refinement, safety, theory of mind, tool usage, and multilingual ability.
Synthetic trajectories: Synthetic trajectories pair the true state with Aura's belief distribution and selected action, using probabilities over bored, confused, frustrated, and engaged states.The belief vector is represented as [B]ored, [C]onfused, [F]rustrated, and [E]ngaged.
User-study tasks: The study used 18 classification tasks spanning difficulty levels 1: Low to 3: High, with one representative sample and four novel instances per task.Tasks 1–9 and 10–18 are listed in Tables 9 and 10.
Experimental interface: All three experimental conditions used the same chat layout and synchronized character-by-character streaming to reduce visual and interaction-velocity differences.The conditions were Llama-3-8B Base, GPT-4o, and Aura.
User-study tasks: After chatbot interaction, participants completed the binary classification interface containing a domain-specific problem statement and two competing hypotheses.Figure 3 shows the post-interaction classification screen.