PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-Aware Emotionally Adaptive Voice Assistants: A Multimodal Framework for Empathetic Human-Agent Interaction

Tapon Kumer Ray, Rajkumar Yesuraj

arXiv:2609.16417v1cs.HC

TL;DR

Voice-assistant interruption research often separates context awareness, affect recognition, and adaptive policy, despite the importance of activity, state, setting, and agency to proactive behavior. EmpathicVA connects multimodal affect estimates to a reinforcement-learning interruption policy, achieving strong held-out recognition performance and improved user-reported outcomes in a six-week deployment.

  • Problem

    Proactive voice-assistant behavior depends on user activity, state, setting, and agency, while related systems often separate context sensing, affect recognition, and adaptive policy learning.

    Source: §I · p. 1

  • Method

    EmpathicVA fuses physiological, vocal, and contextual streams into affect probabilities and uses a Double DQN policy that combines them with context and interaction history to select interruption behavior.

    Source: §I · p. 1, A. Closed-Loop Architecture · p. 2

  • Results

    92.3% accuracy and 0.922 macro F1 on held-out data; in a six-week study, the framework increased satisfaction, usability, trust, and timing appropriateness while decreasing intrusiveness and physiological stress.

    Source: A. Affect Recognition and Policy… · p. 4, §VII · p. 7

  • Takeaways & Limitations

    The findings indicate that interruption management involves timing restraint as well as emotionally appropriate wording, with the learned policy incorporating more delay and silence during stress.

    Source: §VI · p. 5

    The six-week Android deployment included mainly educated English speakers, and RL training was conducted in simulation rather than with users.

    Source: §VI · p. 6

Abstract

from arXiv · show

Voice-assistant interruptions tend to be intrusive because existing systems fail to consider the affective state, cognitive load and situational context of the user when deciding when and how to interrupt.Voice-assistant interruptions tend to be intrusive, since existing systems do not consider the affective state, cognitive load or situational context of the user when determining when and how to interrupt. In this paper, EmpathicVA, a closed-loop framework integrating physiological sensing, vocal-affect analysis, contextual modeling and reinforcementlearning interruption policy, is introduced. A hierarchical fusion model involves integrating HRA, EDA, respiration, acousticprosodic features, linguistic embeddings, and contextual cues and computing the probabilities of five affective states. A Double Deep Q-Network selects immediate response, brief or extended delay, empathetic response, or silent mode based on these probabilities, context and interaction history. The multimodal model obtained an accuracy of 92.3% and an F1-score of 0.922 at the macro level on a held-out test set, outperforming the highest accuracy unimodal model by 6.0 percentage points. Comparing the six-week within-subject field study with 48 participants with a baseline and context-only assistants, there was a corresponding increase in satisfaction, trust, and appropriateness of timing, as well as a large reduction in interruption-related stress episodes. The results suggest that affect-aware timing and restraint are both important in voice interaction in addition to the response wording.

I. INTRODUCTION

Proactive VA behavior depends on user activity, state, setting, and agency, yet related research has often separated context, affect recognition, and adaptive policy learning. EmpathicVA asks whether affect estimates can inform interruption policies that reduce intrusiveness while maintaining usefulness.

I. INTRODUCTION

  • Voice-assistant proactive behavior remains a challenging HCI problem, and its suitability depends on user activity, state, setting, and agency.Inappropriate or premature interventions can contribute to intrusion and loss of trust.Source: §I · p. 1

A. Context, Interruptibility, and Voice Interaction

  • Related work often separates context-aware activity sensing, affect classification, and dialogue policy learning, leaving affect unlinked to policy and physiological intrusion costs out of adaptive dialogue.Context alone may be ambiguous: the same setting can accompany relaxed, frustrated, or acutely stressed states.Source: §I · p. 1, A. Context, Interruptibility, and… · p. 1

I. INTRODUCTION

  • EmpathicVA integrates affect, context, and interaction history in a real-time architecture and tests whether affect estimation can inform interruption policies that reduce intrusiveness while maintaining usefulness.The paper evaluates the framework on held-out multimodal data and a 48-user, six-week deployment.Source: §I · p. 1

B. Multimodal Affect Recognition

The paper links multimodal affect recognition to an adaptive interruption policy in a closed loop. The design combines physiological, vocal, and contextual streams with situation and interaction history to select behavior and incorporate feedback.

B. Multimodal Affect Recognition

  • Multimodal affect research combines prosodic, spectral, linguistic, and physiological cues, but correct classification alone does not determine whether an assistant should respond, delay, acknowledge emotion, or remain silent.Fusion can improve robustness while remaining vulnerable to dataset bias, sensor noise, missing modalities, and cross-subject variation.Source: B. Multimodal Affect Recognition · p. 2

C. Reinforcement Learning for Adaptive Interaction

  • EmpathicVA applies a Double DQN to conversational policy learning, including affect probabilities in the state and physiological stress changes in the reward.The approach connects affect recognition to adaptive interaction decisions.Source: C. Reinforcement Learning for Adaptive… · p. 2

A. Closed-Loop Architecture

  • The closed loop sends physiological, vocal, and contextual streams to MARM for five-class affect probabilities, combines them with context and past interaction results in the RL agent, and returns user and physiological feedback.The response layer carries out the selected action.Source: A. Closed-Loop Architecture · p. 2, Figure 1 · p. 2

B. Multimodal Affect Recognition

The affect model combines physiological, vocal, linguistic, and contextual evidence, while the RL policy uses affect probabilities, context, and interaction history to choose among interruption behaviors. Its reward accounts for satisfaction, timing, emotional congruence, and physiological intrusiveness.

B. Multimodal Affect Recognition

  • Physiological, acoustic, linguistic, and contextual inputs include HRV, EDA, respiration, prosody, language features, and cues such as time, location, application activity, notifications, and meetings.Physiological windows overlap, sensor noise and motion are filtered, and subject-specific z-scoring reduces baseline variation.Source: B. Multimodal Affect Recognition · p. 2
  • The model encodes physiological sequences and spectrograms with convolutional-recurrent branches, encodes context separately, and uses cross-modal attention to output probabilities for five affective states.Training uses Adam, early stopping, stratified five-fold validation, and physiological augmentation.Source: B. Multimodal Affect Recognition · p. 2

C. RL State, Actions, and Reward

  • The policy state combines affect probabilities, context, and the last ten action-feedback pairs, while its five actions range from immediate response to delay, empathetic response, or silent suppression.Retaining probabilities rather than hard labels lets the policy account for uncertainty.Source: C. RL State, Actions, and Reward · pp. 2–3
  • The reward combines explicit satisfaction, inferred timing quality, affective congruence, and physiological intrusiveness, measured using response latency, acceptance or dismissal, and post-interruption HRV and EDA changes.The Q-network uses 37 inputs and Double DQN training with prioritized replay and a target network.Source: C. RL State, Actions, and Reward · p. 3

D. Execution of Response and Personalization

The response layer translates policy choices into observable assistant behavior, while interaction feedback supports personalization over repeated encounters. The loop updates physiological features every 15 seconds and vocal and contextual features at utterances or relevant device events.

D. Execution of Response and Personalization

  • The response layer implements immediate or delayed delivery, brief task-relevant emotional acknowledgement, or silence for nonurgent proactive notifications.Silent mode preserves user-initiated commands and safety-critical notifications, and the separation allows integration with an existing VA backend without retraining its language model.Source: D. Execution of Response and… · p. 3
  • Accepted interactions strengthen the value of similar future actions, while rejected interactions weaken it, allowing preferences and stress buildup to inform personalization.A deployed system would need periodic recalibration as affective baselines and schedules change.Source: D. Execution of Response and… · p. 3
  • Physiological windows update every 15 seconds, while vocal and context features update at each utterance or relevant device event; local processing reduces network reliance and transmission of raw biosignals or audio for policy selection.Feature extraction, rather than the small DQN, mostly determines inference latency.Source: D. Execution of Response and… · p. 3

IV. EXPERIMENTAL METHODOLOGY

The methodology combines benchmark and naturalistic affect data with a six-week, within-subject comparison of EmpathicVA, context-only, and baseline assistants.

A. Data and Model Evaluation

  • The corpus combines WESAD data from 15 participants with 33 additional naturalistic participants and 4,320 utterances across five affect classes.The utterances include 1,440 RAVDESS samples and 2,880 naturalistic commands from the same participants; self-reports, benchmark labels, and temporally close events support ground-truth alignment.Source: A. Data and Model Evaluation · p. 3
  • To avoid identity leakage, data were partitioned 70% training, 15% validation, and 15% held-out testing; the RL environment was calibrated using a one-week diary study and 100,000 simulated episodes of common and rare affect-context combinations.Five runs with random seed numbers assessed convergence, defined as a 2% difference in rolling average reward over 1,000 training episodes. Field feedback was gathered after a specific policy was implemented, without unsafe online exploration.Source: A. Data and Model Evaluation · p. 3

B. Field Study

  • The six-week crossover compared baseline, context-only, and EmpathicVA, with counterbalancing and a washout week to restrict carryover effects.Week 1 covered baseline measurement and sensor calibration, followed by two week-long experimental periods separated by one week of baseline washout.Source: B. Field Study · pp. 3–4
  • Daily ratings, weekly SUS, interaction logs, interviews, and post-interruption physiology measured user experience and behavior across conditions.Physiology was compared over the 30 seconds after an interruption versus the 30 seconds before it; participants could opt out of storing data and gave permissions for individual data streams.Source: B. Field Study · pp. 3–4

C. Outcome Definitions

The outcome definitions cover interruption events, user ratings, and a comparison designed to separate affective sensing from ordinary context adaptation.

C. Outcome Definitions

  • Unnecessary interruption included premature intervention, ignored or dismissed prompts, quick silencing, or a physiological stress reaction without task engagement.Satisfaction, timing appropriateness, emotional understanding, trust, and intrusiveness used five-point scales; SUS used its standard 0–100 scale.Source: C. Outcome Definitions · p. 4
  • EmpathicVA versus baseline was the primary comparison, with context-only serving as an ablation of affective sensing.The evaluation combined self-report, behavior, and physiology to reduce dependence on any single channel; qualitative themes required repeated coded evidence, with Cohen’s κ used for inter-rater reliability.Source: C. Outcome Definitions · p. 4

V. RESULTS

EmpathicVA achieved strong held-out affect recognition and learned state-dependent interruption behavior; in the field study, it improved subjective outcomes and reduced interruption-related stress.

A. Affect Recognition and Policy Learning

  • 92.3% accuracy, 0.922 macro F1, and 0.967 AUC-ROC were achieved by multimodal fusion, with accuracy gains of 6.0, 7.6, and 31.1 points over physiological-only, vocal-only, and context-only models.Neutral and stressed states were most reliably recognized (F1=0.948 and 0.938), while excited was hardest (F1=0.869); errors clustered among affectively adjacent classes.Source: A. Affect Recognition and Policy… · p. 4
  • The Double DQN converged after 6,847 ± 432 episodes, with final average reward 0.847 ± 0.023 and 94.7% action consistency in the final 1,000 episodes.Immediate responses dominated positive and excited states, while stress shifted actions toward delay, empathetic response, and silence; negative affect had the highest empathetic-response rate at 31.2%.Source: A. Affect Recognition and Policy… · p. 4, Table IV · p. 4
  • EmpathicVA outperformed both comparison conditions across primary subjective measures, including satisfaction (4.51 versus 3.24 at baseline), timing appropriateness (4.29 versus 2.89), and trust (4.32 versus 3.45).Emotional understanding rose from 2.12 to 4.18, intrusiveness fell from 3.78 to 2.14, and SUS rose from 68.4 at baseline to 82.7; comparisons were significant at p < .001.Source: Table V · p. 4, A. Affect Recognition and Policy… · p. 5
  • Interruption-related stress episodes fell from 89 to 12, an 86.5% reduction, and unnecessary interruptions decreased by 73%.Under EmpathicVA, post-interruption changes in heart rate, EDA, and HRV RMSSD were small and not statistically significant, unlike baseline changes.Source: A. Affect Recognition and Policy… · p. 5
  • EmpathicVA logs showed 47% more interactions, 23% longer sessions, 31% more multi-step commands, and 58% fewer abandoned interactions.Privacy concern was reported by 31%; the observed relationship and behavior changes should not be interpreted as clinical efficacy.Source: A. Affect Recognition and Policy… · p. 5

C. Cross-Condition Interpretation

Context-only adaptation improved some outcomes but remained between baseline and EmpathicVA, while affect-dependent policy choices balanced responsiveness with restraint.

C. Cross-Condition Interpretation

  • The context-only assistant reduced stress episodes from 89 to 42 and raised SUS from 68.4 to 74.2, but emotional understanding rose only from 2.12 to 2.34.EmpathicVA reached 4.18 for emotional understanding, indicating that users distinguished context adaptation from behavior responsive to affect.Source: C. Cross-Condition Interpretation · p. 5
  • Emotional understanding had the largest subjective effect (d = 2.31), while trust improved more moderately (d = 0.95).The authors note that trust also depends on reliability and privacy; physiological findings were small and nonsignificant under EmpathicVA but showed sympathetic activation at baseline.Source: C. Cross-Condition Interpretation · p. 5
  • The policy combined delay, empathetic response, and silence during stress while preserving immediate responses for most positive and excited states.This state-dependent action shift is described as the mechanism connecting affect recognition to reduced unnecessary interruption.Source: C. Cross-Condition Interpretation · p. 5

VI. DISCUSSION

Empathic interaction depends on timing and restraint as well as wording: the learned policy delays or stays silent under stress, while multimodal signals provide complementary affect cues. The discussion also limits the claims to interruption management and highlights deployment and interpretation risks.

VI. DISCUSSION

  • Stress-appropriate delay and silence matter alongside empathetic wording, and user ratings and physiological changes rose together in the study.The learned policy incorporated more delay and silence during stressful periods.Source: §VI · p. 5
  • Physiological arousal, transient vocal changes, and context help distinguish ambiguous affect patterns, and fusion's advantage over unimodal alternatives suggests complementary information.The discussion notes coherent class-level errors alongside the fusion advantage.Source: §VI · p. 5
  • Affective adaptation may sustain engagement and trust beyond immediate satisfaction, but it raises governance responsibilities around privacy, trust, bias, autonomy, and inappropriate inference.The discussion recommends local processing, granular consent, visible explanations, revocation, and data minimization.Source: §VI · p. 5
  • The study's six-week Android deployment and mainly educated English-speaking sample, plus simulation-trained reinforcement learning, constrain generalization; the authors call for replication across settings.The naturalistic extension remains small for training high-capacity multimodal models, and simulation-to-field mismatch is possible.Source: §VI · p. 6
  • Wearable HRV and EDA improve stress discrimination but impose charging, comfort, and adherence costs; voice-only sensing can also fail in noisy or shared settings.The discussion recommends treating modality availability and confidence as inputs rather than assuming a fixed sensor set.Source: §VI · p. 6
  • The five affect categories are simplified correlates, not facts about feelings or clinical predictions; EmpathicVA is intended to assess interaction risk and choose conservatively under uncertainty.The discussion says its strongest evidence concerns interruption management rather than emotional intelligence generally.Source: §VI · p. 6
  • The proposed design principles combine complementary signals, state-dependent policies, concise emotional acknowledgment, repeated-interaction timing personalization, and transparency about inference and data use.Source: §VI · p. 6

A. Deployment and ethical implications

Deployment guidance emphasizes privacy, transparency, consent, and fairness, while future work calls for longer-term evaluation, cross-cultural validation, privacy-preserving learning, and independent replication.

A. Deployment and ethical implications

  • Affect-aware systems should minimize raw-data exposure and retention, support modality-specific consent and inference control, and make delayed or absent inferences understandable.The discussion also warns against training objectives that maximize engagement at the expense of autonomy.Source: A. Deployment and ethical implications · p. 6
  • Because physiological baselines, vocal prosody, emotional-display norms, and interruption attitudes vary across people and cultures, deployment should audit demographic and linguistic groups and use conservative fallbacks when calibration is poor.Subject normalization does not ensure equal performance.Source: A. Deployment and ethical implications · p. 6

B. Future Work

  • Future evaluation should measure longer-term policy effects, validate classifiers and action appropriateness across cultures and languages, quantify privacy-preserving training's calibration impact, and replicate independently.The reported corpus and deployment were generated within a single study pipeline.Source: B. Future Work · p. 6

C. Practical Design Guidance

Practical guidance calls for confidence-aware, reversible, and proportionate adaptation, while the conclusion reports improved user experience and reduced intrusiveness and physiological stress in a six-week study.

C. Practical Design Guidance

  • When sensing is unavailable or speech is noisy or disorganized, confidence and emotional labeling should decrease without reducing timing priority for safety alerts or user-chosen requests.Affect may change wording and timing only within message-priority limits.Source: C. Practical Design Guidance · p. 6
  • Personalization should be reversible through reset controls, visible preference histories, and regular confirmation; emotional acknowledgment should stay brief and proportionate.The guidance cautions that repeated or overstated emotion comments can feel patronizing or expose inferences in shared spaces.Source: C. Practical Design Guidance · p. 6
  • Evaluation should track calibration, false alarms, missed stress, interruption acceptance, stress costs, task completion, and user control, not classification accuracy alone.The authors characterize their evaluation as more robust than benchmark-only comparison but call for further replication and sampling.Source: C. Practical Design Guidance · pp. 6–7

VII. CONCLUSION

  • EmpathicVA achieved 92.3% affect-recognition accuracy on held-out data and, in a six-week study, increased satisfaction, usability, trust, and timing appropriateness while decreasing intrusiveness and physiological stress.The conclusion argues that voice assistants need both accurate perception and disciplined restraint in interrupting.Source: §VII · p. 7
Loading 2609.16417v1…