PaperlayerView exact revision on arXiv

Source-linked AI summary

EmoSay: Artificial Intelligence-Driven Text-to-Emotional-Speech System for Affective Communication in Extended Reality

Sikiru Ademola Adewale, Sunday D. Ubur, Nikitha Donekal Chandrashekar, Onyeka Emebo, Denis Gračanin

arXiv:2608.26566v1cs.HC

TL;DR

Current neural TTS often lacks contextually appropriate emotional expression, limiting affective communication in XR. EmoSay combines emotion-conditioned synthesis with XR integration, and evaluation reports a SUS score of 74.76 and a positive correlation (r = 0.74) between perceived vocal naturalness and user engagement.

  • Problem

    Many neural TTS systems provide limited, implicitly learned prosody control and lack emotional expression aligned with linguistic meaning in XR.

  • Method

    EmoSay combines discrete emotion-conditioned neural TTS, a Unity-based XR interface, and mixed-method evaluation of speech generation and user perceptions.

  • Results

    A SUS score of 74.76 and a positive correlation (r = 0.74) between perceived vocal naturalness and user engagement support EmoSay’s usability and effectiveness.

  • Takeaways & Limitations

    EmoSay provides a foundation for more inclusive and empathic human-computer interaction in XR through expressive, spatialized audio.

  • Takeaways & Limitations

    Emotion synthesis relies on categorical labels, although human emotion often comprises blended states.

Abstract

from arXiv · show

While contemporary neural text-to-speech (TTS) systems have achieved high levels of intelligibility, they frequently lack the emotional nuance required for authentic affective communication. This limitation is particularly critical in Extended Reality (XR), where the absence of emotionally expressive audio can diminish user presence and spatial immersion. We present EmoSay, an Artificial Intelligence-driven Text-to-Emotional-Speech (TTES) system designed to bridge the semantic-affective gap in immersive environments. EmoSay modulates a neural synthesis pipeline using discrete emotional prompts, delivering the output through a Unity-based interface featuring high-fidelity spatialized audio. The system was evaluated through a comprehensive user study focusing on perception, engagement, and the subjective sense of empathy. Our results demonstrate that EmoSay significantly enhances the immersive experience, achieving a System Usability Scale (SUS) score of 74.76, indicating strong usability and seamless integration within the XR workflow. Subjective assessments reveal a high degree of perceived naturalness and a strong positive correlation between emotional expressiveness and user engagement. Regression analysis identifies vocal naturalness as the strongest of the tested predictors of user satisfaction, suggesting that EmoSay's affective prosody helps meet the heightened expectations for realism in immersive settings. This work contributes a scalable, affect-aware framework for inclusive XR design and demonstrates the role synthetic emotion can play in fostering human-computer rapport through voice-first interaction.

1 Introduction

Existing neural TTS systems achieve fluent, intelligible speech but often neglect contextually appropriate emotional expression. EmoSay addresses this semantic-affective gap by conditioning neural TTS on emotional prompts and delivering expressive speech in immersive XR.

  • Motivation: EmoSay addresses TTS systems’ emphasis on linguistic accuracy and prosodic naturalness, which often leaves speech insufficiently expressive for empathic HCI interactions [9] [44].The motivating limitation is that emotional expression is frequently treated as secondary to segmental accuracy and fluency.
  • System overview: EmoSay lets users specify text and a target emotion, generating emotion-consistent expressive speech rendered in an immersive Extended Reality environment.The system is designed for applications including accessible reading support, language learning, interactive storytelling, and therapeutic use.
  • System design: The system conditions neural TTS on discrete emotional prompts and integrates it with a Unity-based XR front end.This design combines emotional speech generation with an immersive interface for affective XR interaction.
  • Motivation: XR enables emotional speech to be heard and felt as an immersive, spatial, interactive phenomenon, potentially amplifying empathy, engagement, and accessibility.The motivation particularly includes learners with reading challenges and individuals using assistive technologies who may benefit from multimodal emotional cues.
  • Research questions: The study examines how emotional awareness affects engagement and perceived empathy, and how expressive TTS can remain consistent with linguistic and semantic content.These questions frame the study’s focus on emotion-aware HCI in virtual environments.

2 Related Work

Prior work has advanced controllable and adaptable TTS, including emotion-conditioned and multimodal approaches, but XR applications often lack integrated emotional and contextual speech. Existing research demonstrates the value of vocal affect for engagement, empathy, trust, and social presence while leaving open challenges in contextual appropriateness, inclusivity, and reciprocal emotional interaction.

  • Neural TTS: General-purpose TTS has improved speech quality, efficiency, multilinguality, multi-speaker support, and adaptation, but typically offers limited prosody control and implicit emotional expression.Major architectures separate text encoding, acoustic modeling, and vocoding, while newer frameworks broaden deployment to mobile and embedded XR devices.
  • TTS in XR: In XR, TTS mainly supports narration, instructions, and virtual-agent dialogue, yet conventional pipelines rarely integrate emotional or contextual cues deeply enough for immersive environments.This creates a mismatch between the richness of XR settings and the affective capabilities of synthetic speech.
  • Emotion-aware TTS: Emotion-aware TTS conditions acoustic generation on categorical or dimensional affect and learned style representations, using methods such as global style tokens, prosody encoders, and emotion-classification multitask learning.Specialized acted-emotion datasets, including EMO-DB, RAVDESS, and CREMA-D, support supervised learning, while multimodal prompts improve contextual alignment and controllability.
  • Applications and effects: Emotionally expressive speech can improve engagement, comprehension, accessibility, empathy, trust, rapport, and social presence across education, assistive systems, affective computing, HCI, social robotics, and XR.Vocal affect and emotional prosody shape perceptions of agent empathy and credibility, while expressive prosody can convey emphasis, urgency, nuance, and attitudes.
  • Research gaps: Existing XR and empathetic-agent studies remain limited by missing auditory expression, reciprocal emotional feedback, multimodal emotional translation, real-human interaction, and robust handling of cultural or contextual appropriateness.Emotional Voice Puppetry links emotional voice to facial animation without reciprocal feedback; CAEVR uses biosignals without auditory expression, while other systems often rely on simulations or face-based methods affected by HMDs.

3 System Design

EmoSay is a modular text-to-emotional-speech system combining emotion-conditioned neural synthesis, evaluation feedback, and Unity-based XR integration. Users provide text and a discrete emotion, which conditions prosody and produces spatialized speech for immersive affective communication.

  • Architecture: EmoSay combines text and emotion input, an emotion-conditioned TTS backend, evaluation and feedback, and XR integration in a modular architecture.The system interaction flow converts textual content and emotional parameters into synthesized speech for an immersive virtual scene.
  • Emotion Conditioning: Users select discrete emotions such as neutral, happy, sad, angry, fearful, disgusted, or surprised, which are mapped to learned embeddings injected into the acoustic model.The taxonomy aligns with labels in datasets including RAVDESS, CREMA-D, and EMO-DB.
  • Speech Synthesis: The neural TTS acoustic model uses emotion conditioning to modulate pitch, energy, and duration, while a neural vocoder converts generated melspectrograms into waveform audio.During training, emotion embeddings are associated with characteristic acoustic patterns in emotional speech datasets.
  • Evaluation and Feedback: An auxiliary emotion classifier evaluates natural and synthesized speech for alignment between target and produced emotion, with potential future use as an adversarial or multi-task training signal.The classifier currently supports evaluation, while expanded training uses are proposed for future work.
  • XR Integration: The Unity XR front end supports text entry, discrete emotion selection, and spatialized playback on head-mounted or desktop immersive setups.Users submit text and emotion to the backend and hear the resulting speech anchored through spatial audio.

4 Method

EmoSay is evaluated through a mixed-method XR study combining acoustic-prosodic analysis of benchmark emotional-speech datasets with user-centered assessments of perception, engagement, and empathy. Participants interact with the system through Unity-based XR or desktop configurations and complete structured ratings and qualitative feedback analyzed with descriptive, nonparametric, correlational, reliability, and regression methods.

  • Study design: The mixed-method design combines objective analysis of emotional-speech generation with user-centered evaluation of perception, engagement, and empathy in XR.Emotional consistency is examined using acoustic-prosodic patterns from RAVDESS, CREMA-D, and EMO-DB.
  • Participants: Approximately 30 participants with diverse ages, genders, and educational backgrounds interact with EmoSay and report usability, satisfaction, emotional perception, engagement, and empathy.Participants are recruited from a university community and surrounding area, screened for XR discomfort, and need only basic digital-device familiarity.
  • Analysis: Study data are summarized with descriptive statistics and analyzed using Kruskal-Wallis tests, correlation analysis, Cronbach's alpha, and regression models.Analyses examine demographic, emotional, and participant-group differences; relationships among emotion recognition, naturalness, empathy, and engagement; and predictors of engagement or empathy.
  • Implementation: The Unity-based XR application runs on Meta Quest Pro or desktop with high-fidelity stereo headphones, enabling natural-language input and discrete emotion selection for speech generation.The study materials include emotionally driven text prompts, a discrete emotion-selection interface, and a mixed-method questionnaire covering naturalness, emotional appropriateness, empathy, engagement, and spatial presence.
  • Procedure: Participants enter text, select emotional parameters, rate naturalness, clarity, and emotional appropriateness, then assess empathy, engagement, and overall system utility.The single-scenario session follows formal orientation and system interaction before questionnaire completion.

5 Results · 5.1 Descriptive Statistics · 5.2 System Usability and Reliability

The user study evaluated EmoSay’s usability, perception-metric reliability, and relationships among naturalness, human-likeness, engagement, and satisfaction in an XR environment. Results indicate good usability, excellent internal consistency, and strong positive associations among perceived speech qualities and engagement, while noting opportunities to refine interface consistency and responsiveness.

  • 5.1 Descriptive Statistics: The study included 30 participants, providing the demographic basis for evaluating user perceptions of EmoSay in the XR environment.The results section combines user-study findings with standard-dataset analysis, focusing on usability, perceived emotional-speech quality, and factors associated with satisfaction.
  • 5.1 Descriptive Statistics: 63.33% of participants were aged 25–34, and the gender distribution was 56.67% male and 43.33% female.The sample captured varied user perspectives but was skewed toward the 25–34 age group.
  • 5.2 System Usability and Reliability: 74.76/100 SUS placed EmoSay at Grade C (Average/Good) on the Sauro-Lewis scale, indicating functional and accessible usability.The findings also identify opportunities to improve interface consistency and technical responsiveness.
  • 5.2 System Usability and Reliability: α = 0.942 demonstrated excellent reliability for Likert measures of naturalness, realism, and engagement.The coefficient indicates strong internal consistency among the perception items used in the study.
  • 5.2 System Usability and Reliability: r = 0.74 indicated a strong positive relationship between vocal naturalness and user engagement in the perception-metric correlation matrix.Figure 3 summarizes the relationships among user perception metrics and overall satisfaction.
  • 5.2 System Usability and Reliability: r = 0.90 linked naturalness with human-like qualities, while human-likeness correlated with engagement at r = 0.84.These relationships suggest that more realistic emotional nuances were perceived as more authentically human and associated with greater engagement.

5.3 User Perception of Emotional Speech

Users responded most positively to the system’s enjoyment and engagement aspects, while emotional speech enhanced their sense of immersion in the virtual environment.

  • User Perception of Emotional Speech: Users responded most positively to enjoyment and engagement, and reported that emotional speech enhanced immersion in the virtual environment.These findings are summarized in Table 2 using mean scores on a 1–7 scale.

Desired Emotional States in XR · 5.4 Drivers of User Satisfaction

EmoSay users preferred high-arousal positive emotions and rated engagement and naturalness highly. Naturalness was the strongest descriptive satisfaction predictor, but no individual vocal attribute was statistically significant, despite its strong correlation with engagement and consistent satisfaction across demographics.

  • Desired Emotional States in XR: Happiness was selected most often (n = 26), followed by Surprise (n = 22) and Excitement (n = 21), indicating preference for high-arousal positive emotions in XR.Anger and Fear were selected least frequently.
  • Desired Emotional States in XR: Engagement and naturalness both had median scores of 6 on the Likert scale, while human-likeness had a lower median of approximately 5.5 and the widest interquartile range.The distributions were reported in Figure 5 using a 7-point Likert scale.
  • 5.4 Drivers of User Satisfaction: R² = 0.452, and Naturalness had the largest standardized coefficient (β = 0.202), but no individual vocal attribute significantly predicted satisfaction at p < 0.05.Only the model constant was significant (p = 0.021), so the predictor-level finding should be interpreted cautiously.
  • 5.4 Drivers of User Satisfaction: Naturalness and engagement were descriptively more associated with satisfaction than human-like accuracy, although none of these differences were statistically significant.Naturalness and engagement were treated as stronger descriptive associations, not statistically established drivers.
  • 5.4 Drivers of User Satisfaction: r = 0.74, p < 0.001, for the correlation between perceived naturalness and engagement, suggesting natural-sounding prosody may help maintain user interest.The correlation was observed separately from the regression analysis.
  • 5.4 Drivers of User Satisfaction: p = 0.5157 in the Kruskal-Wallis test indicated no statistically significant satisfaction difference across age groups.Figure 6 reports average satisfaction scores with standard-error bars across age groups.
  • 5.4 Drivers of User Satisfaction: 74.76 was the SUS score, while α = 0.942 indicated high internal reliability; satisfaction was also consistent across demographic groups.The demographic consistency was reported alongside Figure 6 and a Kruskal-Wallis test.
  • 5.4 Drivers of User Satisfaction: The results associate spatialized emotional audio with user presence and suggest potential for reliable, inclusive, and empathic immersive environments.This interpretation is supported by the naturalness–engagement relationship, SUS score, demographic consistency, and internal reliability.

5.5 Qualitative Analysis and User Preferences

The qualitative analysis complements quantitative metrics through thematic analysis of open-ended feedback and frequency analysis of desired emotional states, informing future enhancements to EmoSay’s affective depth.

  • Qualitative Analysis: The study complemented quantitative metrics with thematic analysis of open-ended user feedback.
  • User Preferences: Frequency analysis examined the emotional states users desired from EmoSay.
  • Qualitative Analysis: Together, these findings provide a roadmap for enhancing EmoSay’s affective depth.

Sentiment Analysis · Thematic Analysis of User Suggestions

User feedback was sentimentally balanced: participants acknowledged EmoSay’s functional success while identifying technical shortcomings in emotional delivery. Thematic analysis emphasized greater realism, human-likeness, vocal texture, and a wider range of non-verbal emotional cues.

  • Sentiment Analysis: Sentiment analysis produced a neutral score of 0, indicating balanced user feedback.Users recognized the system’s functional success while also offering constructive technical criticism.
  • Sentiment Analysis: Users specifically criticized the naturalness of the emotional delivery.The feedback combined positive recognition of functionality with concerns about affective realism.
  • Thematic Analysis of User Suggestions: Thematic coding identified increased realism and human-likeness as the most prominent improvement theme.Table 4 summarizes the thematic coding of user suggestions.
  • Thematic Analysis of User Suggestions: Users found the speech intelligible but said it retained a robotic feel associated with synthesized speech.They called for further refinement of the system’s vocal texture.
  • Thematic Analysis of User Suggestions: Respondents requested refinements in vocal texture to make the emotional speech more human-like.The suggestion directly targeted the remaining gap between intelligibility and naturalness.
  • Thematic Analysis of User Suggestions: Users also suggested a broader variety and range of non-verbal emotional cues.This recommendation addressed the system’s emotional expressiveness.

6 Discussion

EmoSay shows that emotion-aware neural speech in XR can improve empathy, engagement, realism, and inclusive affective communication, while revealing limits in categorical emotion modeling and ethical risks requiring future work.

  • Contribution: Unity-based XR makes emotional speech spatial and interactive, supporting immersive affective communication and multimodal accessibility for users with reading or communication challenges.The integration also enhances expressiveness in education.
  • Contribution: EmoSay contributes a scalable framework for inclusive XR design while foregrounding ethical concerns about emotional AI in persuasive or sensitive contexts.The discussion connects affective computing with immersive design and notes limited active initiatives in inclusive XR design, including the hibernation of the XR Access Initiative.
  • RQ2: 90.1% auxiliary-classifier accuracy in recognizing target emotions indicates discrete emotional embeddings preserve acoustic-prosodic integrity across textual contexts [44].The neural pipeline conditions synthesis on discrete emotional embeddings rather than relying solely on global style tokens.
  • Limitations and Future Work: Categorical emotion labels limit granularity because human emotions blend, motivating future zero-shot style transfer, multimodal prompts, and context-aware virtual companions.Future work also proposes larger samples, richer empathy and accessibility data, continuous emotion dimensions, user-specific affect profiles, and multimodal sensing.

7 Conclusion and Future Work

EmoSay bridges the semantic-affective gap in XR through discrete emotion conditioning and high-fidelity spatialized audio, supporting affective communication through expressive speech. Its evaluation indicates solid usability and links vocal naturalness with user engagement, while future work targets richer affect modeling, personalization, physiological adaptation, and longitudinal validation.

  • Conclusion: EmoSay addresses limited vocal emotional nuance in immersive technologies by combining discrete emotion conditioning with high-fidelity spatialized audio.The system is designed to enhance affective communication within XR through expressive audio.
  • Conclusion: 74.76 SUS score demonstrates EmoSay’s solid technical viability and usability, while perceived vocal naturalness correlates positively with user engagement (r = 0.74).The findings position emotional prosody as a functional contributor to immersive realism rather than merely an aesthetic feature.
  • Future Work: Future work will replace categorical emotion labels with valence-arousal-dominance models and add zero-shot style transfer for personalized vocal identities while preserving emotional integrity [49].These changes aim to support more complex and blended affective states.
  • Future Work: The researchers plan closed-loop empathic XR using heart rate variability and electrodermal activity to adapt synthesized speech to users’ affective states.Longitudinal studies will also examine expressive synthetic voices’ long-term effects on social presence and trust in collaborative virtual environments.
Loading 2608.26566v1…