Artificial Emotional Intelligence in Socially Assistive Robots for Older Adults: A Pilot Study

1 INTRODUCTION

• Socially assistive robotics aims to support users, including older adults who may experience loneliness and isolation in senior care facilities.The introduction links social interaction and mental stimulation with older adults’ well-being and describes companionship through conversation as one way SAR addresses isolation.Source: §1 · p. 1
• Prior work suggests that emotional information can enhance satisfaction and that empathic robot behavior is perceived as friendly, understanding, and caring.The introduction presents empathy as especially relevant to health and elder care.Source: §1 · p. 1
• The study compares empathic Ryan, which recognizes emotions and responds through an affective dialogue manager, with a non-empathic version using scripted dialogue without facial expressions.The paper’s contributions include developing the multimodal emotion-sensing and expressive system, integrating it into a physical robot, and studying it with older adults in a senior care facility.Source: §1 · p. 2
• The study focuses on whether emotional intelligence can improve the quality of interaction and engagement with Ryan.The authors identify the effect of emotional intelligence on engagement as a focus not thoroughly examined in their previous study of Ryan’s multiple features.Source: §1 · p. 2
• The paper’s hypothesis is that an emotionally intelligent robot will be perceived as friendlier and positively affect users’ mental well-being compared with a robot without empathic capabilities.Well-being is assessed through changes in depression score and emotional state.Source: §1 · p. 2

2 EMOTIONAL INTELLIGENCE

• Emotional intelligence involves perceiving, understanding, expressing, and managing emotions, while empathy is one aspect of emotional intelligence involving sharing, inferring, and adopting others’ perspectives.The section describes three components: sensing emotions, understanding and modeling them, and using and expressing them.Source: §2 · p. 2
• Earlier work includes social robots for older adults and healthcare, but the cited examples do not focus on emotional intelligence and its effects on users.The section also notes prior research fusing modalities for emotion recognition.Source: §2 · p. 3

2.1 Sensing and measuring emotions

• Robots ideally recognize emotion multimodally because users convey emotions through multiple modalities, while many HRI studies use unimodal recognition.The section names facial expression recognition and sentiment analysis as visual and verbal approaches, respectively.Source: §2.1 · p. 3

2.2 Understanding and modeling emotions

• Ryan combines facial-expression analysis and sentiment analysis to disambiguate sensory information, with a weighted dot product producing a valence value from -1 to +1.The sensitivity vector weights each input modality; the authors say those weights can be determined through an HRI study or from each modality’s measurement accuracy.Source: §2.2 · p. 3

2.3 Using and expressing emotions

• A robot can express emotions through facial expressions, gestures, or voice, based on its internal state, a reaction to the user, or a predefined script.The section summarizes an emotionally intelligent robot as sensing and interpreting users’ emotions, choosing a response, and delivering it through multimodal expression.Source: §2.3 · p. 3

3 RYAN, AN EMOTIONALLY INTELLIGENT ROBOT

• Ryan is designed as a companion robot for older adults, with cameras, a microphone, speakers, an animated face, and a display supporting interaction.Its physical platform also includes a depth camera for body tracking and an actuated neck for tracking users and maintaining eye contact.Source: §3 · pp. 3–4, §3.4 · p. 6

3.1.1 Facial expression recognition

• The FER network classifies 64 × 64 RGB face images into positive, neutral, or negative probabilities using a ResNet50.It was trained on AffectNet and fine-tuned on images of people aged 50+; training accuracy stabilized around 80%.Source: §3.1.1 · p. 4, Figure 3 · p. 4

3.1.2 Emotional state measurement

• Ryan smooths facial-valence estimates over the last 30 frames, weighting recent frames more heavily to reduce noise and misclassification.Measured values are grouped into negative, neutral, and positive ranges for facial-expression mirroring.Source: §3.1.2 · pp. 4–5, Figure 5 · p. 5

3.1.3 Sentiment analysis

• Speech sentiment is scored from -1 to +1 and fused with facial emotional state as a weighted average for directing conversation.The equal weighting was chosen from laboratory tests; the authors say more experiments are needed to find the optimal balance.Source: §3.1.3 · p. 5

3.2 Dialogue generation

• Ryan’s more than 90 minutes of dialogue span 12 topics, with empathic replies selected according to perceived facial expression and response sentiment.Positive, negative, and neutral cues prompt different reply styles, while Ryan can mirror positive expressions or show a compassionate face for negative emotion.Source: §3.2 · p. 5, Figure 6 · p. 5

3.3 Affective dialogue system

• Program-R is a hybrid rule-based and machine-learning dialogue system that combines facial and textual information to respond to users.It uses context and session data to select responses, and can initiate a controlled conversation with the user.Source: §3.3 · pp. 5–6, Figure 7 · p. 6

3.4 Robotic platform

• Ryan’s embodied platform combines an expressive animated face and tracking neck with audio input/output and a touchscreen for multimedia.A chest-mounted Kinect depth camera provides body tracking, while shoulder LEDs indicate whose turn it is to speak.Source: §3.4 · p. 6

4.1 Participants

• Ten residents participated; eligibility required age 60 or older, SLUMS scores of 15–26, English verbal skills, depression-related behavior difficulties, and three-week availability.Participants averaged 77.1 years of age; nine were Caucasian and seven were female.Source: §4.1 · pp. 6–7

4.2 Experimental setup

• Participants completed six approximately 15-minute sessions over three weeks, with randomized groups switching between Emotion ON and OFF after three sessions.All participants experienced both versions, and users were unaware of the condition change.Source: §4.2 · p. 7

4.3 Measurements

• Engagement measures included words per utterance and recorded facial-expression and sentiment outputs; mood was rated on a 0–10 face scale before and after sessions.Exit surveys and interviews assessed likeability and empathy, and researchers also interviewed a caregiver.Source: §4.3 · p. 7

5.1 Quantitative analysis

• Users uttered more words with empathic Ryan (M=4.11, STD=5.372) than with non-empathic Ryan (M=3.71, STD=3.350), and Emotion ON/OFF significantly affected word count.The condition did not significantly affect emotional state or sentiment.Source: §5.1 · pp. 7–8, Table 3 · p. 8
• Median face-scale mood increased from 7 before sessions to 9 afterward, regardless of condition (Z = −5.466, p < 0.001).Figure 9 shows pre/post-session score changes.Source: §5.1 · p. 8, Figure 9 · p. 9
• Participants reacted more positively when Ryan empathized: in the first three dialogue sessions, positive detected facial expressions averaged 25.7% for G1 with emotion disabled and 45% for G2 with emotion enabled.The authors note that session order and conversation topics may have affected participants’ experience and the magnitude of group differences.Source: §5.1 · p. 8
• Seven of ten participants improved by 1–16 points on GDS or 1–6 points on PHQ-9 depression scores after the study.The maximum scores were 30 for GDS and 27 for PHQ-9.Source: §5.1 · p. 9, Table 5 · p. 10

5.2.1 Exit survey questionnaires

• Exit-survey averages exceeded 4.00 on all but the item comparing conversation with Ryan to talking to a person; 73% noticed communication or facial-expression changes after crossover.Participants rated feeling happier with Ryan at 4.5 and Ryan empathizing with them at 4.57 on the five-point scale.Source: §5.2.1 · p. 9, Table 4 · p. 10

5.2.2 Participants’ feedback

• Participants described Ryan as enjoyable, interesting, and friendly, while some also noted interruptions, appearance concerns, or frustrating lights and repetition.One participant said she enjoyed Ryan when she smiled but found her less appealing when she was not smiling.Source: §5.2.2 · pp. 9–10

5.2.3 Caregiver’s feedback

• The caregiver reported apparent mood and cognition improvement, including renewed smiling and community engagement for one participant, while noting that these observations were anecdotal.The caregiver also identified Ryan’s smiling and stimulated human interaction among the reported experiences.Source: §5.2.3 · pp. 10–11

• Nursing-home understaffing and older adults’ loneliness motivate socially assistive robots that are socio-emotionally intelligent enough to engage aging users.Source: §6 · p. 11
• Empathic Ryan recognizes users’ emotions and uses facial expressions and an affective dialogue manager, unlike the scripted non-empathic version.Source: §6 · p. 11
• Face-scale mood measurements showed an overall positive effect with either Ryan version, while word counts and exit surveys suggested empathic Ryan was more engaging and likable.Source: §6 · p. 11
• Depression-score changes suggest Ryan may decrease users’ depression, but verification requires more subjects and long-term studies.Source: §6 · p. 11

• Ryan omits acoustic, gaze, gesture, posture, and breathing cues, which could improve emotion and intention recognition accuracy and reliability.Source: §7 · p. 11
• The study was limited by its small sample, imbalanced participant demographics, and number of sessions.Source: §7 · p. 11
• Older adults’ slow speech and long pauses can trigger end-of-sentence detection, interrupting participants and negatively affecting their perception.Source: §7 · p. 11
• Open-domain dialogue was the study’s most limited area, and the rule-based chatbot could grow its knowledge base and pattern-matching rules.Source: §7 · p. 11
• Ryan currently mirrors only users’ facial expressions; future work could let conversation and context drive its expressions.Source: §7 · p. 11