Facial emotion expressions in human-robot interaction: A survey

1 Introduction

• Social robots are increasingly used in collaborative, assistive, and public-facing roles, creating a need for effective verbal and nonverbal interaction with people.Facial expressions can signal a person's conversational status through backchanneling or rapport.Source: §1 · p. 1
• Facial expressions are presented as indispensable affective signals, while emotionally intelligent responses are important to perceived sociability and robot acceptance.The passage reports that 55% of affective information is conveyed through facial expressions.Source: §1 · p. 1
• The overview asks what has been done on facial emotion expressions in HRI and what remains to be done, following growth in publications from 2000 to 2020.It covers human expression recognition and robotic expression generation, then discusses the state of the art and future research.Source: §1 · p. 2

2 Framework of the overview

• The framework separates human facial-expression recognition from facial-expression generation by robots.These are the overview's two main research streams.Source: §2 · p. 3
• Human-expression recognition is divided into studies using predefined datasets and those recognizing expressions in real time.Source: §2 · p. 3
• Robot-expression generation is divided into hand-coded methods and automated methods that move facial features using machine learning.The features named are the robot's eyes and mouth.Source: §2 · p. 3

3 Method

• The review searched Google Scholar for facial-expression recognition and generation studies related to HRI, covering 2000–2020.Search terms included facial-expression recognition AND HRI, facial-expression recognition, and facial-expression generation AND HRI.Source: §3 · p. 3
• The review excluded non-facial modalities, HRI studies involving people with mental disorders such as autism, and studies focused on a single emotion.175 of 276 studies were rejected.Source: §3 · p. 3
• For predefined-dataset recognition, Table 3 includes studies reporting accuracy above 90%.Source: §3 · p. 3
• For real-time recognition, the review lists all studies conducting facial-expression recognition in an HRI scenario.Source: §3 · p. 3

4 Recognition of human facial expressions

• Conventional recognition pipelines detect and preprocess faces, extract features, and classify expressions; deep learning feeds preprocessed images directly into networks such as CNNs or RNNs.Common conventional features include action units, LBP, and HOG; classifiers include KNN, HMM, RNN, CNN, SVM, and LSTM.Source: §4 · pp. 3, 8, Figure 4 · p. 4, Figure 5 · p. 4

4.1 FER on predefined dataset

• Studies report accuracy above 90% on CK+, Jaffe, and Oulu-Casia datasets for both still images and videos, but these evaluations do not involve robots.Video frames provide temporal information absent from still images.Source: §4.1 · pp. 8, 10, Table 3 · p. 9

4.2 FER in real-time

• Real-time FER studies were conducted under controlled laboratory conditions with little variation in lighting and head pose, limiting the conditions represented by their results.High accuracy is easier on predefined datasets recorded under controlled conditions than with spontaneous real-time movements.Source: §4.2 · p. 10
• Real-time HRI recognition accuracies are comparatively lower than predefined-dataset accuracies, and only two studies exceed 90%; Bayesian and ANN methods achieve the highest accuracy.CNNs are commonly used in real-time recognition.Source: Table 3 · p. 9, §4.2 · p. 10

5.1 Facial expression generation is hand-coded

• Ulkni, a robot head for facial-expression imitation, uses 12 RC servos—four DoFs for gaze, two for neck, and six for eyelids and lips—and makes basic expressions after actuator position commands are sent from a PC.Minimal expression cues were movement of the eyes, eyebrows, mouth, and neck. Another implementation mapped expression direction and degree to component movements following FACS-related action units.Source: §5.1 · pp. 10, 13
• Kismet interpolates expressions across valence, arousal, and stance, while other systems add physiological motion or naturalize human-derived expressions.Park et al. add eye blinking and respiration-related motion; Prajapati et al. transform expressions derived from a human face before rendering them robotically.Source: §5.1 · p. 13
• Robot faces with sufficient eye and mouth degrees of freedom can generate basic static emotions and dynamic emotions.The review summarizes both forms as capabilities of the studied robot faces.Source: §5.1 · p. 13

5.2 Facial expression generation is automated

• Automated methods use machine learning; Leonardo learns a direct mapping from human expressions to robot joint space, whereas Horii et al. estimate emotion before generating it with an RBM.The KANSEI system recognizes human emotion, maps it into an emotion-generation space, and synchronizes the robot’s expression; subjects were found more comfortable and communicated more when synchronization occurred.Source: §5.2 · p. 13
• Nico’s RL-learned expressions could represent mixed emotional states and expressions beyond the learned basic five, though they looked slightly distorted.The review contrasts direct expression mapping and estimated-emotion generation with Nico’s association of learned expressions with conversation context.Source: §5.2 · p. 13

6.1 Summary of the state of the art

• Recognition accuracies on predefined datasets exceed 90%, with reported maxima of 100% on CK+, 99.8% on Jaffe, and 92.89% on Oulu-Casia.Source: §6.1 · p. 13
• Zhang et al. report 98.9% accuracy on CK+ but 55.27% on SFEW, whose varied conditions resemble real-world settings where recognition remains challenging.Source: §6.1 · pp. 13–14
• Recognition research mostly targets basic emotional expressions; situation-specific activities such as talking, nodding, and eating remain understudied.Source: §6.1 · p. 14, §6.2 · p. 15
• Robots can display basic expressions, but limited facial DoFs constrain their range; studies also report ambiguity, including fear being confused with surprise.Source: §6.1 · p. 14
• Automated generation can produce mixed or complex expressions beyond the learned expressions available to hand-coded systems, though relatively few studies examine automation.Source: §6.1 · p. 14

6.2 Future research

• Future recognition work should address real-world variation in pose, focus, occlusion, resolution, and illumination, as well as expressions during speech and side activities.Source: §6.2 · p. 15
• Combining facial data with voice, text, gestures, and physiological signals is proposed, alongside studying which emotions robots should express in response to human emotions.Source: §6.2 · p. 15
• The review proposes comparing facial and bodily expression, enabling partial gestures while speaking, recognizing and generating varying intensities, and moving beyond basic emotions.Source: §6.2 · pp. 15–16

• Recognition performs well on preexisting datasets but less well in the wild, motivating work on non-frontal poses, occlusions, speech, and varying emotion intensities.Source: §7 · p. 16
• Robots can make basic facial expressions, but few studies enable autonomous facial expression generation.Source: §7 · p. 16
• Future work includes comparing facial and bodily expressions, using partial gestures while speaking, varying expression intensity, and extending beyond the basic seven expressions.Source: §7 · p. 16

• The authors declare that the research has no compliance issues.Source: Compliance · p. 16
• The project received funding from the German Research Foundation, ZEVEDI Hessen, and the leap in time foundation.Source: Compliance · p. 16
• The authors declare that they have no conflict of interest.Source: Compliance · p. 16