Source-linked AI summary
Nonverbal Cues in Human-Robot Interaction: A Communication Studies Perspective
Jacqueline Urakami, Katie Seaborn
TL;DR
The paper addresses the need for guidance on established nonverbal codes and their application in HRI. It presents a communication-studies framework organized around five human sensory channels and concludes that several areas remain ripe for further exploration.
Problem
The paper identifies a need for an overview of established nonverbal codes and guidance on how they may be applied in HRI.
Source: §1 · p. 3Method
Results
The paper provides an overview of nonverbal codes for vision, audition, haptics, olfaction, and gustation and translates them into suggestions for HRI research and development.
Source: §5 · p. 19Takeaways & Limitations
The authors identify haptics, olfactics, and chronemics as ripe for HRI exploration, while noting that body movement and gestures have already received attention.
Source: §5 · p. 20The review could provide only a sample of HRI work because communication-studies terminology is often absent or difficult to navigate in that literature.
Source: §5 · p. 20
Abstract
from arXiv · showhide
Communication between people is characterized by a broad range of nonverbal cues. Transferring these cues into the design of robots and other artificial agents that interact with people may foster more natural, inviting, and accessible experiences. In this position paper, we offer a series of definitive nonverbal codes for human-robot interaction (HRI) that address the five human sensory systems (visual, auditory, haptic, olfactory, gustatory) drawn from the field of communication studies. We discuss how these codes can be translated into design patterns for HRI using a curated sample of the communication studies and HRI literatures. As nonverbal codes are an essential mode in human communication, we argue that integrating robotic nonverbal codes in HRI will afford robots a feeling of "aliveness" or "social agency" that would otherwise be missing. We end with suggestions for research directions to stimulate work on nonverbal communication within the field of HRI and improve communication between human and robots.
1 INTRODUCTION
The introduction frames HRI communication from the receiver’s perspective and motivates a communication-studies overview of nonverbal cues. It defines the paper’s goals as establishing an HRI framework and showing how codes can be applied.
1 INTRODUCTION
- In HRI, communication is a person assigning meaning to stimuli a robot transmits intentionally or unintentionally, making a receiver-based definition useful when robot agency is uncertain.The paper uses “robot” broadly, including interfaces and AI-infused environments.Source: §1 · p. 2
- Nonverbal cues include meaningful information conveyed without words, including non-word sounds and vocal qualities such as pitch, tone, and intonation.The receiver may assign meaning regardless of whether the sender intended to transmit it.Source: §1 · p. 2
- HRI research has acknowledged nonverbal cues, but designers need an overview of established communication-studies codes and guidance on their application in HRI.The introduction notes particular relevance for novice users in social settings.Source: §1 · p. 3
- The paper proposes an HRI overview grounded in human communication research, defining nonverbal communication, organizing codes, and illustrating their HRI applications.It also offers research and development ideas and an outlook for further exploration.Source: §1 · p. 3
2 ADAPTING HUMAN NONVERBAL “CODES” FOR HRI
The paper links robotic nonverbal communication to impressions of liveness and social agency, while situating its framework against prior HRI work that often addressed cues separately. It argues for a broader communication-studies perspective.
2 ADAPTING HUMAN NONVERBAL “CODES” FOR HRI
- Robotic physiological cues such as blushing, breathing, heartbeat, and temperature may contribute to impressions of a robot as alive or human-like.The paper describes simulated blushing in Flobi and a bear-style robot with breathing, heartbeat, and varying temperature.Source: §2 · p. 4
- The paper develops the idea that nonverbal modalities can give robots a feeling of “liveness” or “social agency” and may support mutual understanding and coordination.The cited HRI literature includes work on gestures, touch, social distance, gaze, and movement.Source: §2 · p. 4
- Prior HRI work has often examined a limited range of cues in isolation, with less attention to how nonverbal cues influence people’s attitudes and behaviour.The paper takes a step toward addressing these gaps through a comprehensive human communication-studies perspective.Source: §2 · p. 4
3 NONVERBAL COMMUNICATION CHARACTERISTICS
The paper defines HRI nonverbal communication as a receiver assigning meaning to robot stimuli, then outlines characteristics that shape how cues are interpreted. These include ambiguity, unintended signals, cue combinations, continuity, concreteness, and cultural context.
3 NONVERBAL COMMUNICATION CHARACTERISTICS
- HRI nonverbal communication comprises wordless communication, including nonverbal utterances, where a person assigns meaning to robot stimuli whether or not they were intentionally transmitted.The paper focuses on the robot as sender and person as receiver, and identifies seven characteristics relevant to implementation.Source: §3 · p. 5
3.1 Multiple Interpretations
- A robot cue such as eye color can prompt varied reactions, so redundant signals and multiple codes can help clarify the intended message.The paper notes that the same movement away from a robot could signal disinterest, fear, surprise, or the end of an interaction.Source: §3.1 · p. 5
3.2 Unintentional Behavior
- Robot features constrained by technical feasibility may still be interpreted as intentional cues, while mismatched or emotionless vocal affect can confuse receivers or be read as lack of empathy.The voice-related examples include affective qualities that do not match expectations.Source: §3.2 · pp. 5–6
3.3 Multiple Cues
- When multiple cues convey the same message, their combined impact is intensified; contradictory cues can confuse and undermine the message.The paper gives a head-shake for “no” paired with spoken “yes” as an example of contradiction.Source: §3.3 · p. 6
3.4 Immediate Message Transmission
- A robot’s mere presence and behavior, including motor noise, can transmit messages that receivers interpret.The paper describes nonverbal cues as continuously sent, intentionally or unintentionally.Source: §3.4 · p. 6
3.5 Continuous
- Nonverbal cues flow into one another rather than having clear boundaries, and isolated gestures can be difficult to interpret and feel unnatural or uncanny.The paper identifies creating continuous nonverbal cues as a major robotics challenge.Source: §3.5 · pp. 6–7
3.6 Concrete
- Because nonverbal cues are often concrete and representational, HRI cues should synchronize with speech and match its content; abstract gestures can be difficult for novice users to interpret.The paper qualifies this recommendation by the technical limits of robot motion.Source: §3.6 · p. 7
3.7 Contextual and Cultural
- Many nonverbal cues are shared within cultural groups and generational cohorts, so robot cues should be selected for the intended culture and cohort.The paper contrasts this with basic-emotion facial expressions that are interpreted similarly in many places.Source: §3.7 · p. 7
4 THE NONVERBAL CODES
The framework organizes robotic nonverbal codes by the human sensory channels they address. Vision and audition are its main focus, with haptics and olfaction explored and gustation included for completeness.
4 THE NONVERBAL CODES
- The framework uses human sensory channels to organize robotic nonverbal codes, focusing on vision and audition while also addressing haptics and olfaction and including gustation for completeness.Figure 1 relates the HRI nonverbal codes to the five human sensory channels.Source: §4 · p. 7, Figure 1 · p. 8
4.1 Visual Nonverbal Codes
Visual nonverbal codes include appearance, movement, posture, facial behavior, and interpersonal distance. Their design for HRI should account for how context and culture shape interpretation.
4.1.1 Physical Appearance and Object Language
- Robot appearance and object properties can shape receivers’ affective responses, while body form should be considered alongside expectations about a robot’s capabilities and behavior.The paper proposes applying communication-studies frameworks to compare body types across robots, contexts, and user groups.Source: §4.1.1 · p. 8, §4.1.1.1 · p. 9
4.1.1.2 Clothing
- Clothing can signal social attributes or a robot’s role, but its effects on people’s perceptions of robots have not yet been well studied.The paper suggests robot clothing as a potentially quick and easy cue to explore.Source: §4.1.1.2 · p. 9
4.1.1.3 Kinesics: Body Movement and Gestures
- The paper adopts Ekman and Friesen’s whole-body gesture categories for HRI, while cautioning that emblem meanings vary across cultures and contexts.It recommends designing emblems with the relevant cultural group’s needs and expectations in mind.Source: §4.1.1.3 · p. 10, §4.1.1.4 · pp. 10–11
4.1.1.5 Illustrators
- Illustrators can reinforce speech with visual information, but more work is needed to test whether people accurately interpret robot-performed illustrative gestures and find them useful.Examples include gestures that indicate an object’s size or shape.Source: §4.1.1.5 · p. 11
4.1.1.6 Regulators
- Regulators such as nods, gaze, and pointing can manage turn-taking, especially in multi-party HRI, but their timing requires situational awareness.The paper describes a progression from signaling speech boundaries to managing group discussion.Source: §4.1.1.6 · pp. 11–12
4.1.1.7 Adaptors
- Adaptors may convey a robot’s internal state and support impressions of humanlikeness or reciprocity, but few studies have examined them as part of a regular robot gesture repertoire.The paper imagines coordinating small cues, such as breathing and shifting, to convey nervousness.Source: §4.1.1.7 · p. 12
4.1.1.8 Posture and Body Language
- Robot posture and whole-body movement can communicate activation, roles, or emotion, while existing work has largely focused on a limited set of upper-body movements.The paper calls for more research on how movement type and style affect impressions of robots.Source: §4.1.1.8 · pp. 12–13
4.1.1.9 Proxemics
- Proxemic behavior in HRI relates to factors including autonomy, gaze, liking, task, and power distance; the paper also raises questions about robots’ awareness of personal-space boundaries.It suggests evaluating near-future applications such as exoskeletons and exploring whether perceived personal space differs by robot form.Source: §4.1.1.9 · pp. 13–14
4.1.2 Facial Expressions
Facial expressions can communicate emotion, but their interpretation and appropriate display depend on psychological, cultural, and professional context. The paper argues that robot facial displays can also support shared understanding and relationships, not only portray inner affect.
4.1.2 Facial Expressions
- Facial expressions are shaped by context and display rules, so the same expression may not carry the same meaning across people or situations.The passage describes cultural and professional expectations governing when particular expressions are appropriate.Source: §4.1.2 · pp. 14–15
- Robot facial displays could support shared understanding, situational awareness, and empathy, rather than serving only to portray a robot’s inner affect.The paper notes that some people perceive robots that appear to pretend to have emotions as uncanny.Source: §4.1.2 · p. 15
- The paper proposes exploring facial expressions as social responses to situations, such as visibly signaling a task failure, without implying an inner emotional state.Source: §4.1.2 · p. 15
4.1.3 Eye Behavior (Oculesics)
Eye behavior can signal awareness, willingness to engage, approach, or avoidance, though its interpretation depends on context. In HRI, gaze can shape interaction, but findings vary and pupil dilation remains underexplored.
4.1.3 Eye Behavior (Oculesics)
- Mutual gaze can signal awareness and willingness to engage, while direct and averted gaze may be associated with approach and avoidance, respectively.The passage notes that direct gaze may be perceived as either threatening or friendly depending on other cues.Source: §4.1.3 · p. 15
- In HRI, gaze has been associated with favoring one group member and faster handovers, while another study found no effect from gaze paired with verbal messages and gestures.Source: §4.1.3 · p. 16
- Pupillary dilation may offer cues about interest or focus, but it has received little research attention in HRI.The paper also suggests studying more complex situations and subtle gaze switching.Source: §4.1.3 · p. 16
4.2 Auditory Nonverbal Codes
Auditory nonverbal codes include vocal qualities, non-speech sounds, silence, and conversational timing. The paper describes opportunities for robots to use paralanguage and chronemics, while identifying open questions about how people interpret these cues.
4.2.1 Paralanguage
- Paralanguage includes vocal qualities, non-word vocalizations, fillers, pauses, and silence, which can convey affect or carry positive and negative meanings.The passage identifies pitch, intensity, tone, and contour among voice parameters that indicate emotion.Source: §4.2.1 · p. 16
- Robots have used non-linguistic vocal cues, and such cues can avoid the need for natural language processing; backchanneling and laughter remain opportunities for further HRI work.The passage notes that laughter is little explored in robot design despite related work on detection and engagement.Source: §4.2.1 · pp. 16–17
4.2.2 Chronemics
- Chronemics treats conversational timing—including turn-taking, speech time, and interaction openings and closings—as nonverbal communication linked to power.The paper raises questions about robot time orientation, speaking pace, processing pauses, and punctuality across cultures.Source: §4.2.2 · p. 17
- The paper suggests adapting interruption-response taxonomies for HRI, including procedures in which people select appropriate nonverbal responses to interruptions.Source: §4.2.2 · pp. 17–18
4.3 Haptic Nonverbal Codes
Haptic communication uses touch, which can meet psychological needs but is governed by culturally variable social rules. HRI examples and proposals therefore raise both relational opportunities and ethical questions about robotic touch.
4.3 Haptic Nonverbal Codes
- Touch can meet psychological needs and provide comfort, positive feelings, and stress reduction, while touch initiation can direct others’ behavior.People perceived as higher status may initiate touch.Source: §4.3 · p. 18
- Human touch follows culturally variable rules, and violations can carry social or legal consequences, making realistic robotic touch ethically consequential.A robot initiating touch may be perceived as intruding into personal space or as an aggressor, depending on context and expectations.Source: §4.3 · p. 18
- HRI work has explored robots that can be held and stroked, surface warmth, and goosebumps; touch-responsive robots may support therapy or relationship-building.The authors suggest that touch feedback could support autistic people and that haptic feedback when a person touches a robot could foster intimacy.Source: §4.3 · p. 18
4.4 Olfactory Nonverbal Codes
Olfactory cues may influence social interaction, but scent has been little studied in HRI. The authors identify existing sensing and display work as a basis for further research on robot scent.
4.4 Olfactory Nonverbal Codes
- Human odors can signal emotional states, and odors associated with disgust or fear produced matching facial expressions in exposed participants.Source: §4.4 · p. 19
- HRI scent work is virtually nonexistent, though prior work includes robot scent detection for teleoperation and a robot-and-olfactory-display theatre experience.Source: §4.4 · p. 19
- Proposed HRI research includes testing scents by robot form, effects on approach distance or personal boundaries, links with personality and humanlikeness, and scent-based personalization.Source: §4.4 · p. 19
4.5 Gustatory Nonverbal Codes
Taste is included to complete the sensory channels considered for HRI, although it is unlikely to matter in most contexts. Feeding robots are a possible application, with taste displays offering a way to alter food taste.
4.5 Gustatory Nonverbal Codes
- Five taste modalities are distinguished: sweetness, sourness, saltiness, bitterness, and savouriness, also called umami.Source: §4.5 · p. 19
- Taste is unlikely to be relevant in most HRI contexts, but feeding robots for older adults or patients could stimulate gustatory senses.Source: §4.5 · p. 19
- Taste displays using gels can reproduce five basic modalities; integrating them into feeding robots could adjust food taste to preferences and help stimulate appetite and nutrition.Source: §4.5 · p. 19
5 CONCLUSION
The paper maps communication-studies nonverbal codes across five human sensory systems in HRI and offers research directions. Its coverage is a sampling, shaped by terminology and categorization challenges in the HRI literature.
5 CONCLUSION
- The overview maps communication-studies nonverbal codes onto vision, auditory perception, haptics, olfactory, and gustatory perception in HRI.Source: §5 · p. 19
- Body movement and gestures have been notably explored in HRI, while haptics, olfactics, and chronemics are identified as promising areas for further research.Source: §5 · p. 20
- Because HRI papers often omit communication-studies terminology and broad terms resist categorization, the authors provide a sampling based on keywords and literature familiar to them.They describe the work as a mapping of how a communication-studies framework can and does apply to HRI.Source: §5 · p. 20