PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Socially Expressive Mixed Reality Arms for Enhancing Low-Expressivity Robots

Thomas R. Groechel, Zhonghao Shi, Roxanna Pakkar, Maja J. Matarić

arXiv:1911.09713v2cs.ROcs.HC

TL;DR

Many robots have limited physical expressivity, motivating this study of mixed-reality arms as a way to add social gestures. In a user study, the arms were associated with higher perceived emotion, helpfulness, and physical presence, and participants distinguished gesture valence and intent.

  • Problem

    Many robots have limited physical expressivity because of physical, cost, and safety constraints, despite the importance of nonverbal communication.

    Source: §I · p. 1

  • Method

    The authors added mixed-reality social arms to a low-expressivity robot and evaluated them in a mathematics task and a gesture-rating study.

    Source: §I · p. 1, §III · p. 2, D. Measures and Analysis · p. 5

  • Results

    Users with the arms reported greater perceived robot emotion, helpfulness, and physical presence, and participants distinguished gesture valence and intended meaning.

    Source: §I · pp. 1–2, §VI · p. 7

  • Takeaways & Limitations

    Mixed reality has the potential to increase a robot’s expressive range and physical presence.

    Source: §VIII · p. 7

    The study used a limited number of gestures and measures and did not assess gesture arousal.

    Source: §VII · p. 7

Abstract

from arXiv · show

Expressivity--the use of multiple modalities to convey internal state and intent of a robot--is critical for interaction. Yet, due to cost, safety, and other constraints, many robots lack high degrees of physical expressivity. This paper explores using mixed reality to enhance a robot with limited expressivity by adding virtual arms that extend the robot's expressiveness. The arms, capable of a range of non-physically-constrained gestures, were evaluated in a between-subject study ($n=34$) where participants engaged in a mixed reality mathematics task with a socially assistive robot. The study results indicate that the virtual arms added a higher degree of perceived emotion, helpfulness, and physical presence to the robot. Users who reported a higher perceived physical presence also found the robot to have a higher degree of social presence, ease of use, usefulness, and had a positive attitude toward using the robot with mixed reality. The results also demonstrate the users' ability to distinguish the virtual gestures' valence and intent.

I. INTRODUCTION

Physical constraints often leave socially assistive robots with limited ways to communicate internal states and intentions. This work adds mixed-reality arms to a low-expressivity robot and reports user-perceived benefits and gesture recognition results.

I. INTRODUCTION

  • Physical, cost, and safety constraints limit many robots’ expressivity, despite its role in nonverbal communication and its links to trust, disclosure, and companionship.Robots without arms or other expressive features may have especially limited social expressivity.Source: §I · p. 1
  • Mixed-reality arms extend a low-expressivity robot’s social gestures while combining physical-robot embodiment with virtual modalities that avoid physical constraints.The paper presents the design, implementation, and validation of the arms in a mathematics task with a robot.Source: §I · p. 1
  • Users experiencing the arms reported greater perceived robot emotion, helpfulness, and physical presence than users without them.Users reporting greater physical presence also reported higher social presence, ease of use, usefulness, and a more positive attitude toward using the robot with mixed reality.Source: §I · pp. 1–2
  • Participants gave consistent ratings of gesture valence and identified the intended meanings of the virtual gestures.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

Prior AR-HRI work has emphasized functional signaling, while research on social expressivity has examined how robots convey affect and social behavior. This section situates mixed-reality gestures at the intersection of those areas.

A. Augmented Reality Human-Robot Interaction

  • AR-HRI systems can use projectors, tablets, or head-mounted displays, each trading off hands-free interaction, direct input, and screen restrictions.Source: A. Augmented Reality Human-Robot… · p. 2
  • Prior AR-HRI work has focused largely on functional signaling and teleoperation, with comparatively little work on social expression.Examples of social overlays include expressive faces on a Roomba and an avatar on a TurtleBot.Source: A. Augmented Reality Human-Robot… · p. 2

B. Social Expressivity and Presence in HRI

III. ROBOT GESTURE DESIGN AND IMPLEMENTATION

The authors chose the low-expressivity Kuri robot as a platform for mixed-reality arms and developed a balanced set of animated gestures.

III. ROBOT GESTURE DESIGN AND IMPLEMENTATION

  • Kuri was selected because its lack of arms makes it a suitable platform for studying mixed-reality gestures on a low-expressivity robot.The mobile robot is 50 cm tall and has 8 DoF, including three in its base and one in each eyelid.Source: §III · p. 2

A. Implementation

  • The implementation used a HoloLens, Unity3D, and the ROS# library to connect the mixed-reality extension with Kuri.Source: A. Implementation · p. 2
  • The six animations form a balanced set of three positive gestures and three negative gestures, built with interpolated keyframes and a simple inverse-kinematic solver.The positive set includes cheer, clap, and wave-like dance; the negative set includes facepalm, shoulder shrug, and crossed arms.Source: A. Implementation · p. 2

B. Gesture Design

The gesture design draws on social collaboration and animation principles, while the experiment evaluates both the arms and participants’ interpretations of the gestures.

B. Gesture Design

  • Gesture concepts drew on collaborative and emblematic communication, such as pointing to indicate desire and clapping to show approval.Source: B. Gesture Design · p. 2
  • The designs applied squash and stretch, exaggeration, and staging to address rigid physical forms and support social communication and joint attention.Source: B. Gesture Design · p. 3
  • Pilot feedback led the authors to pair each virtual gesture with a matching built-in physical expression to make the arms appear more integrated with Kuri.Positive gestures were paired with Kuri’s happy movement and tone; negative gestures with its sad movement and silence.Source: B. Gesture Design · p. 3

IV. EXPERIMENT DESIGN

  • The between-subjects study compared Kuri with and without arms during a mixed-reality math-puzzle task, measuring subjective perceptions and task completion time.Participants also completed a survey and semi-structured interview.Source: §IV · p. 3
  • In a second experiment part, all participants rated each of six gestures’ valence and described what it conveyed.Ratings ran from very negative (-1.00) to very positive (+1.00).Source: §IV · p. 3

A. Part 1: Mixed Reality Mathematics Puzzles

Participants used mixed reality to solve math puzzles with physical blocks and virtual clues while interacting with Kuri. In the arms condition, Kuri’s virtual gestures accompanied its physical responses.

A. Part 1: Mixed Reality Mathematics Puzzles

B. Ensuring Gesture Presentation Consistency

The experiment balanced gesture exposure across participants, including ensuring that everyone saw negative gestures. Participants then rated and described the gestures after viewing them.

B. Ensuring Gesture Presentation Consistency

C. Part 2: Gesture Annotation

D. Measures and Analysis

The study combined task performance, questionnaires, teammate-presence ratings, and interviews to characterize the conditions. Gesture ratings were also assessed for valence and inter-rater reliability.

D. Measures and Analysis

  • Task efficiency was puzzle completion time, with timed-out participants also recorded; four 7-point scales measured social presence, technology attitude, ease of use, and usefulness.Source: D. Measures and Analysis · p. 4
  • Participants rated Kuri from physical to virtual teammate presence on a 0.00–1.00 scale, to locate it on the Milgram virtuality continuum.Source: D. Measures and Analysis · p. 4
  • Interviews were coded for perceived emotion and helpfulness, and experiment participants were additionally asked whether the arms seemed part of Kuri.Source: D. Measures and Analysis · p. 4
  • Gesture valence ratings used a −1.00 to +1.00 slider, while open responses captured perceived meaning; a two-way random-effects ICC assessed single-rater agreement across six gestures.Source: D. Measures and Analysis · pp. 4–5

V. RESULTS

In the between-condition results, participants with arms were more likely to rate Kuri as a physical teammate. The study found no significant condition differences in task efficiency or the subjective measures.

A. Participants

  • The 34 participants were University of Southern California students aged 18–28, randomly assigned to control (17) or experiment (17).Source: A. Participants · p. 5

B. Arms Vs. No Arms Condition

C. Physical Vs. Virtual Teammate Bins

The physical-versus-virtual teammate groups differed on social presence, attitude toward technology, and ease of use, while gesture ratings distinguished valence with moderate-to-good reliability.

C. Physical Vs. Virtual Teammate Bins

  • Physical-teammate ratings were associated with higher social presence, attitude toward technology, and ease of use than virtual-teammate ratings; usefulness increased only marginally.The respective medians were 4.8 vs. 3.9, 6.7 vs. 5.7, and 6.0 vs. 5.4; usefulness medians were 5.3 vs. 5.0.Source: C. Physical Vs. Virtual Teammate Bins · p. 5, Figure 7 · p. 6

D. Gesture Validation

  • Participants distinguished gesture valence, with agreement reliability classified as moderate to good (ICC(A, 1) = 0.77; 95% CI 0.55–0.95).The ratings differed significantly across gestures (F(5, 190) = 125, p < 0.001).Source: D. Gesture Validation · p. 6
  • Fig. 6 shows ratings clustered on either side of the 0.5 rating.Source: Figure 6 · p. 6

VI. DISCUSSION

Arms did not significantly change task efficiency or subjective measures, while physical-teammate perception was associated with more favorable subjective ratings; gesture ratings generally distinguished valence, with some difficulty for negative gestures.

VI. DISCUSSION

  • The arms-versus-no-arms conditions showed no significant task-efficiency or subjective-measure effects, although participants with arms were more likely to rate the robot as physical.The discussion suggests arms or greater movement may have contributed to that perception, but does not establish either as its cause.Source: §VI · p. 6, Figure 8 · p. 6
  • Physical-teammate ratings aligned with more favorable subjective results, while the authors note that factors beyond the arms may have influenced physical presence.The discussion connects this pattern to prior evidence on embodiment and social presence.Source: §VI · pp. 6–7
  • The flexible mixed-reality arm interface could support adding and adapting gestures for other scenarios, while future work may investigate additional influences on physical presence.Source: §VI · p. 7
  • Gesture ratings distinguished valence with high agreement, but participants had more difficulty rating negative gestures; two participants rated all gestures positively.Those six erroneous marks for the three negative gestures were included in the reported statistics, and may indicate confusion about the rating scale.Source: §VI · p. 7, Figure 9 · p. 7, Table III · p. 7

VII. LIMITATIONS AND FUTURE WORK

The study’s limitations include headset visibility, a small gesture set without arousal ratings, and reports that Kuri’s interactions felt limited. The conclusion proposes combining mixed-reality expressivity with real-time user data in future work.

VII. LIMITATIONS AND FUTURE WORK

  • The first-generation HoloLens’s limited field of view made the virtual arms difficult to see.The authors suggest hand tracking and eye tracking as possible future additions for interaction and autonomous responses.Source: §VII · p. 7
  • Gesture evaluation covered a limited set and measures, and did not ask participants to annotate arousal.The authors suggest arousal or a higher-feature-scale measure could provide further insight into perceived gesture meaning and differences from physical gestures.Source: §VII · p. 7
  • Many participants reported limited interactions, and some perceived Kuri as a referee rather than a peer or teammate.Source: §VII · p. 7

VIII. CONCLUSION

  • The conclusion proposes using real-time data, such as eye gaze, to estimate engagement and inform mixed-reality robot responses.The stated aim is more fluid, expressive, and effective human-robot interaction.Source: §VIII · p. 7
Loading 1911.09713v2…