Source-linked AI summary
Communicating Robot Arm Motion Intent Through Mixed Reality Head-mounted Displays
Eric Rosen, David Whitney, Elizabeth Phillips, Gary Chien, James Tompkin, George Konidaris, Stefanie Tellex
TL;DR
Robot motion intent can be difficult to communicate to humans, and MR HMD overlays offer a way to present planned motion in the real-world view. In a comparison with a 2D display, the MR system produced higher accuracy and faster judgments.
Problem
Communicating intended robot motion efficiently is difficult, although people need to predict robot actions for safe, collaborative work.
Source: §1 · p. 2Method
The system overlays 3D graphics of planned robot-arm motion on the real workspace through a mixed-reality HMD.
Source: §1 · p. 2Results
16% higher accuracy and 62% less task time than the next-best 2D display were reported for the MR condition.
Source: §1 · p. 2, §4 · p. 13Takeaways & Limitations
The findings provide evidence that mixed reality is one way to bridge the communication gap and let robots communicate motion intent to humans.
Source: §5 · p. 14The study addresses robot-to-human visual communication of motion intent, not human-to-robot communication or non-visual and non-motion states.
Source: §4.1 · p. 14
Abstract
from arXiv · showhide
Efficient motion intent communication is necessary for safe and collaborative work environments with collocated humans and robots. Humans efficiently communicate their motion intent to other humans through gestures, gaze, and social cues. However, robots often have difficulty efficiently communicating their motion intent to humans via these methods. Many existing methods for robot motion intent communication rely on 2D displays, which require the human to continually pause their work and check a visualization. We propose a mixed reality head-mounted display visualization of the proposed robot motion over the wearer's real-world view of the robot and its environment. To evaluate the effectiveness of this system against a 2D display visualization and against no visualization, we asked 32 participants to labeled different robot arm motions as either colliding or non-colliding with blocks on a table. We found a 16% increase in accuracy with a 62% decrease in the time it took to complete the task compared to the next best system. This demonstrates that a mixed-reality HMD allows a human to more quickly and accurately tell where the robot is going to move than the compared baselines.
1 Introduction
Robots’ intended motions are difficult for people to predict, while existing communication approaches can divert attention or make 3D paths hard to interpret. The paper tests a mixed-reality HMD visualization against a 2D display and no visualization, finding faster and more accurate judgments with MR.
1 Introduction
- Robot motion is difficult for humans to predict, contributing to safety and efficiency issues and restricting collaboration, especially in fluid or autonomous work settings.Motion plans may be defined in advance, but communicating that intent efficiently remains difficult.Source: §1 · p. 2
- Humanoid gestures and social cues are not available to many robot forms, while 2D displays can divert attention from the workspace and make 3D plans harder to understand.A real-world 3D overlay is proposed as a way to avoid those limitations.Source: §1 · p. 2
- The proposed HoloLens system overlays 3D graphics of intended arm motion on the real workspace; the paper also introduces ROS Reality for importing robots into Unity.The visualization lets participants inspect the planned motion before the arm moves.Source: §1 · p. 2
- 16% higher accuracy, 11% higher precision, and 62% less completion time (7.4 seconds) than the next-best 2D display; usability and workload assessments also improved.The comparison was in a task classifying planned robot-arm motions as colliding or non-colliding with blocks.Source: §1 · p. 2
- Figure 1 contrasts HoloLens overlays, a desk-based 2D visualization, and direct observation of a moving arm with a stop button across 14 trials.The HoloLens shows blue virtual arm poses overlaid on the real world.Source: Figure 1 · p. 3
2 Related Work
Prior work explores robot intent communication through social cues, display indicators, augmented environments, and 3D visualization. These approaches have trade-offs, and VR comparisons offer mixed, task-dependent evidence about when head-tracked displays help.
2 Related Work
- Social cues and animated motion can improve legibility, but they do not transparently communicate complex paths; indicators likewise lack high-fidelity motion detail.Close-proximity teamwork may require detailed knowledge of robot actions before and during motion.Source: §2 · p. 4
2.1 Augmented and Mixed Reality for Human-robot Collaboration
- Augmented environments and tablet overlays can show 3D intent, but special-purpose setups have practical constraints, while tablets occupy the operator’s hands and mismatch eye and camera perspectives.Optical HMDs offer a real-world overlay from the human’s point of view, but robot-intent use remained nascent.Source: §2.1 · pp. 4–5
2.2 3D Spatial Reasoning in Virtual Reality Displays
- VR studies report both HMD advantages and desktop advantages, with prior comparisons conflicting and performance benefits dependent on the task.The literature offers no clear prescriptive guidelines for choosing a display technique.Source: §2.2 · p. 5
3 Experiment
The experiment compared three ways to judge whether robot-arm plans would collide with table blocks. Participants completed the same classification task using no visualization, a monitor, and a HoloLens overlay.
3 Experiment
- Thirty-two novice participants used all three interfaces to classify Baxter arm plans as colliding or non-colliding; measures included completion time, classification rates, usability, likability, and workload.The study used a within-subjects design.Source: §3 · p. 6
3.1 Task
- Each participant classified the same 14 randomized motions over a block-covered table, with blocks arranged to make a complete view from one perspective difficult.Participants could walk around the setup and responded with an Xbox controller as quickly and accurately as possible.Source: §3.1 · p. 6
3.2 Interfaces
- The monitor showed future robot poses and a Kinect point cloud with a movable virtual camera; MR overlaid a similar 3D plan on the real workspace, while the no-visualization condition showed the moving arm.In the MR and monitor conditions, the arm itself did not move during judgment.Source: §3.2 · pp. 6–7
3.3 Experimental Procedure
The procedure presented no visualization first, then alternated monitor and HoloLens order across participants. The study measured objective classification and timing outcomes alongside questionnaire assessments of workload, usability, predictability, and preference.
3.3 Experimental Procedure
- Participants completed the no-visualization condition first, then the monitor and MR conditions in alternating order, with questionnaires after each interface.For the initial condition, participants pressed stop only if they thought the moving arm would collide; in the later conditions they labeled the planned motion.Source: §3.3 · p. 7
3.4 Measurements
- Objective measures included true-positive and true-negative rates and participant labeling strategy; monitor and HoloLens trials also recorded time from viewing the plan to labeling it.The timing measure was used alongside accuracy and precision assessments.Source: §3.4 · p. 7
- Subjective measures included NASA-TLX workload, SUS usability, and ratings of perceived predictability and interface preference.NASA-TLX averaged six subscales without weighted paired comparisons; SUS used ten statements on a 7-point scale and a 0–100 score.Source: §3.4 · pp. 7–8
3.5 Hypotheses
The hypotheses predict that the HoloLens interface will outperform the monitor and no-visualization conditions across task performance and user assessments, and enable faster labeling than the monitor.
3.5 Hypotheses
- The expected overall ordering is HoloLens first, monitor second, and no visualization third for performance and subjective measures.The time prediction specifically compares HoloLens with the monitor.Source: §3.5 · p. 8
- H1 predicts the HoloLens will be easiest to use, with the best task performance, workload, usability, predictability, and preference outcomes.Performance predictions include more true positives and true negatives and fewer false positives and false negatives.Source: §3.5 · p. 8
- H2 predicts the monitor will outperform no visualization across performance, workload, usability, predictability, and preference measures.The predicted performance pattern is more true positives and true negatives and fewer false positives and false negatives.Source: §3.5 · p. 8
- H3 predicts faster motion-labeling times with HoloLens than with the monitor.The hypotheses note that no-visualization labeling time is based on watching the robot move, so it is not directly comparable with the monitor and HoloLens conditions.Source: §3.5 · p. 8
Analysis Techniques
The analysis uses repeated-measures ANOVA and signal detection theory to compare the three interfaces. HoloLens produced higher accuracy and strategy-adjusted sensitivity than either baseline, while monitor performance was not significantly better than no visualization.
Analysis Techniques
- Repeated-measures ANOVA and signal detection theory were used to test whether differences among the three conditions were significant at the 95% confidence level.Signal detection theory was used alongside ANOVA to account for response strategy.Source: Analysis Techniques · p. 9
- Signal detection theory accounts for response preferences; d′ is the standardized difference between hit and false-alarm rates and considers decision strategy.False alarms are treated as costly in this human-robot interaction task because they would slow collaboration.Source: Analysis Techniques · p. 9
Accuracy
- Accuracy was 0.76 with MR, 0.66 with the monitor, and 0.60 with no visualization; MR significantly outperformed both, while monitor versus no visualization was not significant.The reported test was Wilks Λ = 0.619, F(2,30) = 9.244, p = .001, η2 = 0.381.Source: Accuracy · pp. 9–10, Figure 2 · p. 10
- d′ was 1.79 with MR, 0.94 with the monitor, and 0.82 with no visualization; MR significantly outperformed both, while monitor versus no visualization was not significant.The condition difference was significant (Wilks Λ = 0.449, F(2,30) = 18.378, p < .001, η2 = 0.551); the monitor versus no-visualization comparison had p = .42.Source: Accuracy · pp. 9–10
Task Time
HoloLens users labeled robot motions faster than monitor users, with a statistically significant difference in mean labeling times.
Task Time
- The paired-samples t-test showed a significant difference in mean motion-labeling times between HoloLens and monitor conditions.The test result was t(31) = 3.415, p < .001.Source: Task Time · p. 10
- 11.95 seconds with HoloLens versus 19.39 seconds with the monitor; HoloLens labeling time was significantly shorter.The paired-samples test found a significant difference between conditions, t(31) = 3.415, p < .001.Source: Task Time · pp. 10–11
Subjective Workload
HoloLens had the lowest reported workload, significantly lower than the monitor condition but not no visualization. Usability scores were highest without visualization, though not significantly different from HoloLens.
Subjective Workload
- HoloLens had the lowest workload score (M = 35.39, SD = 2.78), significantly below the monitor score (M = 42.32, SD = 14.71), but not significantly different from no visualization.No visualization scored M = 37.11, SD = 2.61; monitor and no visualization also did not differ significantly.Source: Subjective Workload · pp. 11–12
- The monitor had the highest workload score of the three conditions, and its workload was not significantly different from no visualization.The workload ordering was HoloLens, no visualization, then monitor.Source: Subjective Workload · pp. 11–12
Subjective Usability
- No visualization had the highest SUS score (M = 38.91, SD = 1.15), followed by HoloLens (M = 37.88, SD = 1.26); their difference was not significant.Both no visualization and HoloLens had significantly higher SUS scores than the monitor (M = 28.31, SD = 0.99; both p < 0.001).Source: Subjective Usability · p. 12
Subjective Collision Predictability
Participants rated the HoloLens as the strongest aid for predicting collisions, and nearly all preferred it for understanding and completing the task.
Subjective Collision Predictability
- HoloLens predictability ratings were 5.28, versus 4.06 for no visualization and 3.38 for the monitor; HoloLens ratings significantly exceeded both.The monitor score was not significantly different from no visualization (p = .44).Source: Subjective Collision Predictability · p. 12
Subjective Enjoyment
- All 32 participants enjoyed the HoloLens most, 29 selected it for easiest understanding, and 30 preferred it for completing the task.No participants preferred no visualization; two preferred the monitor.Source: Subjective Enjoyment · p. 13
4 Discussion
The MR condition outperformed the monitor and no-visualization conditions on objective performance, while participant responses identified both usability benefits and system limitations.
4 Discussion
- MR increased collision-prediction accuracy by 16% and reduced task time by 62% compared with the monitor, while also outperforming no visualization.Participants almost universally preferred the HoloLens for enjoyment, ease, and assessing motion plans.Source: §4 · p. 13
- Participants cited cumbersome, unintuitive monitor controls, while 34% said moving around the robot made collision judgments faster, easier, and more intuitive.Some also reported overlay misalignment, lengthy setup, and difficulty moving around the robot.Source: §4 · p. 13
- MR workload was not significantly lower than no visualization, and no visualization was not significantly more usable than MR.Mean workload favored MR, and participants perceived its mental demand and ease as comparable to simply watching the robot.Source: §4 · p. 13
- The monitor did not significantly outperform no visualization on objective or subjective measures, and participants reported higher workload and lower usability for the monitor.Participants also agreed least that the monitor could help them predict collisions.Source: §4 · p. 14
4.1 Limitations
- The system addresses robot-to-human visual motion intent, leaving human-to-robot communication and non-visual or non-motion communication beyond its scope.The HoloLens also has a small graphical field of view and does not occlude virtual objects behind real objects.Source: §4.1 · p. 14
5 Conclusion
The authors conclude that MR can help robots communicate motion intent to humans. Their findings support MR over monitor visualization and no visualization in the tested setting.
5 Conclusion
- MR visualization was supported over the monitor and no-visualization conditions by participant performance and perceptions.The conclusion describes MR as one way to bridge the communication gap between robots and humans.Source: §5 · p. 14
- The paper frames effective mutual communication of motion intent as necessary for fluid cooperative work partnerships.Source: §5 · p. 14
- The authors present MR as one way for robots to communicate their motion intent to humans.Source: §5 · p. 14