A Comparison of Visualisation Methods for Disambiguating Verbal Requests in Human-Robot Interaction

1 INTRODUCTION

• The study addresses the still-unknown effects of workspace visualisation for disambiguating verbal requests, alongside approaches using follow-up questions or multimodal cues.Prior work found that verbal interactions increase task time and can influence accuracy.Source: §1 · p. 1
• The system highlights candidate objects and updates the selection as users refine ambiguous verbal requests, comparing projector-based augmented reality, HoloLens mixed reality, and a side monitor.The monitor serves as the baseline condition.Source: §1 · p. 2
• In the YuMi experiment, participants used block shape and colour to verbally identify targets while viewing the robot’s real-time hypothesis.The setup intentionally included blocks that produced ambiguous requests.Source: §1 · p. 2
• Task time did not differ significantly across conditions; mixed reality reduced accuracy but was most engaging, while augmented reality was most preferred.Participants also considered augmented reality more observable, less disruptive, natural, and easy to understand.Source: §1 · p. 2

2 RELATED WORK

• Object-retrieval language is mapped to semantic representations of physical space; language grounding means establishing common ground so both parties refer to the same object or concept.Earlier work explored simple machine-understood object references and multimodal features.Source: §2 · p. 2
• Multimodal studies use gaze and pointing gestures to disambiguate verbal references to objects in shared human-robot spaces.Other approaches include incremental reference resolution, spatial references, uncertainty modelling, and past visual observations.Source: §2 · p. 2
• Follow-up questions can gather information but may lengthen tasks and risk speech-to-text failures; Whitney et al. used a POMDP to balance these trade-offs.That approach may be infeasible with many objects, motivating visualisation of the robot’s current understanding.Source: §2 · p. 2
• Projector-based augmented reality has been used to display robot intentions, with prior studies reporting improved user effectiveness and satisfaction or increased trajectory predictability and transparency.These applications visualised robot activity in car assembly and future motion, rather than the current candidate objects in a verbal retrieval request.Source: §2 · p. 2
• For robot-arm collision detection, a prior study found mixed reality faster, more accurate, and more enjoyable than its comparison interfaces [19].The result concerns that specific task, not verbal-request disambiguation.Source: §2 · p. 3

3 HUMAN-HUMAN PILOT STUDY

• Ten participants in five pairs took turns requesting and fetching Lego blocks to build a model, using verbal instructions without pointing.The pilot was designed to inform a reference-disambiguation system for collaborative object retrieval.Source: §3 · p. 3
• Participants most often used colour and shape to describe blocks; speech recognition produced many incorrect descriptors, so a human wizard controlled language understanding.The researchers suggested non-native English speech may have contributed to transcription errors.Source: §3 · p. 3

4 SYSTEM DESCRIPTION

• For ambiguous requests, the system highlights candidate objects and continues through verbal refinements until the target is unambiguous and ready for pickup.The process is summarised in Algorithm 1, titled “Visualisation of object highlighting extracted by object descriptors from human instructions.”Source: §4 · p. 3
• A human wizard extracts supported colour and shape descriptors from each verbal request; this is the system’s only wizarded component.The pilot found these were the most common descriptors.Source: §4 · p. 3
• The object database stores object attributes and 3D poses, with attributes updated by a Kinect-based vision system using colour segmentation and depth information.The database query returns objects matching the supplied descriptors.Source: §4 · p. 3
• When descriptors leave multiple candidates, the system highlights them and can display numbers; when one matching object remains, its coordinates and pickup command go to the controller.The controller plans and executes the pickup, selecting an arm based on target location.Source: §4 · p. 3
• The evaluated visualisation interfaces were a side monitor, augmented reality, and mixed reality.Figure 4 presents the three methods evaluated.Source: §4 · p. 3, Figure 4 · p. 4

5 EVALUATION

• Three conditions presented candidate-object highlights through a workspace monitor, a projector, or a mixed-reality head-mounted display.The mixed-reality workspace was calibrated with a fiducial marker and continuously tracked using the device’s spatial mapping.Source: §5 · p. 4

5.1 Hypotheses

• The hypotheses predicted slower and less accurate performance with the monitor, greater engagement with mixed reality, and lower disruption and preference for augmented reality.The authors expected monitor-based spatial mapping and attention shifts to increase cognitive load and errors.Source: §5.1 · p. 4

5.2 Participants

• Twenty-nine participants, aged 22–50, took part; eligibility required English fluency, no colour vision deficiency, and not wearing glasses.Glasses were excluded because of difficulties wearing the mixed-reality device.Source: §5.2 · p. 4

5.3 Procedure

• Participants requested Lego blocks using colour and shape descriptors in 15-turn trials, with eight participant requests and seven robot requests per trial.Condition order was counterbalanced with a Latin square, and block arrangements were randomised between sessions.Source: §5.3 · pp. 4–5

5.4 Measurements

• Request time excluded robot action completion and the first two human requests per trial; error rate counted incorrect executions per trial, normalised by turns.Participants also rated engagement, observability, and display interference on seven-point scales and reported modality preferences.Source: §5.4 · p. 5

• The reported results cover objective measures collected in the experiment.Source: §6 · p. 5
• The section also reports subjective measures collected in the experiment.Source: §6 · p. 5
• Both classes of measures are presented as experimental results.Source: §6 · p. 5

6.1 Objective Measures

• Request duration showed no significant condition effect, F(2, 14) = 1.19, p = .33, η2 = .15.The duration figure presents averages by trial number and condition.Source: §6.1.1 · p. 5, Figure 5 · p. 5
• Request duration was higher in trial 1 (M = 20.61, SE = 1.48) than trial 2 (M = 16.84, SE = .53) and trial 3 (M = 14.36, SE = .44), with a significant order effect.The condition-by-trial interaction was not significant; post hoc tests found differences between trials 1 and 3 and trials 2 and 3, but not trials 1 and 2.Source: §6.1.1 · p. 5

6.1.1 Request Time.

• Error rates averaged .01 in AR, .02 in MO, and .04 in MR; the reported condition effect was significant, F(2, 56) = 3.22, p < .05, η2 = .10.The passage reports AR as significantly lower than MR in post hoc tests, but gives p = 1.0 for that comparison; no significant differences were found for MO comparisons.Source: §6.1.2 · p. 5

6.2.1 Engagement.

• Engagement ratings were higher for MR (M = 4.89, SE = .32) than MO (M = 4.25, SE = .27) and AR (M = 4.25, SE = .26).The condition effect was significant, F(2, 54) = 4.93, p < .05, η2 = .15; post hoc comparisons found MR higher than both other conditions.Source: §6.2.1 · p. 6, Figure 6 · p. 6

6.2.2 Observability.

• AR provided the highest perceived observability (M = 6.48, SE = .14), followed by MO (M = 6.07, SE = .19) and MR (M = 5.62, SE = .24).AR was rated significantly above both MO and MR; MO and MR did not differ significantly.Source: §6.2.2 · p. 6

6.2.3 Display interference.

• Display interference was lowest in AR (M = 1.86, SE = .29), then MO (M = 3.45, SE = .40), and highest in MR (M = 4.03, SE = .36).The AR differences from MO and MR were significant; MO and MR did not differ significantly.Source: §6.2.3 · p. 6

6.2.4 Overall Preferences.

• Participants most preferred AR, found it significantly easier than the other conditions, and most often chose it for future work with the robot.The reported tests were χ2 = 36.69, p < 0.001 for preference, χ2 = 18.29, p < 0.001 for ease, and χ2 = 27.79, p < 0.001 for future choice.Source: §6.2.4 · p. 6

7 DISCUSSION

• No condition changed request time; first-trial requests took longer, likely because participants initially used unsupported spatial descriptions.Source: §7 · p. 6
• MR produced more mistakes than MO despite AR having the smallest error rate; authors attribute MR errors mainly to participants losing workspace perspective through limited field of view.Source: §7 · p. 6
• MR was most engaging despite higher errors, though novelty may explain some engagement and freedom to move was another possible contributor.Source: §7 · p. 6
• AR was least disruptive and most preferred; participants linked disruption to MR wearables and MO attention-switching, and described AR as natural and easy.Source: §7 · pp. 6–7
• The experiment compared visualizations against a monitor, not pointing or follow-up questions; all trials required disambiguation because some objects shared shape and color.Source: §7 · p. 7

7.1 Limitations

• Trials used similar ambiguity, a flat surface, and mostly unoccluded viewpoints, so transfer to harder, complex scenes remains untested; MR 3D projections may matter more there.Source: §7.1 · p. 7
• Participants had no MR-device practice before trials; excluding the first two turns did not resolve whether prior interface experience affected performance, which larger samples could clarify.Source: §7.1 · p. 7

7.2 Design implications

• Choice should depend on task and setting: MR may suit focus-heavy, occluded or non-flat work, but its weight and field of view and fixed projector or monitor installations constrain long tasks and mobile use.Source: §7.2 · p. 7

• The controlled comparison tested MR, AR, and monitor visualizations for robot-considered objects, measuring request time, error rate, engagement, observability, interference, and preferences.Source: §8 · p. 7
• Task time did not differ significantly by condition; MR increased errors but engaged participants more, while most preferred AR as easiest and less intrusive in this setting.Source: §8 · p. 7
• Future work will examine mixed reality for irregular surfaces and occlusion, and combine visualizations with cues such as pointing and gaze direction.Source: §8 · p. 8