Multimodal Grounding for Embodied AI via Augmented Reality Headsets for Natural Language Driven Task Planning

1. Introduction

• Industrial robot teams require interfaces that let operators supervise robots while maintaining situational awareness in unstructured environments.Mixed Reality can overlay heat maps, structural weak spots, and team-member locations, including in low visibility.Source: §1 · p. 1
• Speech and gesture offer naturalistic multimodal communication, but their combined intent must be robustly grounded into executable tasks.The paper notes these modalities can reduce ambiguity compared with unimodal HRI interaction.Source: §1 · p. 1
• The authors adapt Embodied AI to AR headsets by injecting headset-derived visual and language information into GPT-3 prompts for co-located human-robot teaming.They describe the industrial-domain and AR-headset demonstration as novel to their knowledge.Source: §1 · p. 2

2. Related Work

• Prior Embodied AI deployments largely use hard prompts together with object detectors to predict an agent’s next action.The paper summarizes prior work in Table 1.Source: §2 · p. 2
• Semantically equivalent prompts can produce performance ranging from pure chance to state of the art, and prompt design also faces preference, sample-selection, and ordering biases.Discrete prompt design can require extensive human engineering.Source: §2 · p. 3
• Multimodal fusion either injects image-to-text information into language prompts or synthesizes visual information downstream; this work instead uses human-generated virtual-reality markers.The cited approaches are [7] for prompt injection and for downstream synthesis.Source: §2 · p. 3

3.1. Large Language Models

• Autoregressive language models maximize next-token log-likelihood conditioned on prior tokens, and self-supervised training includes denoising objectives such as next-token prediction.The objective optimizes parameters θ for observations y conditioned on input x and latent features h.Source: §3.1 · p. 3

3.2. Prompt Learning

• Tuning-free prompt learning searches for a prompting function while keeping the language model’s parameters frozen.The background attributes its use to in-context learning and restricted access to model features and gradients.Source: §3.2 · pp. 3–4
• The prompt search scores two-example permutations of multimodal UMRF decoding examples by validation BLEU accuracy and selects the highest-scoring permutation.The search axes include example ordering, selection, and number, subject to API token and rate limits.Source: §3.2 · p. 4

3.3. Unified Meaning Representation Format

• UMRF is a platform-independent JSON-based task format that decouples robot autonomy from its command interface.Its action graphs support parent-child relations, sequential, concurrent, and cyclical structure, and parameterized inputs and outputs.Source: §3.3 · p. 4

4. UMRF Prompt Design

• The discrete prompt design combines relevant world context and objects with few-shot examples, as in InnerMonologue.The approach is described as similar to InnerMonologue.Source: §4 · p. 4
• The authors incorporate chain-of-thought prompting within in-context examples, drawing similarity to ProgPrompt.Source: §4 · p. 4
• The design uses UMRF rather than Python-like code generation and provides language and visual feedback through the AR headset.Source: §4 · p. 4

5.1. Experiment 1. Greedy & Exhaustive Search

• Figure 2 depicts arbitrary commands parsed into UMRF graphs whose nodes map to executable actions; exhaustive search tested 10P2 prompts on five validation examples, taking roughly 11 hours for over 450 queries.Average OpenAI query time was 1.5 minutes.Source: §5.1 · p. 4, Figure 2 · p. 5
• Top prompts commonly combined example types 1, 4, and 5, and example 5 performed best first in all but one case.The authors suggest its length and information may aid generalization first, but confuse decoding when placed later.Source: §5.1 · pp. 5–6
• Prompt 70 (examples 4+5) averaged BLEU 0.662, while the highest-scoring prompt (examples 1+4) reached BLEU 0.850.The average and maximum scores belong to different prompt designs.Source: §5.1 · p. 6
• The top-performing prompts still generalized poorly to validation, demonstrating prompt non-robustness even on a simple UMRF decomposition task.Figure 4 summarizes the variation among the top ten prompt designs.Source: §5.1 · p. 5, Figure 4 · p. 7

5.2. Experiment 2. Assessing Prompt Fragility

• The Text AutoAugment adaptation was unsuccessful because reward signals were weak and performance was highly sensitive to heuristic augmentations.It searched compositional augmentations using Bayesian optimization to maximize validation BLEU, with a surrogate model intended to make LLM-output exploration cheaper.Source: §5.2 · pp. 6–7
• Random deletions and insertions left GPT-3 relatively robust, while random word swaps and synonym replacements had larger performance effects.Figure 5 presents sensitivity to EDA perturbations at varied magnitudes.Source: §5.2 · p. 7, Figure 5 · p. 7
• Compositional prompt augmentations widened performance variation, leading the authors to recommend constraining Text AutoAugment magnitude below 0.1.Figure 6 shows the compositional-augmentation sensitivity analysis.Source: §5.2 · p. 8, Figure 6 · p. 8
• In the low-data regime, prompt similarity to UMRF examples in GPT-3 pretraining did not correlate with prompt performance.Figure 7 compares 90 greedily generated prompts with UMRF examples in the pretraining dataset.Source: §5.2 · p. 8, Figure 7 · p. 9

6. Demonstration

• The remote-inspection setup combines voice commands with gesture-operated virtual markers to define target poses, while task feedback appears in the operator’s field of view.The robot is a Clearpath Husky with two Universal Robots UR5s and an end-effector camera; the operator uses a HoloLens 2.Source: §6 · p. 9
• The HoloLens combines voice and marker coordinates into a command string; a command-server UMRF parser prompts text-davinci-003, and the returned JSON graph is sent to the robot.The prompt contains five operator-command and UMRF-graph examples; ROS and RoboFleet distribute data between components.Source: §6 · pp. 9–10
• Figure 8 compares prompt semantic similarity to GPT-3 pretraining UMRF examples for an unaugmented policy and one using synonym replacement.The caption describes the policies but does not state which has greater similarity.Source: Figure 8 · p. 10

7. Discussion & Future Work

• The authors report successful industrial inspection demonstrations, but identify prompt fragility and nonrobustness as vulnerabilities for physical systems.In-context examples are not token-space efficient, and developing discrete prompts requires extensive human engineering.Source: §7 · p. 10
• OpenAI API request and rate limits commonly caused no responses or incomplete parses in the demonstrated system.The authors also note technical challenges from relying on a third party to serve the language model.Source: §7 · p. 10
• Future studies are proposed on operator agency and quality of life, marker-versus-detector performance, text-to-speech, and multi-step prompting.The authors suggest first querying a compact task-graph representation, then querying the LLM to fill in node information.Source: §7 · pp. 10–11
• The authors call for comparing task-planning representations by their pretraining frequency and measuring prompt-perturbation magnitude against validation BLEU scores.They speculate that more representative formalisms may enable information-retrieval approaches to prompt robustness.Source: §7 · p. 11

Funding

• Support included the European Social Fund via IT Academy, EXCITE, and AI & Robotics Estonia, co-funded by the EU and Estonia’s Ministry of Economic Affairs and Communications.The paper also names the European Regional Development Fund as funding EXCITE.Source: Funding · p. 12
• The authors also acknowledge support from Los Alamos National Laboratory and the Center for Nonlinear Studies.Source: Funding · p. 12

Notes on contributors

• Selma Wanna’s research interests include robustness of deep-learning systems operating in out-of-distribution domains.She holds B.Sc. and M.Sc. degrees in electrical and mechanical engineering from the University of Texas at Austin, respectively.Source: Notes on contributors · p. 12
• Fabian Parra’s interests include mobile manipulators, supervised autonomy, and human-machine interfaces.He earned a bachelor’s degree in mechatronic engineering and an M.Sc. in Robotics and Computer Engineering.Source: Notes on contributors · p. 12
• Robert Valner’s interests include fault-tolerant and adaptive robotic architectures, multi-robot systems, human-robot interaction, and autonomous systems.He is a junior researcher and Ph.D. candidate at the University of Tartu.Source: Notes on contributors · p. 13
• Karl Kruusamäe’s research interests include human-robot interaction and shared autonomy.He is an associate professor of robotics engineering at the University of Tartu.Source: Notes on contributors · p. 13
• Mitch Pryor’s work includes robotics for hazardous, uncertain environments, including manufacturing and material handling.He co-founded the Nuclear Robotics Group and the Drilling & Rig Automation Group.Source: Notes on contributors · p. 13

• Figure A1 shows a lack of prompt robustness.Source: Figure A1 · p. 17
• Figure A2 shows a lack of prompt robustness toward the chosen training example.Source: Figure A2 · p. 17
• Table A2 lists the structures of the ten best-generalizing prompts, with V/L indicating whether visual information or the natural-language command came first.Source: Table A2 · p. 17
• Figure A3 shows a lack of prompt robustness toward whether visual or natural-language command cues come first.Source: Figure A3 · p. 18