PaperlayerView exact revision on arXiv

Source-linked AI summary

Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models

Ada Yi Zhao, Aditya Gunturu, Ellen Yi-Luen Do, Ryo Suzuki

arXiv:2508.03547v2cs.HC

TL;DR

LLM-based AR guidance often lacks rich visual augmentations that situate instructions in physical space. Guided Reality combines LLM planning, a manual-derived five-type visual strategy, and vision-based spatial embedding; evaluations with learners and instructors found the system usable and effective while identifying platform and generative-model limitations.

  • Problem

    Existing LLM-based AR guidance often lacks rich spatially embedded visual augmentations beyond basic highlights or limited motion guidance.

  • Method

    Guided Reality uses LLMs and vision models to generate task plans, select among five visual guidance types, localize interaction points, and embed guidance in physical space.

  • Results

    Evaluations with 16 participants and four instructors found Guided Reality usable and effective, with users especially praising movement and dynamic widget guidance.

  • Takeaways & Limitations

    Guided Reality supports just-in-time training and personalized learning through visually rich, spatially situated AR guidance.

  • Takeaways & Limitations

    The current mobile interface can disrupt task flow because users shift attention between the screen and physical workspace, while generative 3D models may lack functional orientation and interaction context.

Abstract

from arXiv · show

Large language models (LLMs) have enabled the automatic generation of step-by-step augmented reality (AR) instructions for a wide range of physical tasks. However, existing LLM-based AR guidance often lacks rich visual augmentations to effectively embed instructions into spatial context for a better user understanding. We present Guided Reality, a fully automated AR system that generates embedded and dynamic visual guidance based on step-by-step instructions. Our system integrates LLMs and vision models to: 1) generate multi-step instructions from user queries, 2) identify appropriate types of visual guidance, 3) extract spatial information about key interaction points in the real world, and 4) embed visual guidance in physical space to support task execution. Drawing from a corpus of user manuals, we define five categories of visual guidance and propose an identification strategy based on the current step. We evaluate the system through a user study (N=16), completing real-world tasks and exploring the system in the wild. Additionally, four instructors shared insights on how Guided Reality could be integrated into their training workflows.

1 INTRODUCTION

Guided Reality addresses the lack of rich visual augmentation in LLM-based AR guidance by selecting, generating, and spatially embedding visual aids for task instructions. It combines a manual-derived guidance strategy with a fully automated LLM-and-vision pipeline evaluated with learners and instructors.

  • Existing LLM-based AR guidance often provides only basic highlights or limited motion cues, despite evidence that richer augmentations can improve understanding and task performance.
  • Guided Reality generates visually rich, spatially grounded AR guidance that translates step-by-step instructions into animated and contextual visual aids.
  • The system identifies suitable visuals for each instruction, generates or retrieves the content, and embeds it in the user’s environment without manual intervention.
  • The design framework defines five guidance types: component highlights, movement cues, hand gestures, animated tools, and contextual widgets such as timers.
  • Two evaluations involved 16 participants in controlled and in-the-wild studies plus interviews with four instructors, indicating enhanced learning experiences and future training directions.

2 RELATED WORK

Prior AR guidance research spans machine operations, assembly, maintenance, authoring workflows, and LLM-based assistance, but it has not established which visual types fit different instructions. Guided Reality extends this work with a taxonomy-driven, fully automated pipeline for generating and grounding richer visual guidance.

  • Prior AR systems support domains such as machine operations, object assembly, and industrial maintenance, while rich affordances can enhance task guidance.
  • No prior research had examined which visual guidance types are most suitable for different instructional content.
  • Existing systems often rely on predefined instructions or expert configuration, motivating authoring tools and automated guidance generation.
  • Recent multimodal systems automate parts of AR guidance, including text placement, avatar animation, and progress-aware anchored overlays.
  • Guided Reality expands beyond basic highlights and limited motion guidance by identifying, generating, and embedding richer visual forms using live vision and depth data.

3 STRATEGY OF VISUAL GUIDANCE IN AR

Guided Reality derives a visual-guidance design space by analyzing paired instructions and images from diverse user manuals. The resulting five categories connect instructional actions with visual cues for locating, moving, manipulating, gesturing, and waiting.

  • The authors analyzed paired text and images from manuals across five domains to identify visual strategies supporting comprehension and action.
  • Two independent coders systematically coded the dataset, with team review and collaborative reflection resolving discrepancies before finalizing the scheme.
  • The design space contains five categories: component highlighting, movement indication, hand gestures, tool use, and contextual widgets or information.
  • Component highlights help users locate the target for simple actions, while movement guidance adds directional cues and intended end positions.
  • Hand-gesture illustrations communicate hand position and motion, especially for fine-motor or coordinated two-handed actions.
  • Tool visuals show the tool’s position and orientation, with arrows indicating operations such as rotation or linear movement.
  • Contextual widgets such as timers support steps requiring a wait duration or condition before proceeding.

4 GUIDED REALITY SYSTEM DESIGN

Guided Reality converts a natural-language task query into structured, step-specific AR guidance and anchors generated visuals to physical components. Its pipeline combines LLM planning, vision-based localization and segmentation, depth-aware raycasting, and spatial rendering.

  • Given a user question, the system generates a structured task plan containing text instructions, target components, visual types, and supporting metadata.
  • A 3D-printer reset example demonstrates highlights, animated tools, segmented movement, bounding boxes, a contextual timer, and hand gestures across successive steps.
  • The prototype runs on an iPad Pro using camera access and real-time LiDAR depth, chosen for practical mobility and accessibility despite limited hands-free interaction.
  • For each step, the system captures a new scene image, locates the referenced component with a vision model, and segments it when needed.
  • Raycasting combines image coordinates with depth, camera intrinsics, and camera pose to compute a 3D position that remains anchored as the device moves.
  • The system places and orients visual content according to the guidance type, using spatial information such as surface normals for tool models and 3D arrows for motion.

5 IMPLEMENTATION

Guided Reality combines an iOS AR front end with a local Python back end to generate, localize, and render step-specific visual guidance. The implementation uses structured task plans, vision-model outputs, spatial data, and several guidance-generation procedures, with most steps completing within five seconds.

  • Overall System Architecture: The iOS front end handles interaction, task-plan generation, and AR placement, while a local Python back end performs AI-based visual-guidance generation.The front end uses UIKit, ARKit, and SceneKit; the back end uses GPU-accelerated vision models.
  • Task-Plan Generation: Each structured task-plan step specifies an instruction, visual type, and key component, with contextual component details improving downstream object detection.The system obtains plans from ChatGPT and includes relative positions or visual attributes in key-component descriptions based on preliminary testing.
  • Spatial Localization and Rendering: The system localizes guidance by combining vision-model bounding boxes with saved depth maps and camera poses, then anchors and renders the result in 3D space.For region-of-interest guidance, 2D box corners are projected into 3D; boxes with an edge shorter than 5 cm become centered particle effects.
  • Indicating Movement: Rotational guidance highlights the component and overlays a curved arrow aligned to the identified rotation axis and direction.The system standardizes rotation using the axis and direction viewed from the positive side of that axis.
  • User Query–Driven Result: 65 of 100 evaluated steps produced correct results, while 17 failed during task-plan generation and 18 during visual-guidance generation or placement.Visual-type identification reached 90% accuracy under the study’s strict classification scheme.
  • Visual Type Evaluation: Translational movement reached 80% accuracy, rotational movement 70%, hand gestures 75%, and widget guidance was generated correctly for all evaluated widget steps.Errors involved target-position identification, rotation-direction identification, non-planar gesture placement, small or similar components, and tool-model orientation.
  • Visual Type Evaluation: Most steps completed within five seconds, but missing external tools triggered 3D-model generation and increased latency.

6 USER STUDY

The user study found that Guided Reality reduced task errors while maintaining comparable completion times and received strong usability ratings. Participants especially valued movement and widget guidance, but reported inconsistency, visibility, and recognition issues across some visual types.

  • 6.2.1 System Usability and Visual Consistency: Guided Reality achieved a mean SUS score of 84.38 versus 81.41 for the baseline, although the overall difference was not significant.Users scored Guided Reality higher on technician need and confidence, but slightly lower on consistency.
  • 6.2.2 Completion Time and Error Rate: Error rates decreased from 17.14% to 10.42% for scanning-area cleaning and from 9.8% to 7.7% for 3D-printer resetting.Completion times were slightly longer with Guided Reality: 2:42 vs. 2:32 and 3:33 vs. 3:32.
  • 6.2.3 Effectiveness of Visual Guidance Types: Movement guidance was most favored at M = 4.27, conveying interaction location, direction, and movement manner better than the baseline arrow.Participants found it less effective when the target position was misidentified.
  • 6.2.3 Effectiveness of Visual Guidance Types: Widgets scored M = 3.94 and were praised for clarity, responsiveness, and interactive timing support.Participants suggested automatic step advancement or smart-device integration.
  • 6.2.3 Effectiveness of Visual Guidance Types: Tool guidance scored M = 3.81 and helped users identify and use tools, but mismatched models and imprecise placement still required assistance.Participants requested alignment cues combining tool animation with blue segmentation to mark interaction zones.
  • 6.2.4 In-the-Wild Study: In-the-wild users applied the system across safety, household, creative, software, and musical tasks, indicating broad task versatility.Participants also reported inconsistent visuals, low-contrast targets, and occasional difficulty interpreting hand gestures or static highlights.

7 INSTRUCTOR INTERVIEW

Interviews with four instructors examined how Guided Reality could fit hands-on teaching workflows. Instructors saw value for personalized learning, troubleshooting, and reinforcing standard procedures, while requesting author control and interactive clarification.

  • 7.2.1 Personalized Learning: Instructors viewed Guided Reality as a supplement for personalized learning after group workshops, especially for student-selected procedures.I1 and I4 described using it for topics not covered during standard group sessions.
  • 7.2.2 Troubleshooting: Instructors saw anchored machine highlights as useful for troubleshooting, including locating components and diagnosing incorrectly connected wires.They contrasted this with manually switching between diagrams and error-code pages.
  • 7.2.3 Reinforcing Standard Procedures: Guided Reality was considered useful for reinforcing standard procedures, but less applicable to artistic tasks requiring experimentation and conceptual understanding.Because the system is generative and may vary, instructors wanted to enter text instructions manually.
  • 7.2.4 Bi-Directional Interaction: Instructors requested bi-directional interaction so users and the system could clarify ambiguous steps and refine guidance during task execution.Accurate guidance may also require the system to ask users for missing task-specific choices.

8 LIMITATIONS AND FUTURE WORK

The current system is constrained by its mobile interface, imperfect generative 3D models, and an assumption that relevant components are visible in the camera view. Future work targets head-mounted displays, functionally grounded model generation, and persistent tracking of hidden components.

  • 8.1 Integration with Head-Mounted Displays: The iPad supports mobility and accessibility, but shifting attention between its screen and the workspace can disrupt task flow during two-handed activities.The authors plan to expand support to head-mounted displays such as Meta Quest.
  • 8.2 Generative Models for Functional 3D Guidance: Generative 3D models can produce plausible geometry without reliable functional orientation or interaction context.Future approaches may combine generative geometry with explicit spatial constraints or post-generation pose correction.
  • 8.2 Generative Models for Functional 3D Guidance: Generated visual guidance still requires refinement to be visually convincing, functionally accurate, and contextually grounded.The authors identify richer and more realistic gesture generation as a future direction.
  • 8.3 Guidance for Out-of-View Components: The current pipeline assumes that relevant components are visible when guidance generation begins, excluding hidden or out-of-view targets.Planned solutions include continuous camera sampling, persistent spatial anchoring, and relative spatial instructions.

9 CONCLUSION

Guided Reality combines LLMs and vision models to generate visually enriched, spatially situated AR guidance. Its five visual guidance types supported usability and adaptability across real-world tasks, while instructor feedback highlighted applications in personalized learning, troubleshooting, and procedural training.

  • 9 CONCLUSION: Guided Reality integrates LLMs and vision models to select and embed five visual augmentation types grounded in user-manual analysis.The system generates spatially situated guidance for step-by-step tasks.
  • 9 CONCLUSION: Users especially praised movement and dynamic widget guidance for clarity and relevance, while in-the-wild testing showed adaptability across real-world contexts.Instructor interviews further identified value for just-in-time training, personalized learning, troubleshooting, and reinforcing standard procedures.

A.1 Structure Task Plan Prompt

The prompt requires the model to convert a user query and image into a JSON task plan whose steps pair instructions with visual guidance types and key components.

  • A.1 Structure Task Plan Prompt: Each generated plan uses a top-level "instructions" field containing a list of step objects.Each step includes an instruction, key components, and a visualization type.
  • A.1 Structure Task Plan Prompt: The model must produce short step descriptions and identify the interacted object using distinguishing properties such as position, color, or visible text.The first key component should describe the target component with enough detail to distinguish similar objects.
  • A.1 Structure Task Plan Prompt: Visual type 1 highlights a key component to help users locate the object involved in a simple action.The example identifies an orange Start button for a rice cooker.
  • A.1 Structure Task Plan Prompt: Visual type 2 represents a component’s movement using start and end points, with translation or rotation as the movement specification.The example concerns returning an air-fryer basket to its position.
  • A.1 Structure Task Plan Prompt: Visual type 3 specifies the manipulated component and the hand gesture required, using gestures such as poke, hook, palm press, grip, cylindrical grasp, or pinch.The example pairs pulling filament with a pinch gesture.
  • A.1 Structure Task Plan Prompt: Visual type 4 identifies an external tool and its required movement, including up-and-down, left-and-right, rotational, clockwise, or counterclockwise motion.The example represents mixing ingredients with a rotating whisk.
  • A.1 Structure Task Plan Prompt: Visual type 5 represents waiting with a dynamic widget whose key components specify the relevant area and duration in mm:ss format.The example displays a 30-second wait for food in a mixing bowl.

A.2 2D Bounding Box Prompt

The prompts extract spatial information for static, translational, and rotational guidance by returning component locations, movement targets, or rotation axes and directions.

  • A.2 2D Bounding Box Prompt: The 2D bounding-box prompt asks the vision model to identify the key component and return its name with image coordinates.The output format uses [y_min, x_min, y_max, x_max].
  • A.2 2D Bounding Box Prompt: Together, the prompts provide the spatial outputs needed to place guidance for locating components, translating them, or representing their rotation.The outputs progress from a bounding box to a target position or a rotation descriptor.
  • A.3 Translational Movement Prompt: The translational-movement prompt additionally asks for the component’s end position after the instruction.Its output includes the original bounding box and target_pos [xEnd, yEnd].
  • A.4 Rotation Information Prompt: The rotation-information prompt asks how the key component rotates between images and uses the first image to establish the spatial axis.The axis convention sets x rightward in the photo, y physically upward, and z toward the viewer.
  • A.4 Rotation Information Prompt: Rotation is classified by selecting an x, y, or z axis and determining whether the motion is clockwise or counterclockwise.The prompt instructs the model to view the motion from the positive side.
  • A.4 Rotation Information Prompt: The rotation prompt returns the result as a two-element rotation field containing the axis and direction.The specified format is {{rotation: [axis, direction]}}.
Loading 2508.03547v2…