PaperlayerView exact revision on arXiv

Source-linked AI summary

RegulAR: Graph-Grounded Error Recognition and Assistance for Procedural Tasks in AR

Yi-Lin Ye, Jindu Wang, Hiu Tung Wong, Shuchang Xu, Huamin Qu, Wong Kam-Kwai

arXiv:2608.26715v1cs.HC

TL;DR

Procedural AR guidance often emphasizes step delivery rather than recognizing and recovering from errors. RegulAR grounds MLLM-based egocentric interpretation in a hierarchical dependency graph to track state, classify deviations, estimate downstream impact, and present recovery guidance. In a within-subject study, participants reported better task-structure understanding and recovery support than with an MLLM-only baseline.

  • Problem

    Most AR guidance systems focus on step-by-step instruction delivery, leaving limited support for recognizing and recovering from procedural mistakes.

  • Method

    RegulAR combines a hierarchical dependency graph with MLLM-based egocentric interpretation and an in-situ HUD for progress tracking, error classification, impact estimation, and recovery guidance.

  • Results

    Participants reported better task-structure understanding and recovery support with RegulAR than an MLLM-only baseline in a within-subject study (N=12).

  • Takeaways & Limitations

    Making procedural structure explicit supports state tracking, error recognition, and users’ understanding of what went wrong and how to proceed.

  • Takeaways & Limitations

    The evaluation used a small sample and controlled, short, single-user tasks, providing stronger evidence for perceived recovery support than behavioral recovery itself.

Abstract

from arXiv · show

Errors are inevitable in procedural tasks, yet most AR guidance systems focus on step-by-step instruction delivery rather than helping users recognize and recover from mistakes. We present RegulAR, an AR task assistant for procedural error recognition and recovery. RegulAR models task instructions as a hierarchical dependency graph and combines this structure with a Multimodal Large Language Model (MLLM) to interpret egocentric observations during execution. This enables RegulAR to track progress, identify deviations by error type, estimate their impact on later steps, and deliver appropriately salient interventions through an in-situ head-up display that visualizes task state and recovery guidance. By making procedural structure explicit, RegulAR supports not only next-step guidance, but also reasoning about what went wrong, why it matters, and how users can get back on track. In a within-subject study (N=12), participants reported better task-structure understanding and recovery support with RegulAR than the MLLM-only baseline.

1 Introduction

Procedural AR assistance must address not only next-step guidance but also error recognition, impact assessment, and recovery during execution. RegulAR combines a hierarchical dependency graph with MLLM interpretation to provide structured, situated support.

  • Motivation: Procedural work requires reasoning about prerequisites, current-state validity, correct completion, and safe continuation, not merely remembering the next step.Users execute sequences with dependencies, ordering constraints, and evolving object states while having limited capacity to notice deviations.
  • Motivation: Existing AR and MLLM assistants support in-situ instruction and visual interpretation but provide little assistance after users deviate from nominal execution.
  • Motivation: Users seeking deviation support asked whether their current state was acceptable and what the deviation would affect, revealing relational errors, ambiguous observations, and intervention-timing challenges.The needs-finding study identified three challenges: hidden prerequisite violations, partial streaming observations, and the need to avoid overbearing feedback.
  • RegulAR: RegulAR models instructions as a hierarchical dependency graph and combines it with MLLM-based egocentric interpretation to track progress, classify four active-error categories, and estimate downstream impact.
  • RegulAR: An in-situ HUD presents what happened, why it matters, and how to recover by visualizing task state, affected steps, and recovery cues.
  • Evaluation: A within-subject user study (N=12) provided preliminary evidence for improved state tracking, error detection, task-structure understanding, and perceived error support versus a prompt-only MLLM baseline.The evaluation also surfaced a trade-off between intervention timing and user autonomy.

2 Related Work

Prior AR and MLLM systems improve situated procedural instruction and interaction, but generally lack persistent task-state representations for interpreting deviations and their downstream effects. RegulAR addresses this gap by using dependency graphs for continuous, stateful reasoning during execution.

  • AR Guidance: AR guidance systems present procedural instructions in physical environments, reducing context switching for hands-busy tasks across domains such as cooking and industrial maintenance.
  • Language-Model Assistants: LLM- and MLLM-based assistants add egocentric perception, intent inference, next-action anticipation, context-aware feedback, and mixed-initiative interaction.
  • Gap: These assistants often rely on immediate observations without persistent task-state representations, making long-term progress tracking and error-impact reasoning difficult.
  • Error Detection: Egocentric error detection is complicated by camera motion, partial observability, and viewpoint variability; many learned approaches additionally require anomalous videos and labeled datasets.
  • Structured Representations: Task graphs and dataflow diagrams have modeled procedural dependencies, but prior uses emphasize instruction organization, visualization, or offline reasoning rather than dynamic stateful execution.

3 Preliminary Needs-Finding Study

The needs-finding study examined how users understand and recover from procedural deviations with reactive AR assistants. Its observations motivated explicit task-state representations, error differentiation by downstream impact, and concise progressive assistance.

  • Methods: The study recruited six university participants with sufficient English proficiency and moderate AR familiarity.Participants reported moderate AR familiarity on a 5-point Likert scale (M=2.33, SD=0.52).
  • Methods: Participants completed procedural tasks with a Meta Quest 3, static AR instructions, and optional GPT-4V voice access through the headset camera feed.Tasks included Hybrid Meeting Setup and Mug Cake activities.
  • Observations: Participants needed explicit reasoning about both local validity and global progress because implicit relationships and limited structural awareness made task-state reconstruction fragmented and delayed error recognition.
  • Observations: Errors included omitted steps, incorrect actions, and intentional modifications, but instruction-based support treated them uniformly despite differing causes and consequences.
  • Observations: Participants valued issue identification and resolution but preferred concise indications with explanations available on demand because verbose feedback disrupted hands-busy task flow.
  • Design Requirements: The resulting design requirements call for externalized task state, error characterization by type and downstream impact, and progressive assistance adapted to error significance and user demand.

4 RegulAR

RegulAR represents procedural instructions as a hierarchical dependency graph and combines graph state with batch MLLM interpretation of egocentric frames. It tracks execution, recognizes deviations, estimates downstream impact, and adapts recovery assistance to error significance.

  • Task Dependency Graph Construction: RegulAR parses instructions into a hierarchical dependency graph augmented with perceptual cues, error examples, and risk metadata.The graph captures subgoals, prerequisites, valid execution paths, action status, and skip risk.
  • Task Dependency Graph Construction: The graph separates semantic steps from observable atomic actions and represents prerequisite relations while allowing flexible ordering among independent branches.Actions within a step are order-independent unless instructions specify an explicit dependency.
  • Runtime Monitoring and Error Recognition: At runtime, a batch MLLM analyzes recent egocentric frames with the current graph state to predict actions and statuses, while graph validation maintains consistent progress updates.The system samples a frame window every 5 s at 1 fps and suppresses repeated or graph-inconsistent transitions.
  • Runtime Monitoring and Error Recognition: RegulAR detects modification, addition, and slip errors directly from MLLM output, while omission errors are inferred when a predicted action is not executable.Each action can be marked not_started, in_progress, complete, or error.
  • Impact-Aware Error Intervention: Intervention content depends on error type, whereas intervention saliency depends on estimated downstream impact from affected actions and their structural importance.Structural importance combines downstream reachability with remaining task distance, balanced by α = 0.5.

5 User Evaluation

The within-subject study compared graph-grounded RegulAR with an MLLM-based baseline using the same core task-assistance setup. Participants reported stronger task understanding, progress awareness, error recovery, confidence, and preference for RegulAR without measurable workload increases.

  • Study Design: The within-subject study involved 12 participants completing one task with RegulAR and one with an MLLM-based baseline.Both conditions used the same procedural documents, MLLM, egocentric input, monitoring interval, and AR headset; the primary difference was the explicit task graph.
  • Overall Findings: Overall, participants reported better task-structure understanding, progress awareness, and error-recovery support with RegulAR, alongside higher confidence and preference.The study reports these gains without a measurable increase in workload.
  • Task Understanding and Tracking: 6.00 vs. 4.25 for overall task-structure understanding and 6.17 vs. 4.17 for step dependencies, both p< .01, favored RegulAR over the baseline.Participants attributed these gains to the graph’s navigation-like representation of completed and remaining steps.
  • Task Understanding and Tracking: 6.17 vs. 4.17 for progress tracking, p< .01, favored RegulAR, whose explicit graph state reduced disorientation and structurally inconsistent guidance.The baseline sometimes repeated suggestions for actions participants had already completed, compounding errors over time.
  • Error Interpretation and Recovery: 6.08 vs. 4.00 for error recognition and 5.92 vs. 4.08 for localization, both p< .01, favored RegulAR’s graph-grounded error mode.The error mode highlighted affected task-graph portions, helping participants identify where recovery should begin.
  • Error Interpretation and Recovery: 6.25 vs. 3.92 for understanding downstream consequences, p< .001, and 5.83 vs. 4.17 for urgency assessment, p< .05, favored RegulAR.Its feedback explained what happened, why it mattered, and how to recover, supporting recovery reasoning rather than simple next-step following.
  • Workload and Guidance: NASA-TLX showed no significant differences in mental, physical, temporal demand, or effort, while self-rated task performance favored RegulAR at 5.67 vs. 4.67, p< .05.Participants noted an initial graph-learning cost but expected it to diminish with familiarity.
  • Workload and Guidance: RegulAR reduced guidance noise through impact-based filtering, but six participants reported recognition latency and a continuing proactivity–autonomy tension.Usefulness and disruption ratings did not differ significantly, despite qualitative differences in intervention quality.

6 Technical Evaluation

The technical evaluation examined graph generation, runtime tracking, error recognition, and recovery assistance. Graphs were structurally valid with mostly low audit error rates, while graph-constrained reasoning improved procedural learning and error recognition and enabled impact-sensitive recovery guidance.

  • Evaluation Scope: The evaluation assessed procedural learning, error recognition, and error-assistance quality using graph-generation checks, runtime logs, and user-study examples.The technical evaluation combined structural and semantic graph audits with action, state, and error-recognition measures.
  • Graph Generation: All 10 generated graphs were structurally valid DAGs covering 118 steps and 176 actions.Manual audits found low hallucination and limited step-segmentation, action-node, and dependency-edge errors.
  • Graph Generation: Hallucination averaged 0.7%, while step-segmentation, action-node, and dependency-edge errors averaged 2.1%, 5.2%, and 3.3%, respectively.Completion criteria were correctly captured in 92.2% of cases; remaining issues mainly involved repeated, optional, or concurrent actions.
  • Runtime Evaluation: Runtime evaluation covered 535 RegulAR and 525 baseline ground-truth observations across 12 user-study sessions and 24 task executions.RegulAR had slightly shorter completion times, while the baseline often remained in the same state longer; one outlier reflected network delay.
  • Procedural Learning and Error Recognition: Graph-constrained reasoning improved procedural learning and error recognition compared with textual prompting alone.The dependency graph constrained predictions to contextually valid actions and expected transitions, reducing regressions and omission false positives.
  • Procedural Learning and Error Recognition: RegulAR distinguished unmet prerequisites from valid alternative orderings, producing more balanced precision and recall while moderately improving execution-level error recognition.The baseline more often misclassified harmless reordering or intermediate actions as omissions and hallucinated omission errors.
  • Error Assistance: RegulAR adapted recovery guidance to error impact, explaining the error, its consequences, and recovery steps with intervention urgency scaled to severity.Examples included optional low-urgency tool advice for a minor stirring error and rollback guidance for slip errors.

7 Discussion

The discussion argues that AR assistance should prioritize recoverability, contextual error interpretation, and consequence-aware support rather than next-step delivery alone. It also identifies limitations involving reliance, evaluation scope, latency, visual load, and scalability.

  • Design implications: AR guidance should help users validate state and recover after deviations, not only follow the ideal next step.Participants asked whether their current state was acceptable and whether they could continue, motivating recoverability as a broader design target.
  • Design implications: Procedural errors require contextual interpretation because their meaning depends on prerequisites, state transitions, and effects on later progress.The discussion links this requirement to improved completion and omission decisions when explicit task dependencies matter.
  • Design implications: The dependency graph functions as both a computational constraint on plausible states and a visible state contract for coordinating user and assistant understanding.Users used the graph to inspect completed actions, feasible next actions, affected dependencies, and recovery paths.
  • Design implications: Proactive assistance reduces diagnostic burden but may encourage users to defer their own decision-making to the assistant.The discussion recommends supporting users without replacing judgment when multiple valid responses remain possible.
  • Design implications: Interventions should vary with error type, downstream consequence, user expertise, and preferred control, using layered explanations and recovery guidance.Users preferred concise notifications that signal importance, explain why, and offer recovery actions when needed.
  • Limitations and future work: The evaluation is limited by a small sample, uncertain generalizability, and stronger evidence for perceived recovery support than behavioral recovery.Future evaluations should measure recovery time, unresolved deviations, cascaded errors, and false-positive and false-negative costs.
  • Limitations and future work: Five-second monitoring cycles may miss short actions or delay recognition, while API-based multimodal models can introduce latency and recognition errors.These constraints affect responsiveness and the reliability of real-time assistance.
  • Limitations and future work: Continuously displaying graph guidance may compete with physical-task attention, although users can adjust graph transparency, position, and size.Nine participants found the graph easy to understand, while two found it distracting.

8 Conclusion

RegulAR grounds MLLM-based egocentric reasoning in a hierarchical dependency graph to recognize procedural errors, estimate their downstream impact, and provide layered in-situ recovery guidance. Technical and user evaluations found improved state tracking, error recognition, task-structure understanding, progress awareness, and perceived recovery support.

  • Conclusion: RegulAR combines a hierarchical dependency graph with MLLM-based egocentric reasoning for procedural error detection and recovery.The system tracks progress, recognizes errors, estimates downstream impact, and provides layered in-situ guidance.
  • Conclusion: The structured approach improves objective state tracking and error recognition while strengthening users’ task-structure understanding and progress awareness.Users also reported better support for responding to errors.
  • Conclusion: The findings position recoverability after deviations as a central future direction for AR task assistance.The conclusion extends AR guidance beyond delivering instructions alone.

A Study Materials

The study materials include evaluation questions and procedural instruction sources used across needs-finding, system testing, and user-study tasks.

  • Study materials: Table 2 lists the questions used in the user evaluation.
  • Study materials: Table 3 lists procedural instruction sources for needs-finding, system testing, and user-study tasks.The tasks are grouped as T1–T2, T3–T8, and T9–T10.

B Participants

The participant materials report samples for the preliminary needs-finding study and the user evaluation, including AR familiarity ratings and, for the user evaluation, task familiarity ratings.

  • Participants: Table 4 reports participants in the preliminary needs-finding study and their AR familiarity ratings.AR familiarity was measured on a 5-point scale.
  • Participants: Table 5 reports participants in the user evaluation, including AR familiarity and task familiarity ratings.AR familiarity used a 5-point scale, while task familiarity used a 7-point scale.

C Prompts

The prompts convert procedural instructions into a hierarchical dependency graph, then annotate actions with criteria for progress, completion, skipping risk, and execution errors. They also define structured error categories and graph constraints to support faithful, non-hallucinated task representations.

  • C.1 Task Graph Generation Prompt: The task graph is a directed acyclic graph with high-level Step nodes and atomic Action nodes.Each Step contains one or more Actions, while the DONE step has no action.
  • C.1 Task Graph Generation Prompt: Dependencies encode prerequisite ordering, while steps without explicit dependencies may execute in parallel.Actions within the same Step must remain independent and executable in parallel; linear order must not be assumed.
  • C.1 Task Graph Generation Prompt: Each Step stores a unique identifier, grouped description, prerequisite step identifiers, and its action list.Step descriptions retain non-atomic instructional material such as tips, warnings, explanations, and context.
  • C.1 Task Graph Generation Prompt: Each Action stores a unique identifier, one executable instruction, and the exact source text from which it was derived.The prompt also records tools, materials, and precise material amounts when available.
  • C.1 Task Graph Generation Prompt: The generation prompt splits compound or iterative instructions into separate atomic actions and forbids hallucinated or merely descriptive actions.Relevant source text must remain in fields such as original_text, step_description, or content.
  • C.2 Task Criteria Generation Prompt: For each action, the criteria prompt specifies skip risk, skip impact, status, progress indicators, completion criteria, and visual execution-error criteria.The resulting representation supports judging whether an action is ongoing, complete, skipped, or visibly erroneous.
  • C.2 Task Criteria Generation Prompt: The criteria examples distinguish modification, addition, and slip errors according to how execution differs from the specified task.Examples include using different tools or ingredients, adding unnecessary steps, and executing an action without achieving its intended result.
  • C.2 Task Criteria Generation Prompt: Skip risk is categorized as none, medium, or high according to whether omission affects completion, quality, or subsequent steps.High-risk omissions prevent later steps from functioning, whereas medium-risk omissions reduce quality without blocking progression.
Loading 2608.26715v1…