PaperlayerView exact revision on arXiv

Source-linked AI summary

HelpMeOut: A Neuropsychology-Grounded Benchmark for Error-Aware Monitoring of Household Task Execution

Haoran Wang, Jingwei Huang, Anqing Duan, Ho Seok Ahn, Bruce Macdonald

arXiv:2610.17694v1cs.ROcs.AIcs.SC

TL;DR

Household benchmarks usually assume people execute tasks correctly, leaving error monitoring for cognitively impaired people and older adults underrepresented. HELPMEOUT builds paired episodes grounded in everyday action errors and evaluates monitoring as tasks unfold; the evaluated methods often miss or poorly localize errors.

  • Problem

    Household benchmarks typically assume coherent task execution, while everyday actions can contain structured errors that assistive agents need to recognize from partial activity streams.

    Source: §1 · p. 1

  • Method

    HELPMEOUT creates paired correct and errorful household episodes and evaluates streaming monitors on error discrimination, step-level detection, latency, and type diagnosis.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    Learned monitors, symbolic baselines, and frontier LLMs often fail to detect or localize errors as they happen, with many alarms late or absent.

    Source: §1 · p. 2, §4 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    HELPMEOUT provides a testbed for assistive agents intended to catch execution breakdowns before a task fails.

    Source: §6 · p. 10

    The benchmark uses symbolic action sequences and assumes actions are already recognized, excluding recognition errors and multimodal cues such as hesitation, gaze, and fumbling.

    Source: §5 · p. 9

Abstract

from arXiv · show

We introduce HelpMeOut, an embodied AI benchmark for error-aware monitoring of household task execution. Existing embodied household benchmarks typically assume that humans execute tasks correctly, but this assumption does not hold for cognitively impaired persons or older adults whose everyday activities produce structured execution errors. To address this gap, HelpMeOut operationalizes research on everyday action errors into 20000 paired activity episodes, covering nine neuropsychologically motivated error categories. In each episode, agents observe a streaming human activity trajectory and must monitor whether the execution is unfolding correctly, diagnosing any structured errors as early as possible. We evaluate learned-monitor, symbolic, change-point, and frontier LLM prompting baselines. Results show that these methods perform poorly as error monitors, often missing when execution goes wrong. These results suggest a gap left by standard goal-inference and task-completion benchmarks, making HelpMeOut a testbed for error-aware assistive agents.

1 Introduction

HELPMEOUT treats household assistance as monitoring whether a known task is going wrong, addressing structured everyday errors that standard household benchmarks rarely evaluate. It pairs correct and errorful episodes and tests whether agents can detect, time, and diagnose errors.

1 Introduction

  • Everyday action errors include structured omissions, substitutions, sequence errors, and perseverations, making error detection and timely intervention relevant to assistive robots for people with cognitive impairment and older adults.The introduction notes dementia affects 55 million people and is projected to affect 139 million by 2050.Source: §1 · p. 1
  • Existing household benchmarks generally assume coherent, goal-directed human behavior, while execution-monitoring systems often depend on fixed sensors or predefined scripts rather than scalable evaluation from partial activity streams.The missing capability is step-level monitoring of whether a known task is going wrong.Source: §1 · pp. 1–2
  • Experiments find that learned monitors struggle with step-level detection and diagnosis, symbolic baselines mainly identify incomplete tasks at episode end, and frontier LLMs remain limited in localization and timely alerting.The benchmark evaluates whether, when, and what kind of execution error occurs while a household task unfolds.Source: §1 · p. 2
  • HELPMEOUT generates 20,000 paired correct and errorful episodes from ADL/IADL tasks using a neuropsychological goal-control model calibrated to reported error rates.Error types and frequencies vary by category and degree of cognitive impairment rather than being sampled uniformly at random.Source: §1 · p. 2

2 Related Works

Related work spans clinical accounts of everyday action errors, assistive systems for detecting failures, and household benchmarks and simulators. HELPMEOUT combines scalable household generation with neuropsychologically grounded errorful trajectories and step-level onset labels.

2 Related Works

  • Clinical performance-based tests characterize errors in familiar multi-object tasks, while goal-control accounts link omissions to weak or decaying goals and commission errors to failures to suppress competing goals.These accounts draw on the Supervisory Attentional System and contention-scheduling models.Source: §2 · pp. 2–3
  • Smart-home and robot-assisted systems detect deviations using sensors or predefined scripts, but existing systems and datasets are often constrained by task, sensor setup, or simulated errors.Large-scale data collection from cognitively impaired users also faces privacy and ethical constraints; HELPMEOUT instead generates clinically calibrated, automatically labeled episodes with deterministic onsets.Source: §2 · p. 3
  • Goal and plan recognition methods infer intent from partial observations, whereas HELPMEOUT asks whether, when, and how the execution itself goes wrong.Classical approaches typically assume rational behavior; bounded-rational variants allow suboptimal plans [38].Source: §2 · p. 3
  • Household intent benchmarks and simulators assume competent human actions and correct trajectories; HELPMEOUT adds neuropsychologically grounded errorful trajectories, paired references, and step-level onset labels.Table 1 compares related benchmarks across introduced and absent capabilities.Source: §2 · pp. 3–4, Table 1 · p. 3

3 The HELPMEOUT Benchmark

HELPMEOUT formalizes household-task monitoring as detecting whether, when, and what kind of structured execution error occurs from a streaming action prefix. It builds paired correct and errorful episodes using a neuropsychology-grounded generator and evaluates monitors with prefix-based metrics.

3.1 Preliminaries: Neuropsychology of Everyday Action

  • The generator grounds five recurring error labels in neuropsychological assessments and maps goal-activation failures to omissions and executive-control failures to commission errors.Its goal-control model calibrates category rates to studies of control and clinical populations.Source: §3.1 · p. 4, §3.3 · p. 5

3.2 Basic Monitoring Setting

  • Agents observe the task name, initial symbolic scene, and action prefix, then predict whether execution went wrong, its observable onset, and its error categories.They do not receive formal task requirements or future actions.Source: §3.2 · p. 4, §3.4 · p. 6
  • Observable onset is the earliest step an error can be identified from the prefix and task constraints, but an omitted subgoal is confirmed only at task end.Onset annotations are computed deterministically from the initial state, errorful trajectory, and injected category.Source: §3.2 · p. 5

3.3 Structured Execution Errors

  • The nine categories separate progression-level errors from current-action errors; reordered subgoals count as sequence errors only when they violate a required order.Correct executions may vary in subgoal order and intermediate placements.Source: §3.3 · pp. 5–6
  • Dual-standard validation keeps errorful trajectories physically executable while preserving semantic violations, relaxing only the defining semantic precondition for substitutions.Other error types undergo strict precondition replay; substitution traces retain physical-chain constraints.Source: §3.3 · p. 6
  • The benchmark pairs correct and errorful trajectories across 50 ADL/IADL task templates, with 20,000 episodes and 12,746 annotated onsets.The test set contains 500 episodes; omissions are the most frequent category, with 5,346 instances.Source: §3.3 · p. 6

3.4 Prefix-Based Evaluation

  • The streaming evaluation measures trajectory discrimination, onset-step detection, first-alarm timing, and error-type diagnosis, with separate treatment for omissions confirmed at task end.M2 scores only onset steps as positive, while M4 credits category labels only after an error has occurred.Source: §3.4 · p. 6

4 Experimental Results

Across monitor families, strong aggregate scores often depended on task-end checks or false alarms, while stepwise diagnosis and timely alarms remained difficult; LLM prompting also remained below the human reference.

LLM baselines do not close the monitoring gap.

  • 0.837 vs. 0.967 (M1) and 0.482 vs. 0.748 (M2): best LLM confidence-interval upper bounds remain below Human, and no LLM exceeds SGT's 0.423 on M2.With reasoning enabled, larger models produce more on-target first alarms than SGT on the same subset, but also alarm on 30–77% of correct episodes.Source: Figure 3 · p. 8, §4 · p. 9

Learned and change-point baselines.

  • SGT leads learned monitors (M1 0.778, M2 0.656), but Seq2Seq gains concentrate at task end and CUSUM's M1 0.645 comes with alarms on 81% of correct episodes.Seq2Seq and Seq2Seq+SG detect error steps above chance on M2, but their trajectory-level discrimination stays near chance; all methods remain below Human (M1 0.967, M2 0.748).Source: §4 · p. 8

Symbolic baselines.

  • 0.887 M1 is the re-planner's highest score, largely from end-of-task omission checks; pre-end M1 falls to 0.669 and non-final-step AUPRC to 0.118, while 62% of correct episodes receive alarms.The precondition checker fires on fewer than 5% of errorful episodes; valid human executions need not follow shortest plans.Source: §4 · pp. 8–9

Alarm timing and type diagnosis.

  • 80% of SGT's first alarms fall at error onset, but it misses 52% of errorful episodes; substitutions, action additions, and captures have 70–89% miss rates.CUSUM and the re-planner rarely miss but alarm early in about three quarters of errorful episodes and on 81% and 62% of correct episodes, respectively.Source: §4 · p. 9

5 Limitations and Broader Impact

The benchmark's symbolic, generated episodes and its evaluation do not capture several features of real-world execution or deployment.

Data and representation

  • Symbolic action sequences assume actions have already been recognized from video or sensors, so the benchmark captures neither recognition errors nor multimodal cues such as hesitation, gaze, or fumbling.Results are an upper-bound proxy for symbolic-level monitoring; grounding episodes in video or real ADL recordings is left to future work.Source: §5 · p. 9

Episode generation

  • Generated trajectories are always physically executable and correct routines do not interleave subgoals or include harmless redundancy, making some insertion errors identifiable from surface patterns.Clinical experts judged most generated errors plausible, but this is not patient-level validation.Source: §5 · p. 9

Deployment

  • The benchmark does not measure deployment risks: false alarms may interrupt autonomy and trust, while missed alarms may leave unsafe lapses unnoticed.Thresholds should be set with users and caregivers rather than by benchmark score.Source: §5 · p. 9
  • HELPMEOUT contains only synthetic episodes and no personal data, so results on it are not evidence that a system is safe to deploy.In-home monitoring also raises requirements for informed consent, data minimization, and the person's control over what is recorded.Source: §5 · pp. 9–10

6 Conclusions

HELPMEOUT reframes household assistance as online error monitoring, asking whether, when, and what kind of execution error occurs. The evaluated monitors often detect errors only after an episode ends, rarely localize them as they happen, and perform near chance on errors requiring object or tool knowledge.

6 Conclusions

  • Learned monitors, symbolic references, and frontier LLMs can often detect that an episode went wrong, especially after it ends, but rarely detect and localize errors during execution.Errors requiring knowledge of the task’s intended object or tool remain near chance.Source: §6 · p. 10
  • HELPMEOUT reframes household assistance as online error monitoring rather than goal inference.The task asks whether, when, and what kind of error occurs while a task unfolds.Source: §6 · p. 10
  • The benchmark pairs correct executions with neuropsychologically grounded errorful ones to expose a gap left by goal-inference and task-completion benchmarks.The authors hope it serves as a testbed for assistive agents that catch execution breakdowns before a task fails.Source: §6 · p. 10

Neuropsychological grounding of the error taxonomy

HELPMEOUT grounds its taxonomy and generator in clinical research on everyday action, then represents household tasks in a symbolic domain of objects, actions, and subgoals. Its taxonomy groups errors by whether they disrupt task progression or current action selection.

Neuropsychological grounding of the error taxonomy

  • The taxonomy separates progression-level errors that disrupt subgoal structure from current-action errors that disrupt selection of an object, tool, location, or action.This organizing axis differs from the generator’s goal-activation versus executive-control factorization.Source: Figure 4 · p. 15, §A.1 · p. 15

A.1 Neuropsychology of Everyday Action

  • Naturalistic Action Test and Multi-Level Action Test tasks use standardized categories including omissions, sequence errors, substitutions, additions, and perseverations, establishing recurrent structured failures across clinical populations.The Goal-Control account separates goal-activation failures from executive-control failures, while the taxonomy groups errors by what they disrupt.Source: §A.1 · p. 15

A.2 Object Hierarchy and Symbolic Domain

  • The symbolic household domain contains 187 object types: 156 movable types across four classes and 31 non-movable locations.Its state includes spatial relations, object-state attributes, and type predicates for expressing identity constraints.Source: §A.2 · pp. 16–17

A.3 Action Operators

  • The domain defines 38 PDDL action schemas with preconditions and effects; tool and location constraints make violations such as stirring without a spoon identifiable as tool substitutions.Examples include knife-required slicing, iron-required ironing, and heating limited to a microwave, oven, or stove.Source: §A.3 · pp. 17–18

A.4 Task Templates

  • HELPMEOUT covers 50 ADL/IADL task templates across eight families, anchored in clinical assessments, functional scales, or occupational-therapy frameworks.Templates express first-order subgoals and carry simple, medium, or complex tiers based on accomplishment steps.Source: §A.4 · p. 18, Table 7 · p. 18

A.5 Error Taxonomy

The nine error categories distinguish disruptions to task progression from errors in current action selection. Onsets are labelled when the error becomes recognizable from the observed prefix and task, without requiring the reference execution.

A.5 Error Taxonomy

  • Progression-level errors comprise omission, sequence, anticipation, and perseveration; current-action errors comprise object and tool substitution, spatial misreach, action addition, and capture.The taxonomy pairs correct and errorful micro-examples for all nine categories.Source: §A.5 · p. 20, Figure 6 · p. 21
  • Onsets are labelled from the observed prefix and task; harmless immediate repetition without added effect and movement between task-relevant places are not errors.A task-end event may be required to confirm an omission.Source: §A.5 · p. 20
  • An omission deletes an entire subgoal block and is confirmed at task end unless an earlier observable consequence makes it an anticipation.The omitted step is then not counted as an omission, following MLAT coding.Source: §A.5 · pp. 20–21
  • Sequence errors violate a required order, while anticipation commits an item to its final place before required processing is complete.Other orders that still reach the goal count as correct; anticipation begins at the committing step.Source: §A.5 · p. 21
  • Perseveration repeats a one-time action after an intervening action, and object substitution uses a confusable same-subclass object instead of the required one.Each onset is the repeated action or first use of the substitute, respectively.Source: §A.5 · p. 21
  • Current-action errors include using a confusable substitute tool, adding an unrelated item to the target container, acting on an unrelated object, or leaving an unused storage location empty-handed.Their onsets are the first step using the substitute, adding the item, applying the action to the unrelated object, or leaving the location, respectively.Source: §A.5 · pp. 21–22

A.6 Error-Generation Model

The generator makes goal-activation and commission decisions at each reference subgoal, subject to category feasibility, and calibrates the released error distribution to clinical targets. Validation preserves physical executability while relaxing the semantic constraint that defines substitution errors.

A.6 Error-Generation Model

  • At each reference subgoal, the generator drops the block as an omission when activation falls below threshold; otherwise it may inject a feasible commission category using category weights.Feasibility depends on the scene or trajectory, so observed category frequencies reflect both sampling weights and available configurations.Source: §A.6 · p. 22, Table 9 · p. 23
  • The released error distribution is within 2.3 percentage points of clinical targets for every category except action addition, at 8.4% versus 2.8%.Episodes were selected by category from a larger generated pool because feasibility gates unevenly constrain raw category frequencies.Source: §A.6 · p. 22, Figure 7 · p. 23

A.7 Dual-Standard Validation

  • Seven error categories undergo strict PDDL replay, while tool and object substitutions retain physical preconditions but relax the semantic type constraint defining the substitution.The substituted tool or object is drawn from a hierarchy-constrained confusable set.Source: §A.7 · pp. 23–24

B.1 Corpus Overview

  • Figure 8 summarizes corpus episode composition, task coverage, mean trajectory length, injected errors per trajectory, annotated onsets, and frequent error categories.The caption identifies these summary dimensions without specifying their values.Source: Figure 8 · p. 24, §B.1 · p. 24

B.2 Data Split

The corpus is split into training, validation, and test sets, with a smaller shared test subset used for the more expensive LLM and human evaluations.

B.2 Data Split

  • 20,000 episodes across 50 templates are split into 17,478 training, 2,022 validation, and 500 test episodes.The 60-episode LLM and human subset contains 30 errorful and 30 correct episodes; other baselines are also re-scored on it for comparison.Source: §B.2 · p. 24
  • The 500-episode test set covers all 50 templates and stratifies errorful episodes by category in proportion to the full corpus.The shared subset covers every error category, with three to four errorful episodes per category and correct episodes drawn from different templates.Source: §B.2 · p. 24, Table 10 · p. 25

C.1 Learned Baselines

The section describes learned, change-point, and symbolic monitors, then reports weaknesses in their error detection and diagnosis. The re-planner’s strong trajectory-level score is largely driven by its task-end check.

C.1 Learned Baselines

  • Seq2Seq encodes the task, initial scene, and observed action prefix; Seq2Seq+SG adds a graph of symbolic object and spatial state, while sharing the multi-head prediction setup.The graph has no geometry, and both supervised baselines are reference implementations rather than new model contributions.Source: §C.1 · p. 25
  • Learned monitors frequently name an error category in correct episodes, reducing macro-F1 to 0.50–0.55, while object and tool substitution remain at or below 0.20 F1.Omission and sequence are the best-diagnosed categories (F1 ≥0.74); task-name-only category predictions reach at most 0.20 on errorful-only scoring.Source: §C.1 · pp. 25–26, Table 11 · p. 26
  • SGT reaches step-level AUPRC of 0.814 for sequence errors, 0.771 for spatial misreach, and 0.502 for perseveration, but substitution and action addition remain near chance across learned monitors.For object substitution, tool substitution, and action addition, AUPRC ranges are 0.012–0.038, 0.003–0.020, and 0.005–0.029, respectively.Source: §C.1 · p. 26, Table 12 · p. 26

C.2 Scene-Graph Transformer and CUSUM

  • SGT conditions each step on the replayed scene graph before the current action, while CUSUM flags accumulated surprise in task-conditioned action bigrams and makes no type prediction.The SGT’s causal masking was verified: changing later actions changed outputs at prior steps by at most 2.9 × 10^-6.Source: §C.2 · p. 27

C.3 Symbolic Baselines

  • The precondition checker flags only errors detectable through the relaxed substitution precondition and alarms on fewer than 5% of errorful test episodes.The re-planner additionally uses a template-level goal, but not the episode’s grounded goal; 60 test episodes timed out and were excluded from M1 and M3.Source: §C.3 · p. 27
  • The re-planner reaches M1 0.887, but its score falls to 0.669 without the task-end check; it alarms on 62% of correct episodes because correct executions need not follow shortest plans.Its pre-task-end AUPRC is 0.118, and its weakest categories include spatial misreach and action addition, which can leave the goal satisfied.Source: §C.3 · p. 27

C.4 Additional Results

The additional results report that learned-monitor performance varies with task complexity. The LLM protocol evaluates one observed prefix at a time, requiring current-step error judgments and cumulative error-type probabilities.

C.4 Additional Results

  • SGT M1 falls from 0.844 on simple templates to 0.573 on complex templates, while the re-planner does not degrade with complexity because its end-of-task goal check does not depend on task length.Tables 13–15 provide confidence intervals, complexity breakdowns, and first-alarm distributions.Source: §C.4 · p. 28

C.5 LLM Baselines

  • Each LLM request receives the task name, initial scene, and actions through the current step, then returns a current-step error probability and cumulative probabilities for nine error categories.The protocol excludes future actions, trajectory length, formal goal conditions, reference executions, and labels; all 1,201 prompts per configuration were automatically checked.Source: §C.5 · pp. 28–29
  • LLM calls use temperature 0 with provider and reasoning settings pinned, and all thirteen configurations completed every step at a total cost of US$107.Calls were retried under specified timeout and parsing conditions, and responses, reasoning text when returned, token counts, costs, and serving providers were logged.Source: §C.5 · p. 29
  • The prompt instructs models to treat reasonable but inefficient actions and harmless repetition as correct, and to assign current-step scores only where an error occurs.It specifies that category probabilities remain high after an error has occurred, including at the final task-end step.Source: §C.5 · p. 29

C.6 Full LLM Results

LLM configurations are not reliably rankable on the 60-episode subset, and even their best upper confidence bounds remain below the human reference. Expert review offers initial support for generator fidelity, not patient-level validation.

C.6 Full LLM Results

  • The best LLM upper bounds remain below the human reference on M1 (0.837 vs. 0.967) and M2 (0.482 vs. 0.748), while overlapping intervals preclude ranking configurations.The results are reported on the 60-episode subset; Table 17 gives the configurations and intervals, and Table 18 gives first-alarm distributions.Source: §C.6 · p. 30, Table 17 · p. 31, Table 18 · p. 32
  • With reasoning enabled, larger models reach M2-mid 0.18–0.48 and M4 0.43–0.52; without reasoning, M2-mid is at most 0.15 and M4 at most 0.31.The best reasoning configurations are on par with SGT before task end (0.478 and 0.451 versus 0.455), but M2-end is at most 0.22.Source: §C.6 · p. 30
  • GPT-5.5 low and medium are indistinguishable on every metric, with M2 0.238 versus 0.231, while medium costs 65% more.The LLM results do not support ranking configurations because their intervals overlap.Source: §C.6 · p. 30

C.7 Expert Review

  • Two domain experts judged 10 of 12 injected errors plausible and identified 11 of 12 correct episodes as correct, but this review is not patient-level validation.They matched the generator’s error category on 8 of 12 errorful episodes; further expert validation is planned for release on the project website.Source: §C.7 · p. 32

Related papers

Loading 2610.17694v1…