PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AI

Junlong Li, Junxi Li, Jianjun Gao, Chen Cai, Lap-Pui Chau, Yi Wang

arXiv:2608.24134v1cs.CV

TL;DR

EgoErrorVQA addresses the limited evaluation of egocentric procedural comprehension, especially procedural error detection. It introduces a VQA benchmark, an A2A evaluator, and Ego-ADR for decoupled reasoning. Evaluations reveal persistent model weaknesses, while Ego-ADR achieves state-of-the-art results under comparable conditions.

  • Problem

    Existing benchmarks overlook egocentric procedural comprehension and explicit procedural error detection, despite the importance of recognizing failures in interdependent everyday tasks.

  • Method

    EgoErrorVQA combines open-ended and multiple-choice VQA with an A2A evaluator agent, while Ego-ADR adaptively decouples procedural reasoning.

  • Results

    Agents still struggle with procedural errors and error types, while Ego-ADR achieves state-of-the-art performance on key metrics under comparable conditions.

  • Takeaways & Limitations

    The benchmark provides a standardized setting for studying egocentric procedural error understanding and developing procedural agent skills.

  • Takeaways & Limitations

    The benchmark has unequal source-dataset contributions, no training data, and class imbalance that can reduce open-ended score differentiation and complicate metric interpretation.

Abstract

from arXiv · show

The majority of our everyday activities are procedural and consist of sequences of interdependent steps. However, existing benchmarks for Visual Agents and Visual Language Models (VLMs) overlook the evaluation of their procedural comprehension ability from an egocentric visual perspective, particularly for detecting procedural errors, a critical capability for everyday assistance. To bridge this gap, the EgoErrorVQA task is firstly proposed for egocentric procedural comprehension with explicit procedural errors modeling. Besides, we develop a user-friendly evaluator agent based on the Agent2Agent (A2A) protocol, enabling rigorous and standardized evaluation of visual agents through VQA-based interaction. A range of models are evaluated using both open-ended and multiple-choice questions, revealing persistent weaknesses in handling procedural errors and error types. Moreover, we introduce Ego-ADR, an Adaptive Decoupled Reasoning framework that decouples complex procedural reasoning to enhance models' understanding of procedural errors. It achieves consistent performance gains over the selected baselines and attains state-of-the-art results on several metrics under comparable settings. Code: https://github.com/z1oong/EgoErrorVQA

1 Introduction

EgoErrorVQA targets under-evaluated egocentric procedural understanding by modeling procedural errors across diverse scenarios. It combines VQA-based evaluation, an A2A evaluator agent, and Ego-ADR reasoning decoupling, while experiments show models lag behind humans.

  • Egocentric procedural understanding requires step-level comprehension and temporal reasoning over interdependent actions under ordering constraints.
  • Robust assistance also requires detecting omissions, out-of-order steps, wrong-object usage, and redundant actions.
  • EgoErrorVQA formulates procedural error detection and classification as an egocentric VQA task with an eight-category error taxonomy.
  • EgoErrorVQA-E uses fixed interfaces and A2A interaction to provide open-ended and multiple-choice questions and automate reproducible evaluation.
  • EgoErrorVQA benchmarks procedural error detection and classification across diverse scenarios, with experiments showing current models lag behind humans.
  • Ego-ADR decouples procedural understanding and achieves state-of-the-art performance on key metrics under comparable conditions.

2 Related Work

Prior egocentric video benchmarks mainly assess recognition, simple questions, or limited reasoning, while procedural error detection is emerging as a distinct challenge. EgoErrorVQA systematizes errors across everyday scenarios and evaluates them through VQA interactions.

  • Existing egocentric video benchmarks typically focus on action recognition, object identification, or simple questions with short-term reasoning demands.
  • Some prior work addresses task planning, which probes more complex reasoning and memory than basic recognition tasks.
  • Egocentric procedural error detection is positioned as important for AI assistants supporting daily activities and blind people.
  • EgoErrorVQA systematizes procedural errors across everyday scenarios with an error taxonomy and evaluates understanding through user-like VQA interactions.

3 Open-end VQA in EgoErrorVQA

The open-ended evaluation uses procedure-guided, targeted questions to test whether agents understand specific actions and procedural errors. Its scoring uses LLM-based semantic similarity, while QA construction combines language-model generation with human review.

  • Open-ended VQA evaluates agents using procedures that describe complete workflows and questions probing whether specific steps and their ordering are appropriate.
  • Targeted and deliberately confounding questions reduce reliance on generic judgments and better probe stepwise procedural reasoning.
  • QA-pair generation combines Qwen2.5-7B-Instruct outputs with review by three human annotators and additional manually authored questions.
  • EgoErrorVQA contains 3,560 QA-pairs covering 1,805 samples, with approximately 2,749 LLM-generated and 811 manually authored pairs.
  • Two open-source LLM judges score semantic similarity on a 0–5 scale, and their average is used as the final Sim. score.

4 Multiple-choice VQA in EgoErrorVQA

The multiple-choice evaluation targets error-category recognition with procedures, definitions, and action labels that limit irrelevant distraction. It supports quantitative comparison through standard classification metrics and a taxonomy of eight error classes.

  • The benchmark defines eight classes: Wrong Object, Wrong Action, Wrong Order, Omission, Unintended and Unnecessary Action, Correct Wrong Action, Equipment Failure, and Others.
  • Multiple-choice VQA evaluates whether an error occurs at a specified step and, when present, identifies its type.
  • The setting provides task-specific procedures, error definitions, and action labels to focus agents on procedural deviations.
  • Precision, Accuracy, Recall, F1 Score, confusion matrices, and per-error-type accuracy are used to assess performance and diagnose deficiencies.

5 Ego-ADR

Ego-ADR adaptively decouples egocentric procedural reasoning into stages that separately match procedural steps, narrate video, and classify errors. Its deep and shallow variants are selected according to models’ text-only reasoning capabilities.

  • Ego-ADR targets zero-shot procedural understanding because scenario-specific training may hinder transfer across contexts.
  • The framework factorizes reasoning into key-step matching, video narration, and error classification to reduce the burden of single-pass inference.
  • Deep decoupling: Deep decoupling uses text-only reasoning to retrieve nearby procedural steps, prune redundant context, and support subsequent visual description and error judgment.
  • Shallow decoupling: Shallow decoupling uses TF-IDF cosine matching, Ratcliff/Obershelp fallback below 0.25 similarity, and model-generated video narration when text-only reasoning is unavailable.

6 Experiment

Experiments evaluate closed- and open-source video models and egocentric agents with open-ended and multiple-choice VQA. Models remain substantially weaker than humans, while adaptive decoupling improves error-classification performance across tested models.

  • Benchmark setup: The evaluation includes three closed-source models, three open-source video VLMs, and two egocentric visual agents.
  • Open-end VQA: Open-ended VQA peaks at Avg-Sim. 3.33 for Qwen2-VL-7B-Instruct, versus 3.77 for humans, and more frames alone provide no substantial gains.
  • Multiple-choice VQA: Multiple-choice performance remains below humans: the best accuracy is 66.4% for EgoGPT, while the best precision and F1 are 17.5% for GPT-4o and 20.8% for GPT-4o.
  • Error types: Models perform relatively better on Wrong Action, Wrong Object, and Wrong Order, but struggle with Omission and Unintended Action.
  • Metric interpretation: Accuracy, recall, precision, and F1 must be interpreted jointly because class imbalance and over-labeling can make individual metrics misleading.
  • Ego-ADR results: Ego-ADR improves Precision by 23.2%, 10.6%, and 7.7% for Qwen2-VL, Qwen2.5-VL, and Video-LLaVA, respectively.F1 improvements are 19.7%, 8.1%, and 30.1% for the same models.
  • Ablation studies: Adaptive decoupling is necessary: forcing unsuitable text-only reasoning degrades performance, whereas model capabilities determine whether deep or shallow decoupling is appropriate.
  • Ablation studies: Removing video narration reduces recall by 69.7% and F1 by 16.7%, while Ego-ADR outperforms Chain-of-Thought across all three models.

7 Conclusion

The paper introduces EgoErrorVQA as a benchmark for egocentric procedural error detection and classification, with open-ended and multiple-choice evaluations. It also provides an evaluator agent and Ego-ADR, while reporting that agents continue to struggle with the task.

  • EgoErrorVQA is a VQA benchmark designed to evaluate egocentric procedural error detection and classification in agents and VLMs.
  • The benchmark combines open-ended and multiple-choice evaluations to assess procedural-error understanding from complementary perspectives.
  • Its evaluator agent enables automated, dialogue-based assessment intended to simulate a realistic question-answering assistance scenario.
  • The reported results show that agents still struggle with procedural errors, motivating Ego-ADR’s decoupling of complex procedural reasoning.

Limitations

The benchmark has dataset-balance and evaluation-scope limitations. Correct samples dominate, open-ended score differentiation is reduced, and multiple-choice metrics require cautious interpretation; some clips also require fine-grained recognition.

  • The four source datasets differ in size, so their contributions to the benchmark are not exactly equal.
  • The benchmark contains evaluation data but no training data, although the datasets retain samples for possible future training-set expansion.
  • Correct samples dominate because the source datasets contain more correct than incorrect examples, reducing open-ended score differentiation among models.
  • Accuracy and related multiple-choice metrics should not be interpreted in isolation, and some clips require fine-grained action or object recognition.
  • Ego-ADR is evaluated only on multiple-choice VQA because its quantitative metrics more clearly demonstrate gains and avoid open-ended answer-creativity misjudgments.

Ethical Considerations

The dataset is built from licensed egocentric procedural datasets, with documented ethical safeguards and a controlled, text-based QA-generation process. Its evaluator uses A2A message-based communication to support reproducible end-to-end assessment.

  • Ethical safeguards: The source datasets are licensed for research use, comply with ethical standards, and contain no violent, illicit, harmful, or private content.Annotators were authors, were compensated appropriately, and the dataset uses the Apache License 2.0.
  • Dataset construction: Procedural guidelines are condensed from annotated steps and their order, while source annotations are standardized into unified JSON formats with task identifiers.The guidelines support QA-pair generation for each task.
  • QA validation: The evaluator checks video–QA consistency, verifies error and correct-action labels, focuses on procedural correctness, and removes redundant or noninformative pairs.Questions target specific actions and attributes such as tools, ingredient quantities, and heating duration.
  • Evaluation infrastructure: A2A message-based communication lets researchers evaluate agents by connecting them to the evaluator without modifying existing white-agent code.The protocol supports evaluation across multiple benchmarks on public agent platforms.

C Details of Experiment and Human Performance Evaluation

Experiments use four egocentric procedural datasets and evaluate models and humans with matched video-question settings. Multiple-choice results show substantial model weaknesses, while human precision and F1 exceed model performance.

  • Experimental data: The benchmark draws from CaptainCook4D, EgoOops, Epic-Tent, and Assembly101, covering cooking, handicraft, tent setup, and toy-car assembly tasks.The selected datasets provide procedural tasks with explicit error annotations and action labels.
  • Human evaluation: Human evaluation samples 75 open-ended QA pairs and 100 multiple-choice samples, using the same information, videos, tasks, and scoring procedures as model evaluation.The results are presented in Table 7, Figure 7, and Figure 8.
  • Experimental settings: Models receive procedures and questions for open-ended VQA, while multiple-choice VQA additionally supplies action labels, video segments, and error-type definitions.Multiple-choice evaluation asks whether an error occurred and, if so, which category applies.
  • Results: Humans achieve the best performance on every multiple-choice metric except Recall, with the highest precision and F1 scores.The passage attributes the Recall exception partly to the small human sample, where a few missed errors can substantially lower Recall.
  • Results: Qwen’s high Recall mainly results from overclassifying samples as erroneous, which lowers precision and accuracy.The passage characterizes this as heightened sensitivity to procedural errors rather than an evaluation-framework defect.
  • Qualitative analysis: Qwen3-VL-8B-Instruct produces step-by-step reasoning that compares observed actions with the prescribed procedure and identifies ordering mistakes.Its example explains that stirring should occur after the initial microwaving period.

D Details of Error Type Classification

The benchmark uses a unified, fine-grained taxonomy to distinguish procedural error types across scenarios and evaluates both detection and exact classification. Its metrics and prompts emphasize matched error categories rather than binary error detection alone.

  • Taxonomy motivation: Existing datasets use scene-specific error labels, so EgoErrorVQA introduces a systematic taxonomy intended to cover potential errors across selected scenarios.The taxonomy addresses heterogeneous labels that complicate cross-scenario evaluation and method generalization.
  • Error categories: The eight error types include Wrong Object, Wrong Action, Wrong Order, Omission, Unintended and Unnecessary Action, Correct Wrong Action, Equipment Failure, and Others.The supplied passages define each category operationally, including corrections as acceptable but annotated events.
  • Fine-grained distinctions: Correct Wrong Action denotes an operator’s active correction of a prior mistake and is explicitly annotated as a correction event.The behavior is treated as acceptable while remaining distinct in the taxonomy.
  • Interpretation: Low model precision reflects the benchmark’s requirement for precise error-type identification, making it suitable for fine-grained and interpretable error analysis.The benchmark therefore evaluates more than binary detection.
  • Fine-grained distinctions: Wrong Order covers sequence violations, whereas Wrong Action covers incorrect execution within an otherwise relevant step.Examples include using an incorrect heating time or cutting an item into the wrong number of pieces.
  • Evaluation metrics: The benchmark reports Precision, Accuracy, Recall, F1 Score, confusion matrices, and per-error-type Accuracy to diagnose model deficiencies.Precision requires both error detection and a matched error type, while F1 combines Precision and Recall.
  • Evaluation prompts: Multiple-choice prompts provide the full procedure, current step, and error-type definition, then require a correct-versus-error decision and one error category when applicable.The evaluator compares observed actions against the full task procedure and expected sequence.

F Details for Caculate Cohen’s Kappa, Pearson and Spearman

The evaluation details describe deterministic step matching, decoupled visual reasoning prompts, and automated LLM judging, with reliability checked against human ratings. The procedures specify how expected steps are retrieved, errors are classified, and model answers are scored.

  • Human reliability: Three annotators rate sampled responses, use majority voting or discussion to resolve disagreements, and produce a unified human scoring standard.The sample contains 170 responses covering both judge models and four dataset scenarios.
  • Human reliability: Pearson correlations of 0.851 for Qwen and 0.781 for DeepSeek measure alignment between LLM scores and unified human ratings.These correlations support using the LLM judges as evaluation proxies.
  • Ego-ADR prompting: Ego-ADR first guides the model to identify the relevant step and provide a description before issuing a final procedural-error judgment.This decoupled reasoning format is presented as better aligned with procedural error detection and classification than standard CoT.
  • Step matching: Key-step matching retrieves the expected procedure step with highest TF-IDF cosine similarity, then falls back to character-level matching when similarity is below τ = 0.25.The deterministic matcher also returns expected objects, neighboring steps, and a confidence signal for downstream classification.
Loading 2608.24134v1…