PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoTaskQA: Understanding Human Tasks in Egocentric Videos

Baoxiong Jia, Ting Lei, Song-Chun Zhu, Siyuan Huang

arXiv:2210.03929v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Action localization and future prediction provide indirect measures of task understanding, while detailed action effects, dependencies, goals, and beliefs remain challenging to assess in egocentric videos. EgoTaskQA extends LEMMA with annotations and structured question-answering across these dimensions. Evaluated models show significant gaps from humans, and the benchmark is currently limited to indoor goal-oriented tasks.

  • Problem

    Action localization and future prediction are indirect measures, while detailed task understanding involves action effects and dependencies, goals, and beliefs in partially observed multi-agent activity.

    Source: §1 · p. 2

  • Method

    EgoTaskQA extends LEMMA with annotations of object states, relationships, causal dependencies, and agents’ beliefs, then generates questions across four reasoning types.

    Source: §1 · p. 2

  • Results

    Models show significant performance gaps compared with humans on EgoTaskQA’s challenging normal and indirect splits.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    EgoTaskQA provides a direct question-answering benchmark for studying goal-oriented task understanding in egocentric videos.

    Source: §1 · p. 2, §5 · p. 10

    The activities are limited to indoor goal-oriented tasks, and the evaluation discussion covers six state-of-the-art models.

    Source: §5 · p. 10

Abstract

from arXiv · show

Understanding human tasks through video observations is an essential capability of intelligent agents. The challenges of such capability lie in the difficulty of generating a detailed understanding of situated actions, their effects on object states (i.e., state changes), and their causal dependencies. These challenges are further aggravated by the natural parallelism from multi-tasking and partial observations in multi-agent collaboration. Most prior works leverage action localization or future prediction as an indirect metric for evaluating such task understanding from videos. To make a direct evaluation, we introduce the EgoTaskQA benchmark that provides a single home for the crucial dimensions of task understanding through question-answering on real-world egocentric videos. We meticulously design questions that target the understanding of (1) action dependencies and effects, (2) intents and goals, and (3) agents' beliefs about others. These questions are divided into four types, including descriptive (what status?), predictive (what will?), explanatory (what caused?), and counterfactual (what if?) to provide diagnostic analyses on spatial, temporal, and causal understandings of goal-oriented tasks. We evaluate state-of-the-art video reasoning models on our benchmark and show their significant gaps between humans in understanding complex goal-oriented egocentric videos. We hope this effort will drive the vision community to move onward with goal-oriented video understanding and reasoning.

1 Introduction

Egocentric videos capture goal-oriented behavior under partial observability, but existing work emphasizes action recognition and future anticipation rather than deeper task understanding. EgoTaskQA addresses this gap with questions about actions and their effects, agents’ goals, and beliefs, and reports large model–human performance gaps.

1 Introduction

  • Egocentric activity understanding still lacks detailed task annotations for action dependencies and effects, as well as multi-agent belief modeling.Large lifelog datasets are unsuitable for detailed annotation, while goal-oriented datasets do not address these dimensions.Source: §1 · p. 2
  • EgoTaskQA extends LEMMA with object status, human-object and multi-agent relationships, and causal dependencies between actions.The benchmark uses these annotations to provide ground truth for procedural task understanding.Source: §1 · p. 2
  • The benchmark targets action dependencies and effects, agents’ intents and goals, and beliefs about others through descriptive, predictive, explanatory, and counterfactual questions.Questions use direct and indirect references, and the benchmark includes balanced answers and challenging splits.Source: §1 · pp. 2–3
  • Models show large performance gaps compared with humans on EgoTaskQA.The benchmark complements existing video reasoning benchmarks and includes diagnostic experiments on easier and more challenging areas.Source: §1 · p. 2

2 Related Work

Prior work addresses action understanding, egocentric perception, and video question answering, but often covers atomic actions or narrower reasoning tasks. The reviewed datasets and benchmarks leave goal-oriented task understanding in real-world egocentric, multi-agent settings under-addressed.

Action as Inverse Planning

  • Inverse-planning work links actions to mental states, preconditions, and post-effects, but much of it remains limited to atomic actions.Instructional videos address multi-step activities but omit the task-level noise and partial observability of multi-agent settings.Source: §2 · p. 3

Egocentric Vision

  • Egocentric datasets provide information about interactions and social understanding, yet detailed annotations for human tasks remain largely missing.Existing goal-oriented activity datasets annotate tasks and compositional actions but do not address action dependencies, effects, or multi-agent belief modeling.Source: §2 · pp. 3–4

Video Question-Answering Benchmarks

  • Video question-answering benchmarks cover spatial, temporal, causal, compositional, and social reasoning, but not the full dimensions of goal-oriented activity understanding from real-world egocentric video.The reviewed benchmarks include synthetic and crowdsourced video settings as well as short atomic-action clips.Source: §2 · p. 4

3 The EgoTaskQA Benchmark

EgoTaskQA builds on LEMMA with annotations of object states, relationships, and agents’ beliefs, then uses them to construct questions about goal-oriented tasks. Its question generation and benchmark splits target descriptive, predictive, explanatory, and counterfactual reasoning, including multi-step indirect references.

3 The EgoTaskQA Benchmark

  • EgoTaskQA contains 40K balanced question-answer pairs selected from 368K generated questions over 2K egocentric videos.It targets action effects and dependencies, intent and goals, and multi-agent belief modeling.Source: §3 · p. 4

3.1 Data Collection

  • LEMMA videos provide goal-oriented, multi-tasked activities with rich human-object interactions and action dependencies in single-agent and two-agent collaboration settings.The dataset is augmented with ground truths for object states, relationships, and agents’ beliefs about others.Source: §3.1 · p. 4
  • Annotations cover 48 relationships and 14 object attributes, with before-and-after states used to reconstruct temporal status for interacted objects.For causal dependency, an action’s post-effect is checked against another action’s preconditions; the resulting dependency structure is used for explanatory and counterfactual question generation.Source: §3.1 · pp. 4–5
  • The benchmark annotates object visibility to the helper and whether the actor sees or is aware of the helper’s actions.Actor awareness is determined by a majority vote of three workers.Source: §3.1 · pp. 4–5

3.2 Question-Answer Generation

  • Questions target action preconditions, post-effects, and dependencies, agents’ intents and goals, and beliefs about others across four reasoning types.Descriptive questions query annotated intervals; predictive questions use future annotations; explanatory questions query causes and preconditions; counterfactual questions alter actions in causal traces.Source: §3.2 · pp. 5–6
  • Executable programs query annotations and causal traces to generate answers, with indirect references added to actions and objects.The initial generation process produces 368K question-answer pairs over 2K videos.Source: §3.2 · pp. 6–7
  • Answer balancing limits frequent open-answer responses within each reasoning type and produces a 40K set with a 1:2 ratio of binary to open-answer questions.Binary questions are balanced to equal proportions of each answer within each reasoning type.Source: §3.2 · p. 7
  • The indirect split tests multi-step indirect-reference questions in validation and test, while its training set excludes questions without indirect references and simple indirect references.The normal split is randomly sampled by answer distribution and reasoning type into a 3:1:1 train-validation-test split.Source: §3.2 · p. 7

4 Experiments

The experiments compare six video question-answering models across EgoTaskQA settings and examine how language cues and indirect references affect performance. Models remain well below humans, while results expose both useful textual task associations and challenges in multi-agent and indirect-reference reasoning.

4 Experiments

  • The evaluation measures answer accuracy for six state-of-the-art video question-answering models across question scopes, types, targeting semantics, and answer categories.Source: §4 · p. 7

4.1 Comparative Analysis

  • HME, HCRN, and ClipBERT perform best, yet models trail humans substantially; multi-agent questions are weakest, while state and changing-attribute questions fare relatively well.Source: Table 2 · p. 8, §4.1 · p. 8

4.2 The Effectiveness of Language

  • Reducing object-information obfuscation significantly improves performance across models, suggesting fine-grained action text conveys task-related knowledge.The comparison replaces “something” with the original object in action references.Source: §4.2 · p. 8, Figure 4 · p. 9
  • Removing vision lowers performance in most categories, especially object questions; text-only models nevertheless gain slightly on object-state-change questions through action-attribute associations.Examples include cleanliness with wash, emptiness with pour, and shape with cut.Source: §4.2 · pp. 8–9, Table 3 · p. 9

4.3 Generalizing to indirect references

  • On the indirect split, BERT and language-only HCRN drop 20.8% and 26.3% on “change”; most models also decline, motivating stronger spatial-temporal reasoning for indirect references.Source: Table 4 · p. 9, §4.3 · pp. 9–10

5 Conclusions

The conclusion presents EgoTaskQA as a benchmark for evaluating goal-oriented activity understanding in egocentric video, with annotations and questions spanning states, relationships, actions, and beliefs. The authors report model gaps relative to humans on two challenging splits and identify scope limits and future directions.

5 Conclusions

  • EgoTaskQA evaluates goal-oriented egocentric video understanding with questions covering action dependencies, effects, intents, goals, and belief modeling.The authors report gaps between state-of-the-art models and humans on the normal and indirect splits.Source: §5 · p. 10
  • The benchmark is built on LEMMA, with participant consent and safeguards against personally identifiable information and subjective comments.Videos, annotations, metadata, and question-answer pairs are publicly available under a noncommercial CC BY-NC-SA license.Source: §5 · p. 10
  • The benchmark is limited to indoor goal-oriented tasks, and the study compares six state-of-the-art models.The authors leave model-design exploration to future work and plan broader activity curation.Source: §5 · p. 10
  • Future work targets explicit spatial-temporal grounding for modularized video QA and prompting large-scale pretrained models for domain-specific challenges.The authors note that directly adapting pretrained models to a specific domain is non-trivial.Source: §5 · p. 10
  • The authors see potential applications in imitation learning and knowledge acquisition for assistive robots that coordinate with humans.They also connect the augmented LEMMA dataset to world-model learning and real-world complex-event understanding.Source: §5 · p. 10

A.1 Data Statistics

The data statistics describe the annotated objects, state changes, and relationships used to characterize goal-oriented activity. The annotations span object-state transitions and spatial, action-related, and multi-agent relations.

A.1 Data Statistics

  • Annotations cover 30K before-after object-state and spatial-relationship pairs across 10K LEMMA action segments.The object set starts from semantic-position objects and excludes temporally static objects, which remain general references in action descriptions.Source: §A.1 · p. 18
  • Spatial object relationships are the most frequent annotations, followed by multi-agent relationships such as awareness and looking at others.Action-related relations also form a considerable share, describing links to target objects and tools.Source: §A.1 · p. 18
  • The annotation vocabulary includes time-varying object attributes such as visibility, affordance, and task-dependent status, with an unknown option for unclear cases.Unknown answers are excluded from question generation; the possible attribute values are listed in Table 5.Source: §A.1 · p. 18, Table 5 · p. 18
  • Attribute changes increase in frequency from affordance to visibility to task-dependent status, while indoor recordings yield many containment changes such as opening and emptiness.The paper identifies containment relationships as a potentially useful subject for study.Source: §A.1 · p. 18

A.2 Causal Dependency

The causal-dependency procedure compares pairs of actions through their shared interactive objects and annotated attribute transitions. It distinguishes dependent, related-but-uncertain, and unrelated pairs, then constructs dependency trees from dependent pairs.

A.2 Causal Dependency

  • Two actions are considered causally dependent when they share an interactive object and the first action’s attribute change leaves the status that the second action requires as its precondition.The criterion checks whether the attribute changed in both actions and whether the status after the first equals the status before the second.Source: §A.2 · pp. 18–19
  • Actions with no shared interactive objects are treated as unrelated; shared-object pairs failing the dependency test may instead be labeled related.The related label applies when the first action changes an attribute that does not change during the second, so the annotations cannot establish causality.Source: §A.2 · pp. 18–19
  • Question generation is restricted to dependent and unrelated pairs because related pairs have unverified causal status, and dependent actions are recursively organized into dependency trees.The procedure adds actions dependent on a root action or its dependants to that root’s dependant set.Source: §A.2 · pp. 19–20

B.1 Preprocessing Annotations

The preprocessing step forms video intervals for answer generation and expands them for predictive questions using upcoming actions.

B.1 Preprocessing Annotations

  • Intervals average four to five actions, and predictive-question intervals append the actor’s next three actions, including actions unseen in the video.The corresponding actor and helper actions are gathered with their annotations in a dictionary.Source: §B.1 · p. 20

B.2 Operator and Program Design

EgoTaskQA generates questions by executing templates over annotation dictionaries, with operators for conditional filtering and attribute queries. Indirect references substitute action or object parameters, but are limited to one step.

B.2 Operator and Program Design

  • Nine operators compose template logic over annotation dictionaries, using conditional queries to filter by an attribute value or retrieve an attribute value.The paper denotes filtering with A@B and querying attribute A with A$.Source: §B.2 · p. 20, Table 6 · p. 20
  • Program templates are instantiated with sampled actions, objects, attributes, and values, then exhaustively searched to produce 368K executable questions with answers.A non-executable intermediate operator stops the trial and prompts a new attempt.Source: §B.2 · p. 20
  • Indirect references replace action or object parameters with temporal or status-change references, and the benchmark limits them to one step for understandability.Examples include the action before or after another action and the first object whose status changes.Source: §B.2 · pp. 20, 22

B.3 Question-answer statistics and balancing

The benchmark balances generated question-answer pairs by reasoning type and answer distribution, while its templates span descriptive, predictive, explanatory, and counterfactual questions.

B.3 Question-answer statistics and balancing

  • Balancing yields 40K diverse question-answer pairs, with the top 20% of answers capped at 33% of questions per reasoning type and open-answer to binary questions held at 2:1.The answer cap is intended to smooth answer distributions without deleting too many questions.Source: §B.3 · p. 22, Table 7 · p. 22
  • Question templates cover descriptive, predictive, explanatory, and counterfactual reasoning across world-state, intent, action, and multi-agent scopes.Examples include checking visibility or awareness, anticipating a status change, explaining action effects, and testing whether actions remain executable counterfactually.Source: Table 8
  • Indirect-reference templates query actions before or after another action, extending the benchmark beyond direct action and object references.The templates also encode first or last actions and objects whose status changes.Source: §B.3 · p. 23, Table 9

C Experiment

The experiments evaluate video-question-answering baselines using model-specific implementations and training configurations, with all experiments run on a single NVIDIA A100 GPU.

C Experiment

  • The evaluated baselines include VisualBERT, HGA, HME, PSAC, HCRN, and ClipBERT, with model-specific visual and text feature pipelines.VisualBERT uses 20 sampled frames with ResNet features; other baselines use combinations of VGG, C3D, ResNeXt, or Glove embeddings.Source: C Experiment · p. 24
  • Training configurations vary by baseline, including learning rates from 2 × 10^-5 to 1 × 10^-3 and training durations from 25 to 100 epochs.The authors report reducing ClipBERT's learning rate after observing gradient NaNs and Infs.Source: C Experiment · p. 24
  • All experiments run on a single NVIDIA A100 (80G) GPU, and the authors provide code, checkpoints, and reproduction instructions.Training information and hyperparameters are listed for each model.Source: C Experiment · p. 24, Table 10 · p. 24

D Data Documentation

The documentation describes EgoTaskQA’s data, collection, and intended uses, along with known annotation errors and the absence of formal review processes.

D Data Documentation

  • The benchmark was created to study goal-oriented task understanding because prior work lacked task-related object-state and relationship annotations and a suitable evaluation metric.Question-answer pairs are generated programmatically from LEMMA videos, with AMT workers augmenting the original annotations.Source: D Data Documentation · pp. 25–26
  • EgoTaskQA comprises 40K question-answer pairs over 2K video intervals cropped from LEMMA [11], whose clips record daily indoor activities.Each question-answer instance includes question text, a video interval, scope, type, target answer semantics, and a program.Source: D Data Documentation · p. 25
  • The authors acknowledge that some video annotations and question-answer pairs almost certainly contain errors despite efforts to minimize them.They report that no formal review processes were conducted for data collection or annotation.Source: D Data Documentation · pp. 25–26
  • The dataset is self-contained, will be publicly available through the authors’ website, and is released under a CC BY-NC-SA2 license.The documentation says older versions will be maintained and errors may be submitted by email.Source: D Data Documentation · pp. 25, 27, 28
  • The dataset includes human activity recordings, and actors in the original LEMMA videos can be recognized.The documentation reports no identified subpopulations and answers no to whether the data are sensitive.Source: D Data Documentation · pp. 25–26
  • The annotated videos may support world-model learning, while generated question-answer pairs may support evaluation of compositional reasoning.The authors propose before-and-after object-status annotation as a general protocol for changing world states.Source: D Data Documentation · p. 27
  • The authors limit intended use to activity or task understanding and related downstream tasks such as action recognition, anticipation, state or relationship recognition, and question answering.Source: D Data Documentation · p. 27
Loading 2210.03929v1…