PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions

Songruo Chen, Yuqian Fu, Yu Li, Mingkang Dong, Sifan Zhou, Bin Ren, Xu Zheng, Tianwen Qian, Mohamed Elhoseiny, Yanwei Fu

arXiv:2610.18499v1cs.CVcs.AI

TL;DR

Egocentric benchmarks often study capabilities, modalities, and evaluation formats separately, despite their connection in real-world interaction. EgoOmniWorld structures an evidence-grounded benchmark around the Ego–Action–World loop, and experiments report uneven model capabilities, in-domain improvement, and transfer to existing egocentric benchmarks.

  • Problem

    Egocentric benchmarks address diverse capabilities, modalities, and evaluation formats under separate task definitions and protocols, leaving no common framework for organizing their relationships.

    Source: §1 · p. 2

  • Method

    EgoOmniWorld organizes interactions as Node, Edge, Chain, and Loop structures and generates QA from reconstructed interactions grounded in visual and auditory evidence.

    Source: §1 · p. 2

  • Results

    Experiments find uneven model strengths across interaction structures, in-domain improvement after EgoOmniWorld training, and zero-shot transfer gains of up to 8.08 percentage points on existing egocentric benchmarks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    EgoOmniWorld serves as a unified diagnostic testbed and a source of transferable supervision for egocentric video understanding.

    Source: §1 · p. 3

    The dataset uses Ego4D and HD-EPIC, and training experiments primarily focus on the Qwen-Omni family.

    Source: §5 · p. 9

Abstract

from arXiv · show

Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their tight coupling in real-world interaction. We propose the Ego-Action-World interaction loop as a unifying principle: the wearer perceives the world, forms goals, acts, changes the world, and uses feedback to guide subsequent behavior. Based on this view, we introduce EgoOmniWorld, a benchmark organized around four complementary interaction structures: Node captures ego, action, and world states; Edge captures directed relations among them; Chain captures linked dependencies; and Loop captures feedback-closed interactions. We develop an interaction-first, evidence-grounded data pipeline that extracts structured interactions from egocentric videos, grounds them in visual and auditory evidence, and generates questions according to their interaction structure. Required modalities and single- or multi-turn formats follow naturally from the supporting evidence and dependencies. EgoOmniWorld contains 24,166 QA groups, including 20,284 training and 3,882 test samples. Experiments on representative open- and closed-source multimodal large language models reveal substantial and uneven limitations across interaction structures. Models trained on EgoOmniWorld improve in-domain interaction understanding and generalize to existing egocentric benchmarks. EgoOmniWorld provides a unified foundation for evaluating and advancing egocentric video understanding. Data, code, and models will be released.

1 INTRODUCTION

Egocentric benchmarks cover diverse capabilities, modalities, and evaluation formats but often study them separately; EgoOmniWorld organizes them around the closed Ego–Action–World interaction loop. Its evidence-grounded benchmark supports analysis of uneven model capabilities, in-domain training gains, and zero-shot transfer.

1 INTRODUCTION

  • EgoOmniWorld organizes egocentric understanding into Node states, directed Edge relations, linked Chain dependencies, and feedback-closed Loop interactions.These structures are complementary scopes of interaction, not a hierarchy of task difficulty; modality and dialogue format follow the evidence and dependencies involved.Source: §1 · p. 2
  • The interaction-first data engine reconstructs interactions and grounds them in evidence before generating and validating questions, producing 24,166 QA groups.The dataset includes 20,284 training and 3,882 test samples, with questions derived from local components, linked dependencies, or feedback-closed cycles.Source: §1 · p. 2
  • Training improves in-domain interaction understanding, with Qwen2.5-Omni-7B rising from 36.42% to 47.36% with LoRA and 48.34% with full fine-tuning.The experiments also find markedly different model strengths across interaction structures and semantic categories, which aggregate performance alone can obscure.Source: §1 · p. 2
  • Zero-shot transfer to EgoSound, EgoTempo, and EgoToM yields gains of up to 8.08 percentage points.The transfer evaluation uses EgoOmniWorld-trained models on existing egocentric benchmarks.Source: §1 · p. 3

2 RELATED WORK

Prior egocentric benchmarks address varied reasoning capabilities, modalities, and dialogue formats, but use different task definitions and protocols. EgoOmniWorld presents a common interaction structure for relating these strands.

2 RELATED WORK

  • Existing benchmarks cover long-horizon and temporal reasoning, task understanding, mental-state reasoning, multi-hop reasoning, and complementary settings such as cross-domain, nighttime, and scene-text understanding.The cited examples include EgoSchema and EgoTempo, EgoTaskQA, EgoToM, MultiHop-EgoQA, EgoCross, EgoNight, and EgoTextVQA.Source: §2 · p. 3
  • Audio-visual benchmarks and multi-turn dialogue broaden evaluation, but existing works remain organized around different task definitions, modality settings, and protocols without a common structure connecting them.Table 1 is described as a representative, rather than exhaustive, comparison of existing benchmarks by interaction structure, input modality, and dialogue format.Source: §2 · p. 3, Table 1 · p. 3
  • Recent omni-modal MLLMs support joint video, audio, and language understanding and are used to benchmark holistic egocentric understanding across heterogeneous cues.The passage identifies both proprietary and open-source model families as representative systems.Source: §2 · p. 3
  • Figure 2 depicts the interaction loop, three Ego–Action–World node states and six directed edge relations, and four structures from Node to Loop.The figure also gives example questions for the interaction structures.Source: Figure 2 · p. 4

3 EGOOMNIWORLD BENCHMARK CONSTRUCTION

EgoOmniWorld constructs an interaction-centered benchmark by defining complementary interaction scopes, grounding questions in video evidence, and linking interactions when dependencies carry across events. Its statistics describe coverage across structures, modalities, and dialogue formats.

3.1 CORE DESIGN PRINCIPLES

  • Node, Edge, Chain, and Loop represent individual states, directed relations, linked dependencies, and feedback closure, respectively—not a hierarchy of task difficulty.The three components are Ego (interaction-relevant state), Action (executed behavior and progress), and World (external entities, states, and changes).Source: §3.1 · p. 4
  • Visual or auditory evidence determines each question’s modality requirement, while dependency structure determines whether single- or multi-turn QA is used.Local states or relations can be probed in one turn; multi-turn QA can expose intermediate states and dependencies.Source: §3.1 · p. 4

3.2 DATA ENGINE PIPELINE

  • The four-stage data engine extracts local interactions, grounds multimodal evidence, links episodes and dependencies, then generates and validates evidence-grounded QA.Source videos are drawn from Ego4D and HD-EPIC, and source references and timestamps are preserved for traceability.Source: §3.2 · p. 5
  • Interaction records capture pre-action world state, supported goals and subgoals, outcomes, and wearer feedback; failed and interrupted actions are retained.Goals are recorded only when supported by speech, context, or later consequences.Source: §3.2 · p. 5
  • Evidence grounding timestamps and labels visual observations and auditory evidence, while repeated entities are linked across windows and their states remain distinct.The evidence can include speech and action sounds, object properties, hand–object contacts, and scene layout; modality labels determine question requirements.Source: §3.2 · p. 5
  • Episodes are linked only when a state, instruction, failure, or unfinished task carries over; temporal adjacency or a shared object alone is insufficient.Node and Edge questions use local records, while Chain and Loop questions use linked episodes; multi-turn final turns must combine earlier turns with new evidence.Source: §3.2 · p. 5

3.3 DATASET STATISTICS

  • The benchmark contains 24,166 QA groups: 20,284 training and 3,882 test; Node and Edge account for 41.5% and 39.4%, versus 10.3% Chain and 8.9% Loop.Action Status and World Perception are the most frequent Node and Edge semantic categories; Goal Planning, Intent Attribution, and Intent Formation are less represented.Source: §3.3 · p. 6
  • Visual-only, auditory-only, and joint audio-visual evidence are required by 46.9%, 15.4%, and 37.7% of QA groups, respectively.The dataset has 19,833 single-turn groups (82.1%) and 4,333 multi-turn dialogues (17.9%).Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments assess EgoOmniWorld as both a diagnostic testbed and a training resource, evaluating in-domain capability gaps, training effects, and transfer to existing egocentric benchmarks.

4.1 EXPERIMENTAL SETUP

  • The evaluation covered 11 omni-modal MLLMs, trained two open-source models with LoRA or full fine-tuning, and used shared scoring protocols for in-domain and transfer tests.For multi-turn QA, models conditioned on their own prior responses; predictive questions used only observations before the decision cutoff, and a text-only judge scored responses.Source: §4.1 · p. 7

4.2 IN-DOMAIN EVALUATION AND TRAINING

  • Gemini 3.8 achieved 61.51% overall accuracy, the highest open-source baseline reached 48.15%, and the best 7B-scale model reached 36.42%.Models varied across interaction structures and semantic categories, including action recognition, world-state perception, intent interpretation, and action-effect reasoning.Source: §4.2 · p. 7
  • Qwen2.5-Omni-7B rose from 36.42% to 47.36% with LoRA and 48.34% with FFT, while Qwen3-Omni-Thinking rose from 48.15% to 52.10% and 53.18%.Both Qwen2.5-Omni-7B adaptations improved all four interaction structures, with Edge gains of 13.83 and 15.35 percentage points; Qwen3-Omni-Thinking LoRA slightly reduced Chain and Loop accuracy.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 CROSS-BENCHMARK TRANSFER

  • Accuracy improved from 31.33% to 39.41% on EgoSound, 55.95% to 58.22% on EgoToM, and 20.40% to 24.80% on EgoTempo after LoRA adaptation.No additional training was performed on the target benchmarks.Source: §4.3 · p. 8, Table 3 · p. 8
  • Transfer gains spanned multiple Node and Edge categories, with EgoSound Ego State rising approximately from 40% to 60%, while EgoToM and EgoTempo showed gains in additional categories.The fine-grained comparison uses matched questions within each dataset–category group; empty categories lack evaluated questions, and groups with fewer than three questions are omitted.Source: §4.3 · pp. 8–9, Figure 5 · p. 9

5 CONCLUSION AND LIMITATIONS

EgoOmniWorld organizes egocentric understanding around four interaction scopes and reports improved in-domain performance and transfer. Its current data sources and training focus leave broader coverage and model-family evaluation for future work.

5 CONCLUSION AND LIMITATIONS

  • EgoOmniWorld frames egocentric understanding through Node, Edge, Chain, and Loop, with an evidence-grounded engine supporting visual and auditory evidence and single- or multi-turn QA.Experiments report uneven MLLM capabilities, improved in-domain performance, and transfer to other egocentric benchmarks.Source: §5 · p. 9
  • The dataset currently uses Ego4D and HD-EPIC, while longer recordings, richer annotations, broader data sources, and training beyond the Qwen-Omni family remain future directions.The authors suggest these expansions could further extend interaction-scenario and temporal-dependency coverage and investigate transferability across architectures and training paradigms.Source: §5 · p. 9

AI USE STATEMENT

The authors report using LLMs in benchmark construction and evaluation, and AI-assisted tools to polish the paper; they reviewed the outputs and take responsibility for the final work.

AI USE STATEMENT

  • LLMs supported interaction extraction, evidence grounding, episode aggregation, QA generation and repair, and answer judging; authors reviewed the outputs and take responsibility for the final findings.A random test-set subset underwent manual review, and AI-assisted tools were also used to polish the paper.Source: AI USE STATEMENT · pp. 9–10

ETHICS STATEMENT

EgoOmniWorld uses existing Ego4D and HD-EPIC recordings and is intended for research on egocentric interaction understanding, with privacy and licensing boundaries stated by the authors.

ETHICS STATEMENT

  • Source-video access and redistribution must follow dataset licenses; benchmark performance alone does not establish suitability for privacy-sensitive deployment.The authors identify unauthorized surveillance or profiling from first-person recordings as potential misuse.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper documents where implementation and evaluation details are provided and states that benchmark data, construction and evaluation code, and trained models will be released.

REPRODUCIBILITY STATEMENT

  • Data, construction and evaluation code, and trained models will be released; implementation and evaluation details are documented in the main text and appendices.Section 3.2 covers benchmark construction, Section 4.1 covers evaluated models and protocols, and appendices provide additional configurations and prompts.Source: REPRODUCIBILITY STATEMENT · p. 10

A DATASET STATISTICS AND REPRESENTATIVE EXAMPLES

This appendix reports EgoOmniWorld’s dataset scale, temporal coverage, and dialogue statistics, then illustrates its QA examples and describes construction and interaction-linking details. The data engine preserves evidence-grounded interaction structure, entity identity, and dependencies across activity episodes.

A DATASET STATISTICS AND REPRESENTATIVE EXAMPLES

  • 24,166 QA groups comprise 20,284 training and 3,882 test samples, distributed across Node (10,026), Edge (9,521), Chain (2,480), and Loop (2,139).A complete multi-turn dialogue counts as one group; Table 4 summarizes sample counts by category and split.Source: A DATASET STATISTICS AND… · p. 13, Table 4 · p. 13
  • 92.7% of clips are no longer than two minutes, while the longest lasts approximately 46.7 minutes; the dataset primarily covers short interactions but includes longer-span samples.Duration statistics cover 23,246 samples with clip metadata; deduplication yields 10,664 clips.Source: A DATASET STATISTICS AND… · p. 13
  • The dataset contains 19,833 single-turn groups and 4,333 multi-turn dialogues; all multi-turn dialogues use open-ended answers.The dialogues comprise 11,273 turns and average 2.60 turns each; 40.39% have two turns, 59.06% three, and 0.55% four.Source: A DATASET STATISTICS AND… · p. 13

A.1 REPRESENTATIVE QA EXAMPLES

  • Figure 6 pairs questions and reference answers with clip frames across nine Node and Edge categories, plus Chain and Loop dialogues.Chain examples expose dependencies between earlier and later events; Loop examples show an observed outcome informing a subsequent action, and audio-visual examples refer to sounds not conveyed by still frames.Source: §A.1 · p. 13, Figure 6 · p. 14

B DATA CONSTRUCTION DETAILS

  • The construction appendix details video selection, interaction representations, category-specific QA generation, structural validation, constrained repair, and evidence provenance.Ego4D selection uses narration-based clustering, while HD-EPIC segments use official narrations and action intervals as candidate-localization anchors, not substitutes for audio-video evidence.Source: B DATA CONSTRUCTION DETAILS · p. 15, §B.1 · p. 15

B.2 INTERACTION REPRESENTATION DETAILS

  • Interaction records distinguish persistent intentions, current subgoals, and performed actions, retain supporting observations for latent states, and leave unsupported latent states unspecified.Visual and auditory evidence is associated with temporal intervals and modalities; joint observations also record correspondence rather than assuming temporal proximity implies semantic relevance.Source: §B.2 · p. 15
  • The component graph represents interaction components as nodes and candidate relations as edges; connectivity is a structural hypothesis, not proof of causality.Relations used for QA generation must also pass category-specific evidence checks.Source: §B.2 · p. 15

B.3 ENTITY ASSOCIATION AND EPISODE DEPENDENCIES

  • Object tracks merge observations only when physical identity is reliable, preserve states separately over time, and group related interactions into temporally ordered task episodes.Episodes are split when goals, principal objects, workspace, or activity change substantially; unresolved identity or semantic dependence prevents cross-episode relations from directly generating questions.Source: §B.3 · pp. 15–16

B.4 EGO4D CLUSTERING AND SOURCE SELECTION

This section describes the data-construction pipeline, from narration-based clustering and cluster-stratified Ego4D source selection to evidence-grounded QA generation, multi-turn construction, and bounded repair and review. It also reports observed cluster coverage and specifies validation criteria and limits.

B.4 EGO4D CLUSTERING AND SOURCE SELECTION

  • Clustering combined cleaned summaries and action descriptions from two narration rounds, applying TF–IDF, truncated SVD, L2 normalization, and MiniBatch K-Means to 8,931 videos across 160 clusters.The recorded method used random seed 42; keywords and representative videos aid inspection, but keywords are not validated semantic categories.Source: §B.4 · p. 16
  • Candidates were filtered by annotation completeness, valid identifiers, and narration-span duration, then allocated across clusters and ranked by center distance, narration density, and identifier.The duration proxy is the maximum narration timestamp, not media-file duration; the saved script defaults to a 1,000-video budget, minimum quota of four per cluster, and 180–3,600-second range, but historical run parameters were not confirmed.Source: §B.4 · p. 16
  • 159/160 clusters had selected videos; clusters 7 and 112 each had 12, while cluster 39 had none.Cluster 39 contained one video with a narration-based duration proxy of about 92.3 seconds; exclusion below the script’s default 180-second threshold is consistent with the result, but the historical reason is unknown.Source: §B.4 · pp. 16–17
  • Cluster-stratified selection seeks to balance activity coverage and within-cluster representativeness, but is not an unbiased activity-distribution estimate and may underselect peripheral or rare activities.Narration wording can split similar activities across clusters, and 159/160 coverage refers to original clusters, not mutually exclusive human-defined activity classes.Source: §B.4 · p. 17
  • Selected-video provenance is recorded with each video’s identifier, original cluster index, topic summary, and distance to its cluster center; summaries do not alter assignments, selections, or counts.The listed clusters cover varied activities, but their topic summaries are interpretive only.Source: §B.4 · p. 22

C.1 EVIDENCE BINDING

  • Each candidate QA binds a video, events, time intervals, target, evidence descriptions, and required modalities; earlier QA wording may guide questions but cannot support new answers.A modality is required only when removing it removes information needed to answer; audio-visual questions need complementary evidence from both modalities.Source: §C.1 · p. 22

C.2 SEMANTIC QA CATEGORIES

  • Node categories describe individual states, Edge categories directed relations, and Chain and Loop composed dependencies and feedback closure; modality and dialogue format are separate annotation axes.Answer requirements include evidence-grounded intent attribution, action effects and grounding, intent formation, goal planning before a decision cutoff, and wearer-perspective world perception.Source: §C.2 · pp. 22–23

C.3 MULTI-TURN CONSTRUCTION

  • Multi-turn samples contain two to four turns, with earlier turns establishing intermediate facts and later turns combining them with new evidence; final answers must depend on both.Loop questions require evidence covering goal, action, outcome, and feedback to the ego; cross-loop questions also require inter-loop dependencies, while standalone constituent questions are retained where available.Source: §C.3 · p. 23

C.4 GENERAL QUALITY CHECKS

  • General checks cover completeness, references, timing, evidence-source and modality alignment, duplication, answer consistency, semantic support, leakage, and multi-turn dependencies.Multiple-choice samples also require valid, non-duplicate options and answers consistent with those options.Source: §C.4 · p. 23

C.5 CATEGORY-SPECIFIC STRUCTURAL FILTERING

  • Table 9 presents structural filtering criteria for QA candidates.Source: Table 9 · p. 24

C.6 LANGUAGE SHORTCUTS AND MODALITY AUDITS

  • Text-only tests flag potential language shortcuts, while modality audits identify audio-visual samples answerable from either modality alone as redundant and require cross-modal evidence to participate in the same interaction or complement one another.The modality-necessity evaluation is inspired by VisNec [6].Source: §C.6 · p. 24

C.7 CONSTRAINED REPAIR AND REVALIDATION

  • Constrained repair is limited to the existing evidence pool and original reasoning target, and Loop repairs must preserve the goal-to-action-to-world-change-to-feedback connection and goal confirmation or update.Repair may clarify closure only when evidence already supports it; semantic review is required, since wording, graph connectivity, or a validation pass cannot establish missing closure.Source: §C.7 · p. 25
  • Repairs may resolve ambiguity or leakage and adjust evidence or formatting, but cannot add facts, change targets or cutoffs, or require new evidence or substantial restructuring.Such samples are rejected or referred for review.Source: §C.7 · pp. 25–26
  • After repair, deterministic, category-specific, and independent evidence checks reject or refer samples that remain unsupported, ambiguous, misformatted, or structurally inconsistent.Goal Planning samples also receive pre-decision checks against future-information leakage; the separate validation call does not see the editor’s rationale.Source: §C.7 · p. 26

D HUMAN REVIEW

  • Human inspection sampled complete QA groups without replacement using fixed seed 20260921, and reviewers assessed audiovisual consistency, evidence alignment, modality labels, and reasoning structure.The interface showed clips, questions, answers, evidence intervals, modality labels, and reasoning categories; Chain requires an open dependency path, while Loop requires feedback closure.Source: D HUMAN REVIEW · p. 26
  • Reviewers could accept, request revision, reject, or mark samples uncertain, with edits kept separate for traceability; sampled inspection was not exhaustive test-set verification.Samples with unverifiable media or evidence should remain uncertain rather than be treated as validated.Source: D HUMAN REVIEW · p. 26

E.1 TRAINING CONFIGURATION

The saved configurations specify model-specific LoRA and full-fine-tuning setups, shared optimization and checkpointing settings, and an evaluation protocol for reported results.

E.1 TRAINING CONFIGURATION

  • Qwen2.5-Omni-7B LoRA targets all modules at rank 8 and scaling 16, while FFT updates the Thinker’s visual and audio components without training speech generation.Both use an effective batch size of 16; FFT uses DeepSpeed ZeRO-2.Source: §E.1 · p. 26
  • Qwen3-Omni expert-LoRA targets attention projections and 96 fused expert parameters at rank 8 and scaling 16; FFT uses DeepSpeed ZeRO-3.Both use an effective batch size of 16 and disable thinking during training; later continuation launches differ from these initial configurations.Source: §E.1 · p. 26
  • LoRA and FFT use learning rates of 0.00002 and 0.000002, respectively, with cosine decay, 3% warm-up, BF16, FlashAttention-2, and gradient checkpointing.The saved schedules specify ten epochs, epoch-wise checkpointing, no intermediate validation, and retained supervision on earlier assistant answers; not every reported checkpoint is claimed to complete ten epochs.Source: §E.1 · pp. 26–27
  • The human-review interface combines video, questions, reference answers, evidence intervals, and structural annotations with quality-check and revision controls.The screenshot depicts the interface, not aggregate review outcomes.Source: Figure 7 · p. 27

E.2 EVALUATION PROTOCOL AND REPRODUCIBILITY

  • Evaluation uses test-set manifests with deduplicated sample-and-round records, retained dialogue history, and cutoff-safe clips for predictive questions.Reproduction also requires checkpoint identifiers, input settings, scoring protocols, and evaluation manifests for each model and transfer benchmark.Source: §E.2 · p. 27

F.1 RESULTS BY SEMANTIC CATEGORY

Category-level results show that adaptation can improve some Ego–Action–World capabilities while leaving others unchanged or worse, and transfer gains likewise vary across target tasks.

F.1 RESULTS BY SEMANTIC CATEGORY

  • Qwen2.5-Omni-7B LoRA raises Goal Planning from 30.00% to 60.00% and World Perception from 21.88% to 37.98%, while Action Effects falls from 36.73% to 34.69%.FFT instead raises Action Effects to 57.14%, so overall improvement does not imply uniform gains across interaction relations.Source: §F.1 · p. 27, Table 11 · p. 28
  • Qwen3-Omni-Thinking LoRA improves Action Status from 34.59% to 43.55% and World State from 47.30% to 52.70%.These category-level changes complement the interaction-level results and identify capabilities that benefit from training.Source: §F.1 · p. 27, Table 11 · p. 28
  • For Qwen3-Omni-Thinking, FFT exceeds LoRA in Intent Formation (64.13% versus 60.87%) and World Perception (40.31% versus 39.53%), but trails in World State (40.54% versus 52.70%).These differences motivate retaining category-level evaluation rather than relying exclusively on an overall ranking.Source: §F.1 · pp. 27–28

F.2 TRANSFER RESULTS BY ORIGINAL TASK CATEGORY

  • EgoSound LoRA improves accuracy in all seven categories, with the largest gains in sound-source identification (+21.34 points), spatial localization (+11.12), and sound characteristics (+8.94).Cross-modal reasoning gains 7.45 points; temporal information, counting, and causal/contextual reasoning gain +3.80, +2.44, and +2.22 points, respectively.Source: §F.2 · p. 28
  • EgoToM action prediction rises from 35.21% to 41.57% (+6.36 points) and goal inference from 89.03% to 93.67% (+4.64), while belief inference falls from 44.55% to 38.61% (-5.94).Action and goal gains therefore do not extend uniformly to belief reasoning.Source: §F.2 · p. 28
  • EgoTempo gains are strongest for spatial relations (+22.00 points), object localization (+16.00), and action-specific objects (+12.00), while several temporal, prediction, and counting tasks decline.Object-specific actions rise +4.00 points; temporal event ordering falls 2.00, action and object sequences each fall 4.00, future action prediction falls 6.00, and action and object counting fall 8.00 and 12.00. Action prediction improves on EgoToM but declines on EgoTempo, so similarly named tasks can transfer differently across benchmarks.Source: §F.2 · p. 28

G PROVENANCE AND LIMITATIONS

The appendix records sample provenance and details the data-engine and evaluation prompts. It distinguishes traceability and structural reasoning requirements from perceptual correctness and causal certainty.

G PROVENANCE AND LIMITATIONS

  • Retained samples store questions, answers, categories, reasoning levels, modalities, video and event identifiers, intervals, evidence descriptions, and provenance mappings.Multi-turn samples also retain turn dependencies, inherited and new evidence, and standalone constituent questions.Source: G PROVENANCE AND LIMITATIONS · pp. 28–29
  • The pipeline does not establish that every inferred high-level relation is objectively causal or uniquely correct; some intention and task interpretations remain model-generated annotations.Provenance matching checks source consistency, not perceptual correctness; Node, Edge, Chain, and Loop describe required reasoning dependencies, not causal certainty.Source: G PROVENANCE AND LIMITATIONS · p. 29

H.1 LOCAL INTERACTION EXTRACTION

  • Local extraction records interaction components, outcomes, and feedback, while marking unsupported elements unknown and requiring evidence for a complete loop.The HD-EPIC perception branch records timestamped visual observations and entity references, keeping temporally distinct states separate and narrations subordinate to visual evidence.Source: §H.1 · pp. 29–30, §H.2 · p. 30

H.3 EPISODE FORMATION

  • Episode formation groups records by coherent activity stage, and reconciliation merges duplicate episodes while preserving chronology and distinct goals.Both steps retain supported states, entities, instructions, failures, interruptions, and unfinished work rather than inferring hidden motives or inventing stages.Source: §H.3 · p. 30, §H.4 · p. 30

H.5 CROSS-EPISODE RELATION EXTRACTION

  • Cross-episode links require evidence of transferred state, instruction, constraint, pending task, or feedback adjustment; temporal adjacency or shared objects alone do not suffice.A repair pass rechecks for supported missed dependencies without lowering the evidence standard, and zero supported links remains valid.Source: §H.5 · p. 30, §H.6 · p. 30

H.7 SINGLE-TURN QA: STATE AND ACTION-GROUNDING BRANCH

  • Single-turn questions target supported Ego State, World State, and Action Grounding observations, omitting unsupported categories and avoiding answer leakage or unsupported continuity.Answers include evidence quotes and required modalities.Source: §H.7 · p. 31

H.8 DEPENDENCY-BASED MULTI-TURN QA

  • Multi-turn QA requires a final answer to combine earlier and later evidence, including a new observation, and validation checks dependency and answer support without rewriting candidates.Validation also checks leakage, padding, modality requirements, and standalone questions with unchanged answers and evidence.Source: §H.8 · p. 31, §H.9 · p. 31

I EVALUATION PROMPTS AND METRIC COMPUTATION

  • The evaluation judge receives text inputs rather than source video, first judges binary correctness, then assigns a 0–5 score constrained by that decision.The scoring prompt maps correct answers to 4–5 and incorrect answers to 0–3; aggregation reports accuracy and mean score over successfully judged records, with errors and review cases separate.Source: §I.1 · p. 31, §I.2 · p. 31, §I.3 · p. 32

Related papers

Loading 2610.18499v1…