PaperlayerView exact revision on arXiv

Source-linked AI summary

Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning

Qinchuan Cheng, Zhantao Gong, Pengzhan Sun, Angela Yao, Shijie Li

arXiv:2610.02715v1cs.AI

TL;DR

Ego2World addresses the challenge of evaluating agents’ chosen actions and their consequences in recorded everyday activities. It compiles annotated cooking into executable worlds with task conditions and separate world state and agent belief; planner and memory evaluations show that accepted operations can leave goals unmet, while persistent belief improves action validity and reduces visual queries without a detected completion gain.

  • Problem

    Evaluating agents on recorded activity requires testing the consequences of their chosen actions and checking what those actions achieve.

    Source: §1 · p. 1

  • Method

    EGO2WORLD compiles HD-EPIC cooking annotations into executable episode worlds with source-linked operations, transition records, task conditions, and independently controlled world state and agent belief.

    Source: §1 · pp. 1–2

  • Results

    Accepted operations often leave task goals unmet; in the paired Qwen-Plus study, persistent belief raises action validity by 4.15 percentage points and reduces visual-query attempts by 90.27%, with higher token use and no detected completion gain.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EGO2WORLD provides a reusable testbed for tracing how planning and memory choices relate to execution, observation demand, and task attainment.

    Source: §7 · p. 9, §8 · p. 9

    The curated rules cover symbolic cooking activities, and the memory findings concern Qwen-Plus; the results do not establish kitchen-robot deployment readiness.

    Source: §7 · p. 9, ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Egocentric videos capture how people carry out everyday activities, yet testing an agent requires evaluating the consequences of actions it chooses itself. We introduce Ego2World, a benchmark that turns annotated cooking activities into executable planning environments under partial observation. Its compiler links source steps and objects to symbolic action rules, persistent world states, and explicit task conditions, so researchers can execute an agent's proposed actions and check their outcomes. World state and agent belief are maintained separately, enabling controlled studies of planning and information reuse across continuing tasks. Evaluating six planners on 105 tasks shows that accepted operations often leave task goals unmet. Execution traces and condition checks distinguish interrupted runs, partial attainment, and completed execution without goal attainment. In a separate paired Qwen-Plus study, persistent belief improves action validity by 4.15 percentage points and reduces visual-query attempts by 90.27%, with higher token use and no detected completion gain. Ego2World provides a reusable testbed for tracing how planning and memory choices affect execution, observation demand, and task attainment, connecting recorded human activity to the development and evaluation of interactive agents.

1 INTRODUCTION

Evaluating agents on recorded activity requires consequences for their chosen actions and evidence of what those actions achieve. EGO2WORLD compiles cooking annotations into executable worlds with task checks, then uses planner and memory studies to examine execution and information reuse.

Benchmark

  • EGO2WORLD compiles HD-EPIC cooking annotations into episode worlds, executable operations, transition records, and task conditions that link agent behavior back to source activity.Researchers can run proposed actions and inspect whether they achieve the required outcomes.Source: §1 · p. 1

Interaction

  • Separate world state and agent belief allow researchers to vary planning or memory policies under common world rules and measure execution, observation demand, and task attainment.World state persists across related tasks, while agent information can be retained or reset independently.Source: §1 · p. 2

Experiments

  • Persistent belief raises action validity by 4.15 percentage points and reduces visual-query attempts by 90.27%, with greater token use and no detected completion gain.The paired Qwen-Plus study evaluates the retention policy on 51 terminal-state tasks.Source: §1 · p. 2

Contributions

  • The benchmark links source annotations to typed worlds, 155 action types, transition records, and task conditions, preserving the connection between recorded evidence and executable operations.Source: §1 · p. 2
  • Condition checks, planner baselines, and paired memory results support inspection of execution and measurement of information reuse’s benefits and costs.Source: §1 · p. 2

2 RELATED WORK

Prior egocentric benchmarks assess procedural reasoning from activity evidence, while interactive environments provide executable dynamics and goal-based evaluation. EGO2WORLD connects unscripted source activities to executable symbolic interpretations and inspectable agent traces.

From egocentric activity to planning evaluation

  • EGO2WORLD evaluates decisions by executing them in a persistent symbolic world and checking task outcomes, alongside video benchmarks that assess reasoning from activity evidence.Source: §2 · p. 3

Interactive environments and world modeling

  • Unlike interactive environments with simulated activities or derived interaction sequences, EGO2WORLD links each operation to unscripted source evidence and makes new execution traces inspectable.Source: §2 · p. 3

Planning with feedback and retained experience

  • EGO2WORLD separates persistent world state from agent belief and links accepted operations to frozen task conditions, enabling studies of information retention during continuing execution.The paired study measures execution benefits, observation savings, and context cost.Source: §2 · p. 3

3 CONSTRUCTING THE EXECUTABLE DATASET

The dataset compiles HD-EPIC annotations and scene assets into episode worlds, executable operations, source-linked transition records, and fixed task conditions. Its separately defined planner and memory suites provide the reported evaluation scope.

Organizing source evidence

  • HD-EPIC temporal annotations, object references, and verb–noun descriptions become ordered source steps grouped around operations, with source-step identifiers and temporal spans retained.Grouping and reconciliation use schema-constrained language-model assistance.Source: §3 · p. 3

Normalizing entities and operations

  • A manually constructed vocabulary maps more than 300 source verb classes to 155 executable action types, while entity bindings distinguish operations such as opening a cupboard from opening a bottle.Related tasks retain their order within an episode.Source: §3 · p. 3

Giving annotations executable meaning and checking the compilation

  • Curated preconditions and effects interpret normalized actions in typed world graphs, while deterministic checks, semantic review, and recorded graph transitions support compilation and task validation.Tasks use frozen terminal-state conditions when supported by evidence and represented attributes, or accepted-event conditions otherwise.Source: §3 · p. 4

Example task

  • In Add Squash, the normalized pour binds squash to the glass, and the frozen terminal condition checks the glass’s contents rather than merely accepting preparatory actions.The table caption notes that its two attempts finish all four groups and that the condition checks contents, not narrated quantity.Source: §3 · p. 4, Table 1 · p. 4

Tasks and evaluation cohorts

  • The planner suite contains 105 tasks, 18 complete episodes, four scenes, and 3,441 groups per planner; the memory suite contains 51 terminal-state tasks, 22 episodes, seven scenes, and 3,878 groups per condition.The suites are separately defined cohorts, not random samples of HD-EPIC.Source: §3 · p. 4

4 INTERACTION AND STEP-LEVEL FEEDBACK

The environment checks concrete action proposals against a persistent symbolic world and reports execution feedback. Agents receive local visual and symbolic observations, while their belief can be retained or reset separately from world state.

4.1 EXECUTING ACTIONS IN THE HIDDEN WORLD

  • Accepted actions update the world, rejected actions leave it unchanged, and earlier accepted steps remain in effect if a later step fails.The executor reports either acceptance or the failed checks; accepted operations do not by themselves establish task completion.Source: §4.1 · p. 5

4.2 OBSERVING A CHANGING SYMBOLIC WORLD

  • A local observation pairs a recorded region image with current symbolic changes, and the delta reports changes to selected location, storage, and appliance fields.When image and delta conflict, the delta takes precedence; repeated queries reuse the image with updated local changes.Source: §4.2 · p. 5

4.3 THE EVALUATED AGENT AND ITS BELIEF

  • The agent’s lightweight belief store tracks estimated locations, known container states, and evidence sources, while world state carries over across tasks and belief retention is controlled separately.The common planner pipeline uses local action hints, recent feedback, executor repairs, and replanning under a shared limit.Source: §4.3 · p. 5

5 FROM EXECUTION SIGNALS TO EVALUATION

The evaluation separates attempted actions, accepted operations, task-condition attainment, and execution completion. It combines strict completion with graded condition-level and execution diagnostics so model behavior is not reduced to a single endpoint.

5 FROM EXECUTION SIGNALS TO EVALUATION

  • The evaluation distinguishes attempted reference action types, accepted concrete operations, and whether frozen task conditions were met.An accepted action such as retrieving a glass can be valid without establishing the required contents condition.Source: §5 · p. 6
  • Operational-TSR is the fixed-denominator completion baseline, requiring a finished execution and every required condition.Parse failures, empty plans, and execution blocks remain failures; Strict-TSR is restricted to terminal-state tasks.Source: §5 · p. 6
  • Goal completion ratio measures the fraction of frozen conditions satisfied, preserving partial progress that binary success collapses.A completed run satisfying three of four conditions has GCR 0.75 but binary success zero; non-completed runs receive zero under the failure policy.Source: §5 · p. 6
  • Task-macro metrics keep strict completion, GCR, action validity, and completed-run rate as distinct dimensions on the same 105 tasks.Action validity is the task-level fraction of attempts accepted by the executor; unresolved paired intervals remain ties rather than being collapsed into one score.Source: §5 · pp. 6–7

6 EXPERIMENTS

The experiments validate task conditions and compare six planners on a shared 105-task suite, then test persistent belief against per-group resets in a paired Qwen-Plus study. Across these evaluations, goal attainment remains distinct from execution validity, and retaining belief trades fewer visual queries for more token use without a detected completion gain.

6.1 SHARED EVALUATION PROTOCOL

  • The shared planner protocol evaluates six planners on 105 frozen tasks and 3,441 groups per planner, holding task setup, interfaces, and scoring fixed.The suite uses the first 18 complete episodes in four scenes; seven proxy-limited event-contract tasks remain in the primary denominator.Source: §6.1 · p. 7

6.2 TASK EVIDENCE AND OUTCOME CHECKS

  • Reference preparation satisfies all 105 task contracts while all recorded task-initial states fail them, and validation checks support the conditions’ executable interpretation and selectivity.The checks include 317 predicate-level negative suites and a separate 12-task stress test.Source: §6.2 · p. 7

6.3 GOAL ATTAINMENT AND EXECUTION PROFILES

  • Across the same task IDs and task-macro aggregation, terminal-task Strict-TSR and GCR are 8.82%–17.65% and 17.70%–26.40%, respectively; event-task GCR is 2.28%–6.27%, preserving distinct contract interpretations.Source: §6.3 · p. 8
  • Across the 630 planner records, 78 completed runs attain some but not all conditions, which GCR distinguishes from zero attainment.Figure 2 also records mutually exclusive endpoints, including partial attainment in a completed run; its counts describe outcomes, not causes.Source: §6.3 · p. 8, Figure 2 · p. 8

6.4 PERSISTENT BELIEF IN CONTINUOUS EXECUTION

  • Persistent belief raises task-mean action validity from 41.79% to 45.94%, a +4.15 pp difference, and reduces visual-query attempts by 90.27%, while both conditions attain 6/51 terminal successes.Token use rises from 23.58M to 27.14M; the paired completion difference is 0.00 pp with no detected completion gain.Source: §6.4 · p. 8, Figure 3 · p. 9
  • Across every episode quarter, persistent belief has lower query demand and higher token use, while the profiles remain descriptive rather than varying task difficulty or planning horizon.In task records, persistence and reset each produce two successes among four discordant cases, and local validity does not guarantee goal completion.Source: §6.4 · p. 9

7 DISCUSSION AND LIMITATIONS

EGO2WORLD offers a benchmark for inspecting construction, execution, and task outcomes, with metrics for end-to-end attainment, graded progress, and execution. Its scope is curated symbolic cooking, with broader coverage and isolation left as extensions.

7 DISCUSSION AND LIMITATIONS

  • The strict baseline, task-normalized graded progress, and execution diagnostics provide comparable, interpretable axes for agent development.Source links explain construction, executable rules make choices consequential, and outcome checks show achieved states.Source: §7 · p. 9
  • The planner and memory studies support an inspect, revise, and recompare cycle.This is the stated use of the studies as a development process.Source: §7 · p. 9
  • Rules cover curated symbolic cooking; extending the domain requires new ontologies, evidence, and action semantics.Broader cohorts and matched isolation are natural extensions, and the studies do not separately isolate planning horizon; memory findings concern Qwen-Plus.Source: §7 · p. 9

8 CONCLUSION

EGO2WORLD connects egocentric evidence to executable worlds, agent decisions, and outcome checks. Its strict, graded, and execution metrics provide actionable axes for planning and memory studies.

8 CONCLUSION

  • EGO2WORLD links egocentric evidence, executable worlds, agent decisions, and outcome checks.The conclusion describes these links as the benchmark's central connection.Source: §8 · p. 9
  • Its strict, graded, and execution metrics provide actionable capability axes for planning and memory studies.The metrics are presented as support for evaluating planning and memory studies.Source: §8 · p. 9
  • The benchmark connects recorded human activity with the development and evaluation of interactive agents.The paper characterizes the linked evidence, worlds, decisions, and checks as that connection.Source: §8 · p. 9

ETHICS STATEMENT

EGO2WORLD adds an executable annotation layer over HD-EPIC without adding new human-subject recordings. Its scope and claims are bounded by the source collection and symbolic evaluation setting.

ETHICS STATEMENT

  • EGO2WORLD builds an executable annotation layer over HD-EPIC and adds no new human-subject recordings.The work uses the existing dataset rather than collecting new recordings.Source: ETHICS STATEMENT · p. 10
  • Kitchen, activity, and cultural coverage are limited by the HD-EPIC source collection.Source videos remain subject to the dataset’s access terms.Source: ETHICS STATEMENT · p. 10
  • Symbolic action acceptance does not certify physical safety or establish kitchen-robot deployment readiness.The passage explicitly limits what these results establish about physical safety and deployment.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The manuscript source package provides frozen evidence and scripts for reproducing reported tables and figures. A separate benchmark release is planned, while full agent re-execution requires additional access.

REPRODUCIBILITY STATEMENT

  • Anonymized frozen exports, task-level metrics, hashes, and table-regeneration scripts support reproducing reported tables and figures.The package's scope is documented in Appendix F, and it includes contract-type analysis and all 45 post-hoc diagnostic contrasts.Source: REPRODUCIBILITY STATEMENT · p. 10
  • A separate benchmark release will provide the compiler, executor, rules, initial worlds, source mappings, and frozen task contracts.These materials are intended for the two evaluation suites.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Full agent re-execution additionally requires applicable source-data and provider access.Appendix F distinguishes those requirements from manuscript reproduction.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The construction workflow converts narrations into inspectable executable graph-transition worlds, while AI tools assist selected annotation and research tasks.

A ANNOTATION-TO-WORLD CONSTRUCTION

A.1 SOURCE EVIDENCE AND NORMALIZATION

  • Source-step links retain temporal and object evidence through grouped annotations, while normalization maps more than 300 verb classes to 155 executable action types.Each group records its source steps, temporal span, manipulated objects, and normalized primitives.Source: §A.1 · p. 14

A.2 STRUCTURAL AND SEMANTIC QUALITY CONTROL

  • Deterministic checks and localized semantic review constrain revisions to supported source steps, temporal order, and the executable schema.A 5% sample of annotations across P01–P07 was manually inspected.Source: §A.2 · p. 15
  • Validated groups become goal-aligned task units, and curated symbolic rules define how candidate actions change the initialized world graph.Source videos and annotations provide activity evidence, while human normalization and curated semantics provide executable interpretation.Source: §A.2 · p. 15

A.3 REPLAY AND FROZEN TASK ADMISSION

This section describes replaying source activities into frozen task contracts, the handling of hidden world state and agent belief, and the interfaces and accounting used to execute and validate tasks. It also examines partial attainment alongside strict completion, source-to-outcome tracing, and the limits of transfer and evaluator claims.

A.3 REPLAY AND FROZEN TASK ADMISSION

  • Task admission requires reference replay, goal validation, semantic review, and a non-trivial predicate check before contracts are frozen; the 5% manual annotation inspection estimates neither human error nor inter-annotator agreement.Replay captures reference execution, semantic review checks support in activity evidence, and alternative-order tests check exchanges of independent operations; reference execution resets once per episode and processes source operations in group order.Source: §A.3 · p. 15

A.4 A SOURCE-TO-OUTCOME EXAMPLE

  • In the squash example, persistent belief yields an accepted pour and a true contents condition, while per-group reset leaves the condition UNKNOWN despite all four groups finishing in both runs.The frozen terminal condition checks whether squash is in the glass, not the narrated measure, bottle-cap state, or glass placement.Source: §A.4 · p. 16

B.1 PRIMITIVE-LEVEL RULE SEMANTICS

  • Primitive effects apply only when preconditions hold; invalid primitives leave the input graph unchanged, while the executor alone mutates the world and the agent plans from permitted observations, feedback, and retained belief.Belief updates integrate observations, successful actions, and explicit removal deltas; repeated queries combine recorded visual evidence with updated local symbolic changes. Persistent and reset conditions retain or discard agent-side belief at group boundaries, and terminal or event scoring is separate from reference-action diagnostics.Source: §B.1 · p. 17, §B.2 · p. 17, §B.3 · p. 17

C. EVALUATION DEFINITIONS AND VALIDATION

  • The evaluation fixes 105 tasks across 18 episodes and 3,441 groups per planner; task contracts test frozen terminal predicates or accepted task-local events, and missing evidence fails the binary contract.GCR macro-averages condition truth within each task, assigning zero to non-completed runs; the suite has 34 terminal-state and 71 successful-event contracts. Action validity is also a task-macro mean, while reference coverage tracks attempted key-action types without checking bindings or successful execution.Source: §C.1 · pp. 18–19, §C.2 · p. 19, §C.3 · p. 20

C.4 SAME-TASK REFERENCE CONSISTENCY AND EVALUATOR STRESS TESTS

  • The same-task audit matched all 105 planner IDs, with reference records passing 105/105 contracts, initial states failing 105/105, and all 317 predicate-level negative suites passing expected rejection checks.The audit used both contract tiers and no planner outcomes; the 317 suites were not independent trajectories.Source: §C.4 · p. 22
  • In the separate 12-task stress test, replay, transition witnesses, repeated replay, and invalid-action immutability passed 12/12; initial, partial, wrong-binding, and missing-state cases failed 12/12.Ten tasks allowed an independent update swap, and both orders succeeded in 10/10; these checks do not establish exhaustive semantic equivalence.Source: §C.4 · p. 22

C.5 BOOTSTRAP AND PAIRED COMPARISONS

  • Simultaneous intervals excluded zero for GPT-5.5 versus GLM-5 in validity, GPT-5.5, Gemini, and DeepSeek versus Qwen-Plus and Gemini versus Claude in completed-run rate.These were paired diagnostic comparisons across 105 tasks; the supplied passage reports significance but not the directions or effect sizes.Source: §C.5 · p. 22
  • 78 of 630 model–task records were completed runs with partial attainment that a strict conjunction would classify like zero-attainment runs.The records are repeated evaluations of the same 105 tasks; incomplete runs receive zero under the stated policy.Source: §C.5 · p. 23

C.6 PAIRED MEMORY EFFECTS AND CONTINUOUS RESOURCE PROFILES

  • Figure 3 profiles group-level resource use across relative episode progress, while task metrics and resource rates use episode-cluster bootstrap intervals.Each episode contributes groups to all four quarters; intervals are pointwise, not simultaneous, and the analysis uses frozen records without new model or simulator executions. Relative progress does not hold task difficulty fixed or represent a first-action comparison.Source: §C.6 · p. 25

D.1 RUNTIME CONFIGURATION AND ADAPTER BOUNDARY

  • Six planners shared the task order, prompt structure, action schema, executor, and evaluator; parsing normalized JSON structure while the executor separately checked bindings and preconditions.The runtime used each group’s instruction as its goal, and adapters did not semantically repair planner outputs.Source: §D.1 · p. 26, §D.2 · pp. 26–27

D.3 STATE DELTAS AND OBSERVATION PROVENANCE

  • Belief updates combine fixed-time source-video anchors with local symbolic state changes, giving current deltas precedence when they conflict with the image.The image and state delta have distinct reference points: the image capture time and the initialized world state, respectively.Source: §D.3 · p. 27

D.4 REPAIR AND REPLANNING

  • An empty action queue is marked completed even after rejected attempts, whereas a nonempty queue at the loop limit is execution_blocked; neither status proves contract satisfaction.The runtime can insert navigation before retrieval, execute repair-plan primitives after validation fails, and then retry the original action.Source: §D.4 · p. 27

D.5 VISUAL EXPLORATION AND BELIEF UPDATES

  • Visual queries are triggered for retrieval objects that are neither locally visible nor remembered, and persistent memory retains entries across groups without age-based expiry.Visual-model fusion prioritizes local deltas over conflicting anchor evidence; failed-action messages support replanning but do not provide general attribute-by-attribute belief correction.Source: §D.5 · pp. 27–28

D.6 EVALUATION AND RESOURCE ACCOUNTING BOUNDARIES

  • All 105 assigned tasks remain in the outcome denominator, and primitive validity, reference-action-type attempts, and task-contract satisfaction are distinct measures.Memory experiments also report query attempts separately from successful responses and group-accounted tokens; interrupted-attempt reset tokens are excluded from completed-group attribution, with no cost or latency estimate inferred.Source: §D.6 · p. 28

E.1 STEP-LEVEL EXECUTION ACROSS PLANNERS

  • Each planner’s 3,441 frozen group records was hash-checked; pooled primitive-attempt counts include simulator repairs and are not task-macro validity percentages.All recorded step statuses in this cohort are accepted or invalid; counts describe execution volume and composition, not validity averaged across tasks.Source: §E.1 · pp. 28–29, Table 17 · p. 29

E.2 PAIRED TASK OUTCOMES AND THEIR EXECUTION EVIDENCE

Paired execution records show why accepted actions and completed runs cannot stand in for goal attainment, while traces expose the operations and conditions behind divergent outcomes. The benchmark release supports inspecting tasks in their activity context and comparing planning or information-retention policies.

E.2 PAIRED TASK OUTCOMES AND THEIR EXECUTION EVIDENCE

  • Across 51 tasks, 43 failed in both memory conditions, four succeeded in both, and two succeeded only under each condition.The squash trace contrasts a completed pour and true terminal condition with a reset run lacking a pour; the mushroom run blocks after 80 attempts, so its zero is not an observed dirty state.Source: Table 18 · p. 29, §E.2 · p. 29

E.3 A TRACEABLE EXECUTION EXCERPT

  • Completed groups and reference action types do not establish goal success; local observation can also replace a fine-grained placement target with a coarser region.Table 19 records seven completed groups, but completion status alone does not establish successful goal attainment.Source: §E.3 · p. 30, Table 19 · p. 30

E.4 EXECUTOR REPAIR VERSUS PLANNER ACTIONS

  • A simulator-feedback repair navigates to a drawer after rejected scissor pickups, while later planner-tagged actions continue execution; goal evaluation remains separate from action acceptance.The trace records four rejected pick_up(scissor) attempts before later accepted actions involving mesh.Source: §E.4 · p. 30

E.5 RE-OBSERVING A STATIC ANCHOR AFTER A STATE CHANGE

  • Repeated observations retain the source frame while applying local changes: explicit removal overrides frame presence, and unobserved destinations remain unknown.This temporal interpretation is used across interactions.Source: §E.5 · p. 30

E.6 MEMORY PERSISTENCE AND STALE FACTS

  • With persistence, the logger records 351,114 cross-group memory hits versus zero with reset, while repeated region queries fall from 370 to two.The separate anchor–delta conflict counts are three with persistence and 218 with reset; neither counter measures every stale belief or failed action.Source: §E.6 · pp. 30–31

F ARTIFACT SCOPE AND RESPONSIBLE USE

  • Episode worlds, source-step links, transition records, traces, and frozen conditions support task inspection and policy comparisons; redistribution of source data remains governed by HD-EPIC terms.The benchmark release contains interactive components, while the manuscript package reproduces analyses from recorded evidence; the stated source scale is not full-corpus benchmark coverage.Source: F ARTIFACT SCOPE AND RESPONSIBLE USE · pp. 31–32, Table 20 · p. 31
Loading 2610.02715v1…