PaperlayerView exact revision on arXiv

Source-linked AI summary

Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana, Qinrong Cui, Erland Hilman Fuadi, Zayd M. K. Zuhri, Nanda Ryaas Absar, Ahmed Elshabrawy, Wilfried Ariel Mulyawan, Shoubin Yu, Yue Zhang, Mohit Bansal, Alham Fikri Aji

arXiv:2610.01092v1cs.CV

TL;DR

Existing benchmarks leave open how to evaluate autonomous, multi-step egocentric manipulation from high-level goals when valid action paths differ. EGO2ACT pairs real-world scenes with goals, and EGO2ACTJUDGE scores generated rollouts without reference videos. Models still struggle to align complete execution with physical accuracy, while the judge correlates with human consensus at r = 0.69.

  • Problem

    Existing benchmarks do not adequately test autonomous multi-step egocentric planning, and open-ended goals allow multiple valid action paths that preclude reliance on reference videos.

    Source: §1 · p. 1

  • Method

    EGO2ACT pairs real-world initial scenes with high-level goals, while EGO2ACTJUDGE derives subgoals and evaluates task completion and physical plausibility without reference trajectories.

    Source: §1 · pp. 2–3, §4 · p. 6

  • Results

    Models struggle to align physical accuracy with complete task execution, while EGO2ACTJUDGE achieves Pearson correlation r = 0.69 with human consensus.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The paper identifies complete step execution, persistent object identity and state under occlusion, and physically grounded contact interactions as needs for future world models.

    Source: §6 · p. 10

    EGO2ACTJUDGE often misses physics violations unfolding between frames, so its physics scores are best read as optimistic.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions. While evaluating these capabilities is crucial, existing benchmarks focus mainly on single short actions or step-by-step instructions. This leaves multi-step physical reasoning underexplored, especially in egocentric video generation that requires planning to simulate proper execution to accomplish high-level goals by carrying out multiple real-world manipulations. We introduce Ego2Act, a goal-directed benchmark featuring 2,640 videos from 110 real-world tasks across day-to-day settings, varying object clutter and multi-step complexity. Given an initial scene image and a high-level goal, Ego2Act evaluates whether video generation models can produce realistic egocentric videos of a hand manipulating objects to carry out the task. To support scalable evaluation, we also introduce Ego2ActJudge, a reference-free evaluation pipeline that achieves better task completion and physics plausibility evaluation alignment with human consensus compared to relevant baselines. Our findings reveal that models' generated simulations often skip or partially execute steps, leaving later steps missing dependent states, which leads to unfulfilled goal. Furthermore, models consistently fail at fine-grained physical dynamics, particularly during complex object manipulation and persistent world modeling. We hope Ego2Act provides a rigorous testbed for advancing video models toward physically plausible, goal-directed simulation.

1 INTRODUCTION

Goal-directed egocentric video generation requires models to infer intermediate actions and maintain plausible object-state changes, yet existing benchmarks do not adequately test autonomous multi-step execution. EGO2ACT and EGO2ACTJUDGE address this gap, revealing execution failures and only moderate evaluator alignment with human judgments.

1 INTRODUCTION

  • Existing benchmarks either omit egocentric hand-object dynamics, rely on fidelity metrics, or prescribe step-by-step prompts rather than testing autonomous multi-step planning.Because multiple valid action paths can achieve a goal, evaluation cannot depend on a single reference video and must also check physical constraints.Source: §1 · pp. 1–2
  • Models frequently skip dependent steps or make physical errors, exposing fine-grained execution failures that shorter-horizon settings typically miss.Successful egocentric execution also demands state tracking through hand occlusions and dynamic camera motion.Source: §1 · pp. 2–3
  • EGO2ACT introduces a goal-directed, multi-step egocentric benchmark with 110 real-world cases and 2,640 videos across diverse domains, clutter, and action complexity.Each case pairs a real-world starting scene with an outcome-oriented goal; successful human executions require more than five steps on average.Source: §1 · pp. 2–3
  • EGO2ACTJUDGE evaluates open-ended manipulation rollouts for task completion and physical plausibility without reference videos, outperforming automated baselines in human alignment.On a subset panel, Seedance-2.0 scored 64.0 points overall; rubric agreement was ICC = 0.86 and judge-to-human alignment was r = 0.69.Source: §1 · p. 3
  • The paper analyzes rubric design, egocentric execution failure modes, and the reliability of its automated evaluation pipeline.The benchmark tests whether models can produce physically valid egocentric manipulation videos from an initial scene and target goal.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work uses generated videos for embodied planning and learning, while existing evaluation approaches cover separate parts of the problem. EGO2ACT focuses on generated, multi-step egocentric task completion and physical accuracy without reference trajectories.

2.1 VIDEO GENERATION FOR EMBODIED TASKS

  • EGO2ACT evaluates whether a video model can generate a complete, physically plausible multi-step task execution from a real-world egocentric scene and high-level goal.Related systems include action-conditioned simulation, goal-conditioned policy learning, video-based planning, and generated robot or egocentric rollouts.Source: §2.1 · p. 3
  • Figure 2 summarizes benchmark composition through domain, action-family, duration, object-count, operation-count, and per-case action-family distributions.Its caption defines co-occurrence as joint presence within a case and observed operations as the number of subactions per case.Source: Figure 2 · p. 4

2.2 EVALUATING ACTIONS FROM VIDEO MODELS

  • General video metrics assess visual quality or prompt alignment, while EGO2ACT evaluates multi-step goal completion and physical accuracy without reference trajectories.EgoForge shares the generation interface but is evaluated using reference-based visual fidelity metrics rather than goal completion.Source: §2.2 · p. 4

3 EGO2ACT

EGO2ACT evaluates whether an egocentric video reaches a high-level goal through required intermediate steps and plausible physical interactions. Its collection pairs verified real-world tasks with human recordings and generated model videos.

3.1 TASK DEFINITION

  • A successful EGO2ACT rollout reaches the goal through necessary intermediate steps while maintaining plausible hand-object contact, object persistence, and state changes.Any rollout reaching the goal under plausible physics counts as successful, since the goal does not prescribe a unique action sequence.Source: §3.1 · p. 4

3.2 DATA COLLECTION PIPELINE

  • The benchmark contains 110 cases and 2,640 videos: six human recordings and 18 model-generated videos per case.The human recordings include three successful attempts with distinct action sequences and three unsuccessful controls; six video models each generate three trials.Source: §3.2 · p. 5
  • Each case pairs an initial scene with an outcome-oriented goal and admits at least three distinct human execution sequences.Sequences can differ in sub-action order, tools or resources, or intermediate state transitions; tasks span five everyday categories and three clutter levels.Source: §3.2 · p. 5
  • Cases and recordings are reviewed for task clarity, feasibility, visual quality, viewpoint consistency, and compliance with the egocentric closed-world constraint.Objects may leave view, but recordings that introduce additional objects are excluded.Source: §3.2 · p. 5

4 EGO2ACTJUDGE

EGO2ACTJUDGE uses human-crafted rubrics to score open-ended task execution without training or reference trajectories. It derives subgoals, evaluates task completion and physical plausibility separately, then combines their scores.

4 EGO2ACTJUDGE

  • EGO2ACTJUDGE derives a minimal set of observable subgoals from the goal and initial scene, each with one source object, at most one target, and one action.Across videos, 91% of plans keep the same subgoal count and name nearly the same objects.Source: §4 · p. 6
  • Task completion is scored per subgoal through ordered gates for action initiation, coarse interaction, and the resulting state, stopping at the first failed gate.The subgoal score is the number of gates passed, and the task score averages these scores across subgoals.Source: §4 · p. 6
  • Physical plausibility is scored independently through gates for continuity, valid interaction onset, plausible contact, and post-interaction stability.Unattempted subgoals are marked N/A and excluded from the average physics score.Source: §4 · p. 6
  • The final score is the geometric mean of task and physics scores, so a severe failure on either axis suppresses the overall score.A rollout attempting no subgoals receives a final score of zero rather than being omitted from benchmark averages.Source: §4 · p. 6

5 EXPERIMENTS & RESULTS

Across six video generators, the benchmark exposes failures in contact-rich task execution, while EGO2ACTJUDGE aligns closely with human judgments. Qualitative errors include skipped prerequisites, incomplete outcomes, and violations of object persistence and physical interaction; rubric gating and backbone choice also affect evaluation alignment.

5.1 SETUP

  • The study evaluates six video models using shared prompts and human ratings from three annotators on a 25-case, 600-video panel.Generation is generally set to 480p, 16:9, and a maximum of 15 seconds, with stated model-specific exceptions.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • Attachment/connection subgoals reach 38.6% Task completion and 66.7% Physics validity, versus 51.1% and 75.5% for relocation/arrangement.Task length has only a weak negative correlation with performance (Spearman ρ = −0.29), while action type shows the largest differences in generation quality.Source: §5.2 · p. 8
  • Humans and EGO2ACTJUDGE rank the six generators almost identically (Spearman ρ = 0.94), with only Grok-1.5 and MiniMax-H3 swapping adjacent positions.The rubric-derived judge also aligns more closely with human consensus than evaluated baselines.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 QUALITATIVE ANALYSIS OF SIMULATION FAILURES

  • Task errors include skipped prerequisite actions, incomplete final states, and failed operations involving less familiar or culturally specific objects.Examples include attempting insertion before opening a container and leaving objects partly outside containers.Source: §5.3 · p. 8
  • Physics errors include unexplained object-state or count changes, boundary crossings through closed surfaces, and invalid motion or deformation mechanisms.Examples include duplication, objects passing through closed lids, and articulated parts moving about incorrect axes.Source: §5.3 · pp. 8–9

5.4 RUBRIC & AUTOMATED JUDGE ABLATIONS

  • Removing early gates or scoring gates independently increases error; independent gating also inflates EGO2ACTJUDGE Final scores by 6.8 points and lowers Final r from 0.69 to 0.65.The authors report that strict sequential gating makes fine-grained rubric scores more consistent with human judgment.Source: §5.4 · p. 9, Figure 5 · p. 9
  • Controlled physics corruptions show that the judge often detects within-frame violations but largely misses between-frame violations, making its Physics scores optimistic.The passage identifies denser temporal evidence as the most direct path to a stronger judge.Source: §5.4 · p. 9, Table 4 · p. 9

6 CONCLUSION

EGO2ACT and EGO2ACTJUDGE assess goal-directed egocentric manipulation, and the benchmark finds persistent gaps in task completion and physical accuracy. The judge correlates with human consensus, while the paper concludes that reliable simulation requires complete steps, persistent object states, and plausible contact dynamics.

6 CONCLUSION

  • Across six generators, human evaluation on a 25-case panel gives the top model 64.0, while open-weight models score as low as 3.9.EGO2ACTJUDGE achieves Pearson correlation r = 0.69 with human consensus.Source: §6 · pp. 9–10
  • The benchmark identifies omitted prerequisites, incomplete actions, unstable object identity or state, and implausible contact dynamics as key obstacles to reliable embodied simulation.The authors call for future world models to execute required steps, maintain persistent object identity and state, and ground contact-rich interactions in plausible dynamics.Source: §6 · p. 10

AI USAGE DISCLOSURE

The authors disclose uses of generative AI in implementation, data reformatting, qualitative metadata drafting, figure plotting, editing, and reference formatting. They state that authors checked AI-assisted work and take responsibility for the final content.

AI USAGE DISCLOSURE

  • Generative AI supported coding, data reformatting, checked metadata drafting, plotting, manuscript editing, and reference formatting; authors reviewed the work and take responsibility for the final content.The authors state that AI was not used for dataset generation, conceptual or methodological development, experiment design feedback, or result interpretation.Source: AI USAGE DISCLOSURE · p. 10

REPRODUCIBILITY STATEMENT

The paper documents task construction, video-generation settings, rubrics, judge and baseline procedures, and analysis metrics. It states that results are computed from stored outputs and ratings, and lists materials planned for release upon publication.

REPRODUCIBILITY STATEMENT

  • Appendices specify task verification, generation prompts and settings, rubrics, judge implementation, baseline mappings, annotation protocols, and agreement metrics.The paper says reported tables and figures are computed by scripts from stored outputs and ratings, without additional model calls.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors state they will release the benchmark videos and scenes, EGO2ACTJUDGE code and prompts, human annotations, rubric guidelines, and analysis scripts upon publication.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

This section describes safeguards and controlled procedures for Ego2Act’s recordings, collection, and generation, and outlines Ego2ActJudge’s evaluation of generated videos for goal completion and physical plausibility. It details prompts that decompose goals into executable subgoals and ground judgments of progress, outcomes, and plausibility in observable video evidence, distinguishing required goal conditions from optional conventions.

ETHICS STATEMENT

  • The dataset uses consented, anonymized recordings, removes location and device metadata, and is released for research use despite potential misuse of realistic manipulation videos.The recordings are intended to contain no faces or other personally identifying content; human raters scored anonymized videos without knowing which model produced them, and the project releases no new generator.Source: ETHICS STATEMENT · p. 10

A.1 TASK CONSTRUCTION & VERIFICATION PROTOCOLS

  • Cases must describe feasible everyday tasks with clear end states, multiple valid executions, unambiguous objects, and a visually verifiable, closed-world starting scene.Requirements include at least three valid execution sequences, preserving dependencies while allowing independent subgoals in different orders, and ensuring necessary objects and target locations are visible.Source: §A.1 · p. 15

A.3 COLLECTION WORKFLOW & ANNOTATION PROTOCOL

  • Nine annotators collect cases and six recordings per case; a second annotator checks each case and recording against the construction rubric before acceptance.Each case has three successful and three unsuccessful attempts; successful recordings differ structurally, while unsuccessful attempts keep natural motion but leave the goal unmet. Metadata is drafted by a vision-language model and checked by annotators.Source: §A.3 · pp. 17–18

A.4 VIDEO GENERATION PROMPT

  • Generators receive the same initial scene and verbatim prompt across cases, with only the goal changing, and produce three videos per case where supported.The prompt requires one hand, interaction only with objects visible in the starting frame, and realistic interactions and physical laws; seeds are 101, 202, and 303 where supported.Source: §A.4 · pp. 18–19

B VIDEO SCORING RUBRIC DETAILS; B.1 TASK-COMPLETION RUBRIC

  • Reference-free evaluation separates task completion from physical plausibility; task scoring decomposes goals into subgoals and checks intent, process, then final state.Each subgoal score counts gates passed before the first failure, and the final task score is the mean with no separate ordering penalty; independent placements need not follow a fixed order.Source: B VIDEO SCORING RUBRIC DETAILS · p. 19, §B.1 · p. 19

B.2 PHYSICS RUBRIC

  • Physics scoring checks continuity, causation, interaction possibility, and persistence, but excludes unattempted interactions, so high Physics can mask skipped difficult steps.The final score limits this effect because skipped subgoals lower Task, and Physics should be read alongside Task; unresolved decisions remain unavailable rather than being silently scored as yes, no, or zero.Source: §B.2 · pp. 20–21

B.3 ILLUSTRATIVE SCORING CASES; B.4 AGGREGATION RULE

  • The geometric mean treats Task and Physics symmetrically and gives zero when either axis is zero, preventing a strong score on one axis from compensating for complete failure on the other.Missing or unresolved scores remain unavailable; no-subgoal attempts receive final score zero, while Table 8 illustrates rubric outcomes without assigning new judgments to study videos.Source: Table 8 · p. 21, §B.3 · p. 21, §B.4 · p. 22

B.6 EGO2ACTJUDGE PROMPTS

  • Ego2ActJudge prompts are reproduced verbatim with runtime fields and a JSON schema, including a scene-grounded prompt for decomposing goals into subgoals.The decomposition prompt asks for high-level subgoals grounded in the initial scene and goal.Source: §B.6 · p. 23
  • Goal decomposition covers required actions and end states while keeping distinct source objects separate and recording only explicit or physically necessary dependencies.Plans use ordered subgoal IDs, with prerequisites referencing earlier IDs; customary action order alone is insufficient.Source: §B.6 · pp. 23–24
  • Task gates separately judge directed initiation, substantial performance of the main action, and achievement of the required final state using observable video evidence.Minor imprecision may be acceptable, but a materially incomplete or unverifiable outcome fails the final-state gate.Source: §B.6 · pp. 24–25
  • Physics judgments track changes throughout the interaction, allow ordinary occlusion or brief flicker, and require evidence-based answers rather than invented explanations for sudden changes.Judges should inspect only when it could resolve a specific uncertainty and cite observations with original-video timestamps.Source: §B.6 · pp. 25–26
  • The physics gates separately assess object identity and persistence, visible cause and effect, interaction plausibility, and whether post-action states persist consistently with visible forces.The prompts direct judges to follow the whole scene and assess movement over time, not only first and last frames.Source: §B.6 · p. 26

B.5 EGO2ACTJUDGE IMPLEMENTATION

EGO2ACTJUDGE evaluates task completion and physical plausibility through separate, staged pipelines, then compares automated scores with human ratings and existing evaluators. The study also adapts evaluator mappings and prompts to the egocentric manipulation setting.

B.5 EGO2ACTJUDGE IMPLEMENTATION

  • The judge plans Task subgoals and Physics inspection windows before viewing the video, gathers evidence through sampled frames and optional inspections, and advances only subgoals that pass each gate.Unresolved reached gates withhold the affected axis score; unattempted clips with no inspectable Physics interaction can receive a Final score of zero when Task is zero.Source: §B.5 · p. 22

C.1 TASK/PHYSICS MAPPING DECISIONS

  • Evaluator scores are mapped only where their components overlap the Task or Physics rubric; unsupported axes are not replaced with zero, and native outputs remain separate.For example, RBench’s Task and Physics means map its five aspects to the two axes, while VideoScore uses text-to-video alignment for Task and temporal plus factual consistency for Physics.Source: §C.1 · pp. 26–27

C.2 MODIFICATION COURTESY

  • Baseline runs preserve official scoring rules where possible, with RBench’s robot-arm wording adapted to first-person human manipulation and Simple VQA run as a one-call control.Simple VQA uses no subgoal plan, gates, or inspection tool, and its comparison is not compute-matched to EGO2ACTJUDGE.Source: §C.2 · pp. 27–28

D.1 STUDY PROTOCOL

  • Three annotators scored a case-selected panel of 600 videos, and available ratings were averaged into human Task and Physics references, with Physics N/A kept missing.Annotators were blind to generator identity; human Final scores use the geometric mean of the two axis means, with Task zero yielding Final zero.Source: §D.1 · p. 29
  • Automated evaluators were compared with aggregated human references using separate Task and Physics agreement metrics, with overall scores treated as secondary.Confidence intervals use 1,000 case-level bootstrap resamples to account for videos clustered within cases.Source: §D.1 · pp. 29–30

D.2 ALIGNMENT RESULTS

EGO2ACTJUDGE aligns more closely with human judgments than the evaluated baselines on the 600-video panel, while retaining a positive scoring bias. A held-out backbone comparison and coverage checks qualify the alignment results.

D.2 ALIGNMENT RESULTS

  • EGO2ACTJUDGE leads the evaluated baselines on the 600-video panel with Final-score correlation r = 0.69, rank correlation τ = 0.46, concordance CCC = 0.61, and MAE = 22.0.Its Final-score correlation exceeds external baselines’ maximum of 0.42 and Simple VQA’s 0.32; Task and Physics correlations are 0.63 and 0.55, respectively.Source: §D.2 · p. 30
  • The judge still over-scores on average: its Final-score signed bias is +15.4 points, concentrated in Physics (+24.1) rather than Task (+8.1).Removing Cosmos-3 lowers all correlations, but EGO2ACTJUDGE retains the highest reported correlation among the listed evaluators.Source: §D.2 · p. 30
  • In five-fold case-level validation, Gemini-3.7-Flash is selected in every fold and reaches held-out Final correlation r = 0.69 (MAE 22.1), versus GPT-5.6-Terra at r = 0.61 (MAE 24.7).The backbone selection and reported alignment therefore are not an artifact of choosing the backbone on the evaluation cases.Source: §D.2 · p. 30
  • On the same panel videos, human and judge generator rankings differ only by a swap between Grok-1.5 and MiniMax-H3 (Spearman ρ = 0.94).The full-benchmark ranking differs for Wan-2.7 because it uses a different case set.Source: §D.2 · p. 30, Table 9 · p. 31
  • A Final score is available for 98.7% of judged videos and for at least 96.9% of every group, so withheld scores cannot drive model comparisons.Most missing Physics scores are unattempted videos, concentrated in Cosmos-3 (36.4%), which receive S = 0 rather than being dropped.Source: §D.2 · p. 30, Table 10 · p. 31

E.1 GENERATOR SCORES AND DOMAIN PROFILES

  • EGO2ACTJUDGE scores successful human recordings 25.3 points above unsuccessful ones (95.4 versus 70.1), while VideoScore does not separate them (87.5 versus 88.2 on its original score).Neighboring generator intervals often overlap, so small score differences should not be over-interpreted given judge run-to-run variability.Source: §E.1 · p. 31

E.2 TASK FEATURES AND JUDGE-ERROR DIAGNOSTICS

Across action types, subgoal completion and physical validity vary, but attachment remains relatively difficult across generators. The analyses also report prompt sensitivity, concentrated failure profiles, and limits on interpreting case-level feature associations.

E.2 TASK FEATURES AND JUDGE-ERROR DIAGNOSTICS

  • In the 25-case panel, observed action count and reference duration have weak associations with case-balanced scores (ρ = 0.06 and 0.04); across benchmark cases, longer tasks score lower on Task (ρ = −0.29).Cases within the 15-second budget reach 68.6% Task completion versus 54.8% for longer ones; these are descriptive patterns.Source: §E.2 · p. 33, Figure 6 · p. 33
  • Case-level feature patterns are descriptive rather than controlled: action-family groups overlap, some contain only three cases, and only one of 25 cases is labeled Heavy clutter.No tested feature association survives Benjamini–Hochberg correction; reference duration’s judge-error association is ρ = 0.46, uncorrected p = 0.02, q = 0.09.Source: §E.2 · p. 33, Figure 7 · p. 34, Table 15 · p. 34

E.3 SUBGOAL-LEVEL ACTION-TYPE ANALYSIS

  • Attachment/connection has 38.6% Task completion and 66.7% judgeable Physics validity, below relocation/arrangement at 51.1% and 75.5%; surface treatment is highest at 62.0% and 82.2%.Skipped operations account for 35–43% of subgoals in every family; material transfer/mixing has the largest incomplete-attempt share (T2, 12.3%), and attachment/connection the highest invalid-interaction rate (P2, 9.7%).Source: §E.3 · p. 35, Figure 8 · p. 35
  • Attachment/connection trails relocation/arrangement on Task completion for every generator by 1.6–21.3 percentage points, and on judgeable Physics validity for five of six by 2.4–19.2 points.Cutting/separation and device activation have too few subgoals for reliable estimates.Source: Figure 9 · p. 35, §E.3 · p. 36

E.4 PROMPT-EXPANSION DIAGNOSTIC

  • Expanded prompts raise Wan-2.7’s average Final score from 70.4 to 84.9, improving 13 of 15 prompt–case pairs, while Cosmos-3 remains at 25.2 in both conditions.The five-case study was outcome-selected, and original-prompt scores came from benchmark videos rather than fresh reruns, so it indicates sensitivity rather than a causal effect.Source: §E.4 · p. 36, Table 16 · p. 36

F.1 SUBGOAL-LEVEL GATE PROFILES ACROSS MODELS

  • Across generators, T0 and T2 account for 88.7% of pooled failed Task judgments, while Physics N/A, P0, and P2 make up 93.4% of pooled non-passing or unevaluable judgments.The dominant Task failures are omitted operations or attempts that do not reach the target state.Source: §F.1 · p. 36, Figure 10 · p. 37

F.2 SCOPE AND CASE SELECTION

  • The qualitative analysis selects recurring, clearly localizable failures and assigns each rollout’s primary category by its first observable violation, not an inferred internal cause.Examples illustrate error patterns rather than estimating their prevalence; the taxonomy is not exhaustive.Source: §F.2 · pp. 37–38, §F.3 · p. 38

F.3 OVERVIEW OF ERROR CATEGORIES

  • Task failures concern missing or incomplete goal operations, whereas Physics failures concern inconsistent scene states or physically infeasible transitions.A rollout can show multiple related errors, with secondary or downstream errors described alongside its primary category.Source: §F.3 · p. 38

F.4 SELECTED EXECUTION EXAMPLES

Selected examples show that generated manipulation videos can miss required prerequisites or final outcomes, and can violate physical continuity, access, object relations, and mechanism behavior.

F.4 SELECTED EXECUTION EXAMPLES

  • Attempts sometimes skip prerequisite states or perform an action without achieving its required result, such as wiping that leaves markings or packing that leaves an item protruding.Assembly can also manipulate the right objects without establishing the required relations.Source: §F.4 · p. 38
  • Task-directed videos can still violate physics through duplicated or identity-changing objects, container entry without an access route, or invented hinges.The examples motivate checking continuity and interaction plausibility separately from task completion.Source: §F.4 · pp. 38, 43, 46
  • Access and boundary failures include pouring beans onto a still-seated jar lid and lowering a bulb through a closed bin lid.The examples show why checking whether an opening was established matters for container interactions.Source: §F.4 · pp. 40, 44
  • Some outcomes have incorrect object orientation or relations, including inverted cup stacks, a lid on the wrong container, and headphones placed rather than hung.These examples distinguish manipulating relevant objects from achieving the requested final arrangement.Source: §F.4 · p. 41
  • Rigid objects can deform implausibly during handling, while laptop lids and doll heads may rotate or hinge along incorrect mechanisms.Examples include a rigid e-reader casing opening like a soft pouch and an invented hinge on a nesting doll.Source: §F.4 · pp. 45–46

G RUBRIC ABLATIONS AND REPEATABILITY

The ablations show that sequential rubric gates matter for agreement with human judgments, while controls reveal uneven sensitivity to physical violations. Judge and planning variability also place limits on interpreting individual video scores.

G RUBRIC ABLATIONS AND REPEATABILITY

  • Removing T1 raises Task MAE from 0.292 to 0.497, and removing P1 raises Physics MAE from 0.432 to 0.922 in the human-rubric ablation.The comparisons use fixed annotations rather than rerunning annotation.Source: G RUBRIC ABLATIONS AND REPEATABILITY · p. 47
  • Independent judge gates lift Final scores by 6.8 points on average, while Final r falls from 0.69 to 0.65, CCC from 0.62 to 0.51, and MAE rises on all three axes.Independent scoring credits later checks after an earlier failure, unlike human consensus.Source: G RUBRIC ABLATIONS AND REPEATABILITY · p. 47
  • Task and Physics plans use their case’s most common subgoal count in 91% and 92% of plans, but ordering constraints agree for only about 60% of matched subgoals with prerequisites.Count changes account for about a fifth of run-to-run Task variation; the authors caution that stable plans need not be correct.Source: G RUBRIC ABLATIONS AND REPEATABILITY · pp. 47–48
  • Ghost duplication is detected in 55% of controlled cases, versus 7% for teleport and 10% for swap; no corrupted video scores above its unedited rerun.The Physics pass uses 24 evenly spaced frames, and brief edits may fall between samples.Source: G RUBRIC ABLATIONS AND REPEATABILITY · p. 48
  • A single video’s Final score varies by about 10 points across runs, so the authors recommend aggregate model comparisons rather than single-clip verdicts.Model means average over roughly 330 videos, and the judge ranks six generators almost exactly as humans do (Spearman ρ = 0.94).Source: G RUBRIC ABLATIONS AND REPEATABILITY · p. 49
  • Two judge runs on the same videos correlate at r = 0.93 for Task and r = 0.89 for Final, but less for Physics at r = 0.76.Single-run noise is largest on generated videos and smallest on successful human recordings; averaging model scores reduces random noise but not systematic bias.Source: G RUBRIC ABLATIONS AND REPEATABILITY · pp. 49–50

H LIMITATIONS AND FUTURE WORK

The paper identifies limits in what its judge evaluates, the breadth of its human study, and the comparability of adapted baselines.

H LIMITATIONS AND FUTURE WORK

  • EGO2ACTJUDGE assesses visible subgoal completion and physical plausibility, but does not test whether passing rollouts help downstream learning or planning.The paper describes this visual check as necessary but not sufficient.Source: H LIMITATIONS AND FUTURE WORK · p. 50
  • The 600-video human study covers 25 of 110 cases, so it reflects only part of benchmark diversity.The authors suggest annotating more cases, expanding domains, and studying longer tasks as generation lengths grow.Source: H LIMITATIONS AND FUTURE WORK · p. 50
  • Baseline adaptations may not be the best fit for every evaluator, and the comparison omits EgoForge and EgoTV for stated scope mismatches.The authors propose method-specific adaptations or reference-free egocentric evaluators for broader and fairer comparisons.Source: H LIMITATIONS AND FUTURE WORK · p. 51
Loading 2610.01092v1…