WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions

1 Introduction

• Existing benchmarks do not jointly test action-conditioned predictions against physical execution, particularly across long trajectories with successful and failed behaviors.Source: §1 · p. 2
• WorldOracle tests whether an ACWM’s predicted rollout matches a paired real execution given the same initial observation and low-level action trajectory.Source: §1 · p. 2
• The benchmark evaluates causal obedience, distribution transport, and temporal endurance by replaying actions and testing across action sources, embodiments, and long rollouts.Source: §1 · p. 2
• WorldOracle compares predictions with real executions along physical state correctness, interaction spatial alignment, key interaction area fidelity, and multi-view action coherence.Source: §1 · p. 2, Figure 2 · p. 3
• Its real-world suites cover varied interaction dynamics and embodiments, with long multi-segment episodes that include successful and failed executions.Source: §1 · p. 3
• Across seven representative ACWMs, visually plausible rollouts can misrepresent interaction outcomes, and strong short-horizon performance does not consistently predict reliable long-horizon performance.Source: §1 · p. 3

2.1 Embodied World Models

• Action-conditioned world models use low-level actions to control generated rollouts, with approaches varying in action injection, geometric guidance, memory, and predicted states.Source: §2.1 · p. 4
• WorldOracle lets each ACWM use its native conditioning interface while evaluating predictions against paired real-world execution references.Source: §2.1 · p. 4

2.2 Embodied World Model Benchmarks

• Embodied world-model benchmarks assess video properties such as motion, physical consistency, task completion, and planning, but primarily condition on task-level instructions.Source: §2.2 · p. 4
• Recent execution-oriented benchmarks remain simulation-based or use real-to-sim replay, and do not directly assess coherence with corresponding real-world executions.Source: §2.2 · p. 4

3 The WorldOracle Benchmark

• WorldOracle evaluates a predicted rollout against the paired real-world rollout generated from the same initial observation and recorded action sequence.Source: §3 · p. 5
• The benchmark tasks cover rigid-body, deformable-object, fluid, and articulated interactions, with successful and failed outcomes designed to be visually unambiguous.Source: §3 · p. 5

3.1 Data Curation

• The AgiBot G2 training suite contains 1,779 episodes across four tasks, while YAM uses ABC-130K and human-hand models are evaluated zero-shot.Source: §3.1 · p. 5
• Evaluation includes held-out human-operated episodes across three embodiments and a closed-loop policy-rollout split on AgiBot G2.Source: §3.1 · p. 6
• Episodes are segmented at observable outcomes; a VLM generates physical-state criteria, and frame-wise masks identify manipulated objects and robot arms or hands.Source: Figure 3 · p. 5, §3.1 · p. 6

3.2 The WorldOracle Metrics

• Physical state correctness compares predicted outcome keyframes with criteria annotated from paired real keyframes, averaged across criteria and execution segments.Source: §3.2 · p. 6
• Interaction spatial alignment tracks whether robot arms or hands and manipulated entities remain aligned with their real-world trajectories over sampled frames.Source: §3.2 · p. 6
• Key interaction area fidelity evaluates masked interaction regions, while multi-view action coherence aggregates robot-arm alignment across synchronized camera views.Source: §3.2 · pp. 6–8

4 Results

• 88.1–93.2% of metric-induced pairwise preferences agreed with human judgments, with Spearman correlations of 0.72–0.82.The VLM judge marked all physical criteria satisfied on reference keyframes, yielding 100% criterion–reference agreement; human annotators also checked the criteria against reference images.Source: §4 · p. 8
• Physical state correctness and interaction spatial alignment capture complementary failure modes, while key-area fidelity falls with interaction-region corruption but not background-only corruption.These controlled analyses test whether the metrics respond to errors in their intended evaluation dimensions.Source: §4 · p. 9

4.1 Observations and Challenges

• GE-Sim 2.0 scores 81.40/73.49 on interaction spatial alignment in the 30s/long AgiBot setting and 74.97/59.78 on YAM.It also scores 78.70 on the AgiBot policy-rollout distribution; the authors associate its stronger alignment with pixel-aligned pose maps but do not establish that conditioning alone causes the improvement.Source: §4.1 · p. 10
• WorldOracle-SV scores for DreamDojo and Cosmos 3 fall from 50.56 to 27.08 and 53.99 to 26.60, while GE-Sim 2.0 declines from 73.86 to 63.33.All evaluated models decline as rollout horizons increase, though at different rates.Source: §4.1 · p. 10
• GE-Sim 2.0 scores 73.49 in interaction spatial alignment and 96.96 in multi-view action coherence, but 60.26 in physical state correctness and 56.24 in key interaction area fidelity.The results illustrate that close action alignment can coexist with differences in grasp state, contact, object motion, or task outcome.Source: §4.1 · p. 10

5.1 Analysis across Interaction Dynamics

• GE-Sim 2.0 scores 73.9 in average WorldOracle-SV on human-operated AgiBot G2, versus 54.0 for the strongest alternative, with scores of 81.4 in interaction spatial alignment and 67.6 in key interaction area fidelity.The authors suggest pixel-aligned action conditioning as one possible explanation for its spatial grounding advantage.Source: §5.1 · pp. 10–12, Figure 5 · p. 11
• On YAM, GE-Sim 2.0 leads with average WorldOracle-SV 65.8, but its margin over the strongest alternative narrows from 19.9 points on AgiBot G2 to 6.8 points.It retains the highest interaction spatial alignment (75.0) and key interaction area fidelity (54.6) on YAM; the authors associate the narrower margin with domain shift.Source: Figure 7 · p. 12, §5.1 · p. 14
• On human-hand evaluations, OSCAR balances interaction spatial alignment (47.5) with the highest key interaction area fidelity (27.5), while JointControlVideo leads in alignment (57.2) but has the lowest key-area score (20.5).Cosmos 3 scores 33.7 in alignment and 22.2 in key interaction area fidelity.Source: Figure 8 · p. 12, §5.1 · pp. 14–15
• Models generally perform well on pour water for physical state correctness, key interaction area fidelity, and WorldOracle-SV, and score higher in interaction spatial alignment on pull drawer.The authors suggest the large cup and contact region may simplify grasp localization, while the drawer’s salient appearance and constrained motion may aid localization.Source: §5.1 · pp. 12–13
• Multi-view action coherence declines from 97.5 on human-operated AgiBot trajectories to 64.1 on YAM for GE-Sim 2.0, and from 65.6 to 41.9 for Ctrl-World.The authors link the degradation likely to the domain gap, including differences in tasks, scenes, and camera observations.Source: Figure 9 · p. 13, §5.1 · p. 15

5.3 Temporal Degradation across Cumulative Rollout Horizons

• On AgiBot G2, GE-Sim 2.0’s WorldOracle-SV score falls from 86.1 at 10 seconds to 63.4 at 120 seconds, while Cosmos 3 leads other models through 60 seconds before IRASim overtakes it at 70 seconds.The figure reports cumulative-horizon metrics from 10 to 120 seconds.Source: Figure 11 · p. 14, §5.3 · p. 16

5.2 Entity-Centric Error Decomposition

• Across models, embodiments, and data distributions, object alignment trails robot-arm alignment; on human-operated AgiBot trajectories, IRASim scores 29.1 on objects versus 78.8 on the arm.Human-hand models also score substantially lower than robot models, particularly in key interaction area fidelity; the strongest reaches 15.2 on object fidelity and 39.7 on hand fidelity.Source: Figure 10 · p. 13, §5.2 · pp. 15–16

5.3 Temporal Degradation across Cumulative Rollout Horizons

• YAM results are worse than AgiBot G2 amid a larger train–test domain gap, while human-hand models are evaluated zero-shot and face articulated motion, deformable appearance, and fine-grained contact.Across human-hand models, physical state correctness is more durable than spatial alignment and key-area fidelity; JointControlVideo retains 56.5 at 120 seconds versus 24.9 and 8.4, respectively.Source: §5.3 · pp. 16–17

6.1 Human–Metric Alignment Study

• Human agreement ranges from 88.1% to 93.2%, with pair-level Spearman correlations of ρ = 0.72–0.82 across metric score-gap groups.Agreement remains above chance even for small score gaps.Source: §6.1 · p. 17, Figure 14 · p. 18

6.2 Metric Case Studies

• Physical state correctness and interaction spatial alignment capture complementary failures: aligned motion can yield the wrong state, while a correct outcome can follow a misaligned trajectory.The drawer example scores 80.8 for spatial alignment and 0.0 for physical state correctness; the towel example scores 100.0 and 9.7, respectively.Source: §6.2 · p. 17, Figure 15 · p. 18
• Key interaction area fidelity falls from 100.0 to 43.7 for Pick Fruit and to 27.5 for Pour Water under strongest mask-local blur, while background-only blur leaves scores at 100.0.The blur sensitivity is monotonic as mask-local blur increases.Source: §6.2 · p. 19, Figure 16 · p. 19

6.3 Controlled Metric Selectivity

• Outcome swapping reduces physical state correctness by 67.9% on average, more than key interaction area fidelity (34.3%), interaction spatial alignment (20.0%), or multi-view action coherence (12.3%).The largest response occurs in the intended physical state metric, despite accompanying changes in local appearance and geometry.Source: Figure 17 · p. 19, §6.3 · p. 20
• Delaying one auxiliary view by 0.1–1.0 seconds increases the mean multi-view action coherence decrease from 0.7% to 4.4%, while the three top-head metrics remain unchanged.At a 0.5-second delay, the decrease is 2.9%.Source: §6.3 · p. 20, Figure 18 · p. 20

6.4.1 Nuisance Robustness and Model-Ranking Stability

• Appearance changes leave geometry-centered metrics nearly stable, while key interaction area fidelity responds to visible degradation; rankings generally persist under mask noise and are identical under temporal subsampling.At 0.5× resizing, key interaction area fidelity drops 3.8 points; temporal subsampling to 0.25× preserves rankings (ρ = 1.00).Source: §6.4.1 · pp. 21–22, Figure 19 · p. 21

6.4.2 Hyperparameter and ranking stability.

• Hyperparameter rankings remain stable except when spatial feature-evidence weights are halved, which lowers the mean by 41.8 points and rank agreement to ρ = 0.71; key-area ranking agreement stays at ρ ≥0.96.The original spatial ranking is fully recovered at 0.75θ0 and above.Source: §6.4.2 · p. 22, Figure 20 · p. 22

• For datasets with episode-level metadata, duration is frames divided by nominal frame rate; when only aggregate timing is available, the reported range is used.Source: §7 · pp. 22–23
• Many atomic manipulation datasets operate primarily below 30 seconds, whereas datasets with longer, continuous, or compositional behavior extend beyond that boundary.Source: §7 · p. 23, Figure 21 · p. 23
• The paper treats 30 seconds as an approximate transition and evaluates the first 30 seconds as a complementary short-duration regime alongside full-episode evaluation.Source: §7 · p. 23

8 Conclusion

• WorldOracle evaluates real-world matching through four metrics covering physical outcomes, spatial action alignment, interaction-region fidelity, and multi-view action coherence.Source: §8 · p. 23
• Across seven ACWMs, visually plausible rollouts can misrepresent object dynamics and interaction outcomes even when arm motion is accurately reproduced, and short-horizon strength does not consistently carry over to long-horizon prediction.Source: §8 · p. 23
• Persistent action grounding and accurate modeling of object interactions remain important challenges for improving ACWMs.Source: §8 · p. 23

Task overview

• The four tasks cover distinct interaction dynamics while retaining visually verifiable outcomes.Source: A Benchmark Task Definition · p. 27

Pick fruit

• Pick fruit tests repeated transfers of designated fruits into containers, with failures including empty grasps, drops, collisions, and unreachable placements.Source: A Benchmark Task Definition · p. 27

Hang towel

• Hang towel tests picking up, hanging, removing, or returning towels, with failures including dropped towels and unsuccessful rack placement.Source: A Benchmark Task Definition · p. 27

Pour water

• Pour water tests liquid transfer between cups, with colored liquid making transfers and spills more observable; failures include incomplete transfer and spills.Source: A Benchmark Task Definition · p. 27

Pull drawer

• Pull drawer tests opening or closing a selected tier, with failures including missing the handle or losing contact while pushing or pulling.Source: A Benchmark Task Definition · p. 27

B Additional Details on Data Curation

• Episodes include multiple interaction segments and vary object identities, quantities, sizes, initial configurations, and illumination conditions.Source: B Additional Details on Data Curation · p. 27

B.1 Training Suite

• The AgiBot G2 training split contains 1,779 episodes across four tasks, recorded at 30 FPS for approximately one minute each.Within each task, the robot starts from a fixed initial pose, and only its two arms and grippers are controlled; scene variation is controlled and restricted.Source: §B.1 · p. 27, Table S1 · p. 28

B.2 Human-Operated Distribution

• Human-operated evaluation covers AgiBot G2, YAM, and human-hand embodiments, with episodes lasting approximately two minutes and containing multiple consecutive interaction segments.AgiBot G2 and human-hand splits each have 100 episodes, 25 per task; YAM has 75 episodes, 25 each for pick fruit, hang towel, and pull drawer.Source: §B.2 · pp. 27–28, Table S2 · p. 28
• The evaluation suite contains 335 episodes from human-operated and policy-rollout distributions.Table S2 reports the suite’s total; the supplied passage does not give further episode-count breakdowns.Source: Table S2 · p. 28
• AgiBot G2 evaluation expands training visual variation with held-out objects, scene configurations, and illumination conditions while retaining the same task semantics.The design evaluates generalization of learned action–outcome relationships to new visual configurations rather than reproduction of memorized trajectories.Source: §B.2 · p. 28

B.3 Policy-Rollout Distribution

• The policy-rollout split records closed-loop interactions of a π0.5 policy with the physical AgiBot G2 and contains 60 episodes, 20 each for pick fruit, hang towel, and pour water.The policy was trained using the curated training data; its observations and low-level actions were recorded for evaluation.Source: §B.3 · p. 28

C.1 Physical Criteria Annotation and Evaluation

• GPT-5.5 authors criteria from the initial observation and outcome keyframe, then scores each criterion as visibly satisfied or not in the predicted keyframe.A revision pass normalizes entity names against the initial object inventory without changing asserted states.Source: §C.1 · pp. 28–29
• For fruit tasks, criteria track each identifiable fruit’s location or absence, using initial-frame identities and omitting states that are visually ambiguous.Judging matches objects to the closest inventory item and reserves failures for clear contradictions, not shape or type mismatches alone.Source: §C.1 · pp. 29–30
• For towel tasks, criteria specify each towel’s color, relative position when needed, and state; judging distinguishes table, rod, grasp, and absent states.Identity relies on color and relative position rather than shape, which can change with folds, draping, occlusion, or viewpoint.Source: §C.1 · pp. 30–31
• For pouring tasks, criteria combine each cup’s pose or position with water content, identify source-target relations, and note only clear spills.Judging checks cup pose, grasp, water state, pouring relation, and spill evidence, while tolerating appearance changes but enforcing asserted states and spatial relations.Source: §C.1 · p. 32
• The drawer task concerns opening or closing one of three differently colored drawers, with criteria authored from a keyframe and initial-frame drawer and object identities.The provided part specifies the task setting and image references for annotation.Source: §C.1 · p. 33
• Physical criteria identify drawers by color, score open or closed state strictly, and list any inside object as a separate criterion.Ambiguous objects are omitted during authoring; evaluation requires strong geometric evidence of opening and independently checks drawer state and object identity.Source: §C.1 · pp. 33–34
• GPT-5.5 judged 100% of authored criteria satisfied on their ground-truth outcome keyframes, supporting evaluator consistency for physical-state correctness.The consistency check applies the physical-state judge to each real-world outcome keyframe used for criteria authoring.Source: §C.1 · p. 34

C.2 Interaction Region Annotation

• SAM3 [5] generates per-view temporal masks for manipulated objects and arms or hands; main-view masks support spatial-alignment and key-area metrics, while all views support action-coherence evaluation.Object tracks cover task-relevant entities whose states change under actions; arm tracks represent robot arms and grippers or interacting human hands.Source: §C.2 · p. 34

D.1 Physical State Correctness

• Physical-state correctness uses outcome-keyframe criteria for task states such as object identity, location, grasp, articulation, contact, and liquid transfer.A VLM judges whether each criterion is visibly satisfied; aggregation preserves partial correctness and weights interaction segments equally.Source: §D.1 · p. 35

D.2 Interaction Spatial Alignment

• Interaction spatial alignment matches DINOv3 features inside object and arm masks against predicted observations, scoring correspondence displacement and confidence.The primary top-head view is sampled every 10 frames; median displacement reduces sensitivity to isolated incorrect matches.Source: §D.2 · pp. 35–36

D.3 Key Interaction Area Fidelity

• Key interaction area fidelity evaluates action-relevant regions because whole-frame metrics can overlook errors around objects, arms, and contact regions.Dilated masks define crops for masked PSNR, LPIPS, and V-JEPA similarity, whose normalized scores are combined with a geometric mean.Source: §D.3 · pp. 38–39

D.4 Multi-View Action Coherence

• Multi-view action coherence scores robot-arm alignment across three synchronized views, since objects may be occluded or only partly visible in auxiliary views.The view scores receive equal weight, so a missing or poorly aligned view lowers the frame score.Source: §D.4 · pp. 39–40

E Training and Implementation Details

• IRASim generates 15 future frames at 320 × 256 from one history frame, while Ctrl-World generates 5 at 384 × 256 from six history frames.Both models use a 14D dual-arm action interface.Source: E Training and Implementation Details · p. 40
• DreamDojo generates 12 future frames at 640 × 480 from one preceding frame, while Cosmos 3 generates 16 at 832 × 480 from one preceding frame.DreamDojo groups actions into four-action chunks; Cosmos 3 uses a 20D dual-arm representation padded to the model’s 64D action space.Source: E Training and Implementation Details · p. 41
• GE-Sim 2.0 jointly generates synchronized top-head, left-hand, and right-hand views, with 25 future frames per chunk at 512 × 384 per view.It conditions geometrically using pixel-aligned trajectory maps and camera Plücker ray maps.Source: E Training and Implementation Details · p. 41
• The π0.5 policy is fine-tuned from its DROID checkpoint on the AgiBot training suite, and its 51K-step checkpoint supplies policy-rollout trajectories.Training uses eight NVIDIA H100 GPUs.Source: E Training and Implementation Details · pp. 40–42

F Limitations

• A single real-world execution is the reference, so WorldOracle may penalize alternative plausible outcomes under stochastic contact, fluid, or deformable-object dynamics.Matched repeated executions would be needed to estimate real–real variability.Source: F Limitations · p. 42
• The benchmark covers three embodiments and four tasks, while policy-rollout evaluation is currently limited to AgiBot G2.The authors note that the benchmark does not represent the full diversity of robots, environments, and interaction types.Source: F Limitations · p. 42
• Different training support across embodiments means results should not be interpreted as a controlled comparison of cross-embodiment generalization.AgiBot has dedicated training data, YAM uses public data with a larger domain gap, and human-hand predictions are evaluated zero-shot.Source: F Limitations · p. 42
• Because evaluated models differ in architecture, pretraining data, scale, and action representation, observed performance patterns cannot establish which design choices cause them.The authors call for controlled training and architectural ablations to verify these hypotheses.Source: F Limitations · p. 42