PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation

Peterson Co, Sicheng Hu, Chunxuan Jiao, Hongyang Cheng, Yulin Luo, Yijie Xu, Sixiang Chen, Zhongxia Zhao, Zihao Wang, DaFeng Chi, Peidong Liu, YuTong Chen, Henghua Liu, Zhihao Yuan, Huizhu Jia, Yuzheng Zhuang, Tianle Zhang, Liang Lin, Huajie Tan, Shanghang Zhang

arXiv:2608.09298v1cs.ROcs.AI

TL;DR

ACWM evaluation often provides limited diagnostic evidence about whether supplied actions produce corresponding motion and grounded environment responses. WorldSimProbe formalizes this simulator contract and tests it through five controlled suites; across evaluated models, it reveals systematic realization degradation and recurring interaction failures.

  • Problem

    Faithful simulation requires action-to-motion correspondence across feasible controls and environment responses grounded in realized motion.

    Source: §3.2 · p. 3, §3.1 · p. 3

  • Method

    WorldSimProbe operationalizes the Observable Simulator Contract through five controlled suites testing action realization, interaction grounding, and interaction dynamics.

    Source: §3.2 · p. 3, §3.1 · p. 3

  • Results

    Across six open-source ACWMs and three platforms, fidelity degrades under trajectory and behavior-source shifts, with recurring interaction-grounding and dynamics failures; human judgments and downstream policy performance support these diagnoses.

    Source: §6 · p. 8

  • Takeaways & Limitations

    WorldSimProbe localizes where, when, and how simulator fidelity breaks down beyond coarse, task-directed evaluation.

    Source: §6 · pp. 7–8

    The controlled framework assumes exogenous scene changes are absent or fixed so environment responses can be attributed to supplied actions through realized motion.

    Source: §3.1 · pp. 2–3

Abstract

from arXiv · show

Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation. Realizing this promise requires precise action-conditioned transitions rather than merely plausible outputs. Yet their applicability remains difficult to establish because prevailing evaluations emphasize visual quality, task outcomes, or coarse rollout-level responsiveness without directly testing simulator fidelity. To address this gap, we evaluate ACWMs through the observable capabilities expected of physical simulators. Accordingly, we formalize Observable Simulator Contract, a minimal contract that any action-conditioned physical simulator should satisfy: supplied actions must induce corresponding agent motion, and environment responses must be grounded in that realized motion. To operationalize this contract, we introduce WorldSimProbe, comprising five controlled suites spanning local control sensitivity, global trajectory variation, source-diverse actions, interaction grounding, and dynamics. Suite-specific evaluators assess simulator-relative calibration, dense action-to-motion correspondence, false-interaction grounding, and primitive-level dynamics. We evaluate six open-source ACWMs on more than 18,000 instances across RoboTwin, ManiSkill, and LIBERO. World-SimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics, and benchmark signals consistent with human judgments and downstream outcomes. Together, this capability-based framework provides a transparent, and standardized paradigm for diagnosing ACWM simulator fidelity beyond coarse, task-directed evaluation.

2. Related Work

Embodied world-model evaluation has expanded beyond visual fidelity, but existing approaches provide incomplete evidence about action following and simulator fidelity. WorldSimProbe is positioned as tracing supplied actions through realized motion to environment response.

Output-oriented and Indirect Evaluation

  • Language- or video-oriented evaluations provide coarse or indirect evidence of action following because instructions admit multiple trajectories and inferred actions rely on auxiliary estimators.These approaches assess visual fidelity, semantic alignment, motion correctness, task completion, or physical plausibility.Source: §2 · p. 2

Benchmark comparison

  • Table 1 compares embodied world-model benchmarks by whether they test graded local-to-global controls, multiple action sources, controlled interventions, and separate interaction occurrence from dynamics.Its checks indicate explicit evaluation of each capability.Source: Table 1 · p. 2

Explicit Action-conditioned Evaluation

  • Explicit action-conditioned benchmarks enable interventions on control magnitude, timing, and trajectory, but do not jointly trace fidelity across control scales and distributions through environment response.Source: §2 · p. 2

Downstream-utility Evaluation

  • Downstream-utility evaluations measure application value through policy and data-generation outcomes, but aggregate success offers limited failure localization and task-directed controls can underrepresent counterfactual or out-of-distribution behavior.WorldSimProbe traces supplied action to realized motion and realized motion to environment response.Source: §2 · p. 2

3. Simulator Faithfulness Framework

The framework defines simulator faithfulness as linked consistency between supplied actions, realized agent motion, and grounded environment responses. It also motivates testing action realization across feasible controls, not only task-associated trajectories.

3.1 Observable Simulator Contract

  • A rollout meets the Observable Simulator Contract only when supplied actions produce corresponding motion and environment changes are supported by that motion and its contact events.The controlled evaluation holds exogenous scene changes absent or fixed, allowing responses to be attributed through realized motion.Source: §3.1 · pp. 2–3

3.2 Feasible Motion Coverage for Action Realization

  • Figure 2 projects 12-dimensional arm-motion trajectories, excluding gripper dimensions, to illustrate feasible-action coverage for task-conditioned, counterfactual, and source-diverse actions.The example is the RoboTwin handover_mic task with 1,500 instances.Source: Figure 2 · p. 3
  • Action realization requires correspondence for each supplied action and coverage across controls executable in the current state, while ACWMs often cover narrower task-associated subsets.Task success or failure diversity within task-protocol support does not establish coverage of the full feasible-action distribution.Source: §3.2 · p. 3

4. WorldSimProbe: A Diagnostic Benchmark

WorldSimProbe uses five controlled suites to test action realization and environment response at distinct stages of the simulator chain. Its evaluators target local calibration, trajectory and source-specific motion, unsupported interactions, and primitive dynamics.

Benchmark overview

  • The five suites progress from local calibration through global and source-specific action realization to interaction grounding and dynamics, enabling stage-specific failure localization.Figure 3 presents the operational overview and suite-specific evaluators.Source: §4 · p. 3, Figure 3 · p. 4

Task 1: Local Action Calibration

  • Task 1 varies one non-gripper action dimension at two magnitudes while holding the task and outcome fixed, then compares within-model response scaling with the simulator’s scaling.Its score rewards matching simulator-relative scaling and decreases as the relationships diverge; reversed or degenerate scaling receives zero. Figure 4(a) plots mean perturbation–response curves across 50 RoboTwin tasks, and panel (b) marks first persistent failure in a ManiSkill StackCube case study.Source: §4 · pp. 3–4, Figure 4 · p. 5

Task 2: Global Trajectory Coverage

  • Task 2 tests global trajectory coverage through cross-task receiver–donor replay and compares generated motion with simulator-reference flow at active reference-arm pixels.RMFA is sensitive to motion direction, magnitude, and spatial alignment; the donor actions expand evaluated support while retaining simulator-grounded references.Source: §4 · p. 4

Task 3: Action-Source Behavior Preservation

  • Task 3 tests preservation of source-specific behavior using successful and unsuccessful human-teleoperator and policy-checkpoint trajectories as simulator-referenced controls.Lower RMFA scores indicate reversion to canonical execution rather than preserving differences in speed, smoothness, corrective behavior, or strategy.Source: §4 · p. 4

Task 4: Interaction Grounding

  • Task 4 constructs simulator-validated no-contact cases that retain interaction-like cues, then flags object displacement beyond a movement threshold as unsupported interaction.The cases vary object position or action streams, or add duplicate objects; preliminary analysis found failures dominated by contact hallucination rather than missed supported contact.Source: §4 · pp. 4–5

Task 5: Interaction Dynamics

  • Task 5 evaluates eight interaction primitives using a VLM judge given specifications of expected agent motion and object response, retaining simulator labels confirmed by human annotators.On the human-verified set, the judge agrees with majority human labels on 94–97% of instances across simulators.Source: §4 · p. 5

5. Experiments and Results

Across six open-source ACWMs and three platforms, WorldSimProbe finds that simulator fidelity varies by platform and chain stage, and degrades under unfamiliar controls. Interaction failures show recurring grounding and primitive-specific dynamics patterns, while human judgments and downstream policy results support the benchmark’s diagnoses.

5.1 Experimental Setup

  • The evaluation covers six representative open-source ACWMs on more than 18,000 controlled rollouts from RoboTwin, ManiSkill, and LIBERO.Models share initial observations, native action trajectories, and three diffusion seeds per instance.Source: §5.1 · p. 5

5.2 Overall Simulator Faithfulness (RQ1)

  • Ctrl-World leads overall only on LIBERO, while LingBot-VA leads RoboTwin and ManiSkill; cross-platform rankings are consistent (mean pairwise Spearman ρ = 0.695).Suite-level leaders vary: Ctrl-World leads Interaction Grounding on RoboTwin and LIBERO and the cross-platform macro score, while LingBot-VA leads on ManiSkill.Source: §5.2 · p. 6

5.3 Robustness of Action-to-Motion Correspondence (RQ2)

  • Fidelity declines as receiver–donor motion mismatch increases (average ρ = −0.433), and all six models score 10.8–16.1 points higher on late- than early-policy checkpoints.For local calibration, both LingBot-VA and Cosmos-3-Nano lag the simulator failure boundary in the StackCube case study.Source: §5.3 · pp. 6–7, Figure 5 · p. 6

5.4 Structure of Agent–Environment Interaction Failures (RQ3)

  • Grounding is strongest for distractors (75.0) and proximity (54.5), but falls to 38.6 for false contact; dynamics fidelity varies more by primitive than by model.Tap is strongest (40.8–56.9), while shake is nearly absent (0.0–1.2); model averages remain in a narrow 17.7–21.8 range.Source: §5.4 · p. 7, Table 3 · p. 7, Figure 6 · p. 7

5.5 Benchmark Validity and Downstream Relevance (RQ4)

  • For Diverse/OOD controls, RMFA aligns with human judgments (ρ = 0.750), and OOD policy success follows the counterfactual-fidelity ordering: Ctrl-World 53%, BWM 34%, and Cosmos-3-Nano 21%.Standard-control policy success clusters at 78–86%, while VLM agreement with humans is 0.450 on Diverse/OOD samples.Source: §5.5 · p. 7

6. Conclusion and Future Work

WorldSimProbe reframes ACWM evaluation around simulator capabilities rather than plausible or task-successful rollouts. Its five suites reveal weaknesses missed by coarse protocols, with human judgments and downstream policy performance supporting the diagnoses.

6. Conclusion and Future Work

  • WorldSimProbe tests the chain from supplied action to agent motion and grounded environment response.The framework standardizes evaluation around capabilities expected of physical simulators.Source: §6 · p. 7
  • Across six open-source ACWMs and three platforms, fidelity degrades under trajectory and behavior-source shifts, and interaction failures recur across contact cues and dynamics primitives.Human judgments and downstream policy performance support these diagnoses.Source: §6 · p. 8
  • The current evaluation focuses on simulation; future extensions include real-world rollouts, longer-horizon closed-loop prediction, and deformable or multi-object interactions.The authors also identify targeted data collection and model refinement as possible uses.Source: §6 · p. 8

Supplementary Material

The supplement provides construction, evaluator, experimental, robustness, human-evaluation, and downstream-study details that support reproduction. It also describes the platforms and reference-data configuration used by WorldSimProbe.

Supplementary Material

  • The supplement documents benchmark construction, evaluator implementation, experimental settings, robustness analyses, human evaluation, and the downstream study.The main paper is described as self-contained; the supplement provides supporting evidence and reproduction details.Source: Supplementary Material · p. 9

A.1 Platforms and Task Coverage

  • WorldSimProbe uses three platforms selected for task and interaction diversity across single- and dual-arm embodiments.Table S1 summarizes task coverage and reference-data configuration.Source: §A.1 · p. 9

A.2 Task Data Generation

The data-generation pipeline creates suite-specific interventions, validates them against simulator references, and records accepted rollouts. It standardizes model inputs and temporal alignment while retaining instance-specific horizons.

A.2 Task Data Generation

  • Each task uses a dedicated simulator-API script to generate suite-specific interventions, execute them, apply reference-side checks, and record accepted rollouts and metadata.Test trajectories are generated independently of the released training demonstrations.Source: §A.2 · p. 9
  • For T1, one non-gripper action dimension is perturbed at two magnitudes while seed, dimension, direction, and time window remain fixed within each triplet.Triplets are retained only when references remain valid and successful and satisfy 0 < D(x0, xs) < D(x0, xℓ).Source: §A.2 · pp. 9–10
  • T2 pairs a receiver scene with a donor action trajectory from another task, retaining the pair only if full simulator replay is executable and valid.The simulator replay provides the counterfactual reference.Source: §A.2 · p. 10
  • T3 forms matched task, episode, and scene groups for expert, human, and policy trajectories, with independent operators imitating expert-rollout videos.Five operators provide the human trajectories; trajectories passing common reference-validity checks are retained as instances.Source: §A.2 · p. 10
  • T4 isolates false-positive interaction grounding with distractor, open-gripper, and spatial-proximity no-contact interventions.The interventions test whether models generate motion or object responses unsupported by controls or contact; selection uses simulator state before inference.Source: §A.2 · p. 10
  • T5 branches from successful trajectories near initial gripper–object contact and replaces remaining controls with one of eight scripted interaction primitives.Accepted instances are balanced by primitive within each platform; LIBERO includes push, pull, tap, and drop.Source: §A.2 · pp. 10–11

A.3 Reference Validation and Filtering

  • Common filtering rejects invalid controls, failed tracking, unintended collisions, incomplete rollouts, and references that do not realize the required condition.The final filtered test manifest contains 18,608 controlled evaluation instances.Source: §A.3 · p. 11, §A.5 · p. 11, Table S2 · p. 12

A.4 Observations, Actions, and Alignment

  • All six models receive the same initial observation, action stream, simulator seed, reference horizon, and three diffusion seeds; evaluation uses the external scene view.The manifest records platform, task, suite, embodiment, action streams, interventions, videos, states, and contact state.Source: §A.4 · p. 11, §A.5 · p. 12

B.2 Task 1 Temporal Alignment and MSE

The supplementary evaluators align generated and reference motion for action-calibration and trajectory measures, then assess interaction grounding and primitive dynamics. VLM and inverse-dynamics baselines provide additional evaluation signals.

B.2 Task 1 Temporal Alignment and MSE

  • Task 1 aligns perturbation rollouts to the generated original’s timestamps over their common duration, using nearest-frame matching to compare responses at matched physical times.The original–small and original–large distances contribute to calibration; a zero generated original–large distance receives score zero.Source: §B.2 · p. 12

B.3 Robot-Masked Flow Alignment

  • RMFA compares robot-region flow on aligned sequences using overlapping 2-second windows with a 1-second stride and fixed reference-calibrated thresholds.Flow is evaluated at 5 Hz on a shared 160 × 120 grid; reference-arm masks combine endpoint masks for each flow pair.Source: §B.3 · pp. 12–13

B.4 Interaction-Grounding Tracking

  • Interaction grounding gates on robot-arm centroid displacement, then flags object displacement above 10 pixels as unsupported interaction among gate-passing rollouts.Motion-gate or tracking failure receives zero; the thresholds are fixed from simulator-reference diagnostics.Source: §B.4 · p. 13

B.5 Interaction-Primitive Evaluator

  • An instance scores 1 only when the predicted primitive matches the reference and both agent- and object-motion judgments pass; invalid or unparsable primitive labels receive zero.Three annotators independently labeled a stratified sample of simulator-reference Task 5 rollouts; at least two agreed on the primitive for every rollout, yielding 100% consensus coverage.Source: §B.5 · p. 13

B.6 VLM and IDM Baselines

  • VLM ratings assess action-specification following, while platform-specific inverse-dynamics models predict actions from adjacent frames and are compared with reference actions using Micro-MSE.Human agreement for the inverse-dynamics baseline is measured by Spearman’s ρ between negative Micro-MSE and mean human rating.Source: §B.6 · p. 13

C.1 Model Training and Inference

Models are trained from official platform splits and evaluated with shared inputs and generation seeds. The visual-quality robustness diagnostic compares generated-video MUSIQ with matched simulator references.

C.1 Model Training and Inference

  • Each model–platform pair uses one checkpoint trained on its official split, and models receive the same observation, action trajectory, simulator seed, horizon, and three generation seeds.LingBot-VA and Cosmos-3 remain action-conditioned but use video-only decoding.Source: §C.1 · pp. 13–14

C.2 Compute and Software

  • Experiments ran on three identically configured nodes, each with eight NVIDIA H100 80GB GPUs, two Intel Xeon Platinum 8462Y+ CPUs, and 2 TiB RAM.The nodes ran Ubuntu 22.04.5.Source: §C.2 · p. 14

D.1 Visual-Quality Robustness Across Tasks

  • Visual quality is not a benchmark objective; the robustness diagnostic reports generated-video MUSIQ minus matched-reference MUSIQ, with more negative values indicating greater degradation.Task 2’s greatest multi-model degradation motivates testing whether RMFA remains sensitive to the affected visual evidence.Source: §D.1 · p. 14

D.2 RMFA Sensitivity in Task 2

Controlled corruption tests show that RMFA responds to degradation of robot-motion evidence, with monotonic decreases under robot-region corruption in most tested trajectories.

D.2 RMFA Sensitivity in Task 2

  • The test applies Gaussian blur or texture removal inside the robot mask or to an equal-area background region while keeping action trajectories, timestamps, and uncorrupted pixels identical within each pair.Strongest settings are Gaussian blur with σ = 12 and complete texture removal with α = 1; confidence intervals use paired bootstrap resampling of 50 reference trajectories.Source: §D.2 · p. 14
  • 48/50 trajectories show monotonically decreasing RMFA as robot-region corruption increases.An equal-area background control estimates response to generic image degradation; paired differences isolate the additional effect of corrupting robot-motion evidence.Source: §D.2 · p. 14, Table S5 · p. 15

E Human Evaluation

Human evaluation rates action following against matched simulator-reference videos, with scores distinguishing robot-motion agreement from agreement in both motion and environment response.

E Human Evaluation

  • The human-evaluation set covers all 750 rollouts pooled across six ACWMs, split equally among expert, early-policy, and cross-action rollouts.The set’s success/failure composition is shown in Table S6.Source: E Human Evaluation · p. 15, Table S6 · p. 15
  • Three annotators independently rate each matched generated/reference video from 1–5: scores 1–2 indicate action-following failure, 3 preserves robot action following despite differing environment response, and 4–5 indicate closer agreement in both.The three ratings are averaged; presentation order is randomized.Source: E Human Evaluation · p. 15

F Downstream Synthetic-Data Study

The expanded synthetic-data study compares six ACWMs on one RoboTwin task, finding compressed success under standard controls but a wider range under OOD controls. The authors frame it as a controlled case study, not universal downstream validation.

F Downstream Synthetic-Data Study

  • The study constructs matched synthetic training sets from expert task trajectories and simulator-validated counterfactual trajectories.It extends the three-model downstream study to all six evaluated ACWMs.Source: F Downstream Synthetic-Data Study · p. 15
  • Success is 78–89% under standard controls versus 21–53% under OOD controls for policies trained on ACWM-generated synthetic data.The expanded comparison covers all six evaluated ACWMs on the same RoboTwin task.Source: F Downstream Synthetic-Data Study · p. 15, Table S7 · p. 15
  • The authors characterize the downstream experiment as a controlled case study of practical relevance, not universal validation.The supplied passage does not specify broader task coverage.Source: F Downstream Synthetic-Data Study · pp. 15–16
Loading 2608.09298v1…