PaperlayerView exact revision on arXiv

Source-linked AI summary

Pelican-Sim 1.0: A General World Model Simulator for Embodied Intelligence

Shilong Zou, Shilin Zhang, Yingji Zhang, Yuhang Huang, Yi Zhang, Zeyuan Ding, Han Dong, Junwei Liao, Yong Dai, Jian Tang, Xiaozhu Ju

arXiv:2609.12036v1cs.ROcs.AI

TL;DR

Action-conditioned world models can predict future observations from visual context and robot actions, but heterogeneous-robot simulation needs both precise configuration information and explicit geometric guidance. Pelican-Sim combines a 28-dimensional numerical interface, URDF-rendered action videos, and sparse MoE processing; it reports improved benchmark PSNR and downstream RoboTwin results.

  • Problem

    Action-conditioned world models need to interpret numerical commands, account for embodiment geometry, and predict their effects across diverse scenes, while numerical and visual representations each leave part of this information implicit.

    Source: §1 · p. 2

  • Method

    Pelican-Sim pairs a unified 28-dimensional robot-configuration trajectory with synchronized, camera-aligned, URDF-rendered whole-arm skeleton videos and uses sparse MoE layers for heterogeneous dynamics.

    Source: §1 · p. 2, §3.2 · p. 6, §3.3 · p. 8

  • Results

    PSNR improves over the strongest evaluated baselines by 4.636 dB on AgiBotWorld Beta, 2.080 dB on RoboMIND, and 10.343 dB on RoboTwin.

    Source: §1 · p. 3

  • Takeaways & Limitations

    On RoboTwin, adding 500 generated trajectories to 50 demonstrations per task raises policy success from 70% to 93%; evaluation correlation is 0.994, with relative success gains of 47.7% for action selection and 20.3% for policy improvement.

    Source: §1 · p. 4

    The training corpus is targeted for expansion across embodiments, object interactions, sensing conditions, and long-horizon behaviors, with the effects on transfer yet to be assessed.

    Source: §5 · p. 23

Abstract

from arXiv · show

In this technical report, we propose Pelican-Sim 1.0, a general world model simulator for embodied intelligence that predicts future observations from visual context and robot actions to support downstream learning and decision making. The model incorporates four key design features: (1) Unified action representation: a 28-dimensional action value space covering most mainstream embodiments, keeping one model valid across heterogeneous devices. (2) Action-visual injection: URDF- and camera-rendered action videos bridge actions and pixels, giving markedly better controllability across embodiments, scenes, and tasks (PSNR +0.904 over alternative fusion baselines). (3) Sparse mixture-of-experts (MoE): sparse MoE layers add capacity for heterogeneous dynamics and absorb the action modality while reducing inter-modality conflict (FVD -6.530 vs. the dense backbone). (4) Efficient rollout generation: causal adaptation and few-step distillation yield a four-step autoregressive simulator, achieving a 5.67-fold speedup over the 35-step model. Benefiting from these designs, we train on approximately one million real-world and simulated trajectories and obtain large gains in action controllability and video quality: PSNR improves over the strongest evaluated baselines by 4.636 on AgiBotWorld Beta, 2.080 on RoboMIND, and 10.343 on RoboTwin, with the adapted EWMBench DYN score up 0.426 on RoboTwin. Relying on this, four downstream applications on RoboTwin succeed: 500 generated trajectories added to 50 demonstrations per task raise policy success from 70% to 93%; policy evaluation reaches a Pearson correlation of 0.994 across five checkpoints; and relative success gains reach 47.7% for action selection and 20.3% for policy improvement. Qualitative generalization across trajectory, scene, object, embodiment, and viewpoint shifts highlights its potential as a general-purpose world model simulator.

1. Introduction

Pelican-Sim 1.0 pairs unified numerical robot configurations with camera-aligned visual motion guidance, then adds sparse expert processing and faster rollouts for heterogeneous embodiments. Its reported evaluations cover video prediction and downstream policy uses across several datasets and tasks.

Motivation

  • The central challenge is to preserve precise robot configuration while providing transferable geometric guidance across diverse embodiments and scenes.Numerical actions retain joint and gripper states but lack explicit image-space motion guidance; visual representations provide geometric cues but may not retain full numerical configuration.Source: §1 · p. 2

Unified cross-embodiment action conditioning

  • A unified 28-dimensional action value and camera-aligned, URDF-rendered whole-arm skeleton video combine precise configurations with explicit image-space motion guidance.Interleaved action-value and action-video pathways improve PSNR by 0.904 dB over late fusion on AgiBotWorld Beta.Source: §1 · pp. 2–3

Heterogeneous dynamics modeling and efficient rollouts

  • Sparse MoE layers reduce FVD by 6.530 versus the dense backbone, while causal adaptation and few-step distillation produce a four-step simulator with a 5.67× speedup over the 35-step model.The speedup is reported for the 21-frame benchmark.Source: §1 · p. 3

Comprehensive evaluation and downstream utility

  • PSNR improves over the strongest evaluated baselines by 4.636 dB on AgiBotWorld Beta, 2.080 dB on RoboMIND, and 10.343 dB on RoboTwin.The model is trained on approximately one million real-world and simulated trajectories.Source: §1 · p. 3
  • On RoboTwin, adding 500 generated trajectories to 50 demonstrations per task raises policy success from 70% to 93%; policy evaluation reaches a Pearson correlation of 0.994 across five checkpoints.Policy evaluation uses 1,000 task-specific adaptation rollouts.Source: §1 · p. 4
  • Action selection and policy improvement yield relative success gains of 47.7% and 20.3%, respectively, over single-sample execution and supervised initialization.Source: §1 · p. 4

2. Related Work

Related work spans action-conditioned world models, cross-embodiment action interfaces, and sparse expert architectures. Pelican-Sim combines these threads in a simulator that pairs numerical and visual action conditions with shared and input-dependent processing.

2.1. World Models and Action-Conditioned Video Generation

  • Prior world models support planning, predictive rewards, and action-conditioned simulation, while Pelican-Sim targets predictive simulation across heterogeneous robots and downstream policy uses in one framework.Source: §2.1 · p. 4

2.2. Cross-Embodiment Learning and Visual Action Interfaces

  • Visual interfaces make projected geometry explicit, but generally do not retain full numerical configuration; Pelican-Sim pairs configurations with synchronized, camera-aligned, URDF-rendered whole-arm skeleton videos.Source: §2.2 · p. 4

2.3. Diffusion Transformers and Sparse Mixture-of-Experts

  • Sparse MoE architectures use conditional expert selection, and Pelican-Sim applies shared and input-dependent processing to joint training on diverse real-world and simulated robot trajectories.Source: §2.3 · p. 4

3. Method

Pelican-Sim predicts future video from visual context and frame-aligned robot actions using complementary numerical and rendered visual conditions in a shared Video DiT. The method combines sparse expert layers with causal distillation for efficient autoregressive rollout.

3.1. Action-Conditioned Video Generation

  • The model predicts future observations from an initial frame, frame-aligned robot configurations, and an optional task instruction using a target-only flow-matching objective.At inference, it integrates the learned velocity field from noise to generate the future target latent, then decodes the complete clip.Source: §3.1 · pp. 5–6, §3 · p. 5

3.2. Complementary Action Representations

  • A unified 28-dimensional bilateral action value pairs precise robot configurations with a synchronized URDF- and camera-rendered skeleton video.The action value preserves configuration information, while the action video makes projected whole-arm motion explicit.Source: §3.2 · p. 6, Figure 4 · p. 7

3.3. Dual-Branch Conditioning and Sparse MoE Layers

  • Numerical features modulate odd Video DiT blocks, visual action features add residuals at even blocks, and sparse MoE layers combine shared and token-routed experts.The experiments use one shared expert, eight routed experts, and select two routed experts per token.Source: §3.3 · pp. 7–8

3.4. Causal Few-Step Distillation for Efficient Rollout

  • Progressive distillation first adapts the bidirectional model to block-causal attention, then trains a four-step student against the frozen teacher with self-forcing.A small fraction of teacher-forced updates is retained during self-forcing to stabilize training and preserve action controllability.Source: §3.4 · p. 9
  • 2.2 seconds per 21-frame clip and a 5.67× speedup over the original 35-step model are measured for the accelerated four-step causal student.Block-causal attention enables key-value caching, avoiding repeated computation over historical tokens.Source: §3.4 · p. 9

4. Experiments

Pelican-Sim 1.0 is evaluated from video prediction through downstream uses, with benchmark comparisons on three held-out datasets. The setup combines real and simulated robot trajectories and a task-conditioned VLM evaluator.

4. Experiments

  • The evaluation progresses from prediction quality to data generation, policy assessment, action selection, policy improvement, and generalization under controlled distribution shifts.The shifts include trajectory, scene appearance, embodiment, object, and viewpoint.Source: §4 · p. 9

4.1. Experimental Setup

  • The corpus combines real and simulated single-arm and bimanual trajectories and contains approximately 1.0 million curated trajectories totaling 8,000 hours.Video prediction is evaluated on held-out within-source splits, rather than entirely unseen datasets; a fine-tuned VLM supplies task-conditioned downstream feedback.Source: §4.1 · pp. 9–10

4.2. Comparison with State-of-the-Art World Models

  • PSNR improves over the strongest metric-specific baselines by 4.636 dB on AgiBotWorld Beta, 2.080 dB on RoboMIND, and 10.343 dB on RoboTwin.The model also leads on all five video-quality metrics and adapted EWMBench overall scores on each dataset; several individual EWMBench metrics remain below the best baselines.Source: §4.2 · p. 10

4.3. Qualitative Generation Results

Representative rollouts show action-conditioned predictions across real and simulated sources, with varied embodiments and object interactions.

4.3. Qualitative Generation Results

  • Predicted videos closely follow commanded whole-arm trajectories and gripper-state transitions across the illustrated settings.The examples include single-arm and bimanual systems, parallel grippers, and dexterous hands.Source: §4.3 · p. 10
  • Predictions preserve temporal coherence and scene appearance while the action-video condition and ground truth are shown alongside them.The qualitative comparison uses frame-aligned action videos and ground-truth videos for each task.Source: §4.3 · p. 10
  • Predicted object motions are consistent with robot actions across the diverse interactions shown.Source: §4.3 · p. 10

4.4. Ablation Studies and Design Analysis

Ablations test mixed-domain training and sparse MoE, complementary action inputs, and alternative injection architectures.

Mixed-domain training and sparse MoE

  • Replacing dense feed-forward layers with sparse MoE under mixed-domain training yields further gains of 1.214 dB in PSNR and 6.530 in FVD.Adding simulated trajectories first improves PSNR by 0.548 dB and reduces FVD by 3.030; that comparison also changes corpus size and diversity.Source: §4.4 · pp. 10, 14, Table 4 · p. 14

Contribution of complementary action conditions

  • Combining numerical actions and action videos raises PSNR over action video alone by 1.000 dB on AgiBotWorld Beta and 0.748 dB on RoboMIND.The combined conditions also reduce LPIPS from 0.0158 to 0.0125 and from 0.0132 to 0.0115, respectively.Source: §4.4 · p. 14, Table 5 · p. 14

Injection architecture

  • Interleaved injection performs similarly to dual injection, whose PSNR advantage is only 0.075 dB on AgiBotWorld Beta and 0.051 dB on RoboMIND.The selected interleaved architecture alternates numerical and visual conditioning branches across Video DiT blocks.Source: §4.4 · p. 14

4.5. Data Generation

Pelican-Sim 1.0 generates complete task trajectories faster than the compared collection methods and its rollouts support policy training, including with limited demonstrations.

Generation efficiency

  • Pelican-Sim 1.0 achieves 2.50× throughput versus manual collection and 2.89× versus RoboTwin simulation for complete trajectories at matched horizons.Trajectory generation takes 24 seconds versus 60 seconds for manual collection, and 19 seconds versus 55 seconds for RoboTwin; reported throughput excludes manual reset time.Source: §4.5 · p. 15, Table 7 · p. 15

Cross-domain augmentation

  • Cross-domain augmentation achieves a 72.0% mean policy success rate across five tasks, compared with 70.0% for direct simulator collection.Augmentation uses 50 clean demonstrations and 500 generated rollouts per task; direct collection uses 50 randomized trajectories per task.Source: §4.5 · pp. 15–16, Table 8 · p. 15

Scaling with limited demonstrations

  • Success gains are 36.0, 30.0, and 23.0 percentage points with 10, 30, and 50 original trajectories per task, respectively.With augmentation, 10 original trajectories per task reach 64.5%, compared with 70.0% for 50 originals without augmentation.Source: §4.5 · p. 16, Table 9 · p. 16

4.6. Policy Evaluation and Ranking

The section tests whether a Pelican-Sim 1.0–VLM pipeline can reproduce RoboTwin policy outcomes and rankings for matched action trajectories. With 200 world-model adaptation rollouts, it reproduces the ranking of all five evaluated checkpoints; larger budgets improve success-rate estimates.

Validation of the VLM evaluator

  • The evaluator achieves 92.6% terminal accuracy on simulator videos and 91.4% on generated videos, with generated-video progress MAE of 0.078 and validity AUROC of 0.953.These results assess judgment of depicted outcomes separately from world-model prediction accuracy.Source: §4.6 · p. 16, Table 10 · p. 17

Policy evaluation and ranking protocol

  • The evaluation compares matched policy-predicted action trajectories on 200 held-out RoboTwin initial conditions per policy, varying adaptation from 0 to 1,000 task-specific rollouts.Simulator success is determined by the RoboTwin checker and generated-video success by the frozen VLM; this tests fixed trajectories, not closed-loop execution under generated observations.Source: §4.6 · p. 16

Policy evaluation and ranking results

  • Spearman ρ = 1.0, Kendall τ = 1.0, and MMRV = 0 with 200 adaptation rollouts reproduce the RoboTwin ranking of all five checkpoints.Success-rate MAE falls from 4.0 percentage points at 200 rollouts to 2.9 at 500 and 2.4 at 1,000; the 500-rollout variant is used subsequently.Source: Table 11 · p. 17, §4.6 · p. 17

4.7. Action Selection

The section tests whether VLM-guided selection among Pelican-Sim 1.0-predicted candidate outcomes improves RoboTwin task success. The selector outperforms the reported baselines, while increasing candidate count trades additional success for greater latency and lower selection recall.

Candidate generation and selection

  • A frozen policy proposes candidate action sequences, Pelican-Sim 1.0 predicts their outcomes, and the VLM evaluator ranks the resulting videos.The selected candidate is executed in RoboTwin open-loop without replanning.Source: §4.7 · p. 17

Action selection results

  • 63.8% success from Pelican-Sim 1.0 selection exceeds Single sample by 20.6 and Ctrl-World selection by 11.5 percentage points across five RoboTwin tasks.The selector reaches 93.7% selection recall, 4.3 points below the oracle.Source: Table 12 · p. 18, §4.7 · p. 18

Test-time scaling

  • Increasing candidates from 8 to 10 adds 1.3 percentage points of success while latency rises from 10.7 to 15.9 seconds.From 1 to 10 candidates, success rises from 43.2% to 63.8%, while selection recall falls from 100% to 93.7%.Source: Table 13 · p. 18, §4.7 · p. 18

Evaluation protocol and baselines

  • Methods share Diffusion Policy candidate pools and 200 held-out initial conditions per task, with task-checker success, improvement over Single sample, and oracle gap reported.Any-success@M is the fraction of pools containing a successful candidate; selection recall is success conditional on such a pool.Source: §4.7 · p. 18

4.8. Policy Improvement

The section evaluates policy improvement using closed-loop imagined rollouts from Pelican-Sim 1.0 and rewards from a fine-tuned VLM evaluator. The full pipeline improves mean RoboTwin success over the tested learned-policy baselines, though it remains below the simulator-feedback oracle.

Closed-loop world-model rollouts

  • The policy samples action chunks from predicted observations, and Pelican-Sim 1.0 predicts the next observation segment for closed-loop optimization without additional RoboTwin interaction.The accelerated four-step causal world model and VLM evaluator remain frozen during policy optimization.Source: §4.8 · pp. 18–19

VLM-based dense reward

  • The frozen evaluator scores rollout prefixes for task progress and visual validity and assigns terminal-success confidence to the full trajectory.The return combines progress increments with terminal feedback weighted by visual validity; invalid or insufficiently variable rollout groups are excluded or resampled.Source: §4.8 · p. 19

GRPO policy optimization

  • GRPO normalizes returns within each rollout group to obtain trajectory-level advantages, which are applied to action chunks in a clipped policy objective.Training compares initial SFT, Iter-SFT, binary terminal-reward GRPO, and the full reward under matched tasks and optimization budgets.Source: §4.8 · p. 19

Baselines and evaluation

  • 75.4% mean success with the full world-model–VLM pipeline exceeds initial SFT, Iter-SFT, and terminal-only GRPO by 12.7, 8.4, and 6.9 percentage points, respectively.The full method remains 8.4 points below RoboTwin-GRPO; the terminal-only comparison evaluates the combined reward design rather than isolating its components.Source: §4.8 · p. 19

4.9. Generalization under Distribution Shifts

Controlled studies vary trajectory, scene appearance, robot embodiment, manipulated object, and camera viewpoint while holding task semantics and action horizon fixed. The reported qualitative results suggest broad generalization across these shifts, while remaining bounded to the tested variations.

Trajectory OOD

  • Reversed action-conditioned rollouts remain consistent with the reversed commands across real-world and simulated tasks, without assuming exact time reversibility in contact-rich dynamics.The test reverses frame-aligned robot configurations and uses the terminal observation as the new conditioning frame.Source: §4.9 · p. 20, Figure 7 · p. 20

Scene/appearance OOD

  • With the same action trajectory, rollouts remain temporally coherent and consistent with folding behavior across six edited scene variants and the original scene.The variants change background, illumination, texture, and local object appearance; the test is broader than an isolated background shift.Source: §4.9 · p. 20, Figure 8 · p. 21

Robot-embodiment OOD

  • Retargeted block-stacking rollouts follow the source Panda trajectory for held-out Sawyer, IIWA, and UR5e embodiments.The authors interpret these qualitative results as retaining task-level motion semantics while accommodating changed robot kinematics and projected geometry.Source: §4.9 · p. 20, Figure 9 · p. 21

Object OOD

  • Under the same pick-and-place command, generated motions target five edited object variants, move them toward the table center, and largely preserve their visual identity.The edited objects differ in appearance, shape, and scale; the experiment does not assume identical physical properties.Source: §4.9 · p. 22, Figure 10 · p. 22

Viewpoint OOD

  • Across unseen camera elevations, azimuths, and distances, generated sequences remain consistent with the commanded motion and intended bowl-to-plate interaction.Action videos are rendered using calibration for each novel viewpoint.Source: §4.9 · p. 22

Generalization summary

  • The controlled studies report qualitative generalization across temporal, visual, embodiment, and geometric shifts, supporting Pelican-Sim 1.0 as a general-purpose simulator for diverse manipulation settings.The authors attribute the observed transfer to the unified numerical and visual action representations.Source: §4.9 · p. 22

5. Conclusion and Future Work

Pelican-Sim 1.0 combines unified numerical actions, URDF-rendered action videos, sparse MoE layers, and causal distillation for robotic manipulation across heterogeneous embodiments and environments. The conclusion reports improved video quality and action consistency, downstream applications, and task-consistent motion under controlled shifts, while identifying coverage and evaluation priorities for future work.

5. Conclusion and Future Work

  • Pelican-Sim 1.0 combines unified 28-dimensional robot configurations, URDF-rendered action videos, and sparse MoE layers to model manipulation across heterogeneous embodiments and environments.The action pathways provide numerical specifications and image-space motion guidance; MoE layers model heterogeneous robot dynamics.Source: §5 · p. 22
  • Causal distillation reduces rollout sampling from 35 to four steps for rollout-intensive applications.Source: §5 · p. 22
  • Held-out real-world and simulated dataset tests demonstrate improved video quality and action consistency, and validate the action representations, mixed-domain training, and MoE design.Source: §5 · p. 22
  • The model retains task-consistent motion under controlled changes in trajectory, scene appearance, embodiment, object identity, and viewpoint.Viewpoint OOD rollouts include the original camera and five unseen viewpoints.Source: §5 · p. 23, Figure 11 · p. 23
  • Future work targets broader training coverage, quality-efficiency trade-offs in distillation, and more precise evaluation of generalization under defined distribution shifts.Proposed coverage includes embodiments, object interactions, sensing conditions, and long-horizon behaviors.Source: §5 · p. 23

6. Contributions

The contributions section lists contributors and their roles in developing Pelican-Sim 1.0. It identifies core contributors and additional contributors.

6. Contributions

  • The section lists contributors and their roles in developing Pelican-Sim 1.0.Source: §6 · p. 27

6.1. Core Contributors

  • Shilong Zou contributed to data preparation, model architecture, world-model training, and downstream policy training; Shilin Zhang contributed to data preparation, downstream policy training, and visualization.Source: §6.1 · p. 27

6.2. Contributors

  • Yingji Zhang, Yuhang Huang, Yi Zhang, Zeyuan Ding, Han Dong, and Junwei Liao are listed as contributors.Source: §6.2 · p. 27

Related papers

Loading 2609.12036v1…