PrimeSim: A Generalist World Model Simulator from Large-Scale Cross-Embodiment Demonstrations

System overview

• PrimeSim conditions on task prompts, unified robot actions, and three camera views, then predicts future frames using autoregressive generation with selected KV-cache memory.The figure describes four denoising steps for this generation process.Source: Figure 1 · p. 2

Motivation

• A robot simulator must follow commanded actions, keep camera views consistent, and remain coherent beyond its training window.The paper frames world models as a way to make rollouts parallel and repeatable while avoiding the danger and sample cost of online hardware training.Source: §1 · pp. 1–2

Midtraining

• Midtraining on 30,110 hours of demonstrations across 22 platforms combines a shared action representation, frame-aligned dual-path conditioning, and view-aware interleaved attention.Explicit camera identities support cross-view information exchange while preserving camera-specific representations.Source: §1 · pp. 2–3

Posttraining

• Fast-video, slow-memory posttraining combines causal generation, self-rollout training, and hierarchical temporal memory to support stable autoregressive simulation beyond the training window.The memory retains the initial scene, sparse long-term history, and dense recent motion within a bounded KV cache.Source: §1 · p. 3

Results

• 6.09 overall on EWMBench (best), and first on four of five WorldSimProbe suites against five open-source action-conditioned world models.PrimeSim also generates coherent rollouts up to 105 seconds at 10.1 FPS and supports GRPO-based policy optimization.Source: §1 · p. 3

2 Data Engine

• 30,110 hours of open-source, proprietary, and simulated demonstrations are pooled across embodiments; midtraining uses the full corpus, while posttraining uses task-specific datasets with complete episodes.The resulting corpus spans 20 dataset families and 22 robot platforms.Source: §2 · pp. 3–4, Figure 2 · p. 4

2.1.1 Stage 1: Video Quality Filtering

• Video filtering removes corrupted clips, excludes episodes shorter than one second, and controls motion-distribution balance by downsampling near-static clips within each platform.Motion is assessed with dense optical flow, interpreted within a platform because camera movement differs across fixed, handheld, and wrist-mounted views.Source: §2.1.1 · p. 4

2.1.2 Stage 2: Action Quality Filtering

• Action filtering removes corrupted or discontinuous signals and drops data with missing action features that cannot be recovered through forward kinematics.Legged locomotion and datasets with dexterous hands are filtered out, while wheeled humanoids are retained.Source: §2.1.2 · pp. 4–5

2.1.3 Stage 3: Data Alignment

• Data alignment resamples demonstrations toward 10 Hz, fixes batches to three camera slots, supplies task prompts when absent, and converts datasets to LeRobot format.When exact 10 Hz resampling is not possible within the timestamp tolerance, the closest achievable frequency is used.Source: §2.1.3 · p. 5

2.1.4 Stage 4: Action Alignment

• Action alignment converts heterogeneous controls into shared end-effector poses, full-body joint angles, gripper commands, and chassis velocity, using a common pose reference for each window.For missing or poorly defined labels, proprioceptive data may supply the next-timestep pose as the action; gripper commands and chassis velocity remain unchanged when expressing relative actions.Source: §2.1.4 · pp. 5–6

3.1 Pretrain

• PrimeSim starts from Wan2.2-TI2V-5B [75], using its natural-video visual and motion priors as a foundation for later training.The authors perform their own training during midtraining and posttraining.Source: §3.1 · p. 7

3.2 Midtrain

• Midtraining jointly learns action-conditioned multiview prediction with a conditional latent flow-matching objective on 30,110 hours of demonstrations spanning 22 platforms.All camera views are generated jointly, with actions incorporated into the velocity prediction network.Source: §3.2 · p. 7, §3.2.4 · p. 9

3.2.1 Action Representation

• A shared 31-dimensional action representation encodes embodiment-specific pose changes, while first-frame-anchored relative actions handle missing features and cancel absolute origin offsets.The representation covers end-effector poses, grasp flags, head/torso/waist angles, and planar chassis velocity where applicable.Source: §3.2.1 · pp. 7–8

3.2.2 Action Conditioning

• Frame-aligned dual-path conditioning binds actions to their corresponding frames through shifted-diagonal attention and a direct action–frame adapter.The attention path selects the action governing each frame; the adapter directly injects that action signal into the frame representation.Source: §3.2.2 · p. 8

3.2.3 Multiview Adaption

• View-aware interleaved attention alternates view-restricted and global attention, while camera-identity tokens support cross-view exchange without losing view-specific representations.Two of every three DiT blocks restrict attention to a camera view; every third enables exchange across all three views.Source: §3.2.3 · p. 9

3.3.1 Stage 1: Autoregressive Modelling; 3.3.2 Stage 2: Causal Consistency Distillation

• Posttraining first enables chunk-causal autoregressive generation, then distills a four-step causal sampler to make repeated rollout generation tractable.Chunk-causal attention blocks access to future chunks; causal consistency distillation reduces the 50 denoising steps otherwise required per chunk.Source: §3.3.1 · p. 10, §3.3.2 · p. 11

3.3.3 Stage 3: Clip-Level Self-Rollout with Asymmetric DMD

• Clip-level self-rollout conditions later chunks on the student’s own generated history and uses asymmetric distribution matching to address exposure bias and preserve generation quality.The bidirectional midtrained model serves as the reference for supervising the autoregressive student.Source: §3.3.3 · pp. 11–12

3.3.4 Stage 4: Episode-Level Long-Horizon Adaptation

• Episode-level self-rollout exposes the model to long-horizon accumulated errors, while hierarchical temporal memory preserves scene information across multiple timescales using its own rollouts.The memory organizes the KV cache into global sink, slow memory, and local memory.Source: §3.3.4 · pp. 12–13
• Hierarchical memory combines the initial scene, dense recent latents, and downsampled long-term history, using synchronized camera indices and the same self-generated memory updates during training and inference.The slow-memory bank merges previous local entries with existing history and downsamples them to fixed capacity while preserving temporal positions.Source: §3.3.4 · p. 13
• Latent-space continuation carries history through a persistent KV cache without decoding and re-encoding each segment, or supplying continuation windows with new first-frame latents or clean ground-truth prefixes.This keeps autoregressive generation conditioned on the model’s own predictions without repeatedly reinitializing visual context.Source: §3.3.4 · p. 13
• Amortized supervision lets the student roll out complete episodes while reference and fake-score networks assess local transitions within fixed windows, without extending their temporal context.The preceding segment is decoded once, and only its final frame is re-encoded for score-network conditioning outside the student’s gradient path; it is never fed back to the student.Source: §3.3.4 · p. 13
• Actions remain relative to the episode’s initial reference pose across generation windows, preserving a consistent coordinate frame and avoiding ambiguity at continuation boundaries.Source: §3.3.4 · p. 13
• Four-step distillation, episode-level self-rollout, and hierarchical memory enable fast, coherent long-horizon simulation; inference uses only a four-step causal student and bounded KV cache, without future ground-truth frames.The student generates future multiview observations conditioned on robot actions.Source: §3.3.4 · pp. 13–14

4.1.1 WorldSimProbe Evaluation

• PrimeSim leads four of five WorldSimProbe suites, including T2 (67.9), T3 (72.3), T4 (85.7), and T5 (37.1), but trails LingBot-VA on T1 (77.0 vs 70.2).T4 also exceeds Ctrl-World (77.5); the authors link interaction grounding performance to training memory on the model’s own rollouts.Source: §4.1.1 · pp. 14–15, Table 1 · p. 15

4.1.2 EWMBench Evaluation

• PrimeSim scores 6.09 overall on EWMBench, exceeding action-conditioned GE-Sim (LTX, 5.82) and the action-conditioned Thinker-WM variant (5.03).The episodes are in-domain, and the compared models are fine-tuned on these datasets; instruction- and text-conditioned results are not iso-protocol with PrimeSim.Source: §4.1.2 · p. 15, Table 2 · p. 16

4.1.3 Inference Speed Comparison

• PrimeSim is fastest in the reported inference benchmark at 30.35 ViewFPS (10.12 TempFPS), rendering a 33-frame, three-view window at 288 × 384 in 3.16 seconds.The comparison uses released inference recipes on an A100, with unmatched resolution, horizon, and denoising budgets; PrimeSim’s quoted throughput is short-window steady state.Source: §4.1.3 · p. 16, Table 3 · p. 17

4.2 What Midtraining Learns

• On three held-out platforms, the midtrained model reaches an action-following rollout within 100–200 steps, while the pretrained backbone fails to do so even after 2,000 steps.The platforms are xArm 7, ViperX 300, and Stanford UMI; neither initialization receives an embodiment identifier or embodiment-specific parameters.Source: Figure 7 · p. 18, §4.2 · pp. 18–19
• On held-out XVLA scenes and camera placements, zero-shot transfer loses 3.97 dB PSNR and 0.105 LPIPS versus a matched in-distribution baseline; 200-step finetuning recovers most of the gap.The robot embodiment is familiar, while the setting and camera poses are unseen; zero-shot cloth state drifts as motion becomes contact-rich.Source: Table 5 · p. 19, §4.2 · p. 19, Figure 8 · p. 20

4.3 Posttraining Analysis

• Four-step Stage 3 distillation improves throughput by approximately 9.0× over the 50-step causal Stage 1 teacher, while hierarchical memory improves long-horizon quality over rolling memory.Under the same 21-slot budget, hierarchical memory reduces ΔPSNR from 8.427 to 6.008 dB, ΔSSIM from 0.120 to 0.079, and ΔLPIPS from 0.233 to 0.159.Source: §4.3.1 · p. 20, Table 6 · p. 20, §4.3.2 · p. 21

4.4.1 Policy Evaluation Fidelity

• PrimeSim matches the simulator’s ranking of eight fixed policies (Spearman ρ = 1.00, Pearson r = 0.99) with a 2.8-point mean absolute success-rate gap.Its estimate stays within 7.0 points for every policy; Wan2.2-FT has a ρ of 0.90 and an 8.3-point mean absolute gap.Source: §4.4.1 · p. 22

4.4.2 RL Post-training inside PrimeSim

• After 200 GRPO steps trained entirely inside PrimeSim, policy success in the LIBERO simulator rises from 61.7% to 73.2%, while PrimeSim estimates 60.1% to 71.9%.The physics simulator is used only to evaluate the resulting policy; PrimeSim supplies closed-loop rollouts and classifier-based sparse rewards during training.Source: §4.4.2 · p. 22, Figure 9 · p. 22
• RL post-training is presented as a test of whether closed-loop policy optimization inside PrimeSim works, not as an attempt to maximize LIBERO success.The experiment uses a short training budget and an off-the-shelf success classifier, so its scores do not establish the best attainable performance.Source: §4.4.2 · pp. 22–23

4.5 Ablation Study

• Ablations isolate individual components, mostly from the pretrained checkpoint, and use single-embodiment settings that do not test cross-embodiment behavior.The distillation variants instead start from the midtrained model; the single-kinematic-frame setup may understate components whose value depends on cross-embodiment transfer.Source: §4.5 · p. 23

4.5.1 Action Representation

• The action-representation ablation compares absolute poses, previous-frame delta poses, and first-frame-anchored relative action chunks using PSNR, SSIM, and LPIPS.The variants start from the pretrained checkpoint; Table 9 identifies the comparison scope, but the supplied passages do not report its metric values.Source: §4.5.1 · p. 23, Table 9 · p. 23

4.5.2 Action Conditioning

• PrimeSim ranks first on WorldSimProbe T3 (72.3) and T2 (67.9), and second on T1 (70.2) behind LingBot-VA (77.0).The quantitative comparison supports strong trajectory-scale action following, while fine-grained local calibration remains a weakness; counterfactual rollouts also show commanded motion with stable static scene elements.Source: §4.5.2 · pp. 23–24, Figure 10 · p. 24

4.5.3 Multiview Consistency

• The multiview design uses one transformer stack with camera identity tokens and periodically integrates information across three views.Each of 30 DiT blocks attends within its camera view, while every third block spans all views; an incremental ablation adds view tokens and intra-view attention, and joint rollouts respond to one action sequence.Source: §4.5.3 · p. 24, Figure 11 · p. 25, Table 11 · p. 25

4.6 Qualitative Results

• Qualitative results show temporal memory under occlusion, cloth rollouts, and EWMBench generations.In the occlusion example, a shoe reappears at the same position, scale, and orientation after leaving view; each window uses selected KV-cache memory rather than only the preceding frame.Source: §4.6 · p. 25, Figure 12 · p. 25

Video Generation in Robotics

• Video generation expands limited teleoperation data with diverse visual trajectories spanning environments, objects, and tasks [47] [57] [95] [99].Another line uses video models’ learned representations as backbones for action prediction and planning [5] [10] [41] [50] [58] [60] [88].Source: §5 · pp. 25–26

Qualitative rollouts

• Figure 13 samples seven frames from three AgiBot G2 head-camera cloth episodes, showing garment deformation while remaining distinct from the underlying surface.The scenes include white cloth on a bamboo table and blue cloth on bedding.Source: Figure 13 · p. 26
• Figure 14 shows seven sampled frames from open-loop EWMBench bottle-packing and bread-placement rollouts at the 480 × 640 scoring frame size.The figure is the qualitative companion to Table 2.Source: Figure 14 · p. 26

Interactive World Models

• Robot-centric world models emphasize multiview consistency, continuous physically grounded controls, and geometric and physical consistency, unlike game-oriented systems [29] [84].Robot-based models serve as learned simulators for action evaluation, policy learning, and planning [25] [28] [39] [52] [68] [69] [72].Source: §5 · p. 26

Limitations

• At 192 × 256 per view, PrimeSim’s claims concern simulator fidelity rather than pixel-level realism, with no like-for-like visual-fidelity comparison against higher-resolution models.The evaluation is broad but not exhaustive.Source: §6 · p. 27
• Adapting to a new platform requires 100 to 200 gradient steps on that platform’s own data, so the demonstrated transfer is few-shot rather than zero-shot.The authors expect more midtraining compute, tighter corpus action/state alignment, and a larger, more diverse corpus to close this gap, but have not demonstrated that outcome.Source: §6 · p. 27
• Most quantitative and all closed-loop evidence is from simulation; whether rollouts can improve policies on real hardware remains open.Real-platform results show faithful video and action following on XVLA and, after few-shot adaptation, xArm 7, ViperX 300, and a UMI gripper.Source: §6 · p. 27

Conclusions

• PrimeSim leads action-realization and interaction-dynamics axes against five open-source models, runs at 10.1 FPS, and transfers action-following rollouts to three held-out platforms within 100 to 200 gradient steps.At inference it uses actions and the task prompt without an embodiment identifier; the pretrained video prior given ten to twenty times the adaptation budget does not transfer similarly.Source: §6 · p. 27

Corpus Composition

• The corpus pipeline includes available data of reasonable quality and converts heterogeneous sources into a unified format.The paper describes conversion steps, filtering criteria, and representation choices.Source: Corpus Composition · p. 36
• In-house teleoperation data comes from AgiBot G2, AgiBot G2X, and PrimeBOT H2 humanoids performing household tasks such as laundry and room tidying.Source: Corpus Composition · p. 36

Training data mixture

• 20 dataset families and 1,773 LeRobot repositories make up the training-data mixture.Hours are derived as frames/fps/3600, consistent with frame counts by construction.Source: Table 12 · p. 37
• The mixture contains 1.52 million episodes across 22 distinct robot platforms.Source: Table 12 · p. 37
• Real trajectories account for 84% of the mixture by trajectory time.Real and Sim split each family by data source, and two families contribute both.Source: Table 12 · p. 37

• Table 13 presents PrimeSim’s model architecture.Source: Table 13 · p. 37

A.2 Training Hyperparameters

• The architecture starts from Wan2.2-TI2V-5B [75], retains its backbone dimensions, and adds lightweight action-conditioning modules and three learnable view tokens.The trainable parameter count is described as close to the backbone’s.Source: §A.2 · pp. 37–38
• Midtraining uses cosine-decayed learning rates with a short warmup and small minimum rate to avoid late-stage loss spikes on long runs.Action classifier-free-guidance dropout is 0.1 to enable inference guidance without retraining.Source: §A.2 · p. 38

A.3 Distillation Hyperparameters

• The Stage 3 student trains only LoRA adapters, while a trainable critic tracks its generated distribution against the frozen midtrained teacher’s real-data score.The critic is updated five times per generator update, and teacher guidance uses a scale of 3.0.Source: §A.3 · p. 38, Table 17 · p. 40
• Stage 1 adapts the midtrained PrimeSim checkpoint to chunk-causal generation using teacher forcing.Source: Table 15 · p. 39
• Stage 2 distills a 50-step teacher into a four-step causal student using one online teacher ODE step per iteration.Source: Table 16 · p. 39
• Stage 4 specifies episode-level rollouts and hierarchical temporal memory, with cache capacity measured in temporal latent slots.Source: Table 18 · p. 40

• The appendix describes the timing environment, per-model warm-up and chunk semantics, measured-system configurations, and a latent-token proxy for fairness discussion.Source: B Latency Benchmark Details · p. 40

B.2 Warm-up, Repeats and Chunk Semantics

• Table 19 reports timing with all numbers measured on a single GPU at batch size one; Table 20 gives per-model warm-up, timed repetitions and aggregation, and Table 21 specifies chunk semantics.Source: Table 19 · p. 40, Table 20 · p. 41, Table 21 · p. 41

B.1 Environment and Timing Scope

• Timing covers one autoregressive future-frame chunk at each model’s native chunk size, including output materialization when part of its public API.For PrimeSim, the timed call is run_episode; checkpoint loading, text encoding, and forward kinematics are excluded.Source: §B.1 · p. 41
• Checkpoint loading, model construction, and first-time CUDA compilation are excluded from timing, while warm-up chunks absorb compilation and cache effects.Source: §B.1 · p. 41

B.3 Per-model Configurations

• 10.115 TempFPS is PrimeSim’s steady-state mean for three-view, 288 × 384 inference with 33-frame windows and a four-step consistency sampler.The run_episode timing excludes loading, text encoding, and forward kinematics.Source: §B.3 · p. 41
• GE-Sim 2.0 uses a distilled Cosmos 2B checkpoint with Pose Expert, three views, four-step sampling, and a steady-state mean of 11.15 s per chunk.Its action conditioning includes an episode band of intrinsics, extrinsics, and trajectory.Source: §B.3 · p. 41
• Fair-aligned BWM uses Wan2.2-TI2V-5B with action AdaLN, single-view 50-step sampling, and a mean chunk time of 6.19 s.The distilled BWM-fast weights were not publicly available at measurement time.Source: §B.3 · p. 41
• Official BWM uses a vendor-default single-view configuration with 50-step sampling and a mean chunk time of 61.71 s, reported as a heavy default point.Its resolution is 672 × 896, with history 9 and future 48.Source: §B.3 · p. 42
• Ctrl-World stacks three views on the canvas height, uses 7D action conditioning and 50-step sampling, and averages 5.78 s per chunk.The canvas is 576 × 320, with 192 × 320 per view.Source: §B.3 · p. 42
• CausalWM generates a flow-to-pointmap-to-RGB chain from text and a first frame rather than being action-conditioned, averaging 37.97 s per chunk.The reported timing uses two chunks with weights preloaded and excludes load time.Source: §B.3 · p. 42

B.4 Latent-token Proxy

• The proxy is a coarse spatial-by-temporal compression grid in the style of Cosmos/Wan tokenizers, not a FLOPs estimate.The passage describes the grid using spatial stride 8 and temporal compression of approximately 4.Source: §B.4 · p. 42
• Table 22’s approximate per-forward counts use effective temporal extent including memory for GE-Sim 2.0, a stacked canvas for Ctrl-World, and an estimated temporal stride for CausalWM.The table is identified as reporting approximate latent tokens per forward.Source: Table 22 · p. 42
• 46,656 latent tokens is PrimeSim’s proxy count for a 33-frame window, using a spatial stride of 8 and temporal compression of approximately 4.The continuous KV cache bounds attention span but does not alter this per-window grid count.Source: §B.4 · p. 42