SimForcing: Distilling Simulation Motion Priors into Real-Domain Robot World Models

1 Introduction

• Precise action responses and realistic, temporally coherent visual dynamics remain challenging to learn jointly from real robot videos.Prior approaches either learn directly from robot videos or add visual structure and priors; simulation's potential for improving real-video prediction without additional priors remains underexplored.Source: §1 · p. 2
• Latent-motion distillation transfers simulation motion priors by aligning temporal differences between teacher and student latents rather than matching appearance.A simulation-trained mixture-of-transformers model initializes and teaches the student, which is jointly trained on paired simulated and real videos.Source: §1 · pp. 2–3
• Multi-block simulation conditioning and condition dropout let the student use predicted simulation trajectories while reducing dependence on potentially inaccurate guidance.Simulation-conditioning CFG balances predictions with and without simulation latents, combining explicit guidance with internalized motion knowledge.Source: §1 · pp. 2–3
• SimForcing outperforms Ctrl-World, EnerVerse-AC, and a GeniWorld reimplementation across PSNR, SSIM, LPIPS, and FVD on Bridge, and improves LIBERO success under action-only training.The world-model experts raise average LIBERO success from 82.6% to 89.2% relative to Wan DiT initialization; InternData-A1 evaluation supports applicability across robot datasets.Source: §1 · pp. 2–3

2.1 Generative World Models for Embodied Interaction

• Generative embodied world models combine action-conditioned video prediction with varied controls and representations, while some methods add 3D or 4D structure.SimForcing instead transfers motion knowledge from a simulation-pretrained teacher through latent differences, without constructing explicit depth, point-trajectory, or semantic-relation targets.Source: §2.1 · pp. 3–4

2.2 World Action Models

• World-action models use predictive video representations for policy learning, jointly model observations and actions, or extend world models to evaluation, improvement, and data synthesis.SimForcing focuses on action-conditioned real-domain video prediction and uses its model to generate simulation guidance without external robot-motion rendering at inference.Source: §2.2 · p. 4

2.3 Simulation Supervision and Real-domain Adaptation

• Prior simulation-conditioned video methods use externally rendered robot motion or scene and object priors; SimForcing generates simulation-domain latents with the same model to guide real-scene prediction.Its simulation-conditioning CFG controls the influence of that guidance to limit reliance on inaccurate predictions.Source: §2.3 · p. 4, Figure 2 · p. 5

3.1 Problem formulation

• The model predicts future visual observations from an initial observation and future actions; proprioceptive inputs are used but omitted from the notation.A video of T+1 frames is paired with T aligned actions and T proprioceptive states, and temporal compression yields fewer future latent steps than video frames.Source: §3.1 · p. 5

3.2 Simulation World Model Training

• The simulation world model learns action-conditioned motion from replayed real demonstration trajectories and initializes the student while serving as its frozen dynamics teacher.The backbone combines Video DiT and Action DiT streams through Mixture-of-Transformers attention; trajectory augmentation creates three perturbed variants per trajectory.Source: §3.2 · pp. 5–6

3.3 Sim-to-Real Distillation Training

• Latent-motion distillation aligns adjacent-latent changes from the real-domain student with a frozen simulation teacher, transferring motion without directly matching appearance.The student is initialized from the simulation teacher, and adjacent latent differences define motion; the objective also encourages background temporal consistency and helps suppress flickering.Source: §3.3 · pp. 6–7
• Multi-block conditioning injects simulation latents into every video-transformer block, while condition dropout lets the student train with or without that guidance.Independent zero-initialized projections add condition features before each block’s feed-forward network; latent corruption also trains robustness to imperfect simulation inputs.Source: §3.3 · p. 7
• The same student is jointly trained on simulation and real videos, with simulation prediction disabling simulation-latent conditioning.Both domains share student parameters, and the simulation branch helps preserve simulation prediction capability during joint training.Source: §3.3 · pp. 7–8
• At inference, simulation-conditioning guidance balances predictions with and without simulation latents, and the student generates its own simulation condition without an extra world model.The simulation condition is generated first and held fixed during real-video sampling; intermediate guidance weights balance internalized motion priors and simulation guidance.Source: §3.3 · p. 8

4.1 Experimental Setup

• The evaluation uses Bridge for main results and ablations, InternData-A1 for generalization, and LIBERO task success rates for downstream policy learning.Video predictions are conditioned on an initial observation and action trajectory; PSNR and SSIM assess reconstruction fidelity, LPIPS perceptual similarity, and FID/FVD distributional quality.Source: §4.1 · pp. 8–9

4.2 Video Prediction Evaluation

• Among models without embodied pretraining, SimForcing has the best Bridge PSNR, SSIM, LPIPS, and FVD; Cosmos-Predict2.5 has higher PSNR and SSIM and lower FVD, while SimForcing has lower LPIPS at about five times faster inference.Compared with GeniWorld, SimForcing provides closer agreement with action-conditioned reference videos at the same reported inference time, with slightly higher FID.Source: §4.2 · p. 9
• On Bridge and InternData-A1 examples, SimForcing produces fewer visual hallucinations and motion that more closely follows specified actions than other methods.Figure 3 also examines how simulation prediction quality affects real-video predictions on Bridge.Source: Figure 3 · p. 9, §4.2 · p. 10

4.3 Analysis

• On Bridge, joint flow matching and trajectory augmentation improve all five metrics over real-video SFT; motion distillation with augmentation gives lowest FID and FVD, while adding simulation conditioning gives best PSNR, SSIM, and LPIPS.The ablations indicate a trade-off between prediction fidelity and perceptual similarity versus distributional quality.Source: Table 2 · p. 9, §4.3 · p. 10
• Latent-motion distillation beats latent-value matching on four Bridge metrics; CFG weight 0.6 gives best PSNR, SSIM, and LPIPS, while weight 1 gives lowest FID and FVD.On the real branch, latent-value loss stays approximately constant while latent-motion loss decreases; the two losses supervise different quantities, so their absolute magnitudes are not directly comparable.Source: §4.3 · p. 10, Table 3 · p. 10, Figure 4 · p. 10
• On InternData-A1, the full model beats Ctrl-World, GeniWorld, and EnerVerse-AC across all five metrics and Wan2.2 SFT on all but FID; it has highest PSNR, lowest FVD, and ties for highest SSIM.Joint flow matching alone yields the lowest FID and LPIPS, showing that configurations offer different metric-specific benefits.Source: §4.3 · pp. 10–11, Table 4 · p. 11
• Initializing a VLA with the pretrained world-model parameters improves average LIBERO success over Wan2.2 in all three policy-training settings.The largest gain is under action-only training, from 82.6% to 89.2%, particularly through improvements on Goal and Long.Source: §4.3 · p. 11, Table 5 · p. 11

5 Conclusion

• A simulation world model teaches a real-domain student through latent-motion distillation, while simulation-conditioning classifier-free guidance adds controllable guidance from predicted simulation latents.Inference requires only the final student model.Source: §5 · p. 11
• SimForcing achieves the best prediction fidelity and perceptual similarity among compared methods without embodied pretraining, with downstream experiments providing preliminary evidence of utility for policy learning.Source: §5 · p. 11
• Figure A1 aligns seven temporal snapshots across a baseline, two comparison methods, simulator-domain predictions, real-world predictions, and ground truth.Source: Figure A1 · p. 12

A.1 Additional Qualitative Results

• Four qualitative examples each are shown for Bridge and InternData-A1, with simulator-domain predictions alongside real-world predictions.Source: §A.1 · p. 12
• Each example contains seven uniformly spaced frames from the interval shared by all methods, allowing inspection of appearance, robot motion, and agreement with ground truth.Source: §A.1 · p. 12
• Rows proceed from baseline, EnerVerse-AC, and GeniWorld to simulator predictions, real-world predictions, and ground truth; simulator and real predictions share temporal indices.Source: §A.1 · p. 12

A.1.1 Qualitative Results on Bridge

• Figures A1–A3 show four Bridge examples, with columns corresponding to temporally aligned frames.Source: §A.1.1 · p. 12

• Figures A4–A6 present four qualitative examples on InternData-A1.Source: §A.1.2 · p. 12
• The examples use temporally aligned frames across columns.Source: §A.1.2 · p. 12
• The passage identifies the examples as qualitative comparisons.Source: §A.1.2 · p. 12

Effect of Training-Time Conditioning Probability

• The Bridge validation sweep varies training-time condition retention p across four values and inference-time CFG weight w across four values.Each checkpoint generates its own simulation conditions and real-video predictions, and PSNR and SSIM are reported on the same evaluation samples.Source: §A.2 · pp. 12–13
• Intermediate retention probabilities p = 0.3 and p = 0.7 achieve higher best-observed PSNR and SSIM than p = 0 or p = 1.They retain better performance without conditioning while also benefiting from simulation guidance; p = 1 degrades substantially when conditioning is removed at inference.Source: §A.2 · p. 13, Table A1 · p. 13
• (p, w) = (0.7, 0.6) achieves the highest PSNR and SSIM among the evaluated settings.For p = 0.7, w = 0.6 outperforms both w = 0 and w = 1.Source: §A.2 · p. 13, Table A1 · p. 13

Inference Steps and Simulation Guidance

• 25.090 dB PSNR and 0.8585 SSIM are the highest observed values, at (Ns, Nr, w) = (5, 20, 0.6).Increasing real-video generation from one to five or ten steps substantially improves fidelity, while increasing from ten to twenty adds only small gains; more simulation steps have a much smaller and inconsistent effect.Source: §A.3 · p. 14
• At w = 0.6, one simulation step produces results close to larger simulation budgets.The passage suggests a coarse predicted simulation reference can already provide useful guidance.Source: §A.3 · p. 14
• At w = 0.6, (1, 10) reaches 25.063 dB PSNR and 0.8575 SSIM in 2.008 s per sample, about 47% lower latency than (5, 20, 0.6).Its PSNR is 0.027 dB below the highest observed value.Source: §A.3 · p. 14
• The recorded w = 0 latency includes simulation generation, so it does not represent an optimized configuration that omits unused simulation predictions and the unused conditional branch.Ns = 20 timings came from a separate run, and the sweep has no repeated-run uncertainty estimates; small metric differences are not statistically established.Source: §A.3 · pp. 14–15

A.4 Implementation Details

• The BridgeData V2 training set contains 1,952 paired manipulation trajectories with aligned videos, actions, proprioceptive states, and language instructions.Each sampled window contains 21 frames and 20 aligned actions; evaluation uses 100 paired validation samples.Source: §A.4 · p. 15
• The InternData-A1 Franka subset contains 2,657 paired training trajectories, with evaluation on 80 paired validation samples.Each 33-frame video window is sampled every four time steps and paired with 128 actions.Source: §A.4 · p. 15
• The student is initialized from an action-conditioned simulation world model, while a frozen copy serves as the motion teacher.The video backbone is based on Wan2.2-TI2V-5B; the backbone and action transformer each contain 30 blocks.Source: §A.4 · p. 17
• Motion supervision matches adjacent-frame differences in estimated clean video latents for every training sample.The real-world branch receives paired simulation latents with probability 0.7, while the simulation branch does not receive this auxiliary condition.Source: §A.4 · p. 17
• The training objective jointly optimizes flow matching in both domains and teacher-guided motion supervision, with no action-prediction loss.Source: §A.4 · p. 17

A.5 Downstream Policy Learning on LIBERO

• The LIBERO comparison transfers compatible weights to both the video and action experts, rather than replacing only the visual backbone.Evaluation covers Spatial, Object, Goal, and Long under three downstream training settings.Source: §A.5 · p. 18
• The three LIBERO settings differ in how video and action experts are trained and coupled: action-only freezes the video expert, unconditional training uses video and action objectives, and video + action lets action tokens attend to future video tokens.For unconditional training, “unconditional” refers to the video branch lacking action conditioning; visual, language, and proprioceptive conditioning remain.Source: §A.5 · pp. 18–19
• World-model initialization raises average LIBERO success from 82.6% to 89.2% under action-only training, from 96.5% to 97.5% under unconditional training, and from 98.1% to 98.4% under video + action training.The largest improvement is under action-only training; gains are not uniform across suites in the video + action setting.Source: §A.5 · p. 20