VPTwin: Real-Sim-Real Video Prediction for Robotic Manipulation Planning

• Purely data-driven predictors can accumulate long-horizon errors and generate physically implausible hallucinations, while simulation-only prediction can suffer geometry, friction, and contact reality gaps.Source: §1 · p. 1
• A VLM reconstructs executable digital twins, while Isaac Sim generates rollouts across randomized physical configurations because unobserved physical properties are ill-posed to infer from visual observations.Source: §1 · p. 1
• VPTwin-Core conditions real-domain video prediction on multiple possible outcomes from approximate digital twins to mitigate physically implausible predictions and reduce forward hallucinations.Source: §1 · p. 1, Figure 1 · p. 2
• Synthetic simulation failures augment real demonstrations so VPTwin-Core can predict unsuccessful action outcomes without collecting costly real-world failure data.Source: §1 · p. 2
• At each semantic substage, a VLM-proposed action is evaluated in simulation and visually verified; only predicted outcomes meeting predefined success criteria are executed on the real robot.Otherwise, predicted visual feedback guides iterative refinement.Source: §1 · p. 2
• Evaluations across rigid, deformable, and articulated manipulation tasks show substantial reductions in physical hallucinations and marked gains in task success.Source: §1 · p. 2

• Action-conditioned video prediction progressed from video plans and visual-correspondence control to autoregressive, diffusion, and joint video-action models, but contact-rich settings leave physical parameters unobserved and predictions vulnerable to hallucinations.Examples include UniPi (Du et al., 2023), AVDC (Ko et al., 2024), and later controllable visual-dynamics models (Wu et al., 2024).Source: §2 · pp. 2–3
• Real-to-Sim methods reconstruct rigid, soft, and articulated assets, while VPTwin uses VLM-inferred parameter distributions to generate diverse rollout contexts rather than treating one deterministic twin as ground truth.Source: §2 · p. 3
• Simulation-guided video methods use geometry or motion priors, while VPTwin conditions real-video prediction on multiple randomized rollouts of the identical action sequence and addresses reality gaps through staged planning.Source: §2 · p. 3

3.1 REAL-TO-SIM TWIN AUTHORING AND RANDOMIZED PHYSICS ROLLOUTS

• Because unobserved dynamics are ill-posed to infer from video, the Real-to-Sim pipeline authors executable twins and randomizes plausible physical parameters and object placements across rollouts.The recorded action trajectory and static scene conditions remain invariant across the randomized rollouts.Source: §3.1 · pp. 3–4
• VLM success filtering retains 277 physically consistent episodes across four manipulation families for world-model training.Source: §3.1 · p. 4

3.2 SIMULATION-SYNCHRONIZED VIDEO PREDICTION

• Simulated kinematic perturbations create counterfactual failure trajectories that provide failure supervision without costly or hazardous real-world failure collection.Successful demonstrations use real video supervision, while perturbed-action simulation rollouts serve as pseudo-observation targets.Source: §3.2 · p. 4
• Time-aligned simulation rollouts and action windows condition VPTwin-Core, which predicts terminal latent states through a two-stage training and autoregressive inference pipeline.The model uses ten rollout latent clips as physical context; Stage 1 trains an encoder and decoder, while Stage 2 freezes them and trains the dynamics model.Source: §3.2 · p. 5

3.3 VPTWIN-BASED MODEL PREDICTIVE CONTROL

• The planner decomposes manipulation skills into substages, then proposes, simulates, visually verifies, and refines each candidate before committing it to the real robot.Refinement can run for up to three rounds; if no candidate passes, planning ends without hardware execution.Source: §3.3 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation includes six tabletop tasks across measured-catalog and VLM-authored twins, plus real-robot trials on Grasp Cup and Open and Close Drawer.Real-world planning is benchmarked against SIMPACT and open-loop π0.5; object placements are perturbed across episodes.Source: §4.1 · p. 6, Table 2 · p. 7, Figure 4 · p. 7

4.2 LONG-TERM VIDEO PREDICTION RESULTS

• VPTwin outperforms IWS across family-averaged metrics, with PSNR gains of 2.99 dB in measured-catalog and 3.32 dB in VLM-authored settings.LPIPS decreases from 0.329 to 0.284 and from 0.310 to 0.259, respectively.Source: §4.2 · p. 7

4.3 ACTION PLANNING RESULTS

• VPTwin achieves 3/7 versus SIMPACT’s 2/7 on Grasp Cup and 5/5 versus 2/5 on Open and Close Drawer; its critic has perfect Precision and F1, eliminating false acceptances.Open-loop π0.5 fails across all trials; both planners have Recall of 1.00.Source: §4.3 · pp. 7–8, Table 2 · p. 7

4.4 ABLATION STUDIES

• Synthetic failure co-training raises full-rollout PSNR on Open and Close Drawer by 0.97 dB and 0.86 dB under two initializations, and prevents optimistic failure predictions.The gains are 0.07 dB and 0.29 dB on 30-frame rollouts; Grasp Cup full-rollout gains are 0.08–0.09 dB.Source: §4.4 · p. 8
• With three test-time rollouts, training on ten rather than three rollouts outperforms the three-rollout training configuration across all six measured-catalog tasks.For models trained with ten rollouts, even one inference-time rollout improves PSNR over zero simulation context.Source: §4.4 · p. 8, Table 4 · p. 9
• At k = 1, failed simulation context improves measured-catalog PSNR over no context by 3.1 dB for Grasp Cup, 4.1 dB for Open and Close Drawer, and 3.0 dB for Place Cube into Drawer.Across VLM-authored tasks, drawer articulation benefits consistently, while Wipe Table degrades and Pickup Cup and Cloth improves as k scales.Source: §4.4 · pp. 8–9, Figure 6 · p. 9

5 CONCLUSION AND LIMITATIONS

• VPTwin combines automated digital twins, randomized simulation rollouts, and synthetic counterfactual failures to ground prediction and anticipate candidate action outcomes.As a staged-planning visual verifier, it eliminates false acceptances and improves physical-hardware manipulation success over simulation-only planners.Source: §5 · p. 9
• Twin authoring assumes calibrated camera extrinsics and task-specific reconstruction, while synthetic failure co-training does not capture the full diversity of unmodeled real-world dynamics.The authors also identify larger multi-stage suites and closed-loop online adaptation as future work.Source: §5 · p. 9

A EXPERIMENTAL DETAILS

• The tabletop measures 120 × 60 cm and stands 74.4 cm above the floor.Source: §A.1 · p. 13
• The left and right arm-base centers are (−31.6, −45.4) cm and (+30.1, −45.4) cm, respectively.Source: §A.1 · p. 13
• The camera is positioned at (+0.7, −45.1) cm, 100 cm above the floor, and faces downward at 25° from horizontal.Source: §A.1 · p. 13
• Five-point placements use 2.5 cm red markers, which objects cover without requiring perfect center alignment.Source: Figure 7 · p. 13

A.2 TASK PLACEMENTS

• The five slots are center (0, 0), left (−20, 0), right (20, 0), far (0, 20), and near (0, −20) cm.Source: §A.2 · p. 13
• Objects occupy designated slots rather than continuously sampled positions, and a slot is occupied when the object footprint covers its marker.Source: §A.2 · p. 13
• Placement error is not directly measured, and simulation perturbation ranges are not measured bounds on real placement error.Source: §A.2 · p. 13

A.3 REAL-ROBOT EVALUATION PROTOCOL

• Seven Grasp Cup trials used center, far, near, left, and right placements; five drawer trials used separate executions of the same centered layout.Source: §A.3 · p. 13

B.1 MEASURED-CATALOG TWIN

• The measured-catalog twin uses measured scene geometry and calibration, while object physical properties and their randomization ranges are supplied rather than physically measured.Source: §B.1 · pp. 13–14, Table 5 · p. 14

B.2 PHYSICS RANDOMIZATION

• Measured-catalog seeds 1–9 vary placement and physical parameters by severity group; VLM-authored parameter values are sampled uniformly from authored intervals.Source: §B.2 · p. 14

B.3 VLM TWIN AUTHORING AND FILTERING

• The VLM authors executable assets from demonstration stills, and success-criterion filtering retains 277 seed-0 episodes for the prediction set.Source: §B.3 · pp. 15–16, Figure 8 · p. 16

C FAIL-DEST CONSTRUCTION

• Because the real dataset contains no failure videos, the authors construct 118 failure episodes and label outcomes using object-trajectory success criteria.Source: C FAIL-DEST CONSTRUCTION · pp. 16–17
• Failure trajectories perturb active-arm actions around grasp or drawer events, with action values clipped to robot limits and specified fingertip bounds.Source: C FAIL-DEST CONSTRUCTION · pp. 16–17

D.1 TRAINING SETTINGS

• Both stages use AdamW and five independent training seeds; Stage 1 uses horizon 1, while Stage 2 trains the dynamics model with horizon 10.Source: §D.1 · p. 17

D.2 STAGE-1 ENCODER AND DECODER

• Stage 1 trains an encoder and consistency decoder on real success frames and seed-0 simulated failure frames, while randomized seeds condition Stage 2.Source: §D.2 · p. 17

D.3 ALIGNED STAGE-2 WINDOWS

• Stage-2 windows contain ten time-aligned frames at 15 Hz, with matching temporal indices for observation video, actions, and conditioning rollouts.Source: §D.3 · p. 17

D.4 TERMINAL-ONLY SUPERVISION

• The first H−1 positions provide observation history and the final position is the prediction target, so the effective loss supervises only the terminal latent.Source: §D.4 · pp. 17–18

D.5 AUTOREGRESSIVE INFERENCE

• Inference appends a Gaussian latent, denoises the new slot in one consistency step, carries the updated window forward, and decodes each latent in three steps.Source: §D.5 · p. 18

D.6 EPISODE SPLITS

• The measured-catalog split contains 440/54/54 train/validation/test episodes; the VLM-authored set inherits assignments and contains 224/26/27.Source: §D.6 · p. 18, Table 8 · p. 18

D.7 SUCCESS–FAILURE SPLIT AND SAMPLING

• Failure-aware co-training pairs 118 real successes with 118 simulated failures, uses balanced sampling, and evaluates on three grasp pairs and eight drawer pairs.Source: §D.7 · p. 18

E DETAILS ON REAL-SIM-REAL MODEL PREDICTIVE CONTROL

• A cached task-family motion note is authored from teleoperation stills and supplies ordered waypoint rules, while a second VLM pass defines six substages and their local success tests.The motion note is reused across tests and is not a trained motion model.Source: E DETAILS ON REAL-SIM-REAL MODEL… · p. 19
• A candidate must pass both simulation and video-prediction checks at each of six substages; if any substage fails, deployment stops without hardware execution.The committed prefix and candidate chunk are rolled out under seeds 0–9, and refinement is limited to three rounds.Source: E DETAILS ON REAL-SIM-REAL MODEL… · p. 19
• For critic-agreement evaluation, planning continues through failed substages and the resulting trajectory is executed on hardware while retaining its rejection verdict.This evaluation procedure differs from normal deployment, which stops without hardware execution when a substage does not pass.Source: E DETAILS ON REAL-SIM-REAL MODEL… · pp. 19–20
• VPTwin’s critic accepts only when all six substages pass simulation and video prediction, whereas SIMPACT accepts when a candidate passes its simulation check.The baseline receives the same initial image, slot coordinates, fingertip position, primitive schema, and whole-skill success criteria.Source: E DETAILS ON REAL-SIM-REAL MODEL… · p. 20
• 3/7 Grasp Cup planning trials were accepted and executed, and all three succeeded, yielding a 3/7 acceptance rate and 1.00 precision.All rejected plans fail on hardware in this evaluation.Source: E DETAILS ON REAL-SIM-REAL MODEL… · p. 20

F FID AND FVD EVALUATION PROTOCOL

• FID uses 2048-dimensional Inception-v3 frame features, while FVD uses Kinetics-400 I3D features from 16-frame, stride-8 clips that do not cross episode boundaries.Both models are evaluated on matched full-length test episodes at 128 × 128 resolution.Source: F FID AND FVD EVALUATION PROTOCOL · p. 21
• Scores are averaged over five training seeds before reporting means and standard deviations across six measured-catalog or four VLM-authored task families.Sample counts are shared by both models and the ground-truth reference; overlapping FVD clips are not independent.Source: F FID AND FVD EVALUATION PROTOCOL · p. 21, Table 10 · p. 25
• The dest-count ablation uses three evaluation seeds for k ∈ {1, 3, 5, 7, 9}, one for k = 0, and the existing full-test rollout for k = 10.The failure-only ablation produces one evaluation per k, with metrics computed separately by task family, checkpoint, and evaluation seed.Source: F FID AND FVD EVALUATION PROTOCOL · p. 21, Table 11 · p. 25

G PROMPT EXAMPLES

• Later episode photographs may reveal objects hidden at the initial layout, but they must not be used to determine initial object positions.The prompt treats a later empty table as a later moment, not evidence that hidden objects were absent initially.Source: G PROMPT EXAMPLES · p. 21
• The scene-parsing prompt records object appearance, part motion, softbody type, relations, and per-image visibility and presence.It distinguishes objects visible in a photograph from objects present in the initial layout, including covered objects.Source: G PROMPT EXAMPLES · p. 22
• For task instructions, the VLM groups real episode stills by interaction, describes the interaction and success condition, and requires every initial image to appear in exactly one group.Later stills provide timeline evidence, while similar interactions from different starting positions may be merged.Source: G PROMPT EXAMPLES · pp. 22–23
• Table 9 lists task instructions and success criteria authored from multi-time real-video stills; the appendix tables also cover sample counts, ablations, and full-rollout metrics.The full-rollout tables specify that higher PSNR, SSIM, and UIQI and lower MSE, LPIPS, FID, and FVD are better.Source: Table 9 · p. 24, Table 10 · p. 25, Table 11 · p. 25