PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive World Simulator for Robot Policy Training and Evaluation

Yixuan Wang, Rhythm Syed, Fangyu Wu, Mengchao Zhang, Aykut Onol, Jose Barreiros, Hooshang Nayyeri, Tony Dear, Huan Zhang, Yunzhu Li

arXiv:2603.08546v1cs.ROcs.CVcs.LG

TL;DR

Robot world models remain too slow or unstable for faithful long-horizon interaction, limiting scalable policy training and evaluation. Interactive World Simulator uses consistency models for image decoding and latent dynamics prediction, producing stable simulation and comparable policy outcomes from simulator-generated and real-world data. Its simulator performance also strongly correlates with real-world policy performance across diverse tasks.

  • Problem

    Existing robot video-prediction models are often too slow or brittle for long-horizon interaction, while real-world policy evaluation is costly, time-consuming, and difficult to scale.

    Source: A. Video Prediction Model for Robotic… · p. 2, C. Imitation Policy Evaluation · p. 3

  • Method

    Interactive World Simulator combines a compact latent autoencoder with an action-conditioned consistency-model dynamics predictor for interactive video simulation.

    Source: §I · p. 2, §V · p. 8

  • Results

    Policies trained exclusively on simulator-generated data perform comparably to those trained on real-world expert demonstrations, while simulator and real-world policy performance strongly correlate.

    Source: C. Data Generation for Policy Training · p. 7, §V · p. 9

  • Takeaways & Limitations

    The simulator serves as a scalable surrogate for robotic policy training and reproducible evaluation across diverse manipulation tasks.

    Source: §I · p. 2, §V · p. 9

Abstract

from arXiv · show

Action-conditioned video prediction models (often referred to as world models) have shown strong potential for robotics applications, but existing approaches are often slow and struggle to capture physically consistent interactions over long horizons, limiting their usefulness for scalable robot policy training and evaluation. We present Interactive World Simulator, a framework for building interactive world models from a moderate-sized robot interaction dataset. Our approach leverages consistency models for both image decoding and latent-space dynamics prediction, enabling fast and stable simulation of physical interactions. In our experiments, the learned world models produce interaction-consistent pixel-level predictions and support stable long-horizon interactions for more than 10 minutes at 15 FPS on a single RTX 4090 GPU. Our framework enables scalable demonstration collection solely within the world models to train state-of-the-art imitation policies. Through extensive real-world evaluation across diverse tasks involving rigid objects, deformable objects, object piles, and their interactions, we find that policies trained on world-model-generated data perform comparably to those trained on the same amount of real-world data. Additionally, we evaluate policies both within the world models and in the real world across diverse tasks, and observe a strong correlation between simulated and real-world performance. Together, these results establish the Interactive World Simulator as a stable and physically consistent surrogate for scalable robotic data generation and faithful, reproducible policy evaluation.

I. INTRODUCTION

Interactive World Simulator addresses slow and unstable long-horizon robot video prediction with an efficient action-conditioned model. It supports scalable imitation-data generation and policy evaluation, with simulator-trained policies comparable to real-data policies and simulator performance strongly correlated with real-world performance.

  • Existing action-conditioned video models are computationally expensive or unstable over long horizons, limiting faithful prediction of complex physical interactions.Source: §I · p. 1
  • Interactive World Simulator combines compact latent prediction, consistency-model decoding, and action-conditioned dynamics modeling for efficient long-horizon simulation.The autoencoder is trained first, then frozen while latent dynamics are trained with next-frame supervision.Source: §I · pp. 1–2
  • The simulator supports scalable demonstration collection without additional robot interaction and enables reproducible policy evaluation.These applications target imitation learning and controlled comparison across policies.Source: §I · p. 2
  • 100% world-simulator data produces policy performance comparable to an equivalent volume of real-world data.Source: §I · p. 2
  • Across diverse tasks and checkpoints, simulator and real-world policy performance exhibit a strong correlation.Source: §I · p. 2

A. Video Prediction Model for Robotic Manipulation

The paper situates Interactive World Simulator against slow, inaccessible, or brittle video-prediction and evaluation systems. Its two-stage latent-space design targets efficient, multimodal, physically consistent interaction modeling for scalable robotics use.

  • Many existing models lack robot-action conditioning, require substantial compute, or fail to remain robust during stable long-horizon prediction.These limitations hinder real-time interaction and broad academic access.Source: A. Video Prediction Model for Robotic… · p. 2
  • The method trains a compact latent autoencoder before learning action-conditioned future dynamics with a consistency model.The dynamics model predicts future latent states that are decoded into video.Source: Figure 2 · p. 3
  • Consistency models efficiently represent multimodal future outcomes while supporting long-horizon video generation.Source: Figure 2 · p. 3
  • 10 minutes at 15 FPS on a single consumer RTX 4090 GPU makes the simulator suitable for interactive long-horizon prediction.Source: A. Video Prediction Model for Robotic… · p. 3, Figure 2 · p. 3
  • The simulator addresses costly policy evaluation by offering a scalable framework whose outcomes can be compared with real-world performance.Source: C. Imitation Policy Evaluation · p. 3

III. METHOD

The method formulates next-frame action-conditioned video prediction from episodic robot observations and actions. Training uses a two-stage architecture that first learns image reconstruction and then latent-space dynamics.

III-A. Problem Formulation

  • The interaction dataset consists of episodes containing RGB observations and corresponding robot actions.Source: A. Problem Formulation · p. 3
  • The prediction objective uses history observations and actions to predict the next RGB frame by minimizing squared reconstruction error.Source: A. Problem Formulation · p. 4

III-B. Model Training and Inference

1) Stage 1: Autoencoder Training:

Stage 1 trains an autoencoder that encodes images into 2D latent representations and decodes noisy inputs into high-fidelity reconstructions using a consistency-model decoder.

2) Stage 2: Dynamics Training:

Stage 2 freezes the autoencoder and trains an action-conditioned consistency-model dynamics network to predict future latent frames from history and actions, with robustness to noisy contexts.

3) Inference:

Inference autoregressively predicts latent frames from an initial image, action sequence, and evolving history context. The simulator supports interactive demonstration generation and policy interaction, while baseline rollouts show long-horizon failures that the proposed model avoids.

Inference

  • Given an initial image, the simulator encodes it, denoises a noisy latent using action information, and repeatedly appends predictions to the history context.Source: 3) Inference · p. 4

Data Generation

Policy Evaluation

  • Simulator-based policy evaluation mirrors real-world interaction by feeding predicted frames to policies for subsequent action generation.Source: D. Policy Evaluation · pp. 4–5

Qualitative Comparison

  • Across manipulation tasks, baseline rollouts develop inaccurate dynamics, pose drift, artifacts, or lost details, whereas the proposed model maintains coherent robot–object interactions over extended horizons.Source: Figure 3 · p. 5

IV. EXPERIMENT

The experiments compare the simulator across diverse simulated and real-world manipulation tasks involving multiple object types and interactions. The setup uses ALOHA Bimanual Robot data, constrained action spaces, and lightweight training and inference requirements.

Evaluation Questions

  • The evaluation asks whether the simulator improves realism, speed, and robustness, generates data comparable to real data, and faithfully represents real-world policy performance.Source: §IV · p. 5

Task Setup

Data Collection

Task Setup

Efficiency

B. Video Baseline Comparisons

The Interactive World Simulator is compared with state-of-the-art action-conditioned world models under identical long-horizon conditions. It achieves stronger visual fidelity, temporal consistency, and inference efficiency while maintaining physically plausible interactions.

Quantitative comparison

  • Across aggregated metrics, the proposed method consistently outperforms prior world models in long-horizon video prediction.The evaluation uses identical initial conditions and 192-step rollouts across diverse manipulation tasks.Source: B. Video Baseline Comparisons · p. 6

Qualitative comparison

  • Baseline rollouts exhibit pose drift, inaccurate object dynamics, missing details, or severe artifacts, whereas the proposed model maintains stable, physically plausible interactions.The qualitative comparison covers long-horizon predictions across the evaluated manipulation tasks.Source: B. Video Baseline Comparisons · p. 6

Inference efficiency

  • The simulator runs interactive prediction at up to 15 FPS on a single GPU, improving inference efficiency over prior approaches.This efficiency supports interactive long-horizon prediction rather than only offline rollout generation.Source: B. Video Baseline Comparisons · p. 6

C. Data Generation for Policy Training

The world simulator supports interactive demonstration collection and produces training data that remains competitive with real-world data across mixture ratios and dataset sizes. Policies improve as demonstrations increase, with simulator-generated data showing comparable scaling behavior.

Interactive data collection

  • Users can collect expert demonstrations by controlling the simulator with keyboard or kinematic devices, without physical mugs, plates, or other task objects present.The teleoperation interface maps user commands to real-time simulator frames and robot–object interactions.Source: C. Data Generation for Policy Training · p. 6, Figure 4 · p. 6

Data mixtures

  • With 100 training episodes, policies trained across world-simulator and real-world mixtures show consistently high and comparable task scores.Mixtures span from 100% simulator data to 100% real-world data across multiple manipulation tasks.Source: Figure 5 · p. 7, C. Data Generation for Policy Training · p. 7

Data scaling

D. Sim-to-Real Correlation for Faithful Policy Evaluation

The simulator is evaluated as a proxy for real-world policy performance using identical initial configurations across multiple tasks. Strong positive correlations indicate that simulator rankings closely reflect relative real-world performance, despite a slight score bias on most tasks.

Cross-environment correlation

  • Across four manipulation tasks, simulator and real-world policy scores exhibit strong positive correlations under identical evaluation settings.Policies include final and intermediate checkpoints evaluated from 20 sampled initial configurations.Source: D. Sim-to-Real Correlation for… · p. 8

Sim-to-real gap

  • For tasks other than T pushing, fitted lines show a slight positive bias, with policies tending to score higher in simulation than in the real world.The observed bias represents a sim-to-real gap rather than an absence of correlation.Source: D. Sim-to-Real Correlation for… · p. 8

Policy selection

  • If one policy substantially outperforms another in simulation, the results suggest it is likely to do so in the real world as well.This supports using the simulator to select candidate policies before real-world experiments.Source: D. Sim-to-Real Correlation for… · p. 8

V. CONCLUSION

The Interactive World Simulator provides stable, physically consistent, and efficient long-horizon interaction for robotic policy training and evaluation. Simulator-generated training data performs comparably to real-world demonstrations, while simulator performance strongly correlates with real-world performance; future work targets broader environments and more complex tasks.

Core system

  • The simulator supports over 10 minutes of continuous interaction at 15 FPS while maintaining high-quality pixel-level predictions.The conclusion emphasizes both long-horizon stability and computational efficiency.Source: §V · p. 8

Policy training

  • Policies trained exclusively on simulator-generated data perform comparably to policies trained on real-world expert demonstrations.This establishes the simulator’s utility as a scalable source of imitation-learning data.Source: §V · pp. 8–9

Policy evaluation

  • Strong simulator–real-world performance correlations support using the simulator as a surrogate for policy training and evaluation.The claim is reported across several manipulation tasks.Source: §V · p. 9

Future directions

  • Future work will extend the framework to more diverse environments and increasingly complex manipulation tasks.The authors also identify scaling with interaction data and computational resources as an open direction.Source: §V · p. 9

Related papers

Loading 2603.08546v1…