PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive World Simulator for Robot Policy Training and Evaluation

Yixuan Wang, Rhythm Syed, Fangyu Wu, Mengchao Zhang, Aykut Onol, Jose Barreiros, Hooshang Nayyeri, Tony Dear, Huan Zhang, Yunzhu Li

arXiv:2603.08546v1cs.ROcs.CVcs.LG

TL;DR

Robot world models remain too slow or unstable for faithful long-horizon interaction, limiting scalable policy training and evaluation. Interactive World Simulator uses consistency models for image decoding and latent dynamics prediction, producing stable simulation and comparable policy outcomes from simulator-generated and real-world data. Its simulator performance also strongly correlates with real-world policy performance across diverse tasks.

  • Problem

    Existing robot video-prediction models are often too slow or brittle for long-horizon interaction, while real-world policy evaluation is costly, time-consuming, and difficult to scale.

  • Method

    Interactive World Simulator combines a compact latent autoencoder with an action-conditioned consistency-model dynamics predictor for interactive video simulation.

  • Results

    Policies trained exclusively on simulator-generated data perform comparably to those trained on real-world expert demonstrations, while simulator and real-world policy performance strongly correlate.

  • Takeaways & Limitations

    The simulator serves as a scalable surrogate for robotic policy training and reproducible evaluation across diverse manipulation tasks.

Abstract

from arXiv · show

Action-conditioned video prediction models (often referred to as world models) have shown strong potential for robotics applications, but existing approaches are often slow and struggle to capture physically consistent interactions over long horizons, limiting their usefulness for scalable robot policy training and evaluation. We present Interactive World Simulator, a framework for building interactive world models from a moderate-sized robot interaction dataset. Our approach leverages consistency models for both image decoding and latent-space dynamics prediction, enabling fast and stable simulation of physical interactions. In our experiments, the learned world models produce interaction-consistent pixel-level predictions and support stable long-horizon interactions for more than 10 minutes at 15 FPS on a single RTX 4090 GPU. Our framework enables scalable demonstration collection solely within the world models to train state-of-the-art imitation policies. Through extensive real-world evaluation across diverse tasks involving rigid objects, deformable objects, object piles, and their interactions, we find that policies trained on world-model-generated data perform comparably to those trained on the same amount of real-world data. Additionally, we evaluate policies both within the world models and in the real world across diverse tasks, and observe a strong correlation between simulated and real-world performance. Together, these results establish the Interactive World Simulator as a stable and physically consistent surrogate for scalable robotic data generation and faithful, reproducible policy evaluation.

I. INTRODUCTION

Interactive World Simulator addresses slow and unstable long-horizon robot video prediction with an efficient action-conditioned model. It supports scalable imitation-data generation and policy evaluation, with simulator-trained policies comparable to real-data policies and simulator performance strongly correlated with real-world performance.

  • Existing action-conditioned video models are computationally expensive or unstable over long horizons, limiting faithful prediction of complex physical interactions.
  • Interactive World Simulator combines compact latent prediction, consistency-model decoding, and action-conditioned dynamics modeling for efficient long-horizon simulation.The autoencoder is trained first, then frozen while latent dynamics are trained with next-frame supervision.
  • The simulator supports scalable demonstration collection without additional robot interaction and enables reproducible policy evaluation.These applications target imitation learning and controlled comparison across policies.
  • 100% world-simulator data produces policy performance comparable to an equivalent volume of real-world data.
  • Across diverse tasks and checkpoints, simulator and real-world policy performance exhibit a strong correlation.

A. Video Prediction Model for Robotic Manipulation

The paper situates Interactive World Simulator against slow, inaccessible, or brittle video-prediction and evaluation systems. Its two-stage latent-space design targets efficient, multimodal, physically consistent interaction modeling for scalable robotics use.

  • Many existing models lack robot-action conditioning, require substantial compute, or fail to remain robust during stable long-horizon prediction.These limitations hinder real-time interaction and broad academic access.
  • The method trains a compact latent autoencoder before learning action-conditioned future dynamics with a consistency model.The dynamics model predicts future latent states that are decoded into video.
  • Consistency models efficiently represent multimodal future outcomes while supporting long-horizon video generation.
  • 10 minutes at 15 FPS on a single consumer RTX 4090 GPU makes the simulator suitable for interactive long-horizon prediction.
  • The simulator addresses costly policy evaluation by offering a scalable framework whose outcomes can be compared with real-world performance.

III. METHOD

The method formulates next-frame action-conditioned video prediction from episodic robot observations and actions. Training uses a two-stage architecture that first learns image reconstruction and then latent-space dynamics.

  • III-A. Problem Formulation: The interaction dataset consists of episodes containing RGB observations and corresponding robot actions.
  • III-A. Problem Formulation: The prediction objective uses history observations and actions to predict the next RGB frame by minimizing squared reconstruction error.
  • III-B. Model Training and Inference: Stage 1 trains an autoencoder that encodes images into latent representations and decodes them to reconstruct the inputs.
  • III-B. Model Training and Inference: Stage 2 freezes the encoder and decoder, then trains an action-conditioned dynamics model with next-frame supervision in latent space.

1) Stage 1: Autoencoder Training:

Stage 1 trains an autoencoder that encodes images into 2D latent representations and decodes noisy inputs into high-fidelity reconstructions using a consistency-model decoder.

  • A consistency-model decoder provides efficient, high-quality image generation, trained with a weighted regression loss.
  • The autoencoder encodes each image into a 2D latent representation z and conditions decoding on that latent.
  • The decoder maps a higher-noise input to a lower-noise target conditioned on the latent representation.
  • The resulting encoder and conditional decoder reconstruct high-fidelity images from noisy inputs with a small number of denoising steps.

2) Stage 2: Dynamics Training:

Stage 2 freezes the autoencoder and trains an action-conditioned consistency-model dynamics network to predict future latent frames from history and actions, with robustness to noisy contexts.

  • The dynamics model predicts future latent frames from a context window of past latents and actions after the autoencoder is frozen.
  • Training applies full noise only to the last frame so the model predicts its lower-noise latent from action sequences and history context.
  • Inference appends a noisy latent to the history latents before denoising the predicted frame.
  • The dynamics model is trained with a weighted regression loss in latent space and implemented with 3D convolutional blocks, FiLM modulation, and spatiotemporal attention.
  • Small noise is injected into observation contexts to improve robustness when model predictions become later contexts during long-horizon inference.

3) Inference:

Inference autoregressively predicts latent frames from an initial image, action sequence, and evolving history context. The simulator supports interactive demonstration generation and policy interaction, while baseline rollouts show long-horizon failures that the proposed model avoids.

  • Inference: Given an initial image, the simulator encodes it, denoises a noisy latent using action information, and repeatedly appends predictions to the history context.
  • Data Generation: The simulator provides a scalable surrogate for expert demonstrations without requiring access to physical robots.
  • Data Generation: Human operators can issue control commands in the simulator to generate full demonstration trajectories compatible with standard imitation-learning pipelines.
  • Policy Evaluation: Simulator-based policy evaluation mirrors real-world interaction by feeding predicted frames to policies for subsequent action generation.
  • Qualitative Comparison: Across manipulation tasks, baseline rollouts develop inaccurate dynamics, pose drift, artifacts, or lost details, whereas the proposed model maintains coherent robot–object interactions over extended horizons.

IV. EXPERIMENT

The experiments compare the simulator across diverse simulated and real-world manipulation tasks involving multiple object types and interactions. The setup uses ALOHA Bimanual Robot data, constrained action spaces, and lightweight training and inference requirements.

  • Evaluation Questions: The evaluation asks whether the simulator improves realism, speed, and robustness, generates data comparable to real data, and faithfully represents real-world policy performance.
  • Task Setup: The benchmark includes one MuJoCo simulation task and six real-world tasks spanning rigid, deformable, articulated, and piled objects.
  • Data Collection: All tasks use the ALOHA Bimanual Robot, with about 600 real-world episodes per task and 200 steps per episode.
  • Task Setup: The setup constrains portions of the action space and uses a default image resolution of 128 × 128.
  • Efficiency: The mug-grasping model is 176.02 MB, while stage 1 and stage 2 training typically take around 6 and 12 hours, respectively, on one H200 GPU.

B. Video Baseline Comparisons

The Interactive World Simulator is compared with state-of-the-art action-conditioned world models under identical long-horizon conditions. It achieves stronger visual fidelity, temporal consistency, and inference efficiency while maintaining physically plausible interactions.

  • Quantitative comparison: Across aggregated metrics, the proposed method consistently outperforms prior world models in long-horizon video prediction.The evaluation uses identical initial conditions and 192-step rollouts across diverse manipulation tasks.
  • Qualitative comparison: Baseline rollouts exhibit pose drift, inaccurate object dynamics, missing details, or severe artifacts, whereas the proposed model maintains stable, physically plausible interactions.The qualitative comparison covers long-horizon predictions across the evaluated manipulation tasks.
  • Inference efficiency: The simulator runs interactive prediction at up to 15 FPS on a single GPU, improving inference efficiency over prior approaches.This efficiency supports interactive long-horizon prediction rather than only offline rollout generation.

C. Data Generation for Policy Training

The world simulator supports interactive demonstration collection and produces training data that remains competitive with real-world data across mixture ratios and dataset sizes. Policies improve as demonstrations increase, with simulator-generated data showing comparable scaling behavior.

  • Interactive data collection: Users can collect expert demonstrations by controlling the simulator with keyboard or kinematic devices, without physical mugs, plates, or other task objects present.The teleoperation interface maps user commands to real-time simulator frames and robot–object interactions.
  • Data mixtures: With 100 training episodes, policies trained across world-simulator and real-world mixtures show consistently high and comparable task scores.Mixtures span from 100% simulator data to 100% real-world data across multiple manipulation tasks.
  • Data scaling: From 5 to 100 episodes, policy performance improves consistently for both MuJoCo and world-simulator data sources.This similar scaling behavior persists across the evaluated dataset sizes and policies.

D. Sim-to-Real Correlation for Faithful Policy Evaluation

The simulator is evaluated as a proxy for real-world policy performance using identical initial configurations across multiple tasks. Strong positive correlations indicate that simulator rankings closely reflect relative real-world performance, despite a slight score bias on most tasks.

  • Cross-environment correlation: Across four manipulation tasks, simulator and real-world policy scores exhibit strong positive correlations under identical evaluation settings.Policies include final and intermediate checkpoints evaluated from 20 sampled initial configurations.
  • Sim-to-real gap: For tasks other than T pushing, fitted lines show a slight positive bias, with policies tending to score higher in simulation than in the real world.The observed bias represents a sim-to-real gap rather than an absence of correlation.
  • Policy selection: If one policy substantially outperforms another in simulation, the results suggest it is likely to do so in the real world as well.This supports using the simulator to select candidate policies before real-world experiments.

V. CONCLUSION

The Interactive World Simulator provides stable, physically consistent, and efficient long-horizon interaction for robotic policy training and evaluation. Simulator-generated training data performs comparably to real-world demonstrations, while simulator performance strongly correlates with real-world performance; future work targets broader environments and more complex tasks.

  • Core system: The simulator supports over 10 minutes of continuous interaction at 15 FPS while maintaining high-quality pixel-level predictions.The conclusion emphasizes both long-horizon stability and computational efficiency.
  • Policy training: Policies trained exclusively on simulator-generated data perform comparably to policies trained on real-world expert demonstrations.This establishes the simulator’s utility as a scalable source of imitation-learning data.
  • Policy evaluation: Strong simulator–real-world performance correlations support using the simulator as a surrogate for policy training and evaluation.The claim is reported across several manipulation tasks.
  • Future directions: Future work will extend the framework to more diverse environments and increasingly complex manipulation tasks.The authors also identify scaling with interaction data and computational resources as an open direction.
Loading 2603.08546v1…