PaperlayerView exact revision on arXiv

Source-linked AI summary

PlayWorld: Learning Robot World Models from Autonomous Play

Tenny Yin, Zhiting Mei, Zhonghe Zheng, Miyu Yamane, David Wang, Jade Sceats, Samuel M. Bateman, Lihan Zha, Apurva Badithela, Ola Shorinwa, Anirudha Majumdar

arXiv:2603.09030v3cs.ROcs.AI

TL;DR

Robot video world models still struggle with physically consistent contact-rich interactions, partly because human demonstrations provide narrow, success-biased coverage. PlayWorld trains action-conditioned video models from autonomous robot play, enabling scalable interaction-rich data collection, and reports strong dynamics prediction, policy evaluation, and real-world policy improvement, including up to 65% higher success rates after in-model fine-tuning.

  • Problem

    Current video models remain unreliable for contact-rich robotic manipulation, while human demonstration data narrowly covers successful executions and misses diverse contact dynamics and failure modes.

  • Method

    PlayWorld trains high-fidelity action-conditioned video world models on interaction-rich data collected through autonomous robot play with minimal human supervision.

  • Results

    PlayWorld produces broader interaction coverage and realistic dynamics predictions, supports policy evaluation, and improves real-world success rates by up to 65% through closed-loop in-model reinforcement learning.

  • Takeaways & Limitations

    Autonomous play provides a scalable path toward video world models that support policy evaluation and improvement beyond hardware constraints.

  • Takeaways & Limitations

    Data collection can produce redundant trajectories, depends on a sufficiently capable play policy, and does not eliminate long-horizon hallucinations or dynamic discrepancies.

Abstract

from arXiv · show

Action-conditioned video models offer a promising path to building general-purpose robot simulators that can improve directly from data. Yet, despite training on large-scale robot datasets, current state-of-the-art video models still struggle to predict physically consistent robot-object interactions that are crucial in robotic manipulation. To close this gap, we present PlayWorld, a simple, scalable, and fully autonomous pipeline for training high-fidelity video world simulators from interaction experience. In contrast to prior approaches that rely on success-biased human demonstrations, PlayWorld is the first system capable of learning entirely from unsupervised robot self-play, enabling naturally scalable data collection while capturing complex, long-tailed physical interactions essential for modeling realistic object dynamics. Experiments across diverse manipulation tasks show that PlayWorld generates high-quality, physically consistent predictions for contact-rich interactions that are not captured by world models trained on human-collected data. We further demonstrate the versatility of PlayWorld in enabling fine-grained failure prediction and policy evaluation, with up to 40% improvements over human-collected data. Finally, we demonstrate how PlayWorld enables reinforcement learning in the world model, improving policy performance by 65% in success rates when deployed in the real world.

1 Introduction

PlayWorld addresses contact-rich prediction failures in robot video models by training on broad, autonomous robot-play experience rather than narrow successful demonstrations. The resulting models support scalable data collection, realistic dynamics prediction, policy evaluation, and in-model reinforcement-learning fine-tuning.

  • Research gap: Current video models hallucinate during contact-rich interactions, limiting their use in robotic manipulation.Reported failures include object duplication, disappearance, and unrealistic motion or deformation upon contact.
  • Research gap: Human demonstration datasets narrow training coverage toward successful task executions and underrepresent counterfactual contact dynamics and failure modes.This can bias rollouts toward successful behaviors seen in the data.
  • PlayWorld: PlayWorld uses autonomous play to expand coverage of diverse contact events, state transitions, and failure modes with minimal human intervention.The proposed data paradigm is designed for directed exploration and efficient scaling.
  • PlayWorld: Fully unsupervised overnight operation makes PlayWorld highly scalable for collecting interaction experience.Human involvement is reduced to occasional monitoring and scene resets.
  • Results: PlayWorld produces fine-grained predictions for realistic physical interactions and continues improving beyond the data scale where human-demonstration models saturate.The introduction reports continued gains beyond 5× that saturation regime.
  • Results: PlayWorld supports policy evaluation and closed-loop reinforcement-learning fine-tuning, with up to 65% improvement in real-world success rates over the pretrained policy.Policy evaluation is reported to correlate strongly with real-world performance.

2 Related Work

Related work positions world models as tools for prediction, planning, and policy improvement, while emphasizing that broad interaction coverage is difficult to obtain in the real world. Video-based and play-data approaches address parts of this challenge but commonly rely on costly or manually supervised data collection.

  • World models: World models learn environment dynamics that let agents reason and plan beyond purely reactive policies.Model-based reinforcement learning commonly relies on broad state-action coverage from interaction data.
  • World models: Real-world exploration is difficult to scale, creating a bottleneck for collecting diverse interactions needed by generalizable world models.Large-scale exploration is more readily obtained in simulation than in physical environments.
  • Video world models: Video generation models provide pixel-space future prediction and rich visual priors for policy evaluation, improvement, and other embodied tasks.Their interface can span tasks, embodiments, and downstream objectives.
  • Play data: Prior play-data methods support planning, representation learning, policy learning, and pretrained-model fine-tuning, but typically require human-collected demonstrations or manually supervised policy rollouts.These requirements limit the scalability of play-data collection.

3 PlayWorld: Training Video World Models with Robot Play

PlayWorld trains action-conditioned video world models from autonomous robot play, expanding interaction coverage without human supervision. Its pipeline combines task proposal and execution with curriculum-based training to model diverse contact dynamics.

  • PlayWorld trains action-conditioned video models to predict diverse contact dynamics with fine-grained precision.
  • Autonomous play is designed to expand visitation coverage across diverse objects, instructions, states, and interaction events.The data-collection requirements include diverse object interactions, failure recovery, and generalization to varied language instructions.
  • The system autonomously collects interaction data by combining a task proposer with a task executer.A VLM generates scene-grounded instructions, while a VLA executes them to produce interaction data.
  • Video Model Architecture: PlayWorld uses a pretrained stable video diffusion backbone with per-frame action conditioning for controllable visual prediction.The architecture uses factorized spatial and temporal attention to separate motion signals from frame appearances.
  • Curriculum Learning: Curriculum learning addresses redundant and long-tailed play data by gradually emphasizing rare, difficult interactions.Training begins with frequent free-space motions and static contacts before shifting toward harder transitions.
  • Curriculum Learning: The curriculum ranks play observations by distance to success prototypes computed from CLIP embeddings of successful demonstrations.Smaller distances indicate visually and semantically closer transitions, while larger distances indicate exploratory or less task-relevant interactions.

4 Experiments

Experiments evaluate PlayWorld’s interaction coverage, dynamics prediction, policy evaluation, fine-tuning, and scaling across diverse manipulation settings. Results show broader data coverage and improved modeling of contact-rich interactions, with benefits for policy assessment and adaptation.

  • Evaluation Setup: Experiments compare PlayWorld with human demonstrations and human play across three object sets covering containment, stacking, and towel-folding tasks.PlayWorld uses 30 hours of task-agnostic autonomous play, while the human demonstration baseline uses 6 hours of data.
  • Interaction Diversity: Robot play data covers substantially more diverse contact events, object states, and failure modes than human-collected data.Human-collected trajectories form a more concentrated cluster reflecting mostly successful transitions.
  • Dynamics Prediction: PlayWorld improves prediction quality for contact-rich dynamic interactions, while scaling from 6h to 30h provides further gains.Prediction quality for successful interactions is similar across training mixtures, whereas dynamic interactions benefit consistently from PlayWorld, scaling, and curriculum learning.
  • Policy Evaluation: PlayWorld’s predicted success rates correlate strongly with real-world outcomes across diverse policies, reaching a Pearson correlation of 0.8766.The evaluated policies vary in architecture, training mixture, success rate, and failure behavior.
  • Policy Evaluation: PlayWorld predicts both success and failure distributions more closely than baseline models, whose behavior has larger variance.This evaluation compares predicted and human-annotated failure-mode distributions for representative policies.
  • Policy Fine-Tuning: Up to 65% improvement in real-world success rates is achieved by fine-tuning a policy through interactive rollouts inside PlayWorld.The fine-tuning starts from a base diffusion policy trained with fewer than 10 demonstrations.
  • Data Scaling: PlayWorld continues improving as data scales to 30 hours, whereas baseline models show only marginal gains at comparable scale.
  • Object Generalization: Increasing training-object diversity consistently improves prediction quality on held-out objects across test categories.The result holds despite shifts in object appearance and geometry.

5 Conclusion

PlayWorld is an efficient, autonomous pipeline that trains high-fidelity action-conditioned video world models on interaction-rich robot play data. Its resulting model supports reliable policy evaluation and in-model reinforcement-learning fine-tuning, with gains continuing as data scale and object diversity increase.

  • PlayWorld trains high-fidelity action-conditioned video world models on interaction-rich robot play data through efficient, autonomous data collection.The pipeline includes unattended overnight operation and aims to provide a practical path toward high-quality robot simulators.
  • The resulting world model achieves strong dynamics fidelity on both nominal and off-nominal interactions.This supports modeling interactions beyond success-biased demonstrations.
  • PlayWorld enables reliable fine-grained policy evaluation and in-model reinforcement-learning fine-tuning.These post-training applications use the learned video world model rather than relying exclusively on additional hardware interaction.
  • Increasing data scale and object diversity produces continued gains, suggesting a practical path toward simulators for policy evaluation and improvement beyond hardware constraints.

6 Limitations and Future Work

PlayWorld’s current limitations concern data-collection efficiency, residual long-horizon prediction discrepancies, and extension beyond controlled laboratory settings. Future work targets more selective exploration, better rollout-control alignment, and principled scaling across heterogeneous data.

  • The current collection strategy can produce redundant trajectories because it does not explicitly optimize sample efficiency.Future strategies could prioritize high-uncertainty or underrepresented interactions, while specialized play policies could target rare contact events.
  • PlayWorld does not eliminate hallucinations entirely, as open-loop discrepancies, fixed action horizons, and control-mode mismatches can degrade closed-loop stability over long horizons.Proposed directions include flexible prediction horizons, receding-horizon rollouts, and tighter integration between action conditioning and control representations.
  • Extending PlayWorld beyond a controlled lab environment remains an open challenge across more diverse objects, scenes, robot embodiments, and real-world settings.Heterogeneous data distributions require improved scaling strategies, curriculum design, and data balancing.
  • Principled automatic curricula remain needed to optimize prediction fidelity and robustness to long-tail interactions as data scale increases.The paper also identifies multi-robot datasets and non-robot video as possible sources of broader physical priors.
  • PlayWorld’s demonstrated scaling trends support its promise as a general-purpose simulator for policy evaluation and reinforcement learning.

A Implementation Details

The implementation combines autonomous play-data collection with a robust behavior-diversity objective and curriculum-based video-model training. The system uses multimodal robot observations, task prompts, and progressively harder interaction data to train the world model.

  • Data Collection Design: PlayWorld uses a robust proxy that maximizes behavior diversity while staying close to the policy’s natural action manifold.This avoids directly optimizing world-model uncertainty and supports broad coverage without brittle closed-loop dependence on uncertainty estimates.
  • Data Collection Design: The play policy generates random manipulation and arrangement tasks within a marked square table workspace.Example instructions include placing, removing, positioning, and manipulating objects relative to one another.
  • Data Collection Design: The implementation includes both in-distribution and out-of-distribution task examples, such as placing objects, folding towels, sliding items, and flipping blocks.
  • Video Model Training: The video model jointly predicts three camera views—left, right, and wrist—at 192 × 320 resolution from the current view and Cartesian-space end-effector pose.Training begins with SVD pretraining on the full DROID dataset before curriculum-based fine-tuning.
  • Video Model Training: Training uses a curriculum that progressively shifts sampling toward harder, rarely occurring transitions.The implementation details identify Table 2 as the source of training hyperparameters.

B Play Data Analysis

Autonomous play produces diverse, unstructured interaction data with broader state-space coverage than human-collected alternatives at matched budgets. The analysis frames this diversity as a scalable source of training data, while the reported table and category labels organize comparisons across data mixtures and interaction outcomes.

  • Play Data Visualization: Play data consists of unstructured, task-agnostic interactions that provide broad coverage of interaction modes beyond narrow expert demonstrations.It is presented as a scalable way to convert large volumes of uncurated robot experience into a simulatable environment with minimal human involvement.
  • Play Data Visualization: Autonomous roll-outs produce natural variation in object contacts, viewpoints, and interaction outcomes without a scripted task objective.
  • Data Distribution at Different Scales: Across matched data budgets, play data spans a broader region of the embedding space than human-collected trajectories.Demonstration-based data concentrates in a comparatively narrower region, while the analysis embeds initial trajectory frames using CLIP features.
  • Training Mixtures: The training-mixture analysis organizes interaction outcomes into success, missed grasp, slide, slip, deformation, and collision categories.
  • Concurrent Works: Prior autonomous-play work has shown competitive performance against human demonstrations, while this paper focuses on autonomous play as a scalable source for world-model training.

C Additional Trajectory Replay Results

The trajectory replay benchmark targets dynamic contact events that are difficult to predict and decisive for counterfactual evaluation and task success. It uses policy roll-outs and naturally occurring failures to provide broad failure-mode and trajectory coverage, with normalized LPIPS scores enabling fair scaling comparisons.

  • Benchmark motivation: Dynamic collisions, deformation, and slip are emphasized because they are diverse, difficult to predict, and decisive for counterfactual evaluation and task success.The benchmark is designed around predicting these interaction events rather than nominal static contacts.
  • Benchmark composition: Table 3 summarizes the composition of the trajectory replay benchmark.
  • Benchmark construction: The benchmark is curated from policy roll-outs containing natural failure episodes labeled by failure mode.Each failure mode includes a wide range of trajectories.
  • Scaling analysis: The scaling analysis converts raw LPIPS values into a normalized score using a fixed affine transform.The transform uses s_min = 0.072 and s_max = 0.10 as constants defining the benchmark’s LPIPS range.
  • Scaling analysis: Normalization preserves model ordering while improving separation in scaling plots and making higher scores represent better perceptual fidelity.Pairwise model comparisons remain unchanged after the mapping.

D Additional Fine-Tuning Results

PlayWorld supports reinforcement-learning fine-tuning of diffusion policies inside a learned world model by steering frozen diffusion sampling with learned latent noise and dense progress rewards. The approach yields rapid behavioral improvements and real-world gains, but prolonged optimization can exploit model mismatches and reduce hardware performance.

  • Fine-tuning motivation: A learned world model can support reinforcement-learning fine-tuning that improves real-world success rates on challenging fine-grained manipulation tasks.The evaluation addresses mismatch in contact, actuation, latency, and sensing between simulation and hardware.
  • Fine-tuning challenges: Diffusion-policy fine-tuning is challenging because iterative sampling complicates stable, low-variance actor–critic gradients, while learned-world-model training adds exploration brittleness.
  • DSRL formulation: The DSRL setup samples initial latent noise and passes it through a frozen reverse-diffusion denoiser to generate actions.The reverse process is represented by gθ and remains fixed during fine-tuning.
  • DSRL formulation: Fine-tuning learns a state-conditioned distribution over initial noise, acting as a latent-space adapter while leaving diffusion parameters unchanged.
  • RL interface: The RL trajectory is generated by executing a = gθ(s, w), with the resulting state, noise, reward, and next-state tuples used for learning.
  • Reward design: A progress-based reward model converts predicted progress differences between consecutive time steps into dense per-step rewards.The reward model is trained from a small set of demonstrations.
  • Reward design: Measurable behavioral improvements often appear within the first 10–20 episodes, whereas sparse terminal rewards require substantially more interaction.Vision-language-model rewards were less reliable and often destabilized fine-tuning with false predictions.
  • Experimental setup: Table 5 lists the hyperparameters used for DSRL training and the diffusion policy.

E Final Remarks

The paper argues that large-scale robot play data is a promising, scalable supervision source for video world models and can make learned simulators more practical for policy development. Expanding collection across institutions and hardware could broaden model coverage and robustness.

  • Final remarks: Large-scale play data is presented as a promising and scalable supervision source for learning video world models.
  • Final remarks: In the authors’ lab, many hours of diverse, unlabeled interaction data were collected with relatively low overhead.
  • Final remarks: Scaling collection across institutions and hardware setups could expand coverage of environments, object sets, and embodiments while improving learned-dynamics fidelity and robustness.The paper envisions a shared play-data ecosystem for stronger, more generalizable world simulators.
Loading 2603.09030v3…