Source-linked AI summary
DayDreamer: World Models for Physical Robot Learning
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, Pieter Abbeel
TL;DR
Real-world robot learning requires less interaction than standard reinforcement learning typically provides, and simulator-trained behaviors may not capture the physical environment. This paper applies Dreamer directly to four robots, learning world models from experience and behaviors from imagined rollouts without simulators. Across locomotion and manipulation tasks, Dreamer learns from scratch in the real world, including quadruped walking in 1 hour and visual pick-and-place approaching human performance.
Problem
Real-world robot learning is limited by the large amount of trial and error required by deep reinforcement learning, while simulator-based learning may miss real-world complexity.
Method
Dreamer learns world models from replayed real-world experience and trains behaviors from imagined latent-space rollouts while interacting online without simulators.
Results
Dreamer learns locomotion, manipulation, and navigation across 4 robots without changing hyperparameters, including quadruped walking in 1 hour and visual pick-and-place approaching human performance.
Takeaways & Limitations
Dreamer provides a broadly applicable baseline for online robot learning directly in the real world without simulators.
Takeaways & Limitations
Learning on hardware over many hours causes robot wear that may require human intervention or repair.
Abstract
from arXiv · showhide
To solve tasks in complex environments, robots need to learn from experience. Deep reinforcement learning is a common approach to robot learning but requires a large amount of trial and error to learn, limiting its deployment in the physical world. As a consequence, many advances in robot learning rely on simulators. On the other hand, learning inside of simulators fails to capture the complexity of the real world, is prone to simulator inaccuracies, and the resulting behaviors do not adapt to changes in the world. The Dreamer algorithm has recently shown great promise for learning from small amounts of interaction by planning within a learned world model, outperforming pure reinforcement learning in video games. Learning a world model to predict the outcomes of potential actions enables planning in imagination, reducing the amount of trial and error needed in the real environment. However, it is unknown whether Dreamer can facilitate faster learning on physical robots. In this paper, we apply Dreamer to 4 robots to learn online and directly in the real world, without simulators. Dreamer trains a quadruped robot to roll off its back, stand up, and walk from scratch and without resets in only 1 hour. We then push the robot and find that Dreamer adapts within 10 minutes to withstand perturbations or quickly roll over and stand back up. On two different robotic arms, Dreamer learns to pick and place multiple objects directly from camera images and sparse rewards, approaching human performance. On a wheeled robot, Dreamer learns to navigate to a goal position purely from camera images, automatically resolving ambiguity about the robot orientation. Using the same hyperparameters across all experiments, we find that Dreamer is capable of online learning in the real world, establishing a strong baseline. We release our infrastructure for future applications of world models to robot learning.
1 Introduction
Dreamer addresses data-hungry real-world robot learning by learning a world model from experience and planning in imagination. The paper applies this approach across four robots and releases infrastructure for simulator-free online learning.
- Deep reinforcement learning often requires too much real-world trial and error, while simulators can miss real-world complexity and inaccuracies.
- Dreamer learns from replayed experience and uses imagined latent-space rollouts to reduce trial and error during real-world learning.
- Dreamer is applied to 4 robots across varied action spaces, sensory modalities, and reward structures without introducing new algorithms.
- A quadruped learns to roll over, stand, and walk from scratch in 1 hour, then adapts to pushes within 10 minutes.
- Robotic arms learn visual pick-and-place from sparse rewards, with behavior outperforming model-free agents and approaching human performance.
- The authors publicly release infrastructure supporting different action spaces and sensory modalities for future real-world world-model research.
2 Approach
Dreamer learns a predictive world model from replayed robot experience and trains task-specific behavior through imagined latent-space trajectories. Its asynchronous architecture separates learning from action computation while preserving sensory reconstruction for representation learning and inspection.
- Dreamer learns a world model from replay-buffer experience, trains behavior from model-predicted trajectories, and continuously collects new environmental data.
- The world model predicts future latent representations instead of raw sensory inputs, reducing accumulating errors and enabling large-batch parallel training.
- The encoder fuses sensory modalities into stochastic representations, while dynamics, decoder, and reward networks model transitions, observations, and rewards.
- Actor-critic learning optimizes task-specific behavior from imagined latent-space rollouts without decoding observations, using batches of 16K on one GPU.
- The actor maximizes future predicted rewards, while the critic estimates them beyond the planning horizon of H = 16 steps.
- The critic regresses λ-returns and the actor maximizes them, using reparameterization gradients for continuous control and Reinforce gradients for discrete actions.
- Actor and critic gradients are prevented from updating the world model because that would produce incorrect and overly optimistic predictions.
3 Experiments
Dreamer learns directly on four physical robots across locomotion, manipulation, and navigation tasks with varied sensors, actions, and rewards. Across these experiments, it achieves successful real-world behaviors with a shared training setup and adapts to environmental challenges.
- Experimental Setup: Dreamer is evaluated on four robots spanning locomotion, manipulation, and navigation, with continuous or discrete actions, dense or sparse rewards, image or proprioceptive observations, and sensor fusion.The experiments compare Dreamer with task-appropriate algorithmic and human baselines.
- A1 Quadruped Walking: 1 hour of real-world training enables the A1 quadruped to roll over, stand up, and walk from its back without simulators or resets.The robot rolls over within 5 minutes, learns to stand about 20 minutes later, and develops a pronking gait around one hour.
- A1 Quadruped Walking: 10 minutes of additional online learning enables the quadruped to withstand pushes or quickly roll back onto its feet after being knocked over.SAC learns to roll over but fails to stand up or walk under the same small data budget.
- UR5 Multi-Object Visual Pick and Place: 2.5 objects per minute after 8 hours brings Dreamer’s UR5 pick-and-place performance close to human-level performance, while Rainbow DQN and PPO learn short-sighted same-bin behavior.The task uses sparse rewards, pixel-based object localization, and multiple moving objects.
- XArm Visual Pick and Place: 3.1 objects per minute after 10 hours is comparable to human performance on the XArm, whereas Rainbow fails to learn the task.Dreamer uses RGB, depth, and proprioceptive inputs and sometimes pulls the soft object from a corner with its string.
- Sphero Navigation: 2 hours of training yields consistent Sphero navigation to a fixed goal from RGB images alone, achieving an average distance to the goal of 0.15.DrQv2 achieves similar performance on this continuous-control-from-pixels task.
4 Related Work
Prior robot-learning systems commonly rely on simulated experience, robot fleets, demonstrations, or task priors. End-to-end physical-world learning remains relatively uncommon, and existing visual-prediction approaches face short-horizon or computational limitations.
- Common Data Sources: Robot-learning research commonly uses large simulated datasets with domain and dynamics randomization before real-world deployment.Other approaches collect experience with robot fleets or use human demonstrations and task priors.
- Physical-World Learning: Relatively few works demonstrate end-to-end learning from scratch in the physical world.This positions direct online real-world learning as an uncommon setting in prior work.
- Visual Prediction: Visual Foresight uses online planning with video prediction but is limited to short-horizon tasks and requires computationally expensive image generation during planning.The paper contrasts this with latent dynamics for efficient policy optimization in a compact latent space.
5 Discussion
Dreamer enables sample-efficient physical robot learning across locomotion, manipulation, and navigation without simulators or changed hyperparameters, while leaving hardware wear and longer training as open concerns.
- Dreamer solves locomotion, manipulation, and navigation tasks from scratch in the real world without simulators and without changing hyperparameters.
- 1 hour suffices for a quadruped to roll off its back, stand up, and walk from scratch.
- 8–10 hours suffices for two robot arms to pick and place objects from pixels and sparse rewards.
- Training on hardware over many hours creates robot wear that may require human intervention or repair.
A Adaptation
Dreamer adapts to changing physical conditions without algorithmic changes, recovering from perturbations and major lighting shifts during real-world robot learning.
- Dreamer adapts to changing environmental conditions with no change to the learning algorithm.
- The quadruped learns within 10 minutes to withstand pushes or roll over and stand back up.
- The XArm recovers its original performance in about 5 hours after sunrise changes its visual observations.
- Sunrise produces strong light rays that substantially differ from the XArm’s original training observations.
B Imagination
Dreamer’s latent-space imagination can be inspected by rolling out imagined trajectories and decoding them into visualized policy intent.
- Each row depicts an imagined latent trajectory, with every second frame decoded into an image.
- Latent rollouts are shown for both the UR5 and XArm environments.
- In one UR5 trajectory, a static orange ball becomes a green ball, illustrating a modeled visual inconsistency.
C Detailed Related Work
Related work spans simulation-based locomotion, scalable or demonstration-based manipulation, and model-based reinforcement learning, while Dreamer emphasizes direct physical training with latent dynamics.
- RL for locomotion: Simulation-based locomotion commonly uses domain and dynamics randomization before freezing policies for real-world deployment.
- RL for locomotion: Dreamer trains directly on the physical robot without simulators, reset policies, or a simulation-trained recovery controller.
- RL for manipulation: Manipulation methods scale experience through robot fleets or use expert demonstrations, whereas Dreamer parallelizes one robot’s experience in its learned world model.
- Model-based RL: Dreamer learns latent dynamics for policy learning, avoiding direct high-dimensional visual prediction and dependence on high-quality visual reconstructions.