Source-linked AI summary
Learning Latent Plans from Play
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, Pierre Sermanet
TL;DR
Robotics lacks scalable ways to learn diverse general-purpose skills across a continuum of behaviors. The paper self-supervises control from unlabeled teleoperated play by organizing behaviors into reusable latent plans, and reports broader coverage and strong generalization across 18 visual manipulation tasks. Its scope is limited by assuming that important tasks can be specified by a single goal state and by evaluating play and tasks in the same simulated playroom.
Problem
Learning a diverse repertoire of general-purpose robotic skills remains an open challenge because conventional approaches target discrete tasks with task-specific supervision.
Method
Play-LMP self-supervises a conditional sequence-to-sequence latent-plan model on random experiences from unlabeled teleoperated play, then conditions control on current state, goal state, and a sampled plan.
Results
A single Play-LMP policy outperforms 18 specialized supervised behavioral-cloning policies on 18 user-specified visual manipulation tasks, while play covers 4.2 times more interaction-space regions than expert demonstrations and 14.4 times more than random exploration for equal collection time.
Takeaways & Limitations
Play-supervised learning supports scalable task-agnostic control, organizes latent plans around functional tasks without task labels, and is associated with robustness and retrying-till-success behavior.
Takeaways & Limitations
The method assumes important tasks can be described by a single goal state and evaluates play and test-time tasks in the same playroom environment.
Abstract
from arXiv · showhide
Acquiring a diverse repertoire of general-purpose skills remains an open challenge for robotics. In this work, we propose self-supervising control on top of human teleoperated play data as a way to scale up skill learning. Play has two properties that make it attractive compared to conventional task demonstrations. Play is cheap, as it can be collected in large quantities quickly without task segmenting, labeling, or resetting to an initial state. Play is naturally rich, covering ~4x more interaction space than task demonstrations for the same amount of collection time. To learn control from play, we introduce Play-LMP, a self-supervised method that learns to organize play behaviors in a latent space, then reuse them at test time to achieve specific goals. Combining self-supervised control with a diverse play dataset shifts the focus of skill learning from a narrow and discrete set of tasks to the full continuum of behaviors available in an environment. We find that this combination generalizes well empirically---after self-supervising on unlabeled play, our method substantially outperforms individual expert-trained policies on 18 difficult user-specified visual manipulation tasks in a simulated robotic tabletop environment. We additionally find that play-supervised models, unlike their expert-trained counterparts, are more robust to perturbations and exhibit retrying-till-success behaviors. Finally, we find that our agent organizes its latent plan space around functional tasks, despite never being trained with task labels. Videos, code and data are available at learning-from-play.github.io
1 Introduction
Robotics needs generalist control across a continuum of behaviors, but conventional task-specific supervision is costly and narrow. The paper proposes self-supervising goal-conditioned control from unlabeled play, whose coverage exceeds expert demonstrations and random exploration.
- Robotics seeks single agents that learn many skills, rather than separate specialists for individual behaviors.
- Conventional learning relies on task-specific demonstrations or rewards, requiring substantial labeling, segmentation, resets, or manual reward design.
- A task-agnostic policy must cover the continuum of reachable current-state and goal-state pairs, not merely a discrete task set.
- Play-LMP self-supervises behavioral organization in a latent plan space and reuses sampled plans for user-specified goal-conditioned manipulation.
- Play data is inexpensive and rich because teleoperation requires no task segmentation, labeling, or reset and reflects human interaction knowledge.
- 4.2 times more regions are covered by play than 18 tasks of expert demonstrations, and 14.4 times more than random exploration for equal collection time.
2 Related Work
The related work discusses supervision requirements for robotic skill learning and goal-conditioned policies, alongside figures illustrating the experimental environment, latent plan space, and supervised task demonstrations.
- Supervision for robotic learning: Robotic learning methods generally require supervision through engineered costs or rewards, or through demonstrations.The passage notes that both forms can require considerable human effort, including hand-engineered rewards and demonstration collection.
- Goal-conditioned control: Goal-conditioned policies have been explored for reinforcement learning and inverse-model-based control.The paper frames goal conditioning as a way to distill non-task-specific experience into a general-purpose policy.
- Experimental context: Figure 3 depicts the Playground environment used in the paper.
- Latent representations: Figure 4 visualizes the latent plan space with t-SNE.
- Demonstration supervision: Figure 5 shows a supervised demonstration sequence labeled and segmented for the sliding task.
3 Learning Task-Agnostic Control from Play Data
This section develops self-supervised task-agnostic control from unlabeled play by converting random play sequences into goal-conditioned training data and reusable latent plans. Play-LMP represents diverse ways to reach goals in a latent space and decodes those plans into actions.
- Play Data: Play data is modeled as unsegmented streams of high-dimensional observations and teleoperation actions collected during human operation.The observations include RGB images and the robot’s proprioceptive state.
- Play Data: Random play windows provide self-supervised labels by treating each sequence’s initial state as current, final state as reachable goal, and intervening actions as targets.The final state is guaranteed to be reachable from the initial state under the recorded actions.
- Play-GCBC: Play-GCBC encodes sensory channels into a fused state and trains a goal-conditioned stochastic RNN end-to-end to maximize the likelihood of actions in sampled play sequences.The policy receives the current encoded state and synthetic goal state at each timestep.
- Play-LMP Motivation: Multiple valid behaviors can connect the same current and goal states, creating conflicting action trajectories that challenge unimodal behavioral cloning.The policy therefore needs to represent different high-level behaviors leading to the same outcome.
- Play-LMP Motivation: Play-LMP addresses multimodality by learning compact latent representations of high-level plans conditioned on current and goal states, converting multimodal policy learning into unimodal decoding.The latent plan is sampled and supplied to the action policy so it need not represent all plans implicitly.
- Play-LMP: Play-LMP is a conditional sequence-to-sequence VAE that autoencodes random play experiences into a latent plan space for reusable plan representations and task-agnostic control.Its components include plan recognition from play sequences and a current-goal-conditioned plan prior.
- Play-LMP: The model jointly trains a sequence-based plan recognizer, a current-goal plan prior, and an action decoder using reconstruction and KL objectives.The KL term aligns the prior’s plan distribution with plans recognized from observed play; β<1 helps avoid posterior collapse.
4 Experiments
The experiments evaluate whether play-supervised policies generalize across 18 visual manipulation tasks, improve over baselines, and remain robust to perturbations. They also examine latent task organization and retrying behavior.
- Experimental setup: The study tests single play-supervised policies against specialist and multitask baselines across 18 visual manipulation tasks using pixel and ground-truth state observations.The evaluation compares Play-LMP and Play-GCBC with behavioral cloning models trained from expert demonstrations.
- Task success: Play-LMP outperforms specialized behavioral cloning models despite using no task labels, while its decoupled design yields systematic improvements over Play-GCBC and baselines.Per-task gains over the baselines reach up to 50 absolute percentage points.
- Scalability: 85.5% average success across 18 tasks is achieved by one task-agnostic Play-LMP policy without finetuning, versus 70.3% for 18 expert-trained behavioral cloning policies.With only 30 minutes of play data, Play-LMP still reaches 71.8% average success.
- Robustness: Play-trained models are significantly more robust than expert-demonstration-only behavioral cloning models when the robot’s initial end-effector position is perturbed.The authors attribute this robustness to play data’s broader coverage of the environment than expert demonstrations.
- Unsupervised task discovery: Without task labels, Play-LMP organizes its latent plan space around functional behaviors, with regions corresponding to drawer and button manipulation.This result provides qualitative evidence of unsupervised task discovery in the learned plan representation.
- Emergent retrying: Play-supervised models show qualitative retrying behavior after initial failure, unlike models trained solely on expert demonstrations.The observed behavior consists of making multiple attempts to complete the task.
5 Conclusion
The paper argues for learning a continuum of behaviors from unlabeled play rather than discrete tasks from expert demonstrations. It introduces self-supervised latent plan learning and reports scalability, robustness, and task-semantic organization within the tested playroom environment.
- Conclusion: The paper advocates learning the full continuum of tasks from unlabeled play data instead of discrete tasks from expert demonstrations.This reframes skill learning around the behaviors available in the environment.
- Conclusion: Play-LMP learns a self-supervised plan representation that discovers task semantics without task labels and generates actions for a task-agnostic policy.The approach can train a deep sensory stack from scratch.
- Conclusion: The experiments show that play improves data-collection scalability and robustness in the models trained from it.These conclusions are demonstrated in the playroom setting used for both play data and test-time tasks.
- Future work: Future work includes testing generalization to novel objects or environments and examining imbalance in play-data distributions.The paper explicitly identifies these as directions for future investigation.
A.1.1 Unsupervised Representation Learning of Plans and Control from Play
The paper frames play trajectories as observations generated by latent motor plans, then uses conditional variational modeling to recover goal-conditioned control. Play-LMP implements plan recognition, plan proposal, and plan-conditioned policy modules, while Play-GCBC provides a behavioral-cloning alternative.
- Latent plans: Play trajectories are modeled as state-action sequences generated by latent motor plans.The latent variable represents unobserved high-level behavior in the operator’s repertoire.
- Latent plans: Conditional representation learning uses current and goal states to align latent plans with goal-conditioned control.The model conditions the latent variable on the same current-state and goal-state information used to specify the control problem.
- Play-LMP: Play-LMP comprises recognition, proposal, and generation modules corresponding to plan recognition, plan proposal, and a plan- and goal-conditioned policy.The recognition network maps observed trajectories to latent-plan regions; the proposal network maps current and goal states to latent-plan distributions.
- Training: Play-GCBC and Play-LMP are trained as self-supervised methods using unlabeled play data.The supplied training procedures sample windows from play data and optimize goal-conditioned action learning, with Play-LMP also computing a KL loss.
- Training: Play-LMP training updates parameters by minimizing the goal-conditioned behavioral-cloning loss.The training algorithm initializes model parameters, samples play windows, encodes states and plans, computes KL and action losses, and updates the parameters.
A.1.2 Behavior Cloning
The behavioral-cloning baselines use one policy per task, with the same recurrent architecture used across BC, GCBC, and LMP.
- Behavior cloning: BC trains one behavioral-cloning policy πθ(a|s) for each of the 18 tasks.All BC, GCBC, and LMP policies use an RNN with two hidden layers of size 2048.
A.2 Architecture Details
The architecture combines visual and proprioceptive inputs, supports pixel and state experiments, and represents actions with discretized logistic mixtures. Image goals can omit proprioceptive state.
- Architecture: The model architecture includes vision, plan-recognition, plan-proposal, and policy subnetworks.Figure 9 identifies the layers, sizes, depths, inputs, and activation maps of these components.
- Inputs: Pixel observations combine 299x299x3 RGB images with 8-dimensional proprioceptive state.Proprioception contains end-effector position, orientation, and gripper elements.
- Inputs: State observations include normalized 8-dimensional proprioception, block position and orientation, and continuous door, drawer, and button sensors.The sensor values describe opening, closing, or pressing amounts.
- Actions: The 8-DOF action space contains end-effector position, orientation, and gripper angles, with each element quantized into 256 bins.Stochastic outputs are mixtures of discretized logistic distributions over the quantization bins.
- Goals: In image experiments, only the visual embedding is used as the goal state, enabling goal specification with a goal image alone.Proprioceptive inputs are normalized without an additional encoding network.
A.3.1 Playground Environment
The simulated Playground supports diverse play interactions and evaluation on 18 visual manipulation tasks. The experiments compare play-supervised and expert-trained policies, showing broader interaction coverage, stronger perturbation robustness, and emergent retries, within stated scope limits.
- Playground Environment: The Playground contains an 8-DOF robot, sliding door, drawer, block, three light-control buttons, other interactive objects, and a trash bin.Evaluation success functions are used only to measure specific tasks.
- Play Data: Play data records self-guided teleoperation involving varied object interactions such as picking up, pushing, opening, placing, and dropping objects.The example sequence is shown in Figure 11.
- Evaluation Tasks: 18 tasks evaluate knocking, button pressing, block manipulation, shelf and drawer interactions, and sliding-door control.The task suite includes grasping upright and flat blocks, sweeping, rotating, opening, and closing.
- Robustness: Play-supervised models are more robust than expert-demonstration policies when the initial end-effector position is perturbed.The perturbations range from 0.0 to 0.4 meters, and Figure 12 compares Play-LMP, Play-GCBC, and BC.
- Emergent Retrying Behavior: Play-LMP exhibits retrying behavior after initial failures, including succeeding on the third attempt to close a sliding door.The model also recovers by grasping an object after initially knocking it over, unlike models trained solely on expert demonstrations.
- Coverage Analysis: Interaction-space coverage is measured by counting unique bins across 11 quantized object-interaction dimensions over replayed datasets.Coverage plots use summed collection time on the x-axis and visited-bin cardinality on the y-axis; comparisons include expert demonstrations, play, and random exploration.
- Limitations: The study is limited to a single environment and expects generalization mainly to test tasks close to training tasks in current-goal space.Out-of-distribution transfer across rooms is left for future work.