PaperlayerView exact revision on arXiv

Source-linked AI summary

Dexterity from Supervised Play

Botao He, Yiming Dou, Yiannis Aloimonos, Jitendra Malik, Akash Sharma, Ruoshi Liu

arXiv:2610.22340v1cs.ROcs.AI

TL;DR

Robot play is difficult to formalize with heuristics, despite its potential as a task-agnostic source of interaction for learning dexterous manipulation. DSP behavior-clones a goal-agnostic play policy from human teleoperation, uses it for autonomous data collection and world-model-guided MPC, and optionally distills a goal-conditioned policy through model self-play. On 128 distantgoal trials, play-guided MPC reached 72.7/91.4/96.1% at 0.4/0.6/0.8 rad, versus hindsight relabeling’s 25.8/53.9/71.9%.

  • Problem

    Robot play is difficult to formalize with heuristics, despite its potential as a task-agnostic source of interaction for learning dexterous manipulation.

    Source: §I · p. 1

  • Method

    DSP behavior-clones a goal-agnostic play policy from human teleoperation, uses it for autonomous data collection and world-model-guided MPC, and optionally distills a goal-conditioned policy through model self-play.

    Source: §I · p. 1, B. Supervised Play · p. 3, F. Policy Distillation · p. 4

  • Results

    On 128 distantgoal trials, play-guided MPC reached 72.7/91.4/96.1% at 0.4/0.6/0.8 rad, versus hindsight relabeling’s 25.8/53.9/71.9%.

    Source: B. Results · p. 5

  • Takeaways & Limitations

    The results show that autonomous play data can support world-model learning and goal-directed planning, with prediction improving as the real-robot play dataset grows from 0.5 to 5 hours.

    Source: §I · p. 1

    A provisional hardware summary requires a frozen protocol and uncertainty before it can support a final task-success claim.

    Source: §VI · p. 8

Abstract

from arXiv · show

How can robots autonomously collect the contact-rich experience needed to learn dexterous manipulation? We present DSP: Dexterity from supervised play, a framework in which a robot learns a play policy from minutes of human play through behavior cloning, then deploys it autonomously to collect hours of interaction data to train a world model. During inference, the same play policy then guides model-predictive control (MPC) by proposing candidate action sequences for evaluation under the learned world model. To accelerate inference, we also introduce a method for distilling a closed-loop, goal-conditioned policy through self-play within the learned world model. We evaluate this framework on underactuated pushing and any-to-any in-hand pose reorientation with a multifingered hand, in simulation and real world. In real world experiment, our method outperforms hindsight behavior cloning by 69.6% in success rate and achieves comparable success to reinforcement learning with 490 times less interaction data. These results demonstrate that supervised play can autonomously collect interaction data, guide model-based planning, and enable policy learning without any task-specific demonstrations or sim-to-real transfer.

I. INTRODUCTION

DSP addresses the difficulty of engineering play by learning a goal-agnostic play policy from human demonstrations and using it to collect autonomous interaction data. The framework uses those data to train a world model for play-guided planning, with optional policy distillation.

I. INTRODUCTION

  • Play is difficult to formalize with hand-crafted heuristics, and manually engineered safety constraints require tuning that limits scalability.The paper hypothesizes that play can instead be learned as a goal-agnostic, state-conditioned probabilistic model through behavior cloning.Source: §I · p. 1
  • DSP trains a play policy from minutes of human play, then uses it to collect autonomous interaction data for world-model learning and MPC.The same play policy proposes candidate action sequences, while the learned model predicts their outcomes for goal-conditioned action selection.Source: §I · p. 1, Figure 2 · p. 2
  • Held-out world-model prediction improves monotonically as the real-robot autonomous-play dataset grows from 0.5 to 5 hours.This scaling result is presented as a contribution alongside the framework’s manipulation results.Source: §I · p. 1
  • DSP also introduces optional self-play distillation of a closed-loop, goal-conditioned policy to accelerate inference.The policy is distilled within the learned world model.Source: §I · p. 1
  • 91.4% of trials reached within 0.6 rad and 72.7% within 0.4 rad with DSP, versus 53.9% and 25.8% for hindsight BC.The reported results also say PPO reached 86.7% at both thresholds after 344M counted frames, about 490 times the counted-frame budget.Source: §I · p. 2

II. RELATED WORK

Prior interaction-driven RL pipelines demonstrate the power of planning but depend on well-specified rewards, simulators, or game rules. Intrinsic-motivation signals encourage exploration, yet do not specifically bias it toward safe, object-centric contact.

A. Play and interaction in RL

  • Interaction and planning have produced sophisticated behavior, including game and physical-system performance, but those pipelines depend on well-specified rewards, simulators, or game rules.DSP instead targets contact-rich manipulation using behavior derived from human play rather than reward-driven search.Source: A. Play and interaction in RL · p. 2

B. Curiosity and intrinsic motivation

  • Intrinsic-motivation methods use prediction error or feature novelty to drive exploration, but their heuristic signals do not bias interactions toward safe, object-centric contact.That contact is the kind required for dexterous manipulation.Source: B. Curiosity and intrinsic motivation · p. 2

C. Learning from play

Earlier play-learning approaches largely reuse play as a fixed imitation dataset, while DSP turns a human play seed into a reusable autonomous data-collection mechanism. The pipeline trains a play policy from teleoperation and uses its collected transitions to learn manipulation dynamics.

C. Learning from play

  • Play-LMP reuses latent plans from teleoperated play and later work distills affordances, but these approaches largely treat play as a fixed imitation dataset.DSP instead uses a human play seed to train a policy that can collect data autonomously.Source: C. Learning from play · p. 2

D. Dexterous manipulation

  • Prior dexterous-manipulation RL methods require carefully shaped rewards and hundreds of millions of simulated interactions, whereas DSP uses autonomous-play data without simulation data, task-specific demonstrations, or rewards.DSP scores play-policy proposals with a learned world model under MPC for arbitrary goals.Source: D. Dexterous manipulation · p. 2

A. Overview

  • DSP proceeds from teleoperation-trained play policy to autonomous collection of observation–action transitions, world-model training, and either online MPC or optional policy distillation.Distillation amortizes inference cost by producing a goal-conditioned closed-loop policy from self-play.Source: A. Overview · pp. 2–3

B. Supervised Play

  • A diffusion-transformer play policy is behavior-cloned from goal-free human teleoperation data and samples action chunks conditioned on observation history.The demonstrations capture continued object interaction without prescribed goals or task-specific rewards; safe behavior is implicitly demonstrated rather than explicitly engineered.Source: B. Supervised Play · p. 3

C. Autonomous Play

DSP autonomously collects robot play transitions from a behavior learned through human play, then trains a world model to predict future observations from recent history and action chunks. The model is trained with Diffusion Forcing, which supports causal denoising of future observations.

C. Autonomous Play

  • The robot autonomously rolls out the play policy to collect sensory and proprioceptive transitions with little human supervision, in a process that can be parallelized across robots and environments.The resulting play dataset is described as providing sufficient coverage in the state-transition space.Source: C. Autonomous Play · p. 3

D. World Modeling

  • The world model predicts future observations conditioned on recent observation history and an input action chunk.The context length is C and the prediction horizon is K.Source: D. World Modeling · p. 3
  • Diffusion Forcing assigns independent noise levels across observation time steps and learns to denoise causally, enabling clean-history conditioning and autoregressive future rollouts.The model is trained on the collected play dataset.Source: D. World Modeling · p. 3

E. Model Predictive Control

Play-guided MPC samples candidate action chunks from the play policy and scores their predicted outcomes under a learned world model, with goals entering through the reward. Optional self-play distillation trains a closed-loop goal-conditioned policy to amortize repeated MPC search.

E. Model Predictive Control

  • The play policy supplies action proposals that focus MPC search on demonstrated interaction behaviors, and the world model predicts the candidates’ outcomes.At each step, MPC samples N action chunks and predicts outcomes over K ≤ H steps.Source: E. Model Predictive Control · p. 3
  • MPC selects a candidate by its best predicted goal reward, executes only an action prefix, then replans after receiving a new observation.The goal enters through the reward, while the play policy and world model remain goal-agnostic.Source: E. Model Predictive Control · pp. 3–4
  • Two Sharpa-hand trials approach requested cube orientations despite substantial intermediate increases in angular error, with trajectories finishing near their goals.The figure pairs image sequences and error curves with SO(3) trajectories.Source: Figure 4 · p. 4

F. Policy Distillation

  • Self-play distillation runs MPC in the learned world model on recorded histories and sampled goals, then behavior-clones a policy from the selected action prefixes.At deployment, the goal-conditioned policy predicts prefixes from actual observation history and goal, updates from new observations, and amortizes MPC search.Source: F. Policy Distillation · p. 4

IV. EXPERIMENTS

The experiments assess whether goal-free play supports goal-directed manipulation across tasks, hardware, and observation representations. They also compare play-guided MPC with direct-policy and task-specific reinforcement-learning baselines, and examine design factors affecting prediction and control.

IV. EXPERIMENTS

  • The evaluation asks whether DSP turns goal-free play into goal-directed manipulation across tasks and on real hardware, and whether play-guided MPC is more interaction-efficient than direct policy learning and task-specific RL.It also examines play-policy design, autonomous-play scale, and world-model design.Source: §IV · p. 4

A. Experimental Setup

  • The four tasks cover state-based Push-T under a fixed 10k-frame teleoperation budget, simulated multifingered reorientation, real-hardware arbitrary-goal reorientation, and RGB-input visual reorientation.For visual reorientation, the world model evaluates actions in a learned 64-dimensional latent representation.Source: A. Experimental Setup · p. 4
  • The real-world setup uses a 22-joint Sharpa hand, glove-based teleoperation, camera imaging, and cube-mounted tags for pose tracking, with a 120-second control budget.The setup compares against joint random walk, cube random walk, and the play policy; other simulation baselines are not transferable to hardware.Source: A. Experimental Setup · p. 5
  • The simulated reorientation comparisons include play without a goal, hindsight BC, goal-conditioned BC, and PPO trained with matched or performance-matched frame budgets.Task B uses 128 seeded trials, while the real-world reorientation comparisons exclude simulation-only baselines.Source: A. Experimental Setup · pp. 4–5

B. Results

Play-guided MPC improves goal-directed manipulation on both Push-T and distant-goal reorientation trials. In Push-T, it outperforms the strongest policy baseline under the same data budget, while PPO needs far more interaction to match performance.

B. Results

  • 72.7/91.4/96.1%: play-guided MPC outperforms hindsight relabeling’s 25.8/53.9/71.9% at 0.4/0.6/0.8 rad on 128 distant-goal trials.The results indicate that relabeling play outcomes alone cannot reliably compose motions toward requested orientations.Source: B. Results · p. 5

1) Task A: Push-T:

  • For precise planar goals, play-guided MPC reaches 98% SR@0.90 and 74% SR@0.95, versus hindsight BC’s 74% and 36% under the same amount of data.The advantage grows from 24 to 38 percentage points as tolerance tightens.Source: 1) Task A: Push-T · p. 5
  • PPO-PM requires 344M frames—more than 490× the play-guided controller’s budget—to match performance, while task-specific PPO remains stronger at fine convergence for simulatable tasks.With the same 701K-frame budget, PPO-TM reaches only 0.0/0.0/1.6%.Source: 1) Task A: Push-T · pp. 5–6

2) Task C: Real-World Reorientation:

On real hardware, world-model search raises reorientation success above the learned play policy alone and stays close to simulation performance. The same play-guided planning mechanism also improves reorientation from RGB observations, establishing transfer to a visual representation.

2) Task C: Real-World Reorientation:

  • 20.9/38.3/54.1%: the learned play policy beats cube-space random walk’s 6.3/11.4/17.5% on real-world reorientation thresholds.Because the random walk controls for gross rotation speed, the difference points to useful learned contacts and regrasp motions, not motion magnitude alone.Source: 2) Task C: Real-World Reorientation · p. 6
  • 68.6/90.0/97.1%: real-world play-guided MPC gains 47.7/51.7/43.0 percentage points over play alone and remains within 4.1 points of simulation at every threshold.The corresponding simulation rates are 72.7/91.4/96.1%.Source: 2) Task C: Real-World Reorientation · p. 6, Figure 7 · p. 6

3) Task D: Visual Reorientation:

  • 56.2/78.1/89.1%: adding world-model MPC to the RGB policy raises success by 21.8, 25.0, and 12.5 percentage points at 0.4/0.6/0.8 rad.Task D uses 64 trials and a 3,600-step horizon, so it establishes representation transfer rather than a protocol-matched state-versus-RGB comparison.Source: 3) Task D: Visual Reorientation · p. 6

4) Distilled policy.:

Distillation turns planner-selected actions generated inside the learned world model into a goal-conditioned policy, reducing inference cost while retaining much of MPC’s performance. The analysis also finds that transition coverage and additional autonomous-play data improve important parts of the learning pipeline.

4) Distilled policy.:

  • 40.6/72.7/86.7% SR@0.4/0.6/0.8 rad: the distilled policy is the strongest imitation baseline and recovers over 90% of MPC performance at 0.8 rad.It runs in 21.8 ms, 7× faster than an MPC replan; 1.3M planner labels were generated from 54K recorded states paired with 24 goals each inside the world model.Source: 4) Distilled policy · p. 6

C. Analysis

  • 15–18 Hz: batched candidate evaluation and action chunking let the real-world closed-loop system continue executing while the next plan is computed.Each replan takes 134 ms, including 61 ms to sample 1,024 proposals and 44 ms for batched world-model rollout.Source: C. Analysis · pp. 6–7

1) Task B: Simulated Reorientation:

  • 84.2% versus 55.4%: diffusion play produces greater transition coverage than deterministic play, despite similar held-out command error of 0.181 versus 0.192.The passage links diverse transitions to more action-conditioning options for the world model and more ways for MPC to approach a goal.Source: 1) Task B: Simulated Reorientation · p. 7
  • 34.96° to 28.40°: increasing real-robot play data from 0.54 to 5.11 hours monotonically reduces 50-step rotation MAE by 18.8%.The final model is 26.4% below the 38.60° persistence baseline; most gains arrive in the first two hours, then flatten while continuing.Source: 1) Task B: Simulated Reorientation · p. 7

V. DISCUSSION

DSP treats human play demonstrations as a way to start useful interaction rather than specify task solutions. It separates physical experience from task goals, allowing the same play prior and world model to serve different goals without retraining.

A. DSP vs. behavior cloning

  • Human play specifies how to sustain useful interaction, while autonomous experience teaches action consequences beyond the demonstrations.The supervision supplies behavioral structure for data collection rather than a finished controller or expert task solutions.Source: A. DSP vs. behavior cloning · p. 7
  • Play-policy proposals guide MPC by supplying candidate action chunks, which the world model evaluates for movement toward the target.The figure depicts proposals, predicted T-block trajectories, and the selected candidate action sequence.Source: Figure 9 · p. 7

B. DSP vs. RL

  • DSP separates physical learning from task goals, so the same interaction prior and world model can support different goals without retraining.The passage contrasts this with task-specific RL organized around reward optimization and says DSP does not require a task-specific simulator or subsequent sim-to-real transfer.Source: B. DSP vs. RL · p. 7

C. DSP vs. curiosity vs. random exploration

DSP uses human play to structure autonomous interaction for dexterous manipulation, rather than relying on generic novelty objectives to produce sustained object interaction. The conclusion reports simulation and world-model prediction findings, while noting that a final hardware task-success claim awaits a frozen protocol and uncertainty estimates.

C. DSP vs. curiosity vs. random exploration

D. Behavior foundation models as play policies.

  • A broader pretrained behavior model could supply an initial interaction repertoire, which autonomous experience could ground in a particular robot and environment.The authors identify turning large behavior models into generalizable play policies as future work.Source: D. Behavior foundation models as play… · p. 8

VI. CONCLUSION

  • In simulation, the learned model supports arbitrary-goal reorientation with far fewer interactions than PPO-Match in counted training frames, and a controlled model-stack comparison improves goal acquisition under a fixed planner.Source: §VI · p. 8
  • On real-robot data, held-out prediction improves monotonically as autonomous play grows, but the hardware summary is provisional and final task-success results remain to be inserted.Source: §VI · p. 8

Related papers

Loading 2610.22340v1…