PaperlayerView exact revision on arXiv

Source-linked AI summary

DiscoDemo: Discovering Efficient and Diverse Robot Demonstrations for Imitation Learning

Minho Park, Kinam Kim, Donghu Kim, Byungkun Lee, Dongyoon Hwang, Yongjae Shin, Junha Hyung, Hojoon Lee, Jaegul Choo

arXiv:2610.32093v1cs.RO

TL;DR

Behavior cloning depends on training-data coverage, but physical collection of diverse successful demonstrations is costly, and task-solving RL can still produce narrow behavior. DISCODEMO trains a privileged, latent-conditioned RL generator with a reverse curriculum and success-gated diversity, then filters its demonstrations for downstream imitation learning. Across four manipulation tasks, it improves downstream success over the strongest baseline in simulation and on the real robot.

  • Problem

    Physical data collection is costly, and task-rewarded RL can concentrate successful rollouts around narrow solutions that provide limited coverage for downstream behavior cloning.

    Source: §1 · pp. 1–2

  • Method

    DISCODEMO trains a privileged latent-conditioned RL generator with a parallel reverse curriculum and success-gated diversity, then safety-filters its demonstrations for downstream imitation learning.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    62% versus 40% success in simulation and 78% versus 39% on the real robot, averaged across four tasks, compared with the strongest baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Broader dataset coverage is associated with stronger downstream performance, while excessive diversity can compromise task competence and the gains remain complementary to corrective data collection.

    Source: §5 · p. 9

    The approach assumes access to a simulator with privileged state information and resets to specified states, plus a small set of human demonstrations.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Behavior cloning for robotic manipulation is highly sensitive to the coverage of its training data, yet collecting diverse successful demonstrations remains costly. We study reinforcement learning (RL) not as a deployable controller, but as a data generator for downstream visuomotor imitation learning. Our key premise is that multiple successful solution modes cover a broader range of successful trajectories than repeatedly collecting a single narrow solution. We introduce DiscoDemo, which trains a privileged latent-conditioned RL generator with a parallel reverse curriculum and a success-gated diversity objective to discover diverse solutions while preserving task success. The trained generator produces multiple successful closed-loop behaviors, which are safety-filtered and exported as a static dataset for downstream imitation learning. Across four manipulation tasks, DiscoDemo produces broader successful-state coverage and improves downstream visuomotor policies in both simulation and the real world. Averaged across tasks, downstream success increases to 62% from 40% for the strongest baseline in simulation, and to 78% from 39% on the real robot, respectively. These results suggest that successful solution diversity improves imitation learning for a pretrained flow-matching VLA and remains complementary to corrective data collection. Project page: https://davian-robotics.github.io/DiscoDemo

1 INTRODUCTION

Physical data collection constrains the scale of behavior-cloning datasets, while simulation raises questions about how to generate successful trajectories and what behaviors they should cover. DISCODEMO trains an RL generator to discover multiple successful solutions, improving downstream policy success across four tasks in simulation and on the real robot.

1 INTRODUCTION

  • Physical teleoperation requires robot hardware and operator time for every episode, and large robot datasets contain on the order of a million episodes.The passage contrasts this scale with billions of image–text pairs or trillions of tokens used to train vision and language foundation models.Source: §1 · p. 1
  • Simulation can generate trajectories cheaply and in parallel, but the generation strategy and distribution of successful behaviors determine the supervision available to downstream behavior cloning.Source: §1 · p. 1
  • Task reward alone can concentrate RL rollouts around narrow successful solutions, while perturbation and recovery methods remain anchored to existing behavior.Limited coverage can make behavior cloning brittle when prediction errors move the learned policy beyond states represented in its demonstrations.Source: §1 · p. 2
  • DISCODEMO combines a parallel reverse curriculum with a latent-conditioned, success-gated diversity objective to discover distinct successful solutions for downstream behavior cloning.Demonstration states serve as curriculum resets rather than demonstration actions serving as the target solution.Source: §1 · p. 2
  • 62% versus 40% success in simulation, and 78% versus 39% on the real robot, averaged across tasks, gives DISCODEMO the strongest downstream policy results.The comparisons are against the strongest baseline in each setting.Source: §1 · p. 2

2 RELATED WORK

Prior simulation data-generation work reuses, transfers, or expands existing behaviors, while RL offers a complementary route to generating task-solving trajectories. DISCODEMO focuses on discovering multiple successful solution modes online for downstream behavior cloning.

Simulation Data Generation for Behavior Cloning

  • Planning, demonstration retargeting, and intervention or perturbation methods expand scene, state, or recovery coverage from existing demonstrations or behaviors.Source: §2 · p. 2

RL as a Data Generator

  • RL data-generation methods use privileged state and task rewards, demonstration-driven curricula, pretrained behavior priors, or patterns discovered from successful demonstrations.DISCODEMO instead discovers diverse successful solutions online using a latent-conditioned, success-gated objective.Source: §2 · p. 3

Diverse RL for Solution Discovery

  • Unlike diverse skill-learning methods without task-specific rewards or methods that seek diversity while preserving task performance, DISCODEMO uses trajectory-level success gating to broaden successful trajectories for behavior cloning.Source: §2 · p. 3

3 METHOD

DISCODEMO uses a privileged RL generator to produce diverse, successful demonstrations, filters undesirable interactions, and trains a downstream visuomotor policy by imitation learning. Its training combines a reverse curriculum with latent-conditioned diversity gated on task success.

3.1 PROBLEM SETUP

  • The task setup assumes a simulator with privileged state and specified-state resets, a downstream policy that observes images and proprioception, and a small set of human demonstrations.The demonstrations provide offline RL experience and candidate curriculum reset states.Source: §3.1 · p. 3

3.2 TRAINING A TASK-COMPETENT AND DIVERSE GENERATOR

  • Parallel reverse curricula start near the ends of successful demonstrations and move reset boundaries earlier as success rates reach a threshold, eventually switching to the original task distribution.Demonstration-specific curricula advance concurrently across batched rollouts.Source: §3.2 · p. 4
  • A latent-conditioned generator learns a state representation and rewards state-space displacements aligned with sampled skills to encourage distinct behaviors.Source: §3.2 · p. 4
  • Trajectory success gates the diversity reward, because ungated diversity can favor failure modes and can dominate the sparse task reward.The method also applies a safety penalty for unintended robot–scene contact and, in insertion tasks, excessive force; this softly biases solutions toward physically plausible interactions.Source: §3.2 · p. 4

3.3 GENERATING SUCCESSFUL AND DIVERSE DEMONSTRATIONS

  • The trained generator is frozen, sampled with varied latent skills, and used to collect different task-solving behaviors from the original task distribution.Failed trajectories and successful trajectories that violate safety criteria are discarded before retained observations and actions form the static dataset.Source: §3.3 · pp. 4–5

3.4 DOWNSTREAM IMITATION LEARNING

  • The downstream visuomotor policy uses visual observations and proprioception, without privileged state or latent skills, and is trained by imitation learning solely on the fixed exported dataset.It receives no further interaction with the environment or generator during training.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments test downstream imitation, dataset coverage, diversity–success trade-offs, generation efficiency and safety, and whether diverse data complements corrective collection. Across four tasks, DiscoDemo achieves the highest success rates in simulation and the real world while maintaining competitive completion times.

4.1 EXPERIMENTAL SETUP

  • The setup evaluates four manipulation tasks and trains the same DROID-pretrained π0.5 policy on each data source.Generated-data methods provide 3,000 successful demonstrations; evaluation uses simulation and real-robot trials.Source: §4.1 · p. 5

4.2 DOWNSTREAM IMITATION PERFORMANCE

  • DiscoDemo achieves the highest downstream success rates across tasks in simulation and the real world, with competitive completion times.The evaluation compares the same downstream visuomotor learner across data sources.Source: §4.1 · p. 5, Figure 3 · p. 6, §4.2 · p. 7

4.3 ANALYSIS OF GENERATED DEMONSTRATIONS

  • DISCODEMO produces distinct successful trajectories across all four tasks, while several baselines repeat narrow modes or vary source behaviors.The comparisons use demonstrations generated from the same initial placement, separating solution variation from initial-condition variation.Source: §4.3 · p. 7, Figure 4 · p. 7
  • DISCODEMO achieves the highest average dataset diversity and coverage, while maintaining high generation yield and short, goal-directed episodes.Dataset-level coverage is more closely associated with downstream success than diversity measured from one fixed initial scene.Source: §4.3 · pp. 7–8, Figure 5 · p. 7
  • Increasing diversity strength α raises solution diversity, but downstream success peaks at an intermediate value across tasks.The reported per-task peak ranges from α = 0.1 to α = 1.Source: Figure 6 · p. 8, §4.3 · p. 8

4.4 GENERATOR TRAINING EFFICIENCY AND SAFETY CURATION

  • P-RFCL and DISCODEMO train faster than PLD and ExpertGen, while the diversity objective adds task-dependent optimization overhead.The comparison estimates P-RFCL and DISCODEMO times and records PLD and ExpertGen times.Source: Figure 7 · p. 8, §4.4 · p. 8
  • Safety filtering reduces unsafe interactions more than shaping alone, and combining both yields the lowest illegal contact on both tasks.Shaping raises successful-rollout filter pass rates from 31% to 50% on PnP-Banana and from 24% to 46% on FMB-SqCircle.Source: §4.4 · p. 9, Table 1 · p. 9

4.5 COMPLEMENTARITY WITH CORRECTIVE DATA COLLECTION

  • Corrective collection further improves DISCODEMO, which remains stronger than P-RFCL under the same corrections.The paper frames static solution diversity and correction around off-nominal learner states as complementary forms of coverage.Source: §4.5 · p. 9, Figure 8 · p. 9

5 CONCLUSION

DISCODEMO uses reinforcement learning to generate successful, diverse demonstrations for downstream visuomotor imitation learning, improving a pretrained flow-matching VLA across four tabletop tasks. Its applicability is bounded by simulation and task assumptions, and the appendix documents reproduction details and additional results.

5 CONCLUSION

  • Across four tabletop tasks, DISCODEMO datasets improve a pretrained flow-matching VLA in simulation and on the real robot.The conclusion also reports that broader dataset coverage is associated with stronger downstream performance, while excessive diversity can compromise task competence.Source: §5 · p. 9
  • DISCODEMO relies on accurate simulation, privileged state and reset access, and a well-defined success signal; evaluation covers four simple-scene tabletop tasks with a modest sim-to-real gap.The authors note that more complex settings may require real-world adaptation; safety criteria are task-specific, and diversity does not directly optimize downstream utility.Source: §5 · p. 9
  • The authors report using generative AI tools to polish and draft manuscript text, search related work, and assist with code, with all outputs reviewed by the authors.The authors take responsibility for the final content.Source: AI USE STATEMENT · p. 10

APPENDIX

  • The appendix provides implementation details and additional results, covering task setup, generator and baseline implementations, downstream evaluation, analysis metrics, and full results.Simulated downstream results average three fine-tuning seeds with 50 evaluation episodes each; real-robot results use 20 trials per task and method.Source: REPRODUCIBILITY STATEMENT · p. 10, APPENDIX · p. 13

A TASKS AND EXPERIMENTAL SETUP

The study evaluates manipulation tasks on a Franka Research 3 in reconstructed simulation and on the real robot. It specifies the generator’s privileged inputs, the downstream policy’s observations, task success conditions, demonstrations, and real-robot evaluation protocol.

A TASKS AND EXPERIMENTAL SETUP

  • The generator uses a 45-dimensional privileged state and an 8-dimensional action, while the downstream policy observes exocentric and wrist cameras plus joint and gripper state.Generator actions comprise seven rate-limited relative joint-position increments and a binary gripper command.Source: A TASKS AND EXPERIMENTAL SETUP · p. 14
  • Tasks use a Franka Research 3 arm reconstructed in Isaac Sim; simulation runs at 120 Hz and the policy acts at 20 Hz.The real and simulated setups are shown from the same exocentric camera.Source: A TASKS AND EXPERIMENTAL SETUP · p. 14, Figure 9 · p. 14
  • Generator training rewards and terminates an episode only after the task success condition holds for 20 consecutive control steps, encouraging stable end states.The task table lists each task’s horizon, initial-state distribution, and success condition.Source: A TASKS AND EXPERIMENTAL SETUP · p. 14, Table 2 · p. 14
  • Each task uses 20 teleoperated demonstrations; successful simulation replays form the set for the reverse curriculum and H20, MimicGen, and PGDG.For insertion tasks, the peg starts in the gripper, which is held closed.Source: A TASKS AND EXPERIMENTAL SETUP · pp. 14–15
  • Real-robot evaluation uses 20 trials per task on matched initial placements and zero-shot transfer, with no real-robot fine-tuning.For each method, the first training-seed policy is deployed; success rate is the fraction of trials succeeding, and time is mean completion time over successful trials.Source: A TASKS AND EXPERIMENTAL SETUP · p. 15

B.1 TRAINING SETUP

The generator is trained with parallel simulation, a demonstration-based reverse curriculum, latent-conditioned diversity rewards, and safety shaping and filtering. The diversity reward is gated on successful episodes, while exported trajectories must meet contact and, for insertion, jamming criteria.

B.1 TRAINING SETUP

  • P-RFCL and DISCODEMO use FlashSAC for 50M environment steps in 2,048 parallel environments; PLD and ExpertGen train for up to 24 hours each.The actor and critic use FlashSAC’s residual MLP architecture, and each batch draws half its transitions from demonstrations and half from online experience.Source: §B.1 · p. 15

B.2 PARALLEL REVERSE CURRICULUM

  • The parallel reverse curriculum resets to demonstration states, advances each frontier after at least 10 frontier-start episodes achieve a 0.9 success rate, and switches to task-distribution starts when half the demonstrations are solved.Start steps are sampled from each demonstration’s frontier to its end, with roughly M/N environments serving each demonstration.Source: §B.2 · p. 16

B.3 DIVERSITY OBJECTIVE

  • A latent skill is sampled per episode, and its aligned state-space displacement reward is added only for completed successful episodes; the main comparison uses α = 0.3 while P-RFCL uses α = 0.The reward combines task reward, a safety penalty, and α times the success-gated diversity reward; diversity is active in both curriculum phases.Source: §B.3 · p. 16

B.4 SAFETY SHAPING AND FILTERING

  • Safety shaping penalizes unintended robot-scene contacts and excessive insertion force, while export filtering rejects trajectories that exceed contact thresholds or, for insertion tasks, the jamming threshold.The contact cost covers pairs outside allowed grasp contacts; the jamming cost penalizes peg force beyond axial and lateral deadbands.Source: §B.4 · pp. 16–17

B.5 RFCL AND OUR PARALLEL REIMPLEMENTATION

The authors parallelize RFCL’s demonstration-reset curriculum by pooling outcomes across environments, reporting a substantial wall-clock speedup in an earlier task version. The appendix also specifies how comparison baselines generate or collect demonstrations.

B.5 RFCL AND OUR PARALLEL REIMPLEMENTATION

  • Unlike original RFCL’s single policy and pinned per-demonstration environments, the reimplementation pools frontier-start outcomes across environments to advance the curriculum.Because of the changes, the authors caution that their α = 0 results should not be compared directly with Tao et al. (2024).Source: §B.5 · p. 17
  • 29 minutes versus 8.2 hours, about 17× faster in wall-clock time, for the pooled parallel curriculum on an earlier PnP-Banana version.The speedup is in wall-clock time, not samples: the parallel curriculum uses more environment steps, which parallel simulation makes cheap.Source: §B.5 · pp. 17–18

C BASELINES

  • Generated-data baselines produce 3,000 successful demonstrations under DISCODEMO’s initial-state distribution, success condition, horizon, and camera configuration.H20 instead uses the 20 teleoperated demonstrations directly.Source: C BASELINES · p. 18
  • Motion planning connects inverse-kinematics waypoint targets with jerk-limited trajectories, while MimicGen retargets object-centric demonstration segments to new scenes.The planner does not reason about collisions; MimicGen selects segments from the three demonstrations with object layouts closest to the new scene.Source: C BASELINES · p. 18
  • PGDG reimplements its perturbation stage on MimicGen data and retains successful variants, so its same-sized dataset covers fewer distinct initial scenes than MimicGen’s.No official PGDG implementation was available to the authors.Source: C BASELINES · p. 18
  • PLD trains a privileged-state residual against a frozen vision-based π0.5 policy, while ExpertGen learns to steer that prior’s initial noise; DAgger and DART provide corrective-data comparisons.The DART teacher uses perturbed actions but supplies noise-free action labels; DAgger uses the frozen P-RFCL or DISCODEMO generator as teacher.Source: C BASELINES · pp. 18–19

D DOWNSTREAM POLICY TRAINING AND EVALUATION

The section specifies how each dataset is exported and used to fine-tune the same pretrained π0.5 policy, then defines the shared simulation evaluation protocol.

D DOWNSTREAM POLICY TRAINING AND EVALUATION

  • Exported observations contain two 480 × 640 camera images, seven joint positions, and gripper state; actions are processed joint-position targets and binary gripper commands.DISCODEMO samples a latent skill per episode and holds it fixed throughout the rollout.Source: D DOWNSTREAM POLICY TRAINING AND… · p. 19
  • Each dataset is used to fine-tune the same DROID-pretrained π0.5 policy, with three training seeds per dataset.Training uses a fixed instruction per task and a shared recipe.Source: D DOWNSTREAM POLICY TRAINING AND… · p. 19, Table 5 · p. 19
  • Policies are tested for 50 episodes from the training distribution, with success defined as meeting the condition at any step.Success rate, time to first success, and successes per hour are computed per seed and reported as mean ± standard deviation across three seeds; failures incur the full horizon in the time and throughput measures.Source: D DOWNSTREAM POLICY TRAINING AND… · p. 19

E.1 DIVERSITY AND COVERAGE METRICS

The section specifies trajectory features and embeddings for diversity and coverage evaluation, then reports that dataset coverage tracks downstream success more closely than fixed-scene diversity.

E.1 DIVERSITY AND COVERAGE METRICS

  • Trajectories are represented by object and end-effector poses, resampled to 64 steps, and embedded using eight non-DC DCT-II coefficients per feature dimension.Action features are omitted because their signals differ across generators and could dominate comparison without reflecting distinct solutions.Source: §E.1 · p. 20
  • The Vendi score summarizes kernel-matrix diversity, ranging from 1 for identical trajectories to m, with subset size fixed because the score grows with m.It is used instead of log det(K + ϵI), which can be driven by near-zero eigenvalues and log ϵ for near-duplicate trajectories.Source: §E.1 · p. 20
  • Diversity uses a fixed initial scene and 24 successful rollouts per source, while coverage samples 1,000 trajectories from each source’s training dataset.Diversity averages 200 subsets of 20 trajectories; coverage averages 100 subsets of 200 trajectories, and both metrics are averaged across four tasks.Source: §E.1 · p. 20
  • Coverage rank correlation with downstream success is higher or equal to fixed-scene diversity on every task, supporting coverage as a useful performance indicator.Task-average Spearman correlations are 0.50 for coverage and 0.39 for diversity; with seven generators per task, individual coefficients remain noisy.Source: §E.1 · p. 20

E.2 GENERATION YIELD AND QUALITATIVE TRAJECTORIES

  • Generation yield is the fraction of rollouts from the task’s initial-state distribution that succeed before export filtering.For PGDG, yield combines MimicGen yield and perturbed-candidate success; for PLD and ExpertGen, it is success at the end of RL training.Source: §E.2 · p. 20

E.3 SAFETY METRICS

The section defines safety measures for exported demonstrations and summarizes per-task diversity, coverage, trade-off, results-table, and efficiency analyses.

E.3 SAFETY METRICS

  • Illegal contact measures unintended robot–scene contact forces, while container displacement measures movement of the bowl or fixture board in the specified tasks.Finger–banana contact and finger–peg contact are allowed in their respective tasks; illegal-contact evaluation does not clip large forces.Source: §E.3 · pp. 20–22
  • DISCODEMO has the highest diversity and highest or tied-highest coverage on three of four tasks, while PGDG exceeds it on both metrics on StackCube.Figure 10 reports the per-task dataset analysis.Source: Figure 10 · p. 21
  • For FMB-SqCircle without shaping, only 2,701 successes pass the export filter, so its Tab. 1 row uses those trajectories; unfiltered rows use a fixed random subset of 3,000 successes.Source: §E.3 · p. 22

F.1 MAIN RESULTS — FULL TABLES

F.2 DIVERSITY TRADE-OFF ON EVERY TASK

  • Across four tasks, the diversity–success trade-off has a common shape, but the α where generator success begins to give way varies.Figure 12 reports generator success, diversity, and downstream success per task and on average, averaging five seeds per cell.Source: §F.2 · p. 22

F.3 QUALITATIVE TRAJECTORIES ON STACKCUBE AND FMB-ROUND

  • The qualitative trajectory comparison covers StackCube and FMB-Round, with the remaining task illustrations described as following the same format.Figure 4 shows PnP-Banana and FMB-SqCircle; the passage identifies the remaining tasks as below.Source: §F.3 · p. 22

F.4 EFFICIENCY MEASUREMENT

  • Training-time estimates for P-RFCL and DISCODEMO extrapolate measured curriculum-phase steps to 50M steps, while end-to-end GPU-hours include generator training and generating and rendering 3,000 demonstrations.The GPU-hour comparison excludes π0.5 fine-tuning because it is identical across data sources.Source: §F.4 · p. 22

F.5 QUALITATIVE ROLLOUTS

The rollout figures show data generators and downstream policies over time, with shared initial conditions for comparisons and explicit success or failure endpoints.

F.5 QUALITATIVE ROLLOUTS

  • Each rollout figure uses one row per data source and columns for successive time steps, ending in a completion frame or a marked failure.The displayed interval is shared across rows and figure types; it spans 28 seconds for PnP-Banana and StackCube and 17.5 seconds for insertion tasks.Source: §F.5 · p. 23
  • Generator rollouts share a fixed initial scene per task, while downstream simulation policies are shown from a shared initial state selected from ten tested states.On StackCube, MP and PLD have no successful rollout from the displayed scene, so failed rollouts are shown; PLD and ExpertGen are omitted from the downstream simulation figures.Source: §F.5 · p. 23
  • Real-robot downstream-policy figures include PLD and ExpertGen and show one of three reproduced initial scenes per task.The displayed scene is the one used for completion-speed comparison in the project-page video; unsuccessful runs are marked as failures.Source: §F.5 · p. 23

Related papers

Loading 2610.32093v1…