ScheduleGen: Demonstration Generation via Skill Scheduling for Multi-Arm Imitation Learning

1. Introduction

• Multi-arm embodiments support coordinated tasks and larger workspaces, with the potential to complete tasks more efficiently through simultaneous operation.Source: §1 · p. 2
• Large-scale policy training relies on costly, months-long teleoperation, with special interfaces and greater operator burden exacerbating multi-arm data collection.Teleoperation interfaces can also make desirable coordination characteristics difficult to capture.Source: §1 · p. 2
• Existing generators either struggle with contact-rich tasks and coordination diversity or inherit seed-demonstration coordination and lack planning for collision-heavy scenes.DexMimicGen's generated coordination is dictated by its source demonstrations.Source: §1 · p. 2
• ScheduleGen adapts object-relative contact skills, replans collision-free transitions, and jointly schedules arms under task and collision constraints.The paper evaluates its applicability across long-horizon, contact-rich tasks, challenging resets, cluttered workspaces, and multiple embodiments and coordination strategies.Source: §1 · p. 2
• ScheduleGen policies outperform prior work DexMimicGen on comparable tasks and achieve 50–80% success across eight real-world variants.The paper evaluates ScheduleGen on 28 task variants across three simulation frameworks.Source: §1 · p. 2

2. Related Work

• Teleoperation and large teams have scaled imitation-learning datasets, but data sourcing remains a bottleneck, especially for bimanual robots and humanoids requiring specialized interfaces.Source: §2 · pp. 2–3
• Figure 2 depicts skill adaptation and motion planning, collision checking, joint scheduling with waits, and replay of successful rollouts as demonstrations.Source: Figure 2 · p. 3
• Automated generators span TAMP, demonstration adaptation, and motion-planning methods, but prior approaches have limits in contact-rich, multi-arm, or collision-constrained settings.TAMP-based methods target single-robot prehensile manipulation; SkillGen is single-arm, while independently planned arms can compromise safety and success.Source: §2 · p. 3
• ScheduleGen treats demonstrations as schedulable skills and uses collision-aware scheduling to plan safe, efficient multi-arm manipulation.Prior multi-robot planners factor problems into single-robot subproblems and reconcile pairwise constraints through action-timing backtracking.Source: §2 · p. 3

3. Problem Description

• The goal is an automated planning-and-scheduling system that generates demonstration datasets for multi-robot visuomotor policies trained by imitation learning.Source: §3 · p. 3

3.1. Prerequisites

• Multi-robot manipulation is modeled as a POMDP, with robot configurations and end-effector poses in state and joint targets among the supported actions.The system also supports relative and absolute end-effector pose targets.Source: §3.1 · p. 4

3.2. Skill Adaptation

• A skill assigns a robot and reference object frame to state and action sequences; skills can come from segmented demonstrations, scripted waypoints, or agentic authoring.Adaptation assumes approximate pose invariance and transforms end-effector states and actions relative to the reference object, without changing finger states or actions.Source: §3.2 · p. 4

3.3. Temporal Constraints

• Temporal precedence constraints order specified steps across skills, allowing sequential execution, parallel execution, or coordination through matched start and end steps.The constraints capture dependencies such as completing a pour before another arm grasps the receiving bowl, or synchronizing a coordinated lift.Source: §3.3 · pp. 4–5

4. ScheduleGen

• ScheduleGen repeatedly samples an initial state and source demonstration, plans and schedules robot trajectories, and retains executions that reach the goal state.The loop continues until the target number of successful demonstrations is collected.Source: §4 · p. 5

4.1. Skill Planning

• Skill planning orders skills by precedence, adapts object-relative demonstrations, and connects each skill to its start configuration with motion planning.Failed inverse kinematics or motion planning rejects the candidate for resampling.Source: §4.1 · p. 6

4.2. Skill Scheduling

• Scheduling minimizes makespan while enforcing task timing and collision constraints, with binary decisions selecting the safe ordering for colliding configurations.The formulation also enforces minimum waypoint time steps; acceleration is not explicitly constrained.Source: §4.2 · pp. 6–7

4.3. GPU Acceleration

• GPU-accelerated pairwise collision checking evaluates robot-trajectory configuration pairs, using spherical robot and grasped-object geometry.The method uses cuRobo’s custom self-collision CUDA kernel for sphere-sphere checks.Source: §4.3 · p. 7

5.1. Experimental Setup

• With five annotated demonstrations per task, experiments span Isaac Lab, RoboDojo, robosuite, and multiple robot embodiments, using held-out resets for policy evaluation.The setup compares against DexMimicGen and a tuned variant; RoboDojo also uses Nominal and Perturbed conditions.Source: §5.1 · pp. 7–8

5.2. ScheduleGen Features

• ScheduleGen generates data across 17 Isaac Lab variants, seven RoboDojo tasks, and four robosuite variants, with policies learning successfully across the reported settings.Four-arm cube stacking covers eight settings, with data-generation success of 63.4–95.1% and policy success of 30–90%.Source: §5.2 · p. 8
• All four scheduling-cost modes generate 500 successful demonstrations on Insert Tubes, with 40.0–50.0% policy success over 60 held-out Nominal episodes per mode.Mean steps over successful rollouts range from 237.8 to 240.3.Source: §5.2 · p. 8

5.3. ScheduleGen Analysis

• ScheduleGen policies improve success on eight of nine variants supported by DexMimicGen, matching its best baseline at 94% on Normal Two-Arm Cube Stacking.Policy success rises from 42–78% to 76–100% on Bowl and Dishware Stacking, 0–74% to 70–86% on Cross-Arm and Cross-Bin cube variants, and 20–90% to 48–96% on Humanoid Nut Pouring.Source: Figure 3 · p. 9, §5.3 · p. 12
• On RoboDojo, generated policies average 47.9% success on Nominal layouts and 25.0% under doubled pose perturbations; fixed-path scheduling finds valid schedules for 98.3% of released trajectories and reduces total steps by 14.98%.Across generated RoboDojo training trajectories, mean execution length falls by an average of 7.5% relative to released demonstrations.Source: §5.2 · p. 11, §5.3 · p. 12
• Reusing five Normal demonstrations for Cross-Arm Bowl and Dishware yields 66.7% and 68.0% data-generation success and 80% and 76% policy success, outperforming task-matched DexMimicGen data.ScheduleGen replans and schedules free-space motion while retaining contact skills.Source: §5.3 · p. 11

5.4. Real-World Evaluation

• On real YAM tasks, simulation-only policies reach 75–80% success on Bowl and Dishware Stacking, while real-seeded simulation policies reach 65% on Cube Stacking and 50–75% across Trash variants.All policies are evaluated over 20 real-world episodes without real-world fine-tuning.Source: Table 3 · p. 13, §5.4 · pp. 13–14

6. Conclusion

• Across 28 simulation variants, learned policies outperform prior bimanual methods on most comparable tasks and transfer to eight real-world variants without real-world fine-tuning.The system combines object-centric skill adaptation, motion planning, and joint scheduling while preserving contact-rich behavior and satisfying task-order, synchronization, and collision constraints.Source: §6 · p. 14

A. Planning and Scheduling Details

• ScheduleGen’s generation loop and skill planner connect planning, scheduling, and demonstration adaptation for multi-arm systems.The appendix describes how arm paths are formed and collision constraints are expanded.Source: A. Planning and Scheduling Details · p. 17

A.1. Path Construction

• Skills are processed in precedence order to construct the planning scene, while scheduled arms can still execute in parallel.Adapted states define configurations for planning and collision checking; adapted actions provide replay targets, with demonstrated finger states and actions retained.Source: §A.1 · p. 17

A.2. Collision Constraints and Execution

• For each colliding waypoint pair, a binary variable selects which arm passes through the conflict first; feasible schedules preserve waypoint order and task precedence and synchronization while changing timing.A feasible schedule is replayed in simulation to check contact behavior and task completion before acceptance.Source: §A.2 · p. 17

B. Scheduling Costs

• Scheduling minimizes makespan together with weighted arm end times, while the cost ablation compares secondary costs that change timing without relaxing task, collision, or per-arm ordering constraints.In the ablation, secondary objectives are optimized among schedules with the same minimum makespan, unlike the weighted objective that balances makespan against arm end times.Source: B. Scheduling Costs · p. 17, §B.1 · pp. 17–18

B.2. Secondary Cost Definitions

• Continuity penalizes the longest internal pause, while minimum active span accounts for accumulated gaps between movements and permits launch delays or terminal holds.The speed cost maximizes the slowest moving-edge speed but excludes stationary edges, so it can coexist with long pauses.Source: §B.2 · p. 18

• The appendix examines source demonstration reuse, how released demonstrations become skill seeds, and how scheduling changes coordination and execution length.Source: C. Additional Results and Evaluation · p. 19

C.1. Evaluation Metrics

• Data-generation success measures attempts yielding successful demonstrations after replay, while policy success uses held-out episodes and completion steps use successful rollouts only.Source: §C.1 · p. 19
• On CubeClose D1, no DexMimicGen policy was trained because generation yielded no successful demonstrations; on D0, ScheduleGen has lower generation success but higher policy success.The two rates measure different stages and need not rank methods identically.Source: §C.1 · p. 19

C.2. Source Demonstration Reuse

• Task-matched Cross-Arm demonstrations alone do not recover ScheduleGen’s performance, because changed object layouts can make demonstrated connecting motions and timing unsuitable.ScheduleGen reuses five Normal demonstrations and plans connecting motions and arm coordination for each sampled scene, reducing the need to recollect demonstrations for a new layout.Source: §C.2 · p. 19, Figure 7 · p. 19

C.3. RoboDojo Demonstration Transfer

• RoboDojo demonstrations are calibrated into simulation and checked by physics replay; generated demonstrations must pass replay before entering the training set.The transfer procedure includes scene calibration and skill validation through replay, with five source demonstrations used for each RoboDojo task.Source: §C.3 · p. 20, Figure 8 · p. 20

C.4. Effects of Scheduling Costs

• 17.00 to 15.40 s: continuity and minimum active span reduce the summed active spans while all modes finish their paths in 10.44 s.A shared 25-tick final hold brings each successful replay to 11.44 s.Source: §C.4 · p. 20
• Max-min velocity permits a longer pause, showing that faster moving edges do not necessarily produce continuous motion.An arm’s active span includes internal pauses, and panel (b) sums the spans over both arms.Source: §C.4 · p. 20
• Figure 9 compares longest internal pause and summed active spans, both measured in seconds with lower costs preferred.Summed spans can exceed completion time because the arms move in parallel; pauses and spans use a 10^-3-rad joint-motion threshold.Source: Figure 9 · p. 20, §C.4 · p. 20

C.5. Retiming Released RoboDojo Demonstrations

• The experiment fixes per-arm paths and task order, allowing independent motions to overlap when task and collision constraints permit while dependent motions wait.No new geometric paths are planned; this isolates scheduling on existing paths.Source: §C.5 · p. 21
• 688 of 700 trajectories (98.3%) receive feasible timings, with feasibility defined by Eq. 1 rather than physics-replay task completion.Table 4 compares each successfully scheduled trajectory with its original released version, so both step counts refer to the same paths.Source: §C.5 · p. 21, Table 4 · p. 21
• Retiming reduces mean step counts for every task relative to source coordination, which may already include concurrent arm motion.The comparison is not against fully serial execution.Source: §C.5 · p. 21
• 14.98% fewer steps in total and a 16.44% mean reduction per trajectory summarize paired comparisons on the same 688 trajectories.The pooled reduction weights trajectories by source length, while the mean per-trajectory reduction weights each demonstration equally; failed schedules are excluded from length comparisons.Source: §C.5 · p. 21

D. Policy Training

• YAM policies use three RGB views and a 14-dimensional absolute joint-and-gripper state to predict 14-dimensional absolute joint-and-gripper targets.The views are left wrist, top, and right wrist.Source: D. Policy Training · p. 22
• Table 5 lists the YAM visuomotor policy hyperparameters.The supplied table caption identifies its scope but does not provide the hyperparameter values.Source: Table 5 · p. 22
• State-based BC uses a two-layer LSTM with hidden size 1,000 and sequence length 10, paired with either a five-mode GMM or deterministic MSE head.Both use Adam, batch size 100, and a nominal 1,500-epoch budget.Source: D. Policy Training · p. 22
• Image BC uses a recurrent core with a ResNet-18 and SpatialSoftmax encoder per view, with nominal training budgets of 600 epochs.GMM policies use learning rate 10^-4; MSE policies use 2 × 10^-4, and checkpoints are selected using held-out rollout evaluation.Source: D. Policy Training · p. 22