PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaForge: Planning-Guided Residual Learning for Dynamic Manipulation Demonstration Generation

Yiyang Jin, Yu Zheng, Xiao He, Hesheng Wang

arXiv:2609.25631v1cs.RO

TL;DR

Dynamic manipulation demonstrations remain scarce, and existing generation pipelines largely focus on static or quasi-static settings. DynaForge combines phase-structured planning with learned residual corrections and an implicit curriculum. Across nine simulation tasks, it raises mean demonstration-generation success over the planning prior, while its current scope remains constrained by authored task structure and simulator fidelity.

  • Problem

    Dynamic manipulation demonstrations remain scarce because datasets and automated generation pipelines focus primarily on static or quasi-static tasks.

    Source: §I · p. 1

  • Method

    DynaForge combines low-frequency planning and high-frequency local interaction updates with residual corrections and matched-condition group-relative learning that selects mixed-success groups.

    Source: §I · pp. 1–2, D. Implicit Curriculum Group-Relative… · p. 4

  • Results

    Across nine tasks, mean DGR rises from 41.30±1.00% for the planning prior to 78.37±1.41% with DynaForge.

    Source: B. Dynamic Demonstration Generation (Q1) · p. 5

  • Takeaways & Limitations

    Experiments show higher demonstration yield across nine simulation tasks and improved downstream DP3 performance, with initial transfer to three physical skills.

    Source: §VI · p. 7

    Generation depends on authored task structure and simulator fidelity; low precision-insertion success and modest physical performance limit the present scope.

    Source: §VI · p. 7

Abstract

from arXiv · show

Dynamic object manipulation is essential for robots operating in real-world environments, yet methods for generating high-quality demonstrations remain limited. Methods designed for static tasks do not readily transfer to dynamic settings. Among dynamic demonstration generators, planning-based methods can fail near contact, while DOMINO-style replay simplifies dynamic interactions and may limit the experience available for policy learning. We present DynaForge, a planning-guided framework that learns residual corrections for dynamic manipulation demonstration generation. DynaForge combines low-frequency global planning with high-frequency object-centric inverse kinematics across task phases, and applies a residual policy to correct actions during dynamic interaction. An implicit curriculum groups rollouts under matched conditions and selects mixed-success groups, focusing residual reinforcement learning on the evolving competence frontier. On Can and Bottle, it uses 0.73x as many optimizer steps as vanilla GRPO at the same nominal environment-step budget, with higher observed final success rates. Across nine simulation tasks, DynaForge increases mean demonstration-generation success from 41.30% of the planning prior to 78.37%. With 800 demonstrations per task, DP3 policies trained on DynaForge data achieve 49.11% mean success, compared with 7.07% for DOMINO data. On three real-world dynamic tasks, DynaForge-trained policies achieve 30-60% success, compared with 0-10% for DOMINO-trained policies, showing the ability of DynaForge for sim-to-real transfer.

I. INTRODUCTION

Dynamic manipulation demonstrations remain scarce, while existing generation pipelines largely target static settings and often lack execution-level correction near contact. DynaForge uses planning as a structured prior and learns residual corrections, with an implicit curriculum that focuses updates on matched conditions near the evolving success-failure boundary.

I. INTRODUCTION

  • Dynamic manipulation demonstrations remain scarce because existing datasets and pipelines focus mainly on static or quasi-static tasks, limiting data-driven methods for dynamic manipulation.Short contact windows make teleoperation costly and difficult, while video retargeting faces embodiment gaps and contact discrepancies.Source: §I · p. 1
  • Fixed-rule planning and retargeting pipelines update targets or trajectories but do not learn execution-level corrections from rollout outcomes, a limitation that matters at contact boundaries.The rolling-peach example contrasts planner-only contact loss with DynaForge's residual correction at the contact boundary.Source: §I · p. 1, Figure 1 · p. 1
  • DynaForge treats motion planning as a structured prior: planning constructs global motions under collision and kinematic constraints, while a residual policy learns execution corrections from rollout outcomes.This assigns global motion construction to planning and execution-level correction to learning.Source: §I · p. 1
  • DynaForge combines low-frequency planning in stable phases with high-frequency object-centric IK near dynamic contact boundaries.A residual policy corrects nominal planner actions; the approach groups rollouts under matched perturbations and excludes uniformly successful or failed groups from updates.Source: §I · pp. 1–2
  • The implicit curriculum focuses residual learning on matched-condition rollout groups with mixed outcomes, without requiring an explicit difficulty schedule.The method excludes groups whose rollouts all succeed or all fail.Source: §I · p. 2

II. RELATED WORK

Prior dynamic demonstration-generation methods use state machines, target updates, or replay, while dynamic manipulation research also advances perception, prediction, and execution. DynaForge builds on residual learning over priors to learn execution-level corrections near contact.

A. Scalable Demonstration Generation

  • Dynamic demonstration-generation methods include state machines, online target updates, replay, and augmentation, but generally lack learned execution-level corrections near contact.DOMINO synchronizes robot-motion replay with predefined object trajectories, limiting reactive tracking of moving targets.Source: A. Scalable Demonstration Generation · p. 2

B. Dynamic Object Manipulation

  • Dynamic manipulation methods improve representations, future prediction, or real-time execution, while data acquisition still depends heavily on task-specific experts or predefined motion patterns.The passage identifies a remaining need for autonomous adaptation to diverse dynamic perturbations.Source: B. Dynamic Object Manipulation · p. 2

C. Residual Learning over Priors

  • Residual learning over priors focuses exploration on execution errors not covered by the prior, and group-relative methods compare rollouts under the same condition.DynaForge applies this principle to learn execution-level residual corrections over a planner prior and construct an implicit curriculum on GRPO.Source: C. Residual Learning over Priors · p. 2

III. PRELIMINARIES

DynaForge formulates conditional demonstration generation as an episodic process under scene initialization and exogenous object dynamics. Successful rollouts are retained as demonstrations, with object-centric geometry specifying the end-effector target relative to a moving reference object.

III. PRELIMINARIES

  • Each task follows a phase sequence, and the dynamic condition ξ specifies scene initialization and exogenous object dynamics in a condition-dependent episodic MDP.The simulator state and action spaces, transition function, and sparse task reward define the MDP components.Source: §III · pp. 2–3
  • Only successful rollouts are saved as demonstration data for imitation learning.The overview also identifies simulation evaluation and sim-to-real deployment as downstream uses.Source: §III · p. 3, Figure 2 · p. 3
  • Object-centric geometry expresses the desired end-effector pose in the world frame using a target pose in the object frame and the moving object's world-frame pose.The object pose varies with object movement.Source: §III · p. 3

IV. DYNAFORGE

DynaForge combines phase-structured planning with residual correction during dynamic interaction and retains successful rollouts as demonstrations. Its hybrid planner uses global trajectories in stable phases and online object-centric IK in moving or contact-sensitive phases.

A. Framework Overview

  • DynaForge uses a phase-structured planner for nominal actions, a phase-gated residual policy for dynamic interaction, and successful rollouts for downstream imitation learning.Implicit curriculum group-relative learning focuses optimization near the evolving competence frontier.Source: A. Framework Overview · p. 3

B. Phase-Structured Hybrid Planning

  • Stable free-space phases use a collision-aware global trajectory planned at phase entry, while moving or contact-sensitive phases update the target from the latest object pose and solve IK each control step.The phase program assigns each phase an object-centric target, success predicate, control mode, and residual gate.Source: B. Phase-Structured Hybrid Planning · p. 3

C. Phase-Gated Residual Policy

DynaForge uses a residual policy to correct planner actions during contact-sensitive phases. Training samples residual actions, adds them to the planning action, and gates their use by phase.

C. Phase-Gated Residual Policy

  • The residual policy uses the planning action and a short history of privileged robot and object state as inputs.The history is denoted h_t in the passage.Source: C. Phase-Gated Residual Policy · p. 3
  • Residual actions are sampled from a behavior policy during training and added to the planning action to produce the executed action.The action is constrained by the action limits, and a phase gate enables corrections only when dynamic response is needed.Source: C. Phase-Gated Residual Policy · p. 4
  • At evaluation and demonstration generation, DynaForge removes exploration from the residual action.Source: C. Phase-Gated Residual Policy · p. 4

D. Implicit Curriculum Group-Relative Learning

The implicit curriculum compares rollouts under matched dynamic conditions and updates the residual policy using groups with mixed success and failure. As competence improves, optimization shifts toward harder conditions without an explicit difficulty schedule.

D. Implicit Curriculum Group-Relative Learning

  • Each update samples a condition within each difficulty stratum and shares its initialization and perturbation parameters across K environments.Residual actions are sampled independently across the matched rollouts.Source: D. Implicit Curriculum Group-Relative… · p. 4
  • Returns are evaluated once per rollout using terminal success and the furthest validated task-progress checkpoint.The group-relative advantage uses the group mean return and within-group standard deviation, with a small positive constant.Source: D. Implicit Curriculum Group-Relative… · p. 4
  • Mixed-success groups are retained for optimization, while all-success and all-failure groups are filtered out.The update is skipped when no groups pass the frontier filter.Source: D. Implicit Curriculum Group-Relative… · p. 4
  • The competence frontier advances toward harder strata as group success improves, without a prescribed difficulty schedule.The figure marks likely homogeneous-group regions as filtered and shows the frontier moving from easy to hard strata.Source: Figure 3 · p. 4, D. Implicit Curriculum Group-Relative… · p. 4
  • The frontier-filtered GRPO loss clips policy likelihood ratios asymmetrically, using 0.20 for the lower bound and 0.28 for the upper bound.The wider upper clipping range is used for positive-advantage samples.Source: D. Implicit Curriculum Group-Relative… · p. 4

V. EXPERIMENTS

The experiments address demonstration yield and collection cost, downstream data utility, sim-to-real transfer, and the implicit curriculum’s training efficiency. The supplied experiment materials identify the benchmark and task scope but do not provide outcome values.

V. EXPERIMENTS

  • The experiments ask whether DynaForge demonstrations improve simulation policy success over DOMINO data at matched dataset size and training budget, and how well the policies transfer to three real-world tasks.Source: §V · p. 4
  • Table I is titled as a matched dynamic demonstration-generation benchmark.Its notes define DGR as mean ± standard deviation over three seeds, each with 100 rollouts; the supplied text gives no table values.Source: Table I · p. 5, §V · p. 5
  • The simulation benchmark depicts nine tasks spanning dynamic grasping, catching, clicking, placing, and insertion, with panels showing objectives and red arrows indicating motion patterns.Source: Figure 4 · p. 5

A. Experimental Setup

The evaluation uses nine simulated dynamic-manipulation tasks and measures demonstration yield and downstream success across repeated rollouts. DynaForge improves demonstration-generation rate over the planning prior, with preparation costs also compared under an explicit estimate model.

A. Experimental Setup

  • The benchmark uses nine tasks spanning dynamic grasping, catching, clicking, placing, and insertion, with motion from near-static conditions to approximately 1.5 m/s.Rolling objects evolve through simulated contact dynamics; driven targets retain task-specific motion constraints.Source: A. Experimental Setup · p. 5
  • Simulation comparisons use common task conditions and environment-defined success predicates; failures and timeouts count as unsuccessful attempts.Unless otherwise stated, results use three evaluation seeds with 100 rollouts each; the cost table covers five jointly supported tasks.Source: A. Experimental Setup · p. 5, Table II · p. 6

B. Dynamic Demonstration Generation (Q1)

  • DynaForge raises mean DGR from 41.30±1.00% to 78.37±1.41% across nine tasks, improving on the planning prior in every task.Gains range from 12.00 percentage points on Lemon to 57.67 percentage points on Bottle.Source: B. Dynamic Demonstration Generation (Q1) · p. 5
  • DynaForge's estimated machine total is 6.5 h, including 2.5 h of residual training, below baseline estimates under the stated cost model.The model normalizes collection time to a 4 h DynaForge reference and scales it by the sum of inverse task-wise DGRs, assuming equal cost per attempt.Source: B. Dynamic Demonstration Generation (Q1) · pp. 5–6

C. Downstream Policy Evaluation in Simulation (Q2)

The simulation evaluation compares DP3 policies trained on equal-sized DynaForge or DOMINO demonstration datasets under matched training procedures. DynaForge-trained policies achieve higher success across all nine tasks.

C. Downstream Policy Evaluation in Simulation (Q2)

  • The data-source comparison fixes dataset size to assess demonstration utility independently of generation yield.Both policy types receive point clouds and proprioception as input.Source: C. Downstream Policy Evaluation in… · p. 6
  • 49.11% versus 7.07% mean SR: DP3 policies trained on DynaForge demonstrations outperform DOMINO-trained policies across all nine tasks.Each policy uses 800 demonstrations per task and the same representations, preprocessing, 100,000 training updates, and evaluation procedures.Source: C. Downstream Policy Evaluation in… · p. 6
  • DOMINO uses synchronized replay, while DynaForge adjusts actions to track moving objects and applies learned execution corrections.The reported largest improvements occur on Alarm Clock and Toy car, while Peg remains challenging for both methods.Source: C. Downstream Policy Evaluation in… · p. 6

D. Sim-to-Real Transfer (Q3)

DynaForge-trained policies transfer to three physical dynamic-manipulation tasks, outperforming DOMINO-trained policies. In the implicit-curriculum ablation, mixed-success matched-condition training advances the competence frontier and improves final DGR with fewer optimizer steps and less training time than vanilla GRPO.

D. Sim-to-Real Transfer (Q3)

  • 30%, 60%, and 50% success on dynamic grasping, clicking, and placement, respectively, versus 0%, 0%, and 10% for DOMINO-trained policies.The policies were trained on simulation-generated demonstrations and evaluated on physical tasks.Source: D. Sim-to-Real Transfer (Q3) · p. 6

E. Implicit-Curriculum Ablation (Q4)

  • The ablation compares the DGR of the planner without a residual policy with residual generators trained using vanilla GRPO or the implicit curriculum.The curriculum admits only matched-condition groups containing both successes and failures to optimization.Source: E. Implicit-Curriculum Ablation (Q4) · p. 6
  • Success extends toward strata with lower overall success as training progresses, accompanied by an advancing competence frontier.The frontier is the deepest contiguous rank for which each stratum up to that rank has a 95% Wilson lower bound of at least 50%, smoothed over nine updates.Source: E. Implicit-Curriculum Ablation (Q4) · p. 7, Figure 7 · p. 7
  • On Toy car and Peach, training curves show faster and larger gains than vanilla GRPO.The comparison uses matched-condition groups containing both successes and failures for the implicit curriculum, whereas vanilla GRPO uses all complete rollout groups.Source: E. Implicit-Curriculum Ablation (Q4) · pp. 6–7
  • The implicit curriculum improves final DGR over vanilla GRPO by 1 and 5 percentage points on Can and Bottle, and 24 and 48 percentage points on Toy car and Peach.The gains are positive across all four tasks, although their magnitude varies.Source: E. Implicit-Curriculum Ablation (Q4) · p. 7, Table IV · p. 7
  • The implicit curriculum uses 0.73× the optimizer steps and 0.62× the training time of vanilla GRPO under the same nominal environment-step budget, while achieving higher final DGR.The reduction in optimization work and training time indicates improved optimization efficiency.Source: E. Implicit-Curriculum Ablation (Q4) · p. 7

VI. CONCLUSION

DynaForge uses phase-structured planning as a prior for dynamic demonstration generation, with residual learning correcting execution and an implicit curriculum focusing on mixed-success conditions. Demonstrations yield higher downstream DP3 performance and initial transfer to physical skills, while generation remains dependent on authored task structure and simulator fidelity.

VI. CONCLUSION

  • DynaForge turns phase-structured motion planning into a prior for dynamic demonstration generation, with a residual policy correcting execution and an implicit curriculum focusing learning on mixed-success conditions.Experiments show higher demonstration yield across nine simulation tasks and improved downstream DP3 performance, with initial transfer to three physical skills.Source: §VI · p. 7
  • Generation still depends on authored task structure and simulator fidelity; low success on precision insertion and modest physical performance limit the present scope.Future work targets less manual task specification, better contact-variation coverage, and broader evaluation under changes in physical dynamics and visual observations.Source: §VI · p. 7

Related papers

Loading 2609.25631v1…