PaperlayerView exact revision on arXiv

Source-linked AI summary

DROM: A Language-Guided Diffusion Framework for Multi-Skill Robotic Manipulation

Vincenzo Pomponi, Rocco Felici, Paolo Franceschi, Stefano Baraldo, Oliver Avram, Loris Roveda, Luca Maria Gambardella, Anna Valente

arXiv:2609.37348v1cs.RO

TL;DR

Robotic manipulation from limited demonstrations remains challenging because policies must support diverse skills and long-horizon tasks. DROM augments demonstrations with DMPs and combines language-guided multi-skill diffusion with language-model task decomposition; the reported evaluation shows generalization beyond demonstrated workspaces and execution of long-horizon tasks from natural-language instructions.

  • Problem

    Robotic manipulation policies must learn diverse, long-horizon tasks from limited demonstrations, while current execution does not directly incorporate changing object configurations during a generated trajectory.

    Source: §1 · p. 3

  • Method

    DROM augments demonstrations with DMPs, generates language-conditioned trajectories with a unified diffusion model, and uses a language model to decompose high-level requests into skill sequences.

    Source: §1 · p. 3, §6 · p. 33

  • Results

    DROM represents multiple skills, including orientation-sensitive behaviors, generalizes beyond demonstrated workspaces, and composes skills to execute long-horizon natural-language tasks.

    Source: §6 · p. 33

  • Takeaways & Limitations

    The framework offers a way to add manipulation capabilities from a small number of demonstrations without retraining task-specific planners or manually designing motion strategies.

    Source: §6 · p. 33

    The vision and diffusion modules are separate, limiting continuous integration of visual observations during trajectory generation.

    Source: §6 · p. 34

Abstract

from arXiv · show

Learning robust manipulation policies for diverse, long-horizon tasks from limited demonstrations remains a fundamental challenge in robotics. We present DROM, a language-guided diffusion framework that enables robots to learn, represent, and compose multiple manipulation skills within a single generative policy. DROM leverages Dynamic Movement Primitives (DMPs) to augment a small set of expert demonstrations into expressive multi-skill datasets, substantially reducing data collection while improving spatial generalization beyond the demonstrated workspace. Building upon Motion Planning Diffusion (MPD), we extend the diffusion architecture to support language-conditioned multi-skill trajectory generation through cross-attention, allowing a single model to generate skill-consistent motions for a diverse set of manipulation primitives, including orientation-sensitive behaviors that are difficult to design using conventional motion planning or hard-coded controllers. For long-horizon manipulation, a large language model decomposes high-level operator requests into executable sequences of skills, enabling natural language interaction and autonomous task execution. We validate DROM on a Franka Emika Panda robot, a FANUC CRX25ia robot, and in MuJoCo simulation across a wide range of manipulation tasks. Experimental results demonstrate that DROM outperforms Motion Planning Diffusion and Behavior Cloning baselines, achieves robust multi-skill generalization, and composes learned skills to reliably execute long-horizon manipulation tasks from natural language instructions using only a limited number of human demonstrations. Datasets, simulation environments, and more at https://github.com/automation-robotics-machines/drom.

1 Introduction

Robotic manipulation policies remain difficult to scale across skills and unfamiliar workspace configurations, while language-guided long-horizon systems can bring data and inference costs. DROM combines DMP-based demonstration augmentation with language-conditioned diffusion and hierarchical task reasoning to address these challenges.

1 Introduction

  • Conventional imitation-learning methods often require large task-specific datasets, focus on individual skills, and struggle with workspace or object configurations outside training.These limits constrain scalability and reuse across broader workflows.Source: §1 · p. 3
  • VLA systems can execute long-horizon tasks from instructions, but their massive data requirements and computationally heavy inference can hinder real-time physical deployment.Diffusion models can capture multimodal motion distributions, though existing manipulation policies often remain single-skill and sample-inefficient.Source: §1 · p. 3
  • The paper identifies a gap in meeting diffusion models’ dense-data needs without exhaustive demonstrations while avoiding language-model latency in low-level control.Source: §1 · p. 3
  • DROM uses DMPs to expand a small set of human demonstrations into synthetic multi-skill data and generate trajectories that generalize beyond the demonstrated workspace.A language model interprets instructions before execution to select a primitive or skill sequence, decoupling task reasoning from motion generation.Source: §1 · p. 3

2 Related Works

The related work positions DROM at the intersection of DMP-based trajectory augmentation, diffusion-based motion generation, and language-guided skill composition. It uses DMPs to expand demonstrations, extends multi-skill diffusion planning, and uses language models to sequence skills for long-horizon tasks.

2.1 DMPs and Synthetic Trajectory Generation

  • DMPs support trajectory reproduction and adaptation, but using them to generate datasets of complex, object-specific manipulation behaviors remains largely unexplored.DROM uses DMPs to augment a small demonstration set into diverse trajectories for diffusion learning and workspace generalization.Source: §2.1 · p. 4

2.2 Diffusion Models for Trajectory Planning

  • Prior diffusion work informs DROM’s temporal U-Net and multi-task conditioning, while DROM uses DMPs for trajectory augmentation and diffusion for direct synthesis.Related work also explores predicting DMP parameters within diffusion and hierarchical diffusion policies for manipulation.Source: §2.2 · p. 4

2.3 Skill Composition for Long-Horizon Tasks

  • Motion-planning approaches to skill composition often require extensive pre-programming, while language-based plans offer a route to more flexible interaction.DROM extends MPD with language embeddings and a language model that decomposes high-level instructions into executable skill sequences; its unified model handles 17 primitives, with evaluation in simulation and on two manipulators.Source: §2.3 · p. 5

3 Method

DROM combines DMP-based demonstration expansion, skill-conditioned diffusion, and language-based planning to execute single-skill and long-horizon manipulation tasks. Its pipeline moves from expert demonstrations through synthetic-data generation and model training to deployment in simulation and on real robots.

3 Method

  • DROM uses clustered conformal prediction to route known single-skill instructions to the diffusion model and send uncertain or high-level requests to an LLM for skill-sequence generation.The model generates executable trajectories for each selected skill using the current robot and object states.Source: §3 · p. 6

3.1 Diffusion-based Planning

  • The four-stage diffusion pipeline acquires expert demonstrations, augments them with DMPs, trains a skill-conditioned model, and deploys it in simulation and on real robots.Demonstrations are collected through kinesthetic teaching on the real platforms and with a SpaceMouse in simulation.Source: Figure 2 · p. 6, §3.1 · p. 7, §3.1.1 · p. 7

3.1.1 Data Collection

  • Experts provide at least one fixed-horizon demonstration per skill, adding examples when pose-dependent interactions require greater variation.Trajectories encode end-effector position and quaternion orientation alongside a binary gripper command.Source: §3.1.1 · p. 7

3.1.2 Data Augmentation

  • DMPs generate start- and goal-state variations while preserving each demonstrated behavior, producing 300 synthetic trajectories per skill in the experiments.When a skill has multiple demonstrations, its trajectory budget is divided equally among their DMPs.Source: §3.1.2 · pp. 7–8

3.1.3 Skill-Conditioned Diffusion Model

  • Cross-attention injects a learned skill embedding at each reverse diffusion step, allowing one diffusion model to generate trajectories for multiple skills.The model follows a temporal U-Net architecture; the forward process corrupts data trajectories with Gaussian noise and the learned reverse process recovers trajectories.Source: §3.1.3 · pp. 8–9

3.2 Language-based Planning

  • Language-based planning distinguishes requests for a single manipulation primitive from long-horizon tasks, which an LLM decomposes into ordered executable skills.Single-skill requests map directly to a primitive; the LLM can also identify requests outside the robot’s capabilities.Source: §3.2 · p. 9, §3.2.2 · p. 10

3.2.1 Conformal Prediction for Primitive Classification

  • Clustered conformal prediction uses skill-specific thresholds to classify a query as a known skill, an ambiguous match, or an unscripted or out-of-distribution request.Multiple plausible skills or no matching skill trigger LLM handling, while one match selects that primitive.Source: §3.2.1 · pp. 9–10

3.2.2 Language Model for Skill Sequence Generation

  • When a request is uncertain or out of distribution, the LLM composes an ordered sequence from available skills or returns a token indicating the robot cannot fulfill it.Few-shot prompting and constrained decoding mask skills outside the available set.Source: §3.2.2 · pp. 10–11

4 Design of Experiments

The experimental design evaluates DROM’s data-efficient learning, multi-skill trajectory generation, and language-guided composition of long-horizon tasks. It spans two robotic arms and MuJoCo simulation, using skill-specific demonstrations, evaluation criteria, and implementation choices.

4.1 Skill Description

  • The study evaluates 17 manipulation primitives across the Franka Emika Panda and FANUC CRX25ia, with dedicated success metrics for each skill.The skill library includes orientation-sensitive tasks such as inserting a gripper finger into a bottle handle and transporting a liquid-filled cup without spillage.Source: §4.1 · pp. 11–12

4.2 Training Dataset Generation Details

  • Demonstration counts vary with task complexity: workspace-invariant skills may use one demonstration, while orientation-sensitive skills use multiple demonstrations to capture valid modes.A separate DMP is learned per demonstration; generated trajectories perturb initial and goal configurations while preserving the demonstrated motion pattern, then are validated and resampled into 300 trajectories per skill.Source: §4.2 · p. 14

4.3 Diffusion Model Architecture

  • DROM extends MPD with language-embedding cross-attention at each reverse diffusion step, enabling one model to generate skill-consistent trajectories for multiple primitives.The model disambiguates behaviors that may share geometric configurations but require different motion strategies and object interactions.Source: §4.3 · p. 15, Figure 4 · p. 15

4.4 Language Encoder

  • The language encoder embeds instructions so similarly worded commands lie close together, supporting similarity-based retrieval of the requested manipulation skill.The chosen encoder is described as reducing memory consumption and inference time relative to larger language encoders.Source: §4.4 · p. 15

4.5 Language Model

  • The language-model evaluation tests whether generated skill sequences match valid plans, including plans with different acceptable execution orders.Its benchmark includes varied wording and complexity, from direct commands to objectives requiring implicit reasoning about intermediate steps.Source: §4.5 · p. 16

4.6 Task Description

  • Long-horizon evaluations combine primitives for pick-and-place, pouring, stacking, drawer storage, and cleanup; cleanup requires a variable-length sequence for one to three cubes.The Clean Up sequence opens the drawer, repeats pick-and-place for each cube, then closes the drawer.Source: §4.6 · pp. 16–17

4.7 Implementation Details

  • Implementation uses all-MiniLM-L6-v2 for instruction embeddings and gpt-oss-120b for high-level skill-sequence planning, with tests in MuJoCo and on both robotic platforms.The language model averages 3996±2388 ms per inference but is invoked only for high-level planning; diffusion inference generates 10 candidate trajectories in 281 ± 427 ms.Source: §4.7 · p. 17

5.1 RQ1: Learning Manipulation Skills from Limited Demonstrations

With the dataset fixed at 300 trajectories per skill, increasing demonstration diversity improved performance on multimodal skills. DROM also generalized beyond demonstrated workspace ranges, with expansion varying by axis and skill.

5.1.1 Effect of the number of demonstrations

  • Pick Bottle, Pick Cup, and Pour Liquid success rates rose from 24%, 28%, and 22% with one demonstration to 84%, 86%, and 92% with four, respectively.Button improved from 30% to 82%; Battery and Grab Handle reached 98% and 90% with two demonstrations, respectively.Source: §5.1.1 · pp. 18–19
  • Increasing demonstration diversity, rather than dataset size, enabled the model to learn multimodal strategies and select trajectories according to object configuration.The diffusion model was trained on a fixed dataset of 300 trajectories per skill while the number of expert demonstrations varied.Source: §5.1.1 · pp. 18–19

5.1.2 Generalization outside demonstrations

  • Average workspace expansions were 20.7% along X, 25.2% along Y, and 6.0% vertically; vertical generalization was more limited because training tasks varied little in height.Rotational expansion reached 50% for Franka Cube, 33% for Shelf, and 30% for Slide Nut; orientation-sensitive skills achieved more moderate 5–19% expansions.Source: §5.1.2 · p. 19
  • Stirring Spoon and Slide Nut were not evaluated under rotational perturbations, and Slide Nut was not tested at increased Z positions because its motion is constrained by the table.The reported results indicate extrapolation to unseen positions and orientations while preserving skill-specific constraints.Source: §5.1.2 · p. 20

5.2 RQ2: Multi-Skill Diffusion Policy

DROM’s multi-skill diffusion policy outperformed comparison policies in real-robot and simulation evaluations. Qualitative trials also show it selecting appropriate motion modes for orientation-dependent grasping tasks.

5.2.1 Imitation Learning Performance

  • DROM consistently outperformed MPD across considered skills on both robotic platforms and outperformed MPD and Behavior Cloning across evaluated simulation skills and environments.Real-robot trials assessed 50 validations per skill; simulation results report average success rate and relative standard deviation across three random seeds.Source: Table 4 · p. 20, §5.2.1 · p. 21, Table 5 · p. 22
  • DROM’s advantage was especially evident for object interactions requiring precise orientations and approach strategies, including bottle, cup, pouring, and drawer skills.The authors attribute the gains primarily to explicit primitive conditioning, which guides generation toward coherent, goal-oriented, dynamically consistent trajectories.Source: §5.2.1 · pp. 21–22

5.2.2 Imitation Learning in Simulation

  • In MuJoCo, DROM’s whole-trajectory diffusion representation outperformed Behavior Cloning, which predicts actions in a receding-horizon manner and focuses on local decisions.The authors describe the diffusion model as encoding global temporal structure and past and future motion context at each timestep.Source: §5.2.2 · p. 22

5.2.3 Diffusion Model Multi-Modality

  • The model selected the correct approach mode in all 10 Pick Bottle and all 10 Grab Handle trials, while task execution succeeded in 8/10 and 9/10 trials, respectively.The remaining failures were attributed to physical grasping or handle interaction rather than incorrect mode selection.Source: Figure 6 · p. 23, §5.2.3 · p. 24

5.3 RQ3: Language-Guided Diffusion Manipulation

This section evaluates DROM’s language-guided pipeline, from composing and executing long-horizon tasks to recognizing individual skill requests. Results identify task decomposition and increasingly long or variable sequences as important sources of reduced end-to-end success.

5.3.1 DROM Performance

  • Among two-skill tasks, Pick and Place leads at 86.0%; five-primitive Hide achieves 70.4%, and Clean Up reaches 60.9%, below Pick and Place, Pour, and Hide but comparable to Stack.Source: §5.3.1 · pp. 25–26

5.3.2 Ablation Study: LLM Performance

  • Pick and Place achieves 100.0% skill composition; overall success ranges from 60.9% on Clean Up to 86.0% on Pick and Place.Conditioned on correct sequences, execution success is 80.0% for Pour and 90.0% for Stack; the authors identify complex variable-length decomposition as the primary performance constraint.Source: Table 7 · p. 26, §5.3.2 · pp. 26–27

5.3.3 Failure Analysis

  • Language-model sequence errors affect 16.8% of episodes; grasp failures affect 4.4%, goal-configuration failures 4.0%, placement failures and collisions 2.8% each, and unfeasible trajectories 0.4%.The analysis attributes the main limitation to composing and executing interdependent skills, with decomposition or early manipulation errors propagating through later execution.Source: §5.3.3 · pp. 27–28, Table 8 · p. 28

5.3.4 Ablation Study: Language Encoder Performance

  • The language encoder averages 87.6% precision, 86.2% recall, 86.4% F1, and 86.2% success; confusions cluster among semantically related drawer skills.Grab Handle is confused with Open Drawer in 13% of cases, and Open Drawer with Place in Drawer in 14%.Source: §5.3.4 · pp. 28–29, Table 9 · p. 29
  • Across language-guided requests, recognition is 82%–93%, execution 86%–94%, and request-to-goal success 73.04%–85.56%; Place in Drawer reaches 85.5% overall.The authors attribute reduced end-to-end performance mainly to errors accumulating across sequential modules.Source: Table 10 · p. 29, §5.3.4 · p. 30

5.4 Language uncertainty quantification

The uncertainty module uses conformal prediction to route utterances between an embedding-based skill classifier and an LLM. It recognizes most out-of-scope and long-horizon requests for fallback, but some multi-step requests are misclassified, and routing reduces average latency relative to an LLM-only baseline.

Uncertainty routing

  • Conformal prediction routes recognized single-skill requests to the embedding model and empty or ambiguous prediction sets to the LLM; evaluation uses 900 utterances split into calibration and test sets.The calibration set contains 150 skill utterances; the test set contains 750 single-skill, long-horizon, and out-of-distribution requests.Source: §5.4 · p. 31

Prediction-set outcomes

  • All out-of-distribution utterances receive empty prediction sets, while 20% of long-horizon requests receive singleton sets and are incorrectly treated as in-scope single-skill inputs.Most long-horizon requests receive empty sets and are forwarded to the LLM for skill-sequence generation.Source: §5.4 · p. 31

Classification results

  • CP-certain skill predictions achieve 90.45% accuracy (199/220), while accuracy across all certain scenarios is 86.52%; the LLM correctly classifies 74.81% of uncertain utterances (389/520).The all-certain figure is lower because some multi-step requests are incorrectly classified as confident single-skill inputs.Source: §5.4 · p. 32

Overall accuracy and latency

  • 78.40% overall accuracy combines 199 correct embedding-model predictions and 389 correct LLM predictions; routing reduces average latency by approximately 1234.51 ms versus an LLM-only baseline.The estimated average latency is 2761.49 ms per interaction with routing, compared with 3996 ms for the LLM-only baseline.Source: §5.4 · pp. 32–33

6 Conclusions, Limitations, and Future Work

DROM combines demonstration augmentation, language-guided skill representation, and diffusion-based generation to compose learned manipulation skills. The evaluation reports multi-skill generalization and long-horizon execution, alongside limits in adapting trajectories to changing scenes and integrating visual feedback.

6 Conclusions, Limitations, and Future Work

  • DROM combines DMP-based trajectory augmentation, language-guided skill representation, and diffusion-based generation in one model for skill execution and composition.Source: §6 · p. 33
  • DMP-based augmentation and a single language-conditioned diffusion model support diverse skills, workspace generalization, and long-horizon task execution from natural-language instructions.Reported skills include orientation-sensitive behaviors that are difficult to design with conventional motion planning or hard-coded controllers.Source: §6 · p. 33
  • DROM is presented as a unified approach for adding manipulation capabilities with few demonstrations, without retraining task-specific planners or manually designing motion strategies.The authors identify flexible industrial and collaborative robotics with evolving requirements and rapid deployment needs as suitable applications.Source: §6 · p. 33
  • The framework does not dynamically regenerate trajectories from changing object configurations, and its separate vision and diffusion modules cannot continuously integrate visual changes during execution.Source: §6 · pp. 33–34
  • Future work proposes hierarchical diffusion, multimodal sensory feedback, larger skill libraries, more complex industrial scenarios, online adaptation, and learning-based object pose estimation.Source: §6 · p. 34

Appendix A Dynamic Movement Primitives

The appendix describes DMPs as a way to represent and reproduce demonstrated trajectories while generalizing to new initial and goal states. It outlines the dynamical system, its forcing function, and how its parameters are learned and configured.

Appendix A Dynamic Movement Primitives

  • A DMP models a trajectory as a second-order dynamical system, with the state driven toward goal g and a nonlinear forcing function encoding the demonstrated motion.The state and its first and second derivatives are y(t), ẏ(t), and ÿ(t); α_y and β_y are positive gains.Source: Appendix A · p. 34
  • The forcing term is a normalized weighted sum of N radial basis functions, with learnable weights and Gaussian basis functions.Source: Appendix A · p. 34
  • The experiments use 200 Gaussian basis functions and gains α_y = 25 and β_y = 6.25; the canonical phase system decays monotonically, while DMPs generalize to novel initial and goal states.The stated gains yield a critically damped system for smooth convergence while preserving the demonstrated motion.Source: Appendix A · p. 35
  • DMP weights are learned from demonstrations by deriving a target forcing term and estimating each weight through locally weighted regression.Source: Appendix A · p. 35

Appendix B Prompt structure

The prompt structure specifies the available actions and objects, action parameters, and examples of expected task plans. Its examples show how natural-language requests map to ordered manipulation primitives.

Appendix B Prompt structure

  • The prompt presents available primitives and scene objects, specifies action parameters, and includes examples with inputs and expected outputs.Source: Appendix B · pp. 35–36
  • Example plans decompose stacking and drawer-placement requests into ordered actions, including grabbing and opening the drawer before picking and placing an object.The drawer-placement example includes closing the drawer as an optional final action.Source: Appendix B · pp. 36–37
  • Other examples map opening a drawer to grabbing its handle then opening it, while a close-drawer request maps to a single action.Source: Appendix B · p. 37
  • Requests that do not refer to an available skill or a composition of skills receive no output, and the prompt requires valid JSON only.Source: Appendix B · p. 37
Loading 2609.37348v1…