Source-linked AI summary
Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation
Xiao Ma, Sumit Patidar, Iain Haughton, Stephen James
TL;DR
Robotic manipulation needs policies that are both sample-efficient for long-horizon tasks and capable of producing context-aware, kinematics-valid trajectories. HDP addresses this by combining a language-guided next-best-pose planner with RK-Diffuser, which learns joint and pose trajectories and distills between them through differentiable kinematics. The paper reports stronger performance across challenging simulated tasks and successful real-robot demonstrations, while identifying task-context dependence as a boundary for conventional planners.
Problem
Direct policies can be sample-inefficient, while next-best-pose systems may rely on context-unaware planners and inverse kinematics that violate task or robot constraints.
Method
HDP chains a high-level language-guided next-best-pose agent with RK-Diffuser, which learns pose and joint-position trajectories and distills pose trajectories into joint space via differentiable kinematics.
Results
HDP outperforms flat and alternative hierarchical agents on challenging RLBench tasks and achieves high success on a real-robot oven-opening task using only 20 demonstrations.
Takeaways & Limitations
The factorized policy combines long-horizon task-level decisions with fine-grained, kinematics-aware low-level control across simulated and real robotic manipulation.
Takeaways & Limitations
Sampling-based planners can fail on tasks requiring fine-grained trajectories, while handcrafted task-specific constraints are not generalisable across tasks.
Abstract
from arXiv · showhide
This paper introduces Hierarchical Diffusion Policy (HDP), a hierarchical agent for multi-task robotic manipulation. HDP factorises a manipulation policy into a hierarchical structure: a high-level task-planning agent which predicts a distant next-best end-effector pose (NBP), and a low-level goal-conditioned diffusion policy which generates optimal motion trajectories. The factorised policy representation allows HDP to tackle both long-horizon task planning while generating fine-grained low-level actions. To generate context-aware motion trajectories while satisfying robot kinematics constraints, we present a novel kinematics-aware goal-conditioned control agent, Robot Kinematics Diffuser (RK-Diffuser). Specifically, RK-Diffuser learns to generate both the end-effector pose and joint position trajectories, and distill the accurate but kinematics-unaware end-effector pose diffuser to the kinematics-aware but less accurate joint position diffuser via differentiable kinematics. Empirically, we show that HDP achieves a significantly higher success rate than the state-of-the-art methods in both simulation and real-world.
1. Introduction
HDP addresses the tension between long-horizon task planning and context-aware, kinematics-valid motion generation by combining next-best-pose planning with learned low-level control. Its RK-Diffuser generates flexible trajectories while avoiding common inverse-kinematics constraint violations, and experiments report strong simulation and real-robot performance.
- Direct visual-to-action policies retain flexible control but often have low sample efficiency and poor generalisation, especially on long-horizon tasks.
- NBP agents improve sample efficiency by predicting distant keyframes, but predefined motion planners can fail when tasks require context-dependent trajectories.Opening a box may require understanding hinge resistance and executing a specific curved trajectory.
- HDP combines a high-level language-guided NBP agent with a low-level learned controller for long-horizon decisions and context-aware 6-DoF pose reaching.The high-level agent receives 3D observations and language instructions and predicts a 6-DoF end-effector pose.
- RK-Diffuser learns end-effector pose and joint-position diffusion, distilling pose trajectories into joint space through differentiable kinematics.This design targets accurate trajectory generation while retaining control flexibility and avoiding inverse-kinematics constraint violations.
- Across challenging RLBench tasks, RK-Diffuser generally improves goal-conditioned motion generation, while HDP outperforms flat and alternative hierarchical agents.On a real robot, HDP achieves a high success rate on an oven-opening task using only 20 demonstrations.
2. Related Works
Prior work addresses manipulation efficiency through direct policies, next-best poses, 3D action-value maps, and hierarchical diffusion. HDP’s related-work position is to combine hierarchical planning with diffusion-based low-level control while replacing inverse-kinematics execution with joint-trajectory learning and differentiable kinematics.
- End-to-end visual policies are flexible but tend to be sample-inefficient, motivating next-best-pose and 3D voxel action-value-map approaches.
- Language-conditioned high-level agents can be paired with low-level controllers, but prior hierarchical approaches may be computationally heavy.HDP uses PerAct as its high-level language-conditioned agent and RK-Diffuser as the low-level policy.
- Diffusion policies generate diverse multimodal manipulation trajectories through conditional generation and imitation learning.
- HDP’s RK-Diffuser learns both end-effector pose and joint-position trajectories, refining joint trajectories through differentiable kinematics rather than relying solely on inverse-kinematics solvers.The comparison passage identifies inverse-kinematics errors and kinematic violations as weaknesses of pose-based execution.
3. Preliminaries
Diffusion policies model action trajectories through conditional denoising, while differentiable kinematics provides gradients linking robot joint positions to end-effector poses. These preliminaries support learning and refining kinematics-aware manipulation trajectories.
- 3.1. Diffusion Models: Diffusion models add Gaussian noise through forward steps and learn a reverse process to reconstruct the data distribution.The noise schedule is controlled by variance parameters β1, …, βK.
- 3.1. Diffusion Models: In decision making, diffusion policies model an action trajectory a1:T conditioned on observations such as RGB data, point clouds, and robot states.
- 3.2. Differentiable Kinematics: Differentiable simulation represents environment steps as a differentiable computational graph, allowing system inputs to be optimized through gradients from resulting positions.
- 3.2. Differentiable Kinematics: For a robot with a predefined URDF, differentiable forward kinematics maps joint angles to end-effector poses, enabling pose losses to update joint positions directly.The mapping is expressed as sp = fK(sj).
4. Hierarchical Diffusion Policy
HDP factorises manipulation into a high-level next-best-pose policy and a low-level RK-Diffuser controller, combining task-level decisions with trajectory generation. RK-Diffuser uses differentiable kinematics to produce accurate trajectories while respecting robot kinematic constraints.
- Factorised Hierarchical Policy: HDP factorises the policy into a high-level agent predicting end-effector poses and a low-level controller generating actions conditioned on those predictions.The high-level action includes translation, rotation, and gripper control; the low-level policy generates the robot trajectory.
- Keyframe Discovery: Keyframe discovery identifies frames with near-zero joint velocity and unchanged gripper state, retaining their indices for training both hierarchical levels.The high-level agent is trained on keyframes rather than every trajectory point.
- High-Level Policy: The high-level policy uses visual observations and language instructions to predict a 6-DoF next-best end-effector pose for long-horizon task decisions.PerAct provides the language-conditioned high-level agent, using structured voxel-based action representations.
- Conditional Trajectory Generation: Trajectory inpainting fixes the start and predicted next-best poses during diffusion, ensuring the generated trajectory ends aligned with the high-level output.The low-level network also conditions on the start pose, end pose, RGB-D observation, robot state, and trajectory rank.
- Low-Level RK-Diffuser: RK-Diffuser learns end-effector pose and joint-position diffusion models, distilling pose trajectories into joint trajectories through differentiable robot kinematics.This design addresses the difficulty of mapping 6-DoF poses to potentially many joint configurations in an over-actuated 7-DoF arm.
- Kinematics-Aware Diffusion: RK-Diffuser generates joint trajectories that avoid robot kinematic violations while maintaining high manipulation accuracy.The method is designed to replace inverse-kinematics processing of kinematics-unaware end-effector trajectories.
5. Experiments
Experiments show that HDP improves multi-task manipulation across simulation and real-robot settings by combining task-aware hierarchical planning with kinematics-aware low-level control. Its advantages are largest on tasks requiring accurate, context-dependent trajectories and articulated-object interaction.
- Simulation Experiments: HDP outperforms state-of-the-art methods across RLBench tasks, achieving an overall 80.2% success rate across 11 tasks.
- Simulation Experiments: Hierarchical agents consistently outperform ACT and vanilla Diffusion Policy, which struggle to detect intermediate keyframes and accumulate distribution-shift errors on long-horizon tasks.
- Simulation Experiments: Learned low-level agents outperform motion planners because planners can generate task-inappropriate trajectories despite accurate next-best-pose predictions.
- Ablation Studies: Pose Diffusion has a 24.55% overall IK error rate, with invalid quaternions causing 75% of its failure cases.
- Ablation Studies: Joint position diffusion performs worse without last-joint-position inpainting, especially on challenging tasks such as open oven.
- Real-World Experiments: HDP achieves 100% success on real-world oven opening and 94% on sorting objects into a drawer, using 20 demonstrations overall.
6. Conclusion
HDP combines high-level next-best-pose planning with low-level kinematics-aware joint-trajectory generation. It achieves strong performance in challenging simulated and real-world manipulation tasks, while acknowledging compounding-error risks for longer horizons.
- HDP predicts a next-best end-effector pose at the high level and generates a goal-conditioned joint-position trajectory with RK-Diffuser at the low level.RK-Diffuser distills end-effector pose trajectories into joint-position trajectories through differentiable kinematics.
- HDP achieves state-of-the-art performance on challenging RLBench manipulation tasks.
- On a real robot, HDP solves both oven opening and sorting objects into a drawer.
- Longer-horizon behaviour cloning may accumulate errors, causing distribution shifts and eventual failure despite RK-Diffuser robustness to some out-of-distribution poses.The paper identifies reducing compounding error through more unified structures as future work.
A. Implementation Details
PerAct is the high-level language-conditioned agent, combining language and calibrated multi-view RGB-D observations to predict task actions through voxel-based representations and action-value maps.
- PerAct uses language descriptions and calibrated multi-view RGB-D images as inputs to a language-conditioned multi-task behaviour-cloning policy.Language is encoded with frozen CLIP, while RGB-D observations are converted into a 3D voxel grid.
- PerAct decodes fused language and voxel representations into a 3D action-value attention map.PerceiverIO combines language and voxel tokens using fixed latent vectors before decoding actions.
- The high-level action comprises an end-effector pose and gripper opening or closing action.
- The implementation represents scenes with 1003 voxels and 2048 fixed latent vectors of dimension 512.Training applies random translation and rotation offsets to voxel representations.
- RK-Diffuser replaces PerAct's original collision-prediction head and handles collision checking as the low-level agent.
A.2. RK-Diffuser
RK-Diffuser uses point-cloud-conditioned diffusion models to generate both pose and joint trajectories for goal-conditioned control. Its low-level setup conditions on the start and predicted goal poses alongside robot and environment state.
- RK-Diffuser learns separate diffusion models for joint-position and end-effector-pose trajectories.
- The low-level models condition on the known start pose, predicted next-best pose, robot state, gripper opening amount, and environment point cloud.
- RK-Diffuser uses point clouds rather than voxels because it only needs to understand the environment's 3D configuration, not action values over empty space.Point clouds are also less computationally expensive than voxel representations in this low-level setting.
- The low-level agent uses only the front camera and classifier-free guidance during inference for conditional sampling.
- The goal-conditioned low-level setup is trained in a multi-task setting with 100 demonstrations per task.
B.1. Real-Robot Experiment Results
The real-robot experiment section compares trajectory-generation and planner-selection approaches using ranked trajectories and recorded planning methods. The supplied passages describe the ranking setup but do not report the table's numerical outcomes.
- Trajectories are generated by sampled linear-plus-RRT, Bezier, and learned methods, then ranked according to estimated likelihood of success.
- In the behaviour-cloning setting, trajectory ranking cannot use environment rollouts, so recorded RLBench path-generation methods provide the training signal.
- Figure 6 ranks trajectories during training and encourages RK-Diffuser to generate high-rank, shorter trajectories during inference.Red denotes planner-generated ground-truth trajectories, while blue denotes RK-Diffuser trajectories.
- The path-generation baseline adds a PerAct head that predicts whether sampled or Bezier planning is optimal alongside next-best-pose and gripper outputs.
- When sampled planning is selected, inference tries a linear path before RRT; Bezier planning samples curvature parameters and executes the first successful configuration.
B.3. Trajectory Ranking
RK-Diffuser incorporates trajectory rank as an additional conditional variable, measuring path efficiency relative to the straight-line distance between start and end poses. Figure 6 visualizes this ranking analysis by comparing ground-truth and sampled trajectories.
- Trajectory-rank definition: Trajectory rank rξ is defined as the Euclidean start-to-end distance divided by the travelled distance.It provides a measure of trajectory efficiency based on the relation between direct displacement and actual path length.
- Trajectory-rank definition: An optimal trajectory that ignores robot kinematics constraints should have rξ = 1.This value corresponds to the direct Euclidean distance matching the travelled distance.
- Visualization: Figure 6 compares ground-truth trajectories in red with RK-Diffuser samples in blue.The visualizations are provided to analyze the effect of trajectory ranking.