PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, Shuran Song

arXiv:2303.04137v5cs.RO

TL;DR

Robot action prediction is challenging because actions can be multimodal, temporally correlated, and high precision. Diffusion Policy models visuomotor behavior as a conditional denoising diffusion process with closed-loop action-sequence prediction, consistently outperforming existing methods across benchmarks with an average improvement of 46.9%.

  • Problem

    Robot policy learning must handle multimodal, sequentially correlated, and high-precision action distributions that make mapping observations to actions challenging.

  • Method

    Diffusion Policy generates visually conditioned robot action sequences through conditional denoising diffusion and receding-horizon closed-loop control.

  • Results

    46.9% average improvement, with consistent performance boosts across all benchmarks.

  • Takeaways & Limitations

    Diffusion-based visuomotor policies provide a stable, effective approach for robot behavior across simulation and real-world manipulation tasks.

  • Takeaways & Limitations

    Diffusion Policy can perform suboptimally with inadequate demonstrations and has higher computational costs and inference latency than simpler methods.

Abstract

from arXiv · show

This paper introduces Diffusion Policy, a new way of generating robot behavior by representing a robot's visuomotor policy as a conditional denoising diffusion process. We benchmark Diffusion Policy across 12 different tasks from 4 different robot manipulation benchmarks and find that it consistently outperforms existing state-of-the-art robot learning methods with an average improvement of 46.9%. Diffusion Policy learns the gradient of the action-distribution score function and iteratively optimizes with respect to this gradient field during inference via a series of stochastic Langevin dynamics steps. We find that the diffusion formulation yields powerful advantages when used for robot policies, including gracefully handling multimodal action distributions, being suitable for high-dimensional action spaces, and exhibiting impressive training stability. To fully unlock the potential of diffusion models for visuomotor policy learning on physical robots, this paper presents a set of key technical contributions including the incorporation of receding horizon control, visual conditioning, and the time-series diffusion transformer. We hope this work will help motivate a new generation of policy learning techniques that are able to leverage the powerful generative modeling capabilities of diffusion models. Code, data, and training details is publicly available diffusion-policy.cs.columbia.edu

1 Introduction

Diffusion Policy addresses the multimodal, sequential, and high-precision challenges of robot action prediction by modeling visuomotor behavior as a conditional denoising diffusion process. The paper combines diffusion-based expressivity and stability with receding-horizon control, visual conditioning, and a time-series transformer, and reports a 46.9% average improvement across 15 tasks and 4 benchmarks.

  • Motivation: Robot action prediction is challenging because action distributions can be multimodal, sequentially correlated, and highly precise.These properties distinguish robot policy learning from ordinary supervised regression.
  • Method: Diffusion Policy infers the action-score gradient from visual observations and iteratively denoises actions for K iterations instead of directly outputting an action.It generates behavior through a conditional denoising diffusion process over robot action space.
  • Diffusion advantages: Diffusion Policy expresses arbitrary normalizable, including multimodal, action distributions through score-gradient learning and stochastic Langevin dynamics sampling.This addresses multimodality while maintaining distributional expressivity.
  • Diffusion advantages: Diffusion models support joint inference of future action sequences in high-dimensional spaces, promoting temporal consistency and reducing myopic planning, while gradient learning enables stable training without negative sampling.The method bypasses estimation of an intractable normalization constant required by energy-based policies.
  • Technical contributions: The proposed system combines receding-horizon closed-loop control, visual conditioning, and a time-series diffusion transformer for responsive execution, efficient inference, and high-frequency action control.Visual features are extracted once across denoising iterations, while the transformer reduces over-smoothing relative to typical CNN-based models.
  • Evaluation: 46.9% average improvement was observed across 15 tasks from 4 benchmarks spanning simulated and real-world settings and diverse action spaces, task structures, systems, objects, and demonstrators.The evaluation used behavior cloning and included 2DoF to 6DoF actions, single- and multi-task benchmarks, and fully- and under-actuated systems.

2 Diffusion Policy Formulation

Diffusion Policy formulates visuomotor policies as conditional DDPMs that generate actions through iterative denoising. The formulation supports multimodal action distributions and stable training while incorporating action-sequence prediction and visual conditioning for robot control.

  • 2 Diffusion Policy Formulation: Diffusion policies represent visuomotor robot policies as DDPMs, enabling complex multimodal action distributions and stable training with little task-specific hyperparameter tuning.The paper frames these properties as key advantages of the diffusion formulation.
  • 2.1 Denoising Diffusion Probabilistic Models: DDPM generation starts from Gaussian noise and performs K denoising iterations to produce progressively less noisy outputs until a noise-free action is formed.The intermediate sequence is x_k,x_k−1...x_0.
  • 2.1 Denoising Diffusion Probabilistic Models: The noise-prediction network ε_θ predicts a gradient field, making each denoising update interpretable as a noisy gradient-descent step.The learning rate is denoted γ, while the noise schedule controls α, γ, and σ across iterations.
  • 2.2 DDPM Training: Training randomly selects dataset examples, denoising iterations, and appropriately scaled noise, then trains the network to predict the added noise.The loss also minimizes a variational lower bound on the KL-divergence between the data distribution and DDPM samples.
  • 2.3 Diffusion for Visuomotor Policy Learning: Adapting DDPMs for visuomotor learning requires representing outputs as robot actions and conditioning denoising on observations.These are the formulation’s two major modifications from image-generation DDPMs.
  • 2.3 Diffusion for Visuomotor Policy Learning: Closed-loop action-sequence prediction commits to diffusion-generated actions for a fixed duration before replanning, balancing temporal smoothness with reactions to unexpected observations.At time t, the policy uses the latest T_o observation steps to predict T_p action steps.
  • 2.3 Diffusion for Visuomotor Policy Learning: Visual conditioning models p(A_t|O_t) directly rather than p(A_t,O_t), avoiding future-state inference and accelerating diffusion while improving generated-action accuracy.Excluding observation features from the denoising output further improves inference speed, supports real-time control, and facilitates end-to-end vision-encoder training.

3 Key Design Decisions

Diffusion Policy compares CNN and Transformer noise-prediction architectures, recommending CNNs by default and time-series diffusion Transformers for complex tasks or rapidly changing actions. Its implementation also specifies visual encoding, a square cosine noise schedule, and DDIM-based inference for faster control.

  • Network architectures: Diffusion Policy evaluates CNN and Transformer architectures for the noise-prediction network εθ, independently of the visual encoder.The architecture choice is framed as a comparison of performance and training characteristics.
  • CNN-based Diffusion Policy: CNN-based policies work well on most tasks with little tuning but perform poorly when desired action sequences change quickly and sharply.The limitation is attributed to temporal-convolution inductive bias favoring low-frequency signals.
  • Time-series diffusion transformer: The time-series diffusion Transformer is introduced to reduce CNN over-smoothing by predicting noisy action trajectories with transformer decoder blocks.Observation embeddings provide input features, while diffusion-step embeddings and action tokens are processed by the decoder stack.
  • Recommendations: Transformer backbones achieve most of the best state-based results when task complexity and action-change rates are high, but they are more sensitive to hyperparameters.The paper therefore recommends starting with CNNs and switching to Transformers when complexity or high-rate action changes limit performance, at the cost of additional tuning.
  • Visual encoder: The visual encoder is trained end-to-end, uses separate encoders for different camera views, and concatenates independently encoded timestep images into observation embeddings.The implementation uses an unpretrained ResNet-18 modified with spatial softmax pooling to preserve spatial information.
  • Noise schedule and inference: The Square Cosine Schedule works best empirically for the control tasks, while DDIM decouples training and inference denoising iterations to accelerate closed-loop control.The noise schedule controls how action-signal frequency characteristics are captured, and DDIM enables fewer inference iterations.

4 Intriguing Properties of Diffusion Policy

Diffusion Policy’s advantages arise from its natural multimodal action modeling, high-dimensional action-sequence representation, and stable score-based training. It also supports position-control policies and exhibits interpretable limiting behavior for simple linear systems.

  • Multimodality: Diffusion Policy naturally expresses multimodal action distributions through stochastic initialization and Langevin perturbations that move samples among distinct action basins.In planar pushing, samples can commit to different valid trajectories across rollouts.
  • Position control: Diffusion Policy with position control consistently outperforms its velocity-control version, contrasting with behavior-cloning methods that generally rely on velocity control.The paper attributes this advantage partly to stronger position-control multimodality and reduced compounding-error effects.
  • Multimodality: Diffusion Policy learns both leftward and rightward pushing modes while committing to one mode within each rollout.LSTM-GMM and IBC are biased toward one mode, whereas BET lacks single-mode temporal consistency.
  • High-dimensional sequence prediction: Diffusion Policy represents actions as high-dimensional sequences, scaling with output dimensions without sacrificing expressiveness and improving robustness to idle-action sequences.This avoids manually specifying action-distribution modes and reduces overfitting to pauses common in teleoperation.
  • Training stability: Diffusion Policy improves training stability by modeling the action-distribution score function instead of estimating the EBM normalization constant.Neither inference nor training evaluates the intractable normalization constant, avoiding instability caused by inaccurate negative sampling.
  • Linear-system behavior: For a linear feedback system with one-step horizon, DDIM sampling converges to the global minimum a = −Ks; longer-horizon prediction implicitly requires learning task-relevant dynamics.Trajectory prediction follows the system’s linear dynamics, with disturbance terms averaging to zero.

5 Evaluation

Diffusion Policy outperformed alternative methods across all evaluated tasks and variants, achieving a 46.9% average improvement in simulation benchmarks. The evaluation highlights its handling of multimodality, position control, latency, action-horizon tradeoffs, and training stability.

  • Overall performance: 46.9% average improvement: Diffusion Policy outperformed alternative methods on all tasks and variants with state and vision observations.The simulation study covered benchmarks reported in Tables 1, 2, and 4.
  • Multimodality: Diffusion Policy represented short-horizon multimodality by approaching Push-T contact points equally likely from the left or right.LSTM-GMM and IBC exhibited bias in this case study.
  • Multimodality: 32% improvement on Block Push’s p2 metric: Diffusion Policy handled long-horizon multimodality better than baselines.The evaluation also examined arbitrary sub-goal ordering in Block Push and Kitchen.
  • Control and robustness: Position control significantly outperformed velocity control when used as Diffusion Policy’s action space.The evaluated baselines generally worked best with velocity control.
  • Control and robustness: An action horizon of 8 steps was optimal for most tested tasks, balancing action consistency against slower reaction time at longer horizons.Horizons greater than 1 helped compensate for idle demonstration portions, while excessive horizons reduced performance.
  • Control and robustness: Diffusion Policy maintained peak performance with latency up to 4 steps and was more stable to train than IBC.Receding-horizon position control addressed latency from image processing, inference, and network delay; velocity control was more latency-sensitive.

6 Realworld Evaluation

Diffusion Policy performs strongly across real-world manipulation tasks, matching or approaching human performance while handling multimodal behaviors, perturbations, and challenging action requirements. On Push-T, mug flipping, pouring, and spreading, it substantially outperforms recurrent baselines and benefits from end-to-end visual training.

  • Real-world Push-T: 95% success rate and 0.8 average IoU demonstrate near-human real-world Push-T performance, versus 0% and 20% success for IBC and LSTM-GMM.The task is multi-stage and contains short-term multimodality during fine adjustments and end-zone transitions.
  • Real-world Push-T: End-to-end-trained vision outperforms pretrained encoders: R3M reaches 80% success but produces jittery actions and gets stuck more often.ImageNet features yielded abrupt actions and poor performance, while end-to-end training was the most effective approach tested.
  • Robustness: Diffusion Policy remained functional under camera blockage and block displacement, correcting shifted positions and aborting end-zone motion when necessary.A waving hand blocked the front camera for 3 seconds, yet the policy remained on-course; these corrective behaviors were not demonstrated.
  • Mug flipping: 90% success over 20 mug-flipping trials shows Diffusion Policy can capture multimodal 3D manipulation behaviors and sequence undemonstrated pushes or regrasps when needed.The task involves complex rotations near hardware kinematic limits, while the dataset includes grasp-versus-push and multiple grasp variants.
  • Sauce manipulation: Coverage of 0.74 versus 0.79 on pouring and 0.77 versus 0.79 on spreading indicates close-to-human performance on both sauce-manipulation tasks.Diffusion Policy also reacted gracefully when the pizza dough was moved by hand during either task.

7 Realworld Bimanual Tasks

Diffusion Policy extends beyond single-arm setups to challenging real-world bimanual manipulation, working without hyperparameter tuning after the robot stack was extended for multi-arm teleoperation and control. Across egg-beater, mat-unrolling, and shirt-folding tasks, it achieves 55–75% success over 20 trials.

  • Realworld Bimanual Tasks: Diffusion Policy handled challenging bimanual tasks out of the box without hyperparameter tuning, following extensions for multi-arm teleoperation and control.The observation and action spaces were expanded to represent both end-effectors and grippers, with two scene cameras and two wrist cameras.
  • Egg Beater: 55% success rate over 20 trials was achieved on bimanual egg-beater manipulation using 210 demonstrations.Primary failures involved out-of-domain initial positioning, missing the crank handle, or losing grasp of it.
  • Mat Unrolling: 75% success rate over 20 trials was achieved on mat unrolling using 162 demonstrations.The skill was omnidextrous, unrolling left or right depending on the initial condition; failures primarily involved missed initial grasps and repeated behavior.
  • Shirt Folding: 75% success rate over 20 trials was achieved on shirt folding using 284 demonstrations.The task can require up to nine discrete steps, with collision avoidance important because the grippers approach closely; failures involved missed initial grasps and end-of-task adjustment.

8 Related Work

Related work distinguishes explicit policies, which efficiently regress observations to actions but struggle with multimodal behavior, from implicit policies, which model multimodal action distributions but can be unstable to train. Diffusion Policy applies diffusion models to behavioral cloning, complementing prior work on planning, reinforcement learning, policy augmentation, and simulated diffusion policies.

  • Explicit Policy: Explicit policies map observations or world states directly to actions in one forward pass, but struggle to model multimodal demonstrated behavior and high-dimensional actions.They can be trained with direct regression loss and offer efficient inference.
  • Implicit Policy: Implicit policies assign energies to actions, naturally representing multimodal distributions because multiple actions may have low energy.Action prediction is formulated as finding a minimum-energy action.
  • Implicit Policy: Existing implicit policies can be unstable to train because their Info-NCE loss requires drawing negative samples.
  • Diffusion Models: Prior diffusion-model work studies planning, trajectory inference, state-based reinforcement-learning policies, or augmentation of explicit policies, whereas this work applies diffusion models to behavioral cloning.Diffusion models iteratively refine sampled noise into draws from an underlying distribution and can be understood as optimizing a learned implicit action-score gradient.
  • Diffusion Models: Concurrent studies analyze diffusion-based policies in simulated environments, emphasizing sampling strategies, classifier-free goal guidance, or reinforcement learning, while this work focuses on effective action spaces.The paper states that its empirical findings largely concur with theirs in simulation.

9 Limitations and Future Work

Diffusion Policy inherits behavior-cloning limitations when demonstrations are inadequate and incurs higher computational costs and inference latency than simpler methods. Future work can explore reinforcement learning and diffusion acceleration to use suboptimal data and reduce inference steps, while action-sequence prediction may remain insufficient for high-rate control.

  • Limitations: Diffusion Policy can perform suboptimally when demonstration data are inadequate because it inherits behavior-cloning limitations.Future work may apply Diffusion Policy to reinforcement learning to exploit suboptimal and negative data.
  • Limitations: Diffusion Policy has higher computational costs and inference latency than simpler methods such as LSTM-GMM.Action sequence prediction partially mitigates this issue but may not suffice for tasks requiring high rate control.
  • Future Work: Future work can use diffusion-model acceleration methods to reduce the number of inference steps required.The paper identifies acceleration as a way to address Diffusion Policy’s computational burden and latency.

10 Conclusion … A.4 Hyperparameters

The paper finds diffusion-based visuomotor policies consistently outperform existing methods across 15 simulated and real-world tasks while remaining stable and easy to train. Implementation guidance covers action normalization, rotation representations, visual augmentation, task-dependent hyperparameters, and inference efficiency.

  • 10 Conclusion: Diffusion-based visuomotor policies consistently and definitively outperform existing methods across 15 simulation and real-world tasks while remaining stable and easy to train.The evaluation also identifies receding-horizon action prediction, end-effector position control, and efficient visual conditioning as critical design factors.
  • A.1 Normalization: Independently scaling each action dimension’s minimum and maximum to [−1,1] generally works well, whereas zero-mean unit-variance normalization can make action-space regions inaccessible.When variance is small, shifting to zero mean without scaling prevents the issue.
  • A.2 Rotation Representation: Velocity-control environments use 3D axis-angle rotations, while positional-control environments use Zhou et al.’s 6D rotation representation.Axis-angle singularities and discontinuities usually do not cause problems because velocity commands are close to zero.
  • A.3 Image Augmentation: Training uses random crop augmentation, while inference uses a static center crop of the same task-specific size.The crop size is specified separately for each task in the paper’s hyperparameter tables.
  • A.4 Hyperparameters: Block Push requires substantially different optimal observation and action horizons because its demonstrations come from a Markovian scripted oracle rather than human teleoperation.The corresponding hyperparameters are therefore not representative of the other tasks.
  • A.4 Hyperparameters: CNN-based Diffusion Policy has task-consistent optimal hyperparameters, whereas transformer attention dropout and weight decay vary greatly across tasks.Increasing CNN model parameters always improved performance during tuning, while transformer model-size increases were not uniformly beneficial except on Push-T.
  • A.4 Hyperparameters: Vision-based Diffusion Policy prefers a low observation horizon greater than 1, with 2 a good compromise for most tasks, while state-based policy is not sensitive to horizon.Diffusion Policy also outperforms LSTM-GMM at every training dataset size.
  • A.4 Hyperparameters: Simulation uses iDDPM with 100 denoising iterations for training and inference, while real-world benchmarks use DDIM with 16 inference iterations to reduce latency.Batch sizes are 256 for state-based and 64 for image-based experiments, with cosine scheduling and linear warmup of 500 CNN or 1000 transformer steps.

A.5 Data Efficiency … C.2.1 Demonstrations

Diffusion Policy outperformed LSTM-GMM across training dataset sizes and achieved an average improvement of approximately 46.9% over baseline methods. The experiments also characterize observation-horizon effects, training and evaluation protocols, and demonstration collection for real-world tasks.

  • A.5 Data Efficiency: Diffusion Policy outperformed LSTM-GMM at every training dataset size.This result is reported for the data-efficiency comparison in Fig. 15.
  • B.1 Observation Horizon: State-based Diffusion Policy was insensitive to observation horizon, whereas vision-based variants, especially CNN-based ones, degraded as the horizon increased.An observation horizon of 2 was effective for most tasks with both state and image observations.
  • B.1 Observation Horizon: CNN- and transformer-based Diffusion Policy configurations were specified with separate control, horizon, model-size, optimization, and diffusion-iteration hyperparameters.The corresponding settings are provided in Tables 7 and 8.
  • B.2 Performance Improvement Calculation: 46.9% was the average performance improvement of Diffusion Policy over the strongest baseline method per task.For each task, the comparison used the maximum baseline performance and maximum performance among CNN and Transformer Diffusion Policy variants.
  • C.1 Push-T: 136 demonstrations were collected for Push-T training, with initial conditions varied by randomly pushing or tossing the T block.The operator had prior extensive experience with the task.
  • C.1.1 Demonstrations: Each method used a fixed 12-hour training period and its last checkpoint, except IBC, which used the checkpoint with minimum training-set action-prediction MSE.The exception addressed IBC’s training-stability issue.
  • C.1.2 Evaluation: Each method was evaluated for 20 episodes from matched initial conditions, with episodes ending after 0.5 seconds in the end-zone or at 60 seconds.For Push-T, IoU was computed directly from top-down camera pixels.
  • C.2 Sauce Pouring and Spreading: 50 demonstrations were collected for each sauce task, with 90% used for training and task-specific variations in object locations, sauce shape, and cleanup procedures.Diffusion Policy and LSTM-GMM were trained for 1000 epochs, using the last checkpoint for evaluation.

C.2.2 Evaluation … D.1 Franka Robot Station

Evaluation varies object placements across shared random initial conditions and uses fixed sauce amounts, while real-world experiments span UR5 and Franka stations with distinct control and sensing setups. Franka inference uses non-haptic control, despite a separate haptic teleoperation controller.

  • C.2.2 Evaluation: Each method is evaluated from the same set of random initial conditions, varying the positions of the pizza dough and sauce bowl.The protocol is similar to PushT; the initial shape of poured sauce is not matched, but the sauce amount remains fixed.
  • C.2.2 Evaluation: Evaluation episodes terminate after the spoon moves upward away from the dough for 0.5 seconds or when the operator judges behavior unsafe.
  • C.2.2 Evaluation: Coverage is computed by projecting left- and right-camera RGB images into table space through homography and reporting the maximum camera coverage.
  • D Realworld Setup Details: Push-T experiments are performed on the UR5 robot station.
  • D.0.1 UR5 robot station: The UR5 accepts end-effector positional commands at 125Hz, interpolated from 10Hz demonstrations or policy commands, with velocity and table-region safety limits.The interpolation controller limits end-effector velocity below 0.43 m/s and position within 1cm above the table.
  • D.0.1 UR5 robot station: The UR5 station records 720p RGB videos at 30fps from five RealSense D415 cameras, using two downsampled to 320x240 at 10fps for policy observation.During demonstration, the operator teleoperates with a 3dconnexion SpaceMouse at 10Hz.
  • D.1 Franka Robot Station: Sauce Pouring and Spreading, Bimanual Egg Beater, Bimanual Mat Unrolling, and Bimanual Shirt Folding experiments use the Franka robot station.
  • D.1 Franka Robot Station: For Franka inference, the learned policies use non-haptic control, while haptic teleoperation uses a separate pure torque controller formulated as a 200Hz Quadratic Program.Non-haptic control tracks desired end-effector poses through differential-kinematics QP and joint-level control; collision avoidance is not enabled for the haptic mode.

D.2 Initial and Final States of Bimanual Tasks

The section visualizes initial and final states across four bimanual tasks, marking successful and failed rollouts and using homographic projection for flat objects.

  • Overview: Four bimanual tasks are shown through their initial and final states, with green boxes marking successful rollouts and red boxes marking failed rollouts.The visualizations compare each task’s starting and ending configurations.
  • Visualization: Homographic projection improves visualization of the initial and final states for the particularly flat mat and shirt objects.This projection is used because mat and shirt are very flat.
  • Egg Beater: Egg Beater is documented with separate figures for its initial and final states.These are presented in Figures 18 and 19.
  • Shirt Folding: Shirt Folding is documented with separate figures for its initial and final states.These are presented in Figures 20 and 21.
Loading 2303.04137v5…