PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Policy Policy Optimization

Allen Z. Ren, Justin Lidard, Lars L. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, Max Simchowitz

arXiv:2409.00588v3cs.ROcs.LG

TL;DR

Diffusion policies are effective but their policy-gradient fine-tuning had been conjectured inefficient, limiting reinforcement-learning optimization beyond demonstrations. DPPO embeds diffusion denoising in a policy-optimization framework and reports strong, stable performance, structured exploration, and robust deployment across simulated and real robotic settings.

  • Problem

    Policy-gradient fine-tuning of diffusion-based robot policies was believed inefficient, despite reinforcement learning’s potential to improve policies beyond limited or suboptimal demonstrations.

  • Method

    DPPO fine-tunes pre-trained Diffusion Policies with policy gradients by treating denoising as an MDP and optimizing the resulting diffusion-policy MDP.

  • Results

    DPPO achieves strong training stability and final performance across challenging robotic tasks, including pixel-based simulation and zero-shot sim-to-real hardware deployment.

  • Takeaways & Limitations

    DPPO’s diffusion parameterization supports structured, on-manifold exploration, robustness, and generalization during policy fine-tuning and deployment.

  • Takeaways & Limitations

    DPPO’s structured on-manifold exploration may hinder fine-tuning when aggressive unstructured exploration is desired, and its greatest benefit may require pre-training with sufficient success-mode coverage.

Abstract

from arXiv · show

We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic framework including best practices for fine-tuning diffusion-based policies (e.g. Diffusion Policy) in continuous control and robot learning tasks using the policy gradient (PG) method from reinforcement learning (RL). PG methods are ubiquitous in training RL policies with other policy parameterizations; nevertheless, they had been conjectured to be less efficient for diffusion-based policies. Surprisingly, we show that DPPO achieves the strongest overall performance and efficiency for fine-tuning in common benchmarks compared to other RL methods for diffusion-based policies and also compared to PG fine-tuning of other policy parameterizations. Through experimental investigation, we find that DPPO takes advantage of unique synergies between RL fine-tuning and the diffusion parameterization, leading to structured and on-manifold exploration, stable training, and strong policy robustness. We further demonstrate the strengths of DPPO in a range of realistic settings, including simulated robotic tasks with pixel observations, and via zero-shot deployment of simulation-trained policies on robot hardware in a long-horizon, multi-stage manipulation task. Website with code: diffusion-ppo.github.io

1 Introduction

The paper introduces DPPO, a policy-gradient framework for fine-tuning pre-trained diffusion policies, and shows strong performance across challenging robotic settings. Investigations link its behavior to structured exploration, training stability, robustness, and generalization.

  • DPPO fine-tunes pre-trained diffusion-based robot policies using policy gradient methods in reinforcement learning.
  • Policy-gradient fine-tuning, previously conjectured inefficient for Diffusion Policy, yields robust, high-performing policies with favorable training behavior.
  • DPPO improves final performance and training stability over off-policy Q-learning, weighted regression, demo-augmented RL, and Gaussian policy baselines.
  • Investigations associate DPPO’s success with structured exploration that better uses the training-data manifold and produces greater robustness to perturbations.
  • DPPO is evaluated in simulated and real environments, including pixel observations, sparse-reward long-horizon tasks, and zero-shot simulation-to-hardware deployment.
  • The framework may extend to sequential interactive diffusion-model applications beyond robotics, including human-feedback generation, molecular design, and interactive language modeling.

2 Related Work

Related work covers policy optimization for robotics, diffusion-based policies, methods for improving them, and policy gradients through diffusion models. DPPO builds on these lines by embedding denoising into the environment MDP for continuous-control tasks.

  • Policy optimization updates explicit policy representations through action-likelihood gradients, with algorithms such as DDPG and PPO improving stability or sample efficiency.
  • Diffusion-based policies are commonly trained from demonstrations and model complex, multimodal trajectory distributions with high training stability.
  • Prior methods improve diffusion policies through reward or goal guidance, Q-learning, weighted regression, and online or offline estimation.
  • Earlier work applied PPO to diffusion denoising and showed policy gradients could pass through Diffusion Policy, while DPPO embeds denoising within the environmental MDP.

3 Preliminaries

The preliminaries define the environment MDP, policy-gradient optimization, diffusion-model sampling, and Diffusion Policy. They also note how the formalism extends to pixel observations and how value estimates reduce gradient variance.

  • An environment MDP consists of states, actions, an initial-state distribution, transition probabilities, and rewards.
  • At each timestep, the agent observes a state, samples an action, transitions according to environment dynamics, and receives a reward.
  • Policy gradients improve policy parameters using discounted future rewards and action-likelihood gradients; an advantage estimate can replace returns.
  • The framework can represent partially observed settings in which actions depend on observations such as pixels rather than ground-truth states.
  • Diffusion models generate samples by reversing a Gaussian noising process, with a neural network predicting added noise at each denoising step.
  • Diffusion Policy conditions denoising transitions on state and lacks an explicit likelihood for the final action pθ(a0 | s), unlike standard unimodal Gaussian policies.

4 DPPO: Diffusion Policy Policy Optimization

DPPO embeds diffusion denoising within an environmental MDP, making each denoising transition amenable to policy-gradient optimization. The framework instantiates PPO with denoising-aware advantages and practical sampling, noise, and architecture choices for stable and efficient fine-tuning.

  • Core formulation: DPPO differentiates through action likelihood and applies policy-gradient updates to a diffusion policy.The framework uses PPO as a policy-gradient improvement.
  • Core formulation: The denoising process is modeled as a multi-step MDP embedded within the environmental MDP, forming a larger Diffusion Policy MDP.States and actions alternate between denoising transitions and environment dynamics, with rewards supplied when the denoised action is taken.
  • Core formulation: Each denoising step has a Gaussian likelihood, allowing the full diffusion-policy MDP to be optimized with policy gradient.The denoising process is sampled through the usual Diffusion Policy forward pass, beginning from environmental initial states combined with Gaussian noise.
  • PPO formulation: Denoising-discounted advantages downweight noisier steps, while the value estimator uses only the environmental state component for more efficient and stable training.Because rewards are zero at intermediate denoising steps, values at later denoising states need not be estimated.
  • Best practices: Fine-tuning only the last denoising steps or using DDIM reduces training cost and memory, while preserving final performance in the reported experiments.DDIM can use far fewer sampling steps; training uses η = 1 for exploration and evaluation uses η = 0.
  • Best practices: DPPO improves exploration and stability by clipping diffusion noise to minimum values for action sampling and likelihood evaluation.The reported settings use σexp_min values such as 0.01–0.1 for exploration and σprob_min = 0.1 for likelihood evaluation.

5 Performance Evaluation of DPPO

DPPO is evaluated across locomotion, kitchen, robot manipulation, long-horizon furniture assembly, and sim-to-real settings. Across these comparisons, it generally provides stable fine-tuning and strong performance, including with pixel observations and challenging long-horizon tasks.

  • Evaluation settings: DPPO is evaluated across GYM, FRANKA-KITCHEN, ROBOMIMIC, FURNITURE-BENCH, and real-hardware settings.The evaluation includes state and pixel inputs, sparse and dense rewards, and long-horizon multi-stage manipulation.
  • Diffusion-based RL comparisons: DPPO performs consistently with strong training stability across diffusion-policy fine-tuning tasks, while many diffusion-based baselines are less stable.In ROBOMIMIC, DPPO is strongest especially on Transport; several off-policy methods perform worse or drop at the beginning of fine-tuning.
  • Policy parameterizations: DPPO outperforms Gaussian and GMM policies on challenging ROBOMIMIC tasks, reaching > 90% success on state-based Transport.With pixel inputs, DPPO trains faster and reaches higher accuracy on Square and Transport, while Gaussian remains at 0% pretrained success on Transport.
  • Long-horizon manipulation: DPPO improves performance in all six FURNITURE-BENCH settings, whereas Gaussian-MLP collapses to zero success in several medium-randomness and Round-table conditions.The comparison uses three furniture-assembly tasks and evaluates success over fine-tuning iterations.
  • Sim-to-real transfer: 80% real-world success was achieved by DPPO after zero-shot transfer on One-leg, compared with 0% for the fine-tuned Gaussian policy on hardware.DPPO achieved 16 out of 20 hardware trials; the Gaussian policy reached 88% in simulation after fine-tuning but failed entirely on hardware.
  • Ablations: DPPO’s ablations identify state-dependent value estimation, clipped denoising noise, and appropriate action-chunk configuration as important training choices.The reported studies link these choices to performance, exploration, or stability in challenging settings.

6 Understanding the performance of DPPO

DPPO’s gains arise from structured exploration near the expert-data manifold, progressive multi-step denoising, and robustness to perturbations. These properties support stable fine-tuning and improved deployment behavior, while the benefit depends on the exploration regime and pre-training coverage.

  • Robust and generalizable fine-tuned policy: On hardware, fine-tuning improves insertion reliability and produces corrective behavior compared with pre-trained-only policies.Representative rollouts include a pre-trained failure from imprecise insertion and successful fine-tuned corrective behavior.
  • Structured, on-manifold exploration: DPPO induces structured exploration near the pre-training data manifold, unlike less structured Gaussian and narrower-coverage GMM exploration.In the Avoid environment, DPPO samples trajectories with wide coverage around the expert data manifold during the first fine-tuning iteration.
  • Structured, on-manifold exploration: Diffusion’s repeated noise injections expand coverage while denoising pushes actions toward the expert-data manifold.The diffusion parameterization and action-chunk denoising process make DPPO’s stochasticity structured across action dimensions.
  • Scope of the mechanism: On-manifold exploration may hinder fine-tuning when aggressive, unstructured exploration is needed, as suggested by DPPO’s slight underperformance in Lamp with Low randomness.The authors expect the greatest benefit when pre-training sufficiently covers relevant success modes.
  • Training stability from multi-step denoising: With at least four denoising steps, DPPO remains robust to injected action noise while Gaussian and GMM performance collapses.The comparison uses fine-tuning after pre-training with M2 and gradually increases noise during training.
  • Training stability from multi-step denoising: DPPO progressively updates action distributions through denoising steps, preserving iterative refinement and contributing to training stability.The distribution does not collapse prematurely to optimal actions at only the first or final denoising step.
  • Robust and generalizable fine-tuned policy: DPPO fine-tuned policies are more robust to action perturbations and converge toward near-optimal paths from a larger distribution of initial states.The robustness evaluation perturbs actions after fine-tuning and varies the initial-state distribution.

7 Conclusion and Future Work

DPPO fine-tunes pre-trained Diffusion Policies with policy gradients by optimizing the full denoising chain. The paper reports efficient benchmark performance, strong deployment robustness, and sim-to-real transfer in a long-horizon manipulation task.

  • 7 Conclusion and Future Work: DPPO fine-tunes a pre-trained Diffusion Policy with policy gradients while leveraging the sequential denoising process and entire chain of diffusion MDPs.The framework is presented for fine-tuning diffusion-based robot policies.
  • 7 Conclusion and Future Work: DPPO exhibits structured online exploration, strong training stability, and robustness and generalization at deployment.These properties are reported across the paper’s RL and robotics evaluations.
  • 7 Conclusion and Future Work: DPPO demonstrates effective fine-tuning across RL and robotics benchmarks and strong sim-to-real transfer in a long-horizon, multi-stage manipulation task.The paper reports this transfer from simulation-trained policies to robot hardware.
  • 7 Conclusion and Future Work: The authors identify lower sample efficiency than off-policy methods as DPPO’s main limitation.They suggest combining DPPO with planning or video-prediction tools as future directions for addressing this limitation.

A.1 RL training of robot policies with offline data

Robot RL commonly combines offline behavior-cloning pre-training with online fine-tuning, while other approaches use replay-buffer integration, offline RL, residual policies, or diffusion-specific action updates. DPPO differs by applying policy gradients directly to a diffusion policy without fine-tuning regularization.

  • Offline pre-training and online fine-tuning: A standard offline-to-online pipeline pre-trains a robot policy with behavior cloning and then fine-tunes it online with RL.DPPO follows this pre-training-plus-online-fine-tuning approach.
  • Offline pre-training and online fine-tuning: Regularization is often used to keep fine-tuned policies close to their base policies, but DPPO omits it because on-manifold exploration maintains natural behavior.This distinguishes DPPO from regularized offline-to-online methods.
  • Residual and hybrid approaches: Residual-policy methods learn an RL correction over a frozen base policy, but a cited one-step residual approach lacks DPPO’s structured on-manifold exploration.The residual approach is fully closed-loop but differs in its exploration behavior.
  • Off-policy and offline-RL alternatives: Alternative methods add offline data directly to replay buffers for online off-policy learning or switch from offline RL to online RL while retaining offline samples.Examples include RLPD, Cal-QL, and related replay-buffer approaches.
  • Diffusion-policy RL: DIPO updates sampled actions with critic gradients and fits the actor afterward, while QSM argues that optimizing the full denoising-chain likelihood can be inefficient.DPPO reports contrary findings in its fine-tuning setting.
  • Diffusion-policy RL: Most diffusion-based RL methods update diffusion actors using approximated state-action Q functions, whereas DPPO uses policy gradients despite the multi-step denoising challenge.The paper notes that inaccurate Q estimates can bias actor updates and contribute to collapse.

B Additional details of DPPO implementation

DPPO pre-trains a diffusion policy with behavior cloning, then applies a PPO-style fine-tuning objective with environment-step advantage estimation and careful treatment of denoising steps. Implementation choices address exploration, stability, runtime, and comparisons with demo-augmented baselines.

  • Core training procedure: DPPO takes a behavior-cloned diffusion policy as input and fine-tunes it using a PPO-style loss with denoising-aware treatment.The policy, value function, and rollout buffer are initialized before parallel environment rollouts and updates.
  • Advantage estimation: Generalized Advantage Estimation is computed at environment steps and then discounted across denoising steps for policy updates.The advantage estimator interpolates between one-step temporal-difference estimates and Monte Carlo returns.
  • Core training procedure: DPPO collects denoising transitions during parallel environment interaction and performs repeated minibatch updates governed by the replay ratio.The implementation records denoising-step states, actions, and rewards in the rollout buffer.
  • Fine-tuning details: Fine-tuning uses a clipped policy objective, can update only the last K′ denoising steps, and trains a value function to predict discounted returns.Early denoising steps may remain frozen when only a suffix of the chain is fine-tuned.
  • Stability and efficiency choices: Large batches compensate for gradients spanning both environment and denoising steps, while clipping fractions of 10% to 20% are targeted for efficient training.The authors report that smaller batches can improve sample efficiency on some ROBOMIMIC tasks.
  • Baseline comparisons: Using diffusion policies for RLPD and Cal-QL does not resolve their poor task performance, with both failing to solve the task in the reported comparison.The baseline comparison uses action-chunk sizes T_a = 1 or T_a = 4 and averages results over three seeds.

C.2 Ablation studies on design decisions in DPPO

The ablations identify design choices that improve DPPO’s efficiency, exploration, and stability across control and manipulation tasks. Environment-state-only advantage estimation, moderate denoising discounting, sufficient diffusion noise, and fine-tuning about 10 denoising steps are emphasized.

  • Choice of advantage estimator: Environment-state-only advantage estimation performs best on challenging manipulation tasks, although action-value baselines achieve higher reward on Hopper-v2.The authors attribute this pattern to the difficulty of estimating values for continuous, high-dimensional, stochastic diffusion actions.
  • Denoising discount factor: γ_DENOISE = 0.8 improves efficiency in Hopper-v2 and HalfCheetah-v2, whereas γ_DENOISE = 0.5 slows training.Smaller values downweight earlier denoising steps in the policy gradient.
  • Choice of diffusion noise schedule: Higher minimum diffusion noise preserves the two modes of pre-training trajectories while widening state-space coverage for exploration.The authors report that σ_exp^min = 0.1 is required to prevent policy collapse and that higher noise improves coverage in Square.
  • Choice of fine-tuned denoising steps: Fine-tuning 10 denoising steps provides the best overall efficiency, while fine-tuning too few steps can reduce asymptotic performance and slow convergence.GPU memory usage scales linearly with K′, and comparable behavior appears in Avoid.
  • Design interaction: Denoising discounting acts as a soft counterpart to limiting the number of fine-tuned denoising steps, allowing efficiency to be adjusted flexibly.The preferred setting depends on how much the pre-trained action distribution must change.

C.3 Effect of expert data

DPPO can leverage varying amounts of expert data for fine-tuning and remains effective even without expert-data pre-training. Its advantage is especially visible with limited demonstrations and in comparisons on the Avoid task.

  • Varying expert demonstrations: DPPO better leverages pre-training data and fine-tunes to high success rates across Hopper-v2, Square, and One-leg.On One-leg, DPPO reaches 60% success from only 10 demonstration episodes.
  • Training from scratch: With no expert-data pre-training, Gaussian trains faster than DPPO, but DPPO still attains reasonable final performance in OpenAI GYM tasks.The comparison uses single-action chunks because larger chunks perform poorly when training from scratch.
  • Pre-training variants: In Avoid, DPPO with K′ ∈ {15, 20} achieves the highest performance among the compared methods and trains quickest in environment steps.It also appears to attain the greatest terminal performance after M1 and M2 pre-training.

C.5 Comparing to other policy parameterizations in the easier tasks from ROBOMIMIC

In the easier Lift and Can ROBOMIMIC tasks, DPPO is compared with Gaussian and GMM policies across architectures and state or pixel observations. DPPO achieves similar or better sample efficiency while exact-likelihood policy gradients are less practical.

  • Policy parameterizations: DPPO achieves similar or better sample efficiency than Gaussian and GMM policies across Lift and Can with state and pixel inputs.The comparison spans multiple architectures and observation modalities.
  • Exact action likelihood: Exact-likelihood diffusion policy gradients improve Hopper-v2 but do not outperform DPPO and reach zero success rate in Can.They also require more runtime and GPU memory because they backpropagate through the probability-flow ODE.
  • Exact action likelihood: Exact-likelihood policy gradients do not provide DPPO’s flexibility to fine-tune fewer denoising steps or discount early denoising steps.This limits direct control over the computational and optimization trade-offs explored in DPPO.
  • Structured exploration: Structured exploration with higher noise covers expert data modes without aggressively entering new modes, while final-step-only noise makes trajectories less structured.The degradation is especially apparent after M3 pre-training.
  • Structured exploration: The original DPPO exploration setup converges faster than adding noise only at the final denoising step in Can and Square.This supports structured exploration as a contributor to DPPO’s sample-efficiency advantage over Gaussian and GMM policies.

D Reporting of Wall-Clock Times

DPPO’s wall-clock cost depends on the comparison: it is faster than several diffusion-based RL methods in GYM tasks but slower than Gaussian and GMM baselines in ROBOMIMIC. The reported timings use standardized parallel environments and hardware settings.

  • Diffusion-based RL algorithms: DPPO trains 41%, 37%, and 12% faster on average than DAWR, DIPO, and DQL in GYM tasks.QSM, DRWR, and IDQL train 43%, 33%, and 7% faster than DPPO, respectively.
  • Timing protocol: The timing tables measure one training iteration using parallelized environments, task-specific episode lengths, and specified GPUs.GYM timings use 40 environments and an RTX 2080; ROBOMIMIC timings use 50 environments and an NVIDIA L40.
  • Policy parameterizations: With state input, DPPO-MLP trains 24%, 21%, 24%, and 22% slower on average than baselines across the reported ROBOMIMIC comparisons.The higher cost is attributed to more expensive diffusion sampling.
  • Experimental setup: DPPO fine-tuning is evaluated with common pre-trained policies and online experiences, while implementation choices include MLP, Transformer, UNet, and ViT architectures.The appendix also specifies normalization, optimization, replay, and environment settings used for the comparisons.

E.8 Additional details of FURNITURE-BENCH tasks and training in Section 5.4

FURNITURE-BENCH covers three multi-stage assembly tasks, trained first from simulated SpaceMouse demonstrations and then fine-tuned online. Real-hardware deployment uses AprilTag-based state estimation, domain randomization, and observation-processing safeguards.

  • Tasks: FURNITURE-BENCH includes One-leg, Lamp, and Round-table tasks, each requiring sequential placement, insertion, and screwing operations.The tasks differ in the furniture component assembled and the sequence of manipulation stages.
  • Pre-training: Pre-training uses approximately 50 simulated demonstrations collected with a 6 DoF SpaceMouse at 10 Hz, while no-op actions below ε = 0.05 are discarded.SpaceMouse delta actions are converted into absolute end-effector pose actions before storage for training.
  • Training: Policies use only the current normalized observation, with shared training settings including batch size 256, learning rate 1e-4 decayed to 1e-5, and weight decay 1e-6.Diffusion-based policies are trained for 8000 epochs, whereas Gaussian policies use 3000 epochs.
  • Fine-tuning: Fine-tuning samples online experience from 1000 parallelized IsaacGym environments, and evaluation runs 1000 episodes without extra exploration noise.A smaller exploration-noise setting is needed for pre-trained policies to achieve nonzero success rates at fine-tuning onset.
  • Hardware deployment: Hardware control converts desired end-effector pose sequences into joint targets through differential inverse kinematics, while AprilTags provide object poses in the robot frame.The estimation pipeline detects tags, maps them to the robot frame, applies known object offsets, and calibrates camera pose.
  • State estimation and sim-to-real: Real-world robustness depends on multiple cameras, suitable lighting, temporal filtering, multi-view averaging, cached grasped-part poses, and carefully chosen normalization ranges.Domain randomization matches measured hardware observation noise and adds zero-mean action noise with standard deviation 0.03 during simulation training.
Loading 2409.00588v3…