PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners

Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, Ping Luo

arXiv:2302.01877v2cs.LG

TL;DR

Limited diversity in offline training data restricts diffusion-based planners, particularly on insufficient-data and novel-task settings. AdaptDiffuser self-evolves by generating reward-guided synthetic demonstrations, filtering them with a discriminator, and fine-tuning on selected data. Experiments report improvements on seen benchmarks and adaptation to unseen tasks without additional expert data, while high-dimensional observation spaces and added training computation remain limitations.

  • Problem

    Limited diversity in offline training data restricts diffusion-based planning performance and generalization to new tasks.

  • Method

    AdaptDiffuser generates reward-guided synthetic expert data for goal-conditioned tasks, filters it with a discriminator, and fine-tunes the diffusion model using selected data.

  • Results

    AdaptDiffuser improves performance on existing tasks and adapts to unseen tasks without additional expert data.

  • Takeaways & Limitations

    The method supports self-bootstrapping diffusion-based planners across benchmark and designed KUKA and Maze2D tasks.

  • Takeaways & Limitations

    AdaptDiffuser increases training computation and remains challenged by high-dimensional observation-space tasks.

Abstract

from arXiv · show

Diffusion models have demonstrated their powerful generative capability in many tasks, with great potential to serve as a paradigm for offline reinforcement learning. However, the quality of the diffusion model is limited by the insufficient diversity of training data, which hinders the performance of planning and the generalizability to new tasks. This paper introduces AdaptDiffuser, an evolutionary planning method with diffusion that can self-evolve to improve the diffusion model hence a better planner, not only for seen tasks but can also adapt to unseen tasks. AdaptDiffuser enables the generation of rich synthetic expert data for goal-conditioned tasks using guidance from reward gradients. It then selects high-quality data via a discriminator to finetune the diffusion model, which improves the generalization ability to unseen tasks. Empirical experiments on two benchmark environments and two carefully designed unseen tasks in KUKA industrial robot arm and Maze2D environments demonstrate the effectiveness of AdaptDiffuser. For example, AdaptDiffuser not only outperforms the previous art Diffuser by 20.8% on Maze2D and 7.5% on MuJoCo locomotion, but also adapts better to new tasks, e.g., KUKA pick-and-place, by 27.9% without requiring additional expert data. More visualization results and demo videos could be found on our project page.

1. Introduction

Offline reinforcement learning aims to learn policies from previously collected data without ongoing interaction with the live environment.

  • Offline reinforcement learning learns policies from previously collected data.

I. Offline Trajectories

AdaptDiffuser addresses limited training-data diversity in diffusion-based planning through reward-guided synthetic demonstrations and discriminator-based self-evolution. It improves performance on seen tasks and supports adaptation to unseen tasks without additional expert data.

  • Limited diversity in diffusion-planner training data weakens performance, especially with insufficient expert data and new tasks.
  • AdaptDiffuser generates diverse synthetic demonstrations for seen and unseen goal-conditioned tasks using reinforcement-learning reward gradients during diffusion sampling.
  • A discriminator filters generated demonstrations, and selected high-quality data fine-tunes the diffusion model for self-evolution.
  • AdaptDiffuser improves diffusion-based planning on existing benchmarks and adapts to unseen tasks without additional expert data.
  • Experiments cover D4RL MuJoCo and Maze2D benchmarks plus newly designed KUKA and Maze2D unseen tasks.

2. Related Works

Related work frames offline reinforcement learning as sequence modeling and positions diffusion models as flexible trajectory generators for reward-guided planning. Existing methods provide conditioning and generation capabilities but motivate further adaptation to limited or novel task settings.

  • Offline Reinforcement Learning: Offline reinforcement learning learns behaviors from fixed data, while distribution deviation can cause standard off-policy methods to overestimate values.
  • Reinforcement Learning as Sequence Modeling: Sequence-modeling approaches represent reinforcement learning through trajectories of states, actions, rewards, and values.
  • Reinforcement Learning as Sequence Modeling: Diffuser generates trajectories with diffusion models and supports flexible reward-guided constraints during sampling.
  • Diffusion Probabilistic Model: Diffusion models generate data through iterative denoising and support flexible conditioning and compositional behaviors.

3. Preliminary

The preliminary section formulates offline reinforcement learning as trajectory optimization in a Markov decision process and describes diffusion-based sequence generation as iterative denoising.

  • Markov Decision Process: A Markov decision process is defined by state space S, action space A, transition function T, reward function R, and discount factor γ.
  • Trajectory Optimization: Trajectory optimization seeks an action sequence a*_0:T that maximizes expected return over a planning horizon T.
  • Diffusion-Based Sequence Generation: Diffusion-based sequence generation iteratively denoises trajectories through conditional transitions pθ(τ_i−1 | τ_i).
  • Diffusion-Based Sequence Generation: The sequence distribution factorizes across step-wise denoising transitions, with τ_0 denoting the original noiseless sequence.
  • Diffusion Model Training: Diffusion-model parameters are optimized by minimizing an evidence lower bound, using a simplified surrogate loss based on predicting noise or the Gaussian mean.

4. Method

AdaptDiffuser improves diffusion-based planning by self-evolving the trajectory model with reward-guided synthetic data, discriminator filtering, and iterative fine-tuning. It targets both existing goal-conditioned tasks and unseen tasks without additional expert supervision.

  • Method overview: AdaptDiffuser extends diffusion-based planning to improve performance on existing tasks and adaptability to unseen tasks without expert data.The method is designed for goal-conditioned decision-making and self-evolves through repeated data generation and model updates.
  • Task-oriented diffusion: Reward-guided diffusion generates trajectories conditioned on task objectives, using reward information to adjust state and action gradients during denoising.The formulation supports reward-to-go guidance, trajectory constraints, and single-point goal conditions.
  • Motivation: The method addresses a limitation of conditional guidance: generated trajectory quality also depends on the learned diffusion means and covariance, which depend on training-data quality.If the learned mean deviates substantially from the optimal trajectory, stronger guidance may still produce biased, low-quality results.
  • Self-evolution: Accepted synthetic trajectories are added to the training data and used to fine-tune the diffusion model through repeated iteration rounds.The iterative process is intended to continually improve performance and generalization to new tasks.
  • Self-evolved data: Synthetic demonstrations are generated for unseen tasks with reward guidance, then filtered by a discriminator using reward and dynamics consistency criteria.State sequences are converted into executable actions through inverse dynamics, and trajectories are rejected when revised states differ excessively from generated states.
  • Reward-guided generation: The reward-guidance procedure handles continuous rewards, sparse goal rewards, and combinations of goal and auxiliary constraints.For sparse rewards, goal constraints replace sampled values at diffusion timesteps; combined guidance addresses tasks requiring both goal achievement and intermediate objectives.

5. Experiment

Experiments evaluate AdaptDiffuser on Maze2D and MuJoCo benchmarks, hard navigation cases, reward-guided gold-coin navigation, KUKA pick-and-place, and iterative or data-limited settings. Across these evaluations, AdaptDiffuser improves planning quality, robustness, and adaptation relative to Diffuser and other offline RL baselines.

  • Maze2D benchmark: In hard Maze2D cases, AdaptDiffuser generates shorter, smoother paths and succeeds where Diffuser fails to find a feasible path.These cases use long planning paths in Maze2D-Medium and Maze2D-Large.
  • MuJoCo benchmark: AdaptDiffuser is competitive with or outperforms most offline RL baselines across Hopper, HalfCheetah, and Walker2d locomotion settings.It achieves higher reward than Diffuser in almost all datasets, with especially large improvements on Hopper-Medium and Walker2d-Medium.
  • Unseen Maze2D task: With an added gold-coin reward, AdaptDiffuser changes its route to pass through the coin while avoiding the wall collision produced by Diffuser.Without the additional reward, both methods choose the shorter bottom route; with it, both shift toward the middle route.
  • Unseen KUKA task: AdaptDiffuser greatly outperforms Diffuser on both KUKA pick-and-place setups and performs better when blocks begin stacked at a random location.The two setups differ in whether blocks start randomly on the floor or stacked at a random location.
  • Ablations: A second iterative phase significantly improves Medium-dataset performance, whereas later gains are not obvious for Medium-Expert data.The results attribute the difference to lower original data quality in Medium and broader expert coverage in Medium-Expert.
  • Ablations: Using 50% of the expert data and 25% of the training steps, AdaptDiffuser beats fully trained Diffuser on Maze2D-Large.The comparison uses the 100%D setting for fully trained Diffuser.

6. Conclusion

AdaptDiffuser improves diffusion-based planners through self-evolution, using synthetic data generation and filtering to enhance existing-task performance and adaptability to unseen tasks. The method is effective across offline RL benchmarks and unseen KUKA and Maze2D tasks, but adds training computation and remains challenged by high-dimensional observations.

  • AdaptDiffuser improves diffusion-based planners in offline reinforcement learning through self-evolution.It generates diverse synthetic expert data with reward-guided diffusion and filters infeasible data with a rule-based discriminator.
  • The method enhances performance on existing decision-making tasks, especially goal-conditioned tasks, and improves adaptation to unseen tasks without additional expert data.
  • The experiments validate AdaptDiffuser on offline RL benchmarks and unseen tasks in KUKA and Maze2D environments.
  • AdaptDiffuser increases training computation while adding almost no inference-time cost.
  • The method still faces challenges on tasks with high-dimensional observation spaces.

B. Visualization Results of KUKA Pick-and-Place Task

The KUKA visualization task requires sequentially moving four colored blocks from the right side of a table to the left in a specified order. The section also situates this evaluation alongside Maze2D and standard offline-RL baselines.

  • The KUKA robot must move green, yellow, blue, and red blocks from right to left in the order yellow, blue, green, red.The blocks begin at random positions on the right side of the table.
  • Figures 6–9 visualize the pick-and-place process for the green, yellow, blue, and red blocks, respectively.
  • The Maze2D experiments evaluate AdaptDiffuser on separate seen and unseen tasks.
  • The evaluation compares against IQL and Diffuser implementations across KUKA, Maze2D, and MuJoCo settings.

C.2. Metric Values for Reward Discriminator

AdaptDiffuser filters synthetic trajectories using environment-specific quality criteria, including trajectory length, reward, reward-length ratio, and successful placements. Training and inference use task-specific horizons, diffusion steps, and guidance settings.

  • Maze2D discrimination uses absolute reward, trajectory length, and reward-length ratio to prefer longer or more reward-efficient trajectories.The maximum episode lengths are 300, 600, and 800 for UMaze, Medium, and Large, respectively.
  • Maze2D-UMaze accepts trajectories with L > 200 or L > 50 and R + 1.0 ∗(Maxe −L) > 210.
  • Maze2D-Medium accepts trajectories with L > 450 or L > 200 and R + 1.0 ∗(Maxe −L) > 400.
  • Maze2D-Large accepts trajectories with L > 650 or L > 270 and R + 1.0 ∗(Maxe −L) > 400.
  • For KUKA, the discriminator requires R >= 2.0, meaning at least half of four placements succeed.
  • AdaptDiffuser generates synthetic data during training and performs one denoising process during inference.

D.2. Training-time Characteristic of AdaptDiffuser

AdaptDiffuser’s training time combines synthetic-data generation, diffusion-model training, and fine-tuning, while parallel generation and warm-starting reduce overhead. Its total pre-training time is higher than Diffuser’s, but inference remains comparable.

  • AdaptDiffuser training combines synthetic-data generation time with diffusion-model training time.
  • Fine-tuning starts from a pre-trained Diffuser model and uses one quarter of the usual training steps.
  • 54 hours is AdaptDiffuser’s average total pre-training time, compared with 41 hours for Diffuser.
  • Parallel generation makes collecting ten synthetic trajectories take the same time as collecting one trajectory in the described MuJoCo setting.

E. Comparison with Decision Diffuser

AdaptDiffuser is presented as a general self-evolving algorithm that can complement diffusion-planner improvements and perform on existing and unseen tasks. In MuJoCo, its normalized average return exceeds the compared methods.

  • AdaptDiffuser generates high-quality synthetic data with reward and dynamics-consistency guidance for diverse tasks.
  • AdaptDiffuser is compatible with improvements such as Decision Diffuser rather than conflicting with them.
  • 83.4 normalized average return is reported for AdaptDiffuser, compared with 74.7 for DT, 78.9 for TT, 77.5 for Diffuser, and 81.8 for Decision Diffuser.
  • AdaptDiffuser performs comparably to or better than Decision Diffuser in most datasets.

F.1. Adapt AdaptDiffuser to Maze2D Gold Coin Picking Task with Coin Locating Far from the Optimal Path

AdaptDiffuser adapts Maze2D trajectories to gold-coin locations far from the original optimal path, supported by diverse maze generation and changing start and goal points during training. The section also discusses extensions to high-dimensional observations.

  • Gold-coin adaptation: AdaptDiffuser generates a new route through a gold coin even when the coin is far from the optimal path.A described example places the coin at position (4,2), producing a route that passes through it.
  • Training mechanism: Changing the start and goal points multiple times during training helps cover more of the trajectory space.The authors connect this coverage to generating paths through different gold-coin locations during planning.
  • Scope and extension: For high-dimensional observations, the authors suggest encoding inputs into a latent space before applying AdaptDiffuser.They propose an embedding module such as an MLP, or an autoencoder-style representation, followed by latent-space planning.
  • Maze generation: ChatGPT is used to generate diverse Maze2D layouts, enabling varied path generation.The approach targets many distinct maze layouts, progressing from simple terrain to mazes with dead ends and loops.
  • Maze generation: Few-shot maze examples improve ChatGPT-generated maze quality, while successive prompts produce increasingly complex terrains.The prompts request legal 9*12 mazes with obstacle boundaries and 4-connected empty spaces, followed by additional dead ends, loops, and obstacles.
Loading 2302.01877v2…