PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Conditional Generative Modeling all you need for Decision-Making?

Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, Pulkit Agrawal

arXiv:2211.15657v4cs.LGcs.AI

TL;DR

The paper asks whether conditional generative modeling can directly solve sequential decision-making without the complexities of traditional offline RL. It introduces Decision Diffuser, a return-conditional diffusion policy using classifier-free guidance, and reports stronger performance than existing offline RL approaches on standard benchmarks while also combining constraints and composing skills at test time.

  • Problem

    Offline decision-making must recover reward-maximizing trajectories from sub-optimal logged data, while value estimation and dynamic programming introduce instability and distribution-management challenges.

  • Method

    Decision Diffuser frames offline sequential decision-making as conditional generative modeling with a return-conditional diffusion model and classifier-free guidance with low-temperature sampling.

  • Results

    Decision Diffuser outperforms existing offline RL approaches on standard benchmarks and generates behaviors that flexibly combine constraints or compose skills during test time.

  • Takeaways & Limitations

    Conditional generative modeling provides a simpler decision-making pipeline that can target high returns and support flexible constraint combination and skill composition.

  • Takeaways & Limitations

    The work focuses on offline, state-based sequential decision-making; online fine-tuning and image-based extensions are left for future work.

Abstract

from arXiv · show

Recent improvements in conditional generative modeling have made it possible to generate high-quality images from language descriptions alone. We investigate whether these methods can directly address the problem of sequential decision-making. We view decision-making not through the lens of reinforcement learning (RL), but rather through conditional generative modeling. To our surprise, we find that our formulation leads to policies that can outperform existing offline RL approaches across standard benchmarks. By modeling a policy as a return-conditional diffusion model, we illustrate how we may circumvent the need for dynamic programming and subsequently eliminate many of the complexities that come with traditional offline RL. We further demonstrate the advantages of modeling policies as conditional diffusion models by considering two other conditioning variables: constraints and skills. Conditioning on a single constraint or skill during training leads to behaviors at test-time that can satisfy several constraints together or demonstrate a composition of skills. Our results illustrate that conditional generative modeling is a powerful tool for decision-making.

1 INTRODUCTION

The paper reframes offline sequential decision-making as conditional generative modeling, using diffusion models to stitch trajectories, maximize returns, satisfy combined constraints, and compose skills.

  • Motivation: Conditional generative modeling is motivated by its success in generating high-quality data across images, language, and mathematical solutions.The paper applies this broader generative-modeling success to decision-making using offline trajectory data.
  • Motivation: Offline decision-making seeks reward-maximizing trajectories by stitching sub-optimal reward-labeled trajectories from a fixed dataset.Existing offline RL methods use dynamic programming and value estimation for this stitching problem, but suffer from instability and distributional constraints.
  • Decision Diffuser: Decision Diffuser learns a return-conditional diffusion model and uses classifier-free guidance with low-temperature sampling to generate return-maximizing trajectories.The approach aims to avoid value estimation and dynamic programming while extracting high-performing behaviors from sub-optimal trajectories.
  • Conditioning beyond returns: The conditional diffusion framework can generate trajectories satisfying multiple constraints by combining constraints learned from separate training data.In the concentric-circle navigation example, the model generates trajectories whose final positions lie between the circles.
  • Conditioning beyond returns: Conditioning on skills enables Decision Diffuser to compose skills into novel behaviors during inference, rather than only imitating individual skills.The paper presents skill composition alongside return conditioning and constraint combination as uses of the same framework.

2 BACKGROUND

The background introduces MDPs and offline RL, then describes diffusion models and guidance methods as the generative-modeling foundations for Decision Diffuser.

  • Reinforcement learning: A discounted MDP is defined by initial-state distribution, state and action spaces, transitions, rewards, and discount factor.A stochastic policy generates trajectories of state-action-reward transitions, whose discounted rewards define the return.
  • Reinforcement learning: The standard RL objective is to find a policy maximizing expected trajectory return, while temporal-difference methods estimate optimal Q-values.Q-function estimation conditions on a starting state and action and uses a parameterized value function.
  • Reinforcement learning: Continuous-action RL additionally learns a parametric policy that supplies the maximizing action in the value-based objective.This introduces a separate policy objective that must be maximized.
  • Offline RL: Offline RL learns a return-maximizing policy from a fixed dataset collected by an unknown behavior policy, but naive temporal-difference learning can move policy visitation away from the dataset distribution.Function approximation, off-policy learning, and bootstrapping jointly create practical instabilities.
  • Diffusion probabilistic models: Diffusion models learn a data distribution through a predefined forward noising process and a trainable reverse denoising process.The forward process adds Gaussian noise, while the reverse process models denoising transitions.
  • Guided diffusion: Diffusion models can be conditioned on labels to generate samples with specified attributes, using classifier-guided or classifier-free guidance.Classifier-guided methods require an additional classifier trained on noisy data.

3 GENERATIVE MODELING WITH THE DECISION DIFFUSER

Decision Diffuser formulates offline decision-making as conditional diffusion modeling over trajectories, using return, constraint, or skill information to guide generation. It diffuses state sequences, recovers actions with inverse dynamics, and uses classifier-free guidance to plan and compose behaviors.

  • Conditional trajectory modeling: Decision Diffuser learns a conditional generative model of offline trajectories instead of relying on dynamic programming and value estimation.The model conditions trajectory generation on characteristics such as returns, constraints, or skills.
  • Diffusing over states: The diffusion process models noisy state sequences of length H, rather than directly diffusing actions.The authors motivate this choice by noting that actions can be discrete, high-frequency, and less smooth than states.
  • Diffusing over states: An inverse dynamics model converts consecutive generated states into actions, using the same offline data as the reverse diffusion process.This design is reported to improve performance over jointly diffusing states and actions.
  • Planning with classifier-free guidance: Classifier-free guidance with low-temperature sampling extracts high-likelihood trajectories conditioned on desired returns, without requiring Q-values or dynamic programming.The guidance perturbs the denoising process toward trajectory portions exhibiting the requested condition.
  • Conditional planning: The planning algorithm maintains a history of observed states, denoises a conditioned future state sequence, and applies inverse dynamics to select actions.Classifier-free guidance is applied during iterative denoising while the plan remains consistent with the observed history.
  • Conditioning beyond returns: The same conditional diffusion framework supports composing multiple constraints or skills at test time, provided the requested composition is feasible.The paper notes that strict conditional independence is not required empirically for feasible compositions.

4 EXPERIMENTS

Experiments evaluate Decision Diffuser on offline RL, constraint satisfaction, and skill composition, including ablations of guidance and action modeling. It matches or outperforms offline RL approaches, improves over diffusion baselines, combines constraints, and composes running gaits.

  • Offline Reinforcement Learning: Classifier-free guidance improves CondDiffuser over Diffuser in 2 of 3 environments, while Decision Diffuser improves over CondDiffuser across all 3.The ablation compares state-action diffusion without classifier guidance against Decision Diffuser with inverse dynamics.
  • Constraint Satisfaction: In Kuka Block Stacking, Decision Diffuser generates trajectories satisfying individual constraints and flexibly combines multiple constraints during testing.Training uses 10,000 expert demonstrations per constraint, while combined constraint demonstrations are withheld.
  • Offline Reinforcement Learning: Decision Diffuser either matches or outperforms current offline RL approaches on D4RL tasks using normalized average returns.Table 1 reports means and standard errors over 5 random seeds.
  • Offline Reinforcement Learning: Inverse dynamics for action prediction improves Decision Diffuser over jointly diffusing states and actions in all 3 environments.The paper also reports that CondMLPDiffuser performs worst among the diffusion models.
  • Skill Composition: When conditioned on both bounding and pacing, generated quadruped trajectories transition between the two gait classifications over time.The classifier identifies composed trajectories as bounding for 38.5% and pacing for 60.1% of the time.
  • Skill Composition: Decision Diffuser composes running skills at test time despite training on demonstrations of individual gaits.The Unitree-go-running evaluation uses datasets of 2,500 demonstrations for each gait.

5 RELATED WORK

Diffusion models generate data through iterative denoising and support conditional sampling through classifier-based or classifier-free guidance. This background motivates applying conditional generative modeling beyond image and text generation.

  • Diffusion Models: Diffusion models formulate data sampling as an iterative denoising procedure and can be interpreted through score matching.Conditional sampling may use classifier guidance or classifier-free guidance based on conditional and unconditional score functions.
  • Diffusion Models: Classifier-free guidance uses the difference between conditional and unconditional score functions to facilitate conditional sampling.The passage states that this approach has produced better conditional samples than classifier-based guidance.

6 DISCUSSION

Decision Diffuser uses conditional diffusion to generate high-return trajectories and flexibly compose constraints or skills, while discussion highlights practical assumptions and design choices.

  • Discussion: Decision Diffuser samples for high returns, captures the best dataset behaviors, and outperforms existing offline RL approaches on standard benchmarks.The approach frames offline sequential decision-making as conditional generative modeling and sidesteps reinforcement learning.
  • Discussion: The model can condition on constraints or skills and generate novel behaviors by combining constraints or composing skills at test time.These capabilities extend beyond return conditioning.
  • Discussion: Decision Diffuser can stitch sub-optimal Maze2D trajectories from A to B and B to C into near-optimal trajectories reaching C.The training dataset contains 500 trajectories for each segment, with maximum trajectory length 50.
  • Discussion: Low-temperature sampling with α = 0.5 gives the best average returns, whereas α = 0 eliminates entropy and α = 1.0 increases return variance.The reverse process samples x_k−1 from a Gaussian with reduced variance αΣ_k−1.
  • Discussion: Constraint composition requires at least one feasible trajectory satisfying all conditions; infeasible compositions produce incoherent behavior.The method empirically does not require strict conditional independence of conditioning variables.

E RUNTIME CHARACTERISTIC OF DECISION DIFFUSER

Decision Diffuser requires reverse-diffusion planning for each environment action, creating a performance–runtime trade-off that can be improved by warm-starting.

  • Runtime: Each action requires 100 reverse diffusion steps and takes 1.26s on average in wall-clock time.The measurement comes from planning in the D4RL Hopper-Medium-Expert environment.
  • Runtime: Figure A2 compares normalized average return against wall-clock planning time after warm-starting the reverse diffusion process.The figure is intended to show the trade-off between performance and planning time.

F WHEN TO USE INVERSE DYNAMICS?

Inverse dynamics is more useful when action trajectories are difficult to model directly: torque control produces higher-frequency actions, while position control produces smoother ones.

  • Setup: The Block Push state space has 10 dimensions, and success occurs when the red cube reaches the green circle.The environment includes gripper joint angles, velocities, center-of-mass coordinates, and cube position.
  • Setup: Position-control actions are smooth, whereas torque-control actions contain higher-frequency components.Both control types specify three-dimensional actions, but their temporal structure differs.
  • Setup: The offline datasets contain one million transitions collected from expert and random-action trajectories for each control type.Soft Actor-Critic first trains an expert policy for one million environment steps.
  • Results: Decision Diffuser and CondDiffuser perform similarly under position control, but CondDiffuser performs worse under torque control.The comparison uses the success rate of pushing the red cube to the green circle, averaged over five random seeds with standard error.

G ROBUSTNESS TO STOCHASTIC DYNAMICS

Decision Diffuser becomes less reliable as environment dynamics grow more stochastic because return conditioning can associate high returns with stochasticity rather than good actions.

  • Results: Decision Diffuser’s performance suffers as stochasticity increases, remaining ahead of Diffuser and CQL at p = 0.05 but becoming similar at higher p values.Performance is measured by red-cube success rate over five random seeds.
  • Setup: The stochasticity experiment randomizes each action with probability p ∈ {0, 0.05, 0.1, 0.15}.Otherwise, the policy action is executed.
  • Mechanism: Return-conditioned policies may learn sub-optimal actions when environment stochasticity produces high-return trajectories in the dataset.They cannot distinguish high returns caused by good actions from those caused by stochastic dynamics.
  • Future work: An alternative is to condition on predicted latent future states rather than returns to improve robustness to stochastic dynamics.The passage presents this as a possible direction for future work.

H KUKA BLOCK STACKING

The Kuka experiments test Decision Diffuser on block stacking, rearrangement, and composed running gaits. The method is evaluated on both in-distribution and out-of-distribution stacking configurations, with gait composition additionally verified quantitatively.

  • Kuka block stacking: Kuka stacking trains on demonstrations of sequentially stacking four blocks into a tower.At test time, the stacking task again requires constructing a four-block tower.
  • Kuka block stacking: Rearrangement requires placing blocks in relations that may form two towers, creating an out-of-distribution challenge.
  • Kuka block stacking: 0.0 success rate: goal-conditioned CQL and BCQ fail on block stacking and single-constraint rearrangement.
  • Running-gait composition: Decision Diffuser composes bounding, trotting, and pacing into new quadruped running behaviors at test time.It is trained on expert trajectories demonstrating individual gaits and composes the reverse-diffusion noise models during testing.
  • Running-gait composition: A gait classifier finds that composed trajectories assign the two source gaits the two highest probabilities across different timesteps.The evaluation generates 1000 trajectories for each individual gait and each possible gait pair.

J NOT COMPOSITIONS WITH DECISION DIFFUSER

Decision Diffuser supports NOT composition by perturbing its conditional diffusion noise to avoid a specified condition. In Kuka block stacking, this mechanism satisfies a constraint combined with the negation of its converse, while classifier-free guidance is preferred over Q-function guidance in practice.

  • NOT composition: NOT composition samples trajectories conditioned on avoiding a specified variable yj while using the set of available conditioning variables.
  • NOT composition: Decision Diffuser satisfies BlockHeight(i) > BlockHeight(j) AND NOT BlockHeight(j) > BlockHeight(i) in Kuka block stacking.
  • Guidance comparison: Classifier-free and Q-value-guided diffusion are theoretically equivalent, but classifier-free guidance performs better in practice.
  • Guidance comparison: Offline Q-functions can assign erroneously high values to out-of-distribution actions, a problem that is difficult to correct without online interaction.
  • Guidance comparison: Decision Diffuser composes conditioning variables through their score functions, unlike Decision Transformer’s next-action likelihood model.

M LIMITATIONS OF DECISION DIFFUSER

The paper identifies scope and performance boundaries for Decision Diffuser, including full observability, offline state-based settings, limited composition operators, stochastic dynamics, and limited data.

  • Observability and setting: Decision Diffuser assumes fully observable MDPs; naive extensions to partially observed settings may cause self-delusions.
  • Observability and setting: The experiments use state-based environments, although latent-space diffusion is proposed as a route toward image-based environments.
  • Composition limits: The method supports AND and NOT composition but cannot natively support OR composition without explicit density estimates for conditions.
  • Performance boundaries: In highly stochastic environments, Decision Diffuser loses its advantage and performs similarly to Diffuser and CQL.
  • Performance boundaries: Limited data can cause overfitting because diffusion models are prone to overfit in that regime.
Loading 2211.15657v4…