PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Based Online Decision Making via Generative Trajectory Planning

Haldun Balim, Yilun Du, Na Li

arXiv:2610.18677v1cs.LGcs.AIcs.ROcs.SYeess.SY

TL;DR

Online decision-making needs efficient inference without giving up explicit long-horizon reasoning. GTP uses a diffusion prior to propose action sequences that learned models evaluate and refine; across the reported benchmarks, it matches or exceeds strong baseline families, with stable performance where some baselines degrade.

  • Problem

    Value-based RL relies on short-horizon bootstrapping, while trajectory optimization requires costly high-dimensional optimization at each step, leaving a gap in efficient online long-horizon reasoning.

    Source: §1 · p. 1

  • Method

    GTP treats a learned generative model as a proposal distribution over action sequences, refining sampled candidates with learned value and dynamics models.

    Source: §1 · p. 2

  • Results

    GTP matches or exceeds model-free, diffusion-policy, and model-based baselines on locomotion benchmarks, improves sample efficiency on long-horizon manipulation, and scales to visual manipulation where standard RL makes no progress.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support using generative models as structured proposals for model-based planning rather than as policy replacements.

    Source: §1 · p. 2

    GTP relies on terminal value functions, motivating reduced reliance on them for sparse-reward settings or unreliable value estimates.

    Source: §5 · p. 10

Abstract

from arXiv · show

Online decision-making faces a tension between efficient inference and long-horizon reasoning. Value-based methods rely on short-horizon bootstrapping that limits the propagation of long-term information, while trajectory optimization methods plan explicitly but must solve a high-dimensional optimization problem at every decision step. We propose Generative Trajectory Planning (GTP), a model-based framework that addresses this tension by learning a generative model as a proposal distribution over action sequences. Instead of optimizing sequences from scratch, GTP samples candidate sequences from a diffusion prior and refines them using learned dynamics, reward, and value models, enabling efficient trajectory-level reasoning. Across continuous control and long-horizon manipulation tasks, GTP matches or exceeds state-of-the-art model-free, diffusion-policy, and model-based baselines, and remains stable in regimes where strong baselines degrade. Our results suggest that generative models are best understood not as policies, but as structured proposal distributions for planning, offering a new perspective on how to integrate generative modeling with model-based decision-making.

1 Introduction

Value-based RL and trajectory optimization each leave a gap in online long-horizon decision-making: one relies on short-horizon updates, while the other requires costly sequence optimization. The introduction proposes GTP, which uses generative action-sequence proposals and learned models to combine efficient inference with trajectory-level reasoning.

1 Introduction

  • RL’s short-horizon bootstrapping slows long-term information propagation, accumulates estimation errors, and encourages action-level rather than coherent exploration [32].Model-based RL often uses learned models indirectly, leaving decisions driven by local updates.Source: §1 · p. 1
  • Trajectory optimization enables explicit long-horizon reasoning but requires costly high-dimensional, non-convex optimization and can degrade under accumulated model error.Its optimizer may exploit prediction errors in learned dynamics.Source: §1 · p. 1
  • Online generative planning remains underused because prior work largely treats generative models as policy replacements trained with standard RL objectives [23] [26] [41].The introduction says this leaves their long-horizon and expressive benefits underutilized online.Source: §1 · p. 2
  • GTP samples action sequences from a learned generative proposal and refines them with value and dynamics models, using ensemble uncertainty for trajectory-level optimistic exploration.The contributions describe iterative beam search with diffusion-based refinement and report advantages across locomotion and manipulation benchmarks.Source: §1 · p. 2

2 Preliminaries

The preliminaries frame online decision-making as learning from interaction in an unknown continuous-state, continuous-action MDP, then describe model-predictive planning and diffusion models for action sequences. GTP uses learned models to evaluate and refine proposals from a diffusion prior.

2.1 Online Decision Making

  • The problem is to maximize expected discounted return in an infinite-horizon continuous-state, continuous-action MDP with unknown transition and reward models.The agent learns from online interaction without knowing those models.Source: §2.1 · p. 2
  • Model predictive control optimizes an action sequence from the current state, optionally bootstraps beyond its horizon with a terminal value, executes the first action, and repeats.Sampling-based methods iteratively refine a parametric proposal toward the objective.Source: §2.1 · p. 2
  • Online planning replaces unknown dynamics, reward, and terminal value with learned models, while balancing exploitation against uncertainty in the learned dynamics.The planning objective represents this trade-off through sequence-level uncertainty.Source: §2.1 · p. 3
  • GTP replaces simple parametric proposals with a reusable generative prior over high-return trajectories, then refines sampled candidates using learned dynamics, reward, and value models.The prior is instantiated as a diffusion model to represent high-dimensional, multimodal action distributions.Source: §2.1 · p. 3

2.2 Diffusion Models for Action Trajectory Generation

  • Diffusion generates action sequences by gradually noising them and then applying a learned reverse denoising process conditioned on the current state.Training weights set the target trajectory distribution: uniform weights recover standard diffusion training, while non-uniform weights shift it toward high-quality trajectories scored at the trajectory level.Source: §2.2 · p. 3

3 Generative Trajectory Planning

GTP samples candidate action sequences from a generative prior and evaluates them with learned models. Beam search and online-learning procedures support trajectory-level planning, while the prior update has a conditional convergence result.

3 Generative Trajectory Planning

  • GTP samples state-conditioned H-step action sequences from a generative prior and evaluates them with ensemble dynamics, reward, and terminal-value models.Source: §3 · p. 3

3.1 Planning with Generative Prior

  • The prior restricts the growing action-sequence search to plausible trajectories; beam search scores prefixes with discounted rewards and terminal value while refining future suffixes.Source: §3.1 · p. 4

3.2 Online Model Learning

  • Langevin ascent from planner elites injects novel high-scoring candidates to counter the mode collapse risk of sample-and-imitate prior training.Source: §3.2 · p. 5
  • Ensemble disagreement supplies trajectory-level uncertainty for optimistic exploration, while median, inverse-variance-weighted multi-step estimates provide robust value targets.Source: §3.2 · p. 5

3.3 Training Objectives

  • Dynamics and reward models use supervised regression with bootstrap-diverse ensemble members and a termination head; value learning uses planner trajectories and horizon-weighted targets.Source: §3.3 · pp. 5–6
  • The diffusion prior learns from high-scoring beam-search and Langevin trajectories, weighting elite examples by their planner scores and adapting temperature to maintain target effective sample size.Source: §3.3 · p. 6

3.4 Theoretical Analysis

  • Under a bounded objective, fixed positive temperature, and initial support for every δ-optimal set, idealized updates improve expected return and concentrate on δ-optimal sequences; practice uses beam search, adaptive temperature, and Langevin augmentation.Source: §3.4 · p. 6
  • GTP consistently outperforms baselines on OpenAI Gym MuJoCo evaluation-return curves, with means and standard errors shown over five random seeds.Source: Figure 3 · p. 7

4 Experiments

Experiments evaluate GTP on continuous-control, long-horizon manipulation, and visual manipulation, then examine its design choices, planning horizon, and exploration. Across these settings, GTP shows strong performance and sample efficiency against diverse baselines.

4 Experiments

  • Experiments compare GTP with model-free RL, diffusion-policy RL, trajectory-optimization, and model-based RL baselines across locomotion and manipulation tasks.Methods are trained for 1M environment interactions; GTP uses H = 4 for locomotion comparisons and H = 8 for manipulation.Source: §4 · p. 7, §4.1 · p. 7, §4.2 · p. 8

4.1 Locomotion Tasks

  • GTP-H4 outperforms all baselines across five MuJoCo environments, while GTP-H1 is competitive with diffusion-based RL methods.GTP-H4 also outperforms strong model-based baselines, and reaches high returns faster than all baselines except on Hopper.Source: §4.1 · pp. 7–8

4.2 Long-horizon Manipulation Tasks

  • Across Fetch and PushT-Vision, GTP converges reliably and achieves higher final performance than planning baselines; on PushT-Vision it reaches near-perfect success while learning-based baselines fail in the default regime.The manipulation table reports success rates; Fetch results also show faster convergence and lower variance, while the PushT-Vision results extend to image-based decisions.Source: Table 2 · p. 8, §4.2 · pp. 8–9, Figure 4 · p. 8

4.3 Analysis

  • Removing Langevin augmentation causes mode collapse, while Gaussian augmentation recovers much—but not all—of GTP’s performance; single-batch filtering also substantially underperforms iterative planning.The TD-1 variant has lower final returns than trajectory-level value targets, consistent with slower long-horizon information propagation.Source: §4.3 · p. 9
  • A single dynamics model is catastrophic, and non-conservative updates destabilize value learning; conservative rollouts reduce sample efficiency by suppressing exploration.The conservative update down-weights ensemble disagreement, while the conservative rollout has competitive final returns but trails GTP through most of training.Source: §4.3 · p. 9
  • Increasing the planning horizon to H = 4 improves HalfCheetah and FetchSlide, but longer horizons degrade HalfCheetah and plateau on FetchSlide.The authors attribute the task difference to dynamics accuracy in contact-rich locomotion versus simpler manipulation dynamics; training remains stable at H = 16 on manipulation tasks.Source: §4.3 · pp. 9–10, Figure 7 · p. 9
  • On PushT, GTP’s visitation shifts toward the object’s contour and contact regions, unlike SAC and DPMD’s rare object interaction or TD-MPC2’s broad, indiscriminate exploration.The analysis uses object-centric end-effector visitation heatmaps to compare exploration over training.Source: Figure 6 · p. 9, §4.3 · p. 10
  • The experiments support GTP’s use of generative trajectory priors as planning proposals for structured exploration and long-horizon reasoning across locomotion and manipulation tasks.The authors report that each design choice contributes to performance and that the approach improves sample efficiency and final performance across the evaluated tasks.Source: §4.3 · p. 10

5 Conclusion

GTP combines a learned generative trajectory prior with model-based planning and online learning. The paper identifies reliance on terminal value functions, rich visual tasks, and planning efficiency as directions for further work.

5 Conclusion

  • Across continuous-control and manipulation benchmarks, GTP shows strong sample efficiency and performance, while search-discovered trajectories are distilled into its prior to improve proposals through interaction.Ablations highlight iterative refinement, trajectory-level value learning, and structured exploration.Source: §5 · p. 10
  • Future work includes reducing reliance on terminal value functions, extending GTP to richer visual tasks, and improving planning efficiency and trajectory quality.The paper notes terminal values may be less reliable in sparse-reward settings.Source: §5 · p. 10

Appendix

  • The appendix covers implementation and experimental analyses, including ensemble aggregation, beam-search sensitivity, computational costs, trajectory reuse, and theoretical convergence guarantees with their assumptions.It also details image-observation handling and strategies for reducing inference-time cost.Source: Appendix · p. 14

A Related Work

  • Prior online RL methods remain largely policy-centric, using stepwise value-based decisions or short-horizon planning rather than coherent trajectory-level reasoning; GTP instead reasons over action sequences with a learned prior.The related-work discussion also contrasts local Gaussian exploration with structured variation over sequences.Source: A Related Work · p. 14
  • Offline diffusion work establishes sequence generation as useful when datasets or demonstrations provide supervision, whereas GTP generates its supervision online through planning.The paper cites Diffuser, classifier-free conditioning, and short action-chunk policies as examples of offline or imitation settings.Source: A Related Work · p. 15

B.1 Networks and Training

The appendix specifies GTP’s model architecture and training objectives, adaptive prior learning, planning configuration, and Langevin trajectory augmentation. Together, these choices support model-based evaluation and iterative refinement of generated sequences.

B.1 Networks and Training

  • The diffusion prior is a FiLM-conditioned temporal U-Net with 20 denoising steps; the dynamics ensemble has five residual-predicting MLPs, alongside reward, termination, and clipped double-V critics.The reward model uses two heads, and the critic target is updated by Polyak averaging at τ = 0.005 every two gradient steps.Source: §B.1 · p. 15
  • Each dynamics model is trained on an independent bootstrap resample to encourage ensemble diversity while predicting next states from state-action pairs.The objective is one-step prediction error.Source: §B.1 · p. 15
  • The shared reward and termination model learns rewards by regression and termination logits by binary cross-entropy, so predicted termination can modulate accumulated rewards and terminal-value bootstrapping.The reward and termination heads are trained jointly.Source: §B.1 · p. 15
  • The prior temperature λ is adapted to target a fixed effective sample size over elite weights, avoiding hand-scheduling as beam-search returns tighten while the prior sharpens.The target ESS lies in (1/M, 1], with 1 corresponding to uniform weights.Source: §B.1 · pp. 15–16
  • Training linearly increases the planning horizon from 1 to its final value, letting predictive models stabilize on short horizons before supporting longer plans.The final horizon is reached at 300,000 environment steps for MuJoCo and 350,000 for manipulation.Source: §B.1 · p. 16

B.2 Planning and Sampling

  • Deployment samples 32 candidate sequences through beam search, using a narrower single-beam configuration for HalfCheetah-v4; performance is reported as largely robust to beam-search hyperparameters.For terminating environments, updates use minimum aggregation and rollout selection uses the median; non-terminating HalfCheetah-v4 and manipulation use median for updates and maximum for rollout selection.Source: §B.2 · pp. 16–17

C Augmenting Planned Trajectories via Langevin Dynamics

  • Prior sharpening can collapse elite samples onto the current mode, while isotropic action noise is objective-uninformed and can produce temporally incoherent sequence perturbations.The prior must also retain stochasticity to support sampling-based planning at deployment.Source: C Augmenting Planned Trajectories via… · p. 17
  • Langevin augmentation ascends the planning objective with respect to full action sequences, adding candidates that are distinct from the prior yet score highly under the model.The step size and noise scale are coupled, and the noise standard deviation is scheduled from 0.2 to 0.1 as the horizon grows.Source: C Augmenting Planned Trajectories via… · p. 17

D Aggregation Sensitivity and Failure Modes

On termination-sensitive Ant-v4, optimistic ensemble aggregation degrades performance, while a more pessimistic scheme substantially improves stability and return. The authors interpret aggregation as an implicit risk preference over model uncertainty.

D Aggregation Sensitivity and Failure Modes

  • On termination-sensitive tasks, model errors can compound into unstable states that trigger early termination, while maximum aggregation favors high-return plans supported by only one ensemble member.The default median-update, maximum-rollout configuration degraded performance in Ant-v4.Source: D Aggregation Sensitivity and Failure… · p. 17
  • Pessimistic aggregation substantially outperforms optimistic aggregation on Ant-v4, recovering strong performance with minimum aggregation for updates and median aggregation for rollout evaluation.The figure compares the pessimistic min/median scheme with optimistic median/max aggregation.Source: D Aggregation Sensitivity and Failure… · pp. 17–18, Figure 8 · p. 18
  • Aggregation sets an implicit risk preference: optimistic choices emphasize best-case predictions, whereas minimum aggregation approximates a worst-case objective across ensemble models.The paper relates this interpretation to robust MPC and quantile-based risk-sensitive control.Source: D Aggregation Sensitivity and Failure… · p. 18

E Beam Search Parameters

Under a fixed sampling budget on HalfCheetah-v4, performance is largely insensitive to the tested beam-search settings. Most tested configurations exceeded the strongest reported baseline.

E Beam Search Parameters

  • Seven of eight beam-search configurations exceeded the strongest reported baseline on HalfCheetah-v4 under a fixed budget of 32 trajectories; the remaining configuration was still within range.The table reports best mean returns and standard errors over 200k iterations and five random seeds for different beam widths and beam counts.Source: Table 4 · p. 18, E Beam Search Parameters · p. 18

F PushT-Vision: Environment Details and Extended Training

The extended PushT-Vision evaluation finds persistent failure among five baselines, while TD-MPC2 eventually ties GTP after more interactions. The section also specifies the task and the 3M-interaction evaluation setting.

F PushT-Vision: Environment Details and Extended Training

  • PushT is a planar task where a circular end-effector pushes a T-shaped block from a randomized initial pose to a fixed target pose.Observations are 96 × 96 RGB images, actions are velocity commands, and success requires intersection-over-union above 0.95.Source: F PushT-Vision: Environment Details… · p. 18
  • All baselines and GTP were rerun for 3M environment interactions with 5 seeds, 4 parallel environments, and 100 evaluation trials per seed.Figure 9 shows evaluation returns; Table 5 reports success rates.Source: Figure 9 · p. 19, Table 5 · p. 19
  • 0% success persists for SAC, TD3, QSM, DPMD, and DreamerV3 at 3M interactions, while TD-MPC2 eventually reaches a statistical tie with GTP after substantially more interactions.The authors attribute the gap to GTP’s learned diffusion prior supplying coherent action sequences from the outset.Source: F PushT-Vision: Environment Details… · p. 19, Table 5 · p. 19

G Baseline Implementations

The baseline implementations generally follow original recommendations and shared evaluation settings, with custom details for CEM planners and methods handling images. GTP learns its image representation jointly with predictive models, while other baselines use reconstruction-only latents or native image architectures.

G Baseline Implementations

  • Baselines use recommended hyperparameters and the shared evaluation protocol, with SAC and TD3 implemented in JAX and DPMD, QSM, and DreamerV3 using original code or published settings.TD-MPC2 was reimplemented in JAX while retaining its original architecture, objectives, and hyperparameters.Source: G Baseline Implementations · p. 19
  • CEM-MPC uses horizon H = 20 without terminal value, while CEM-MPC-V adds a learned terminal value and shares GTP’s horizon and value function to isolate the action prior.Both variants share GTP’s learned dynamics and are tuned across sample count, elite fraction, iterations, and noise schedule.Source: G Baseline Implementations · pp. 19–20

H Handling Image Observations

  • GTP handles images by encoding frames into a compact latent jointly trained with dynamics and reward models, adding reconstruction loss to their predictive losses.This lets predictive objectives shape representation learning, rather than relying on reconstruction alone.Source: H Handling Image Observations · p. 20
  • TD-MPC2 and DreamerV3 use their native image architectures; other baselines use the GTP encoder–decoder trained only with replay-buffer reconstruction loss.The encoder–decoder maps images to a latent used in place of the state input.Source: H Handling Image Observations · p. 20

I Computational Cost

GTP’s action selection time is similar to TD-MPC2 at the same horizon, but its training updates cost substantially more. The section characterizes these costs under matched hardware and batch settings.

I Computational Cost

  • Timing measurements used one NVIDIA RTX 5090 GPU, averaged 1,000 steps after warm-up, and matched baselines on batch size 256.The measurements distinguish per-step inference cost from per-update training cost.Source: I Computational Cost · p. 20
  • 34.25 ms per action for GTP (H = 8) versus 36.20 ms for TD-MPC2 at the same horizon, despite GTP’s diffusion denoising and beam search.The generative prior proposes promising trajectories with fewer candidates than Gaussian sampling-based planners.Source: I Computational Cost · p. 20, Table 6 · p. 20
  • 157.15 ms per update for GTP (H = 8) versus 3.99 ms for TD-MPC2 at the same horizon, reflecting the cost of generating elite trajectories through beam search.The authors note this trade-off is relevant when environment interactions are expensive.Source: I Computational Cost · p. 20, Table 6 · p. 20

J Efficient Training via Trajectory Reuse

Trajectory reuse caches planner-generated elite trajectories, refreshing them every 10 gradient updates instead of regenerating them each time. It reduces update cost while preserving comparable learning performance across four manipulation tasks.

J Efficient Training via Trajectory Reuse

  • Trajectory reuse addresses GTP’s update overhead by reusing previously optimized elite trajectories rather than generating fresh elites at every iteration.The overhead arises from beam-search generation of elite trajectories during gradient updates.Source: J Efficient Training via Trajectory… · p. 21
  • Comparable learning curves and final success rates across all four manipulation tasks show that frequent elite regeneration is not essential for effective learning.Figure 10 compares the original training procedure with trajectory reuse.Source: J Efficient Training via Trajectory… · p. 21
  • 157.15 ms to 19.12 ms per GTP (H = 8) update, an 8.2× speedup, with trajectory reuse.The variant refreshes cached elites every 10 gradient updates, reducing elite-generation frequency tenfold while preserving the same training objective.Source: J Efficient Training via Trajectory… · p. 21

K Efficient Inference

GTP tests two ways to reduce inference-time planning: removing explicit planning and executing multiple actions between replanning steps. Planning-free inference reduces action-selection time while maintaining comparable success, and multi-step execution preserves strong performance with limited degradation.

K Efficient Inference

  • 8.61 ms versus 34.25 ms action selection time (4.0× speedup) with comparable success rates when inference-time planning is removed on FetchPick&Place.The evaluation includes a single-candidate variant without beam search.Source: K Efficient Inference · p. 21, Table 7 · p. 22
  • GTP preserves strong FetchPick&Place performance when it executes multiple actions before replanning.The evaluation uses a model trained with planning horizon H = 8.Source: K Efficient Inference · p. 21, Table 8 · p. 22
  • Multi-step execution reduces planning frequency with limited performance degradation at deployment, while per-step replanning remains preferable during online training.Inaccurate dynamics and rapidly changing policies can amplify prediction errors.Source: K Efficient Inference · pp. 21–22

L Convergence Analysis of the Generative Prior

This section analyzes the convergence properties and practical update mechanisms of the GTP prior, including the assumptions and limits of its guarantees. It also documents asset use and reproduction instructions and states that the paper involves neither human-subject research nor LLMs in its core methodology.

L.2 Convergence Analysis

  • Under a fixed planning objective, infinite sampling, and exact diffusion fitting, the idealized update guarantees monotonic improvement and asymptotic convergence toward optimal trajectories.The analysis does not guarantee convergence with finite-budget beam search, imperfect function approximation, or jointly learned dynamics and value models.Source: L Convergence Analysis of the… · p. 22, §L.2 · p. 23, §L.3 · p. 24

L.1 Idealized Prior Update

  • Reweighted diffusion training [23] yields an exponentially tilted prior, equivalent to KL-regularized improvement over trajectory distributions and related to policy mirror descent [37].This characterization assumes a fixed objective, infinitely many sampled trajectories, and sufficient diffusion-model capacity with exact optimization.Source: §L.1 · p. 22

L.3 Relation to GTP

  • Beam search proposals are reweighted by exp(J(a)/λ); distillation achieves an expected planning objective at least as high as the proposals, and is monotonic when their objective is no lower than the current prior’s.This interprets beam search as an improvement mechanism whose discoveries are distilled into the generative prior.Source: §L.3 · pp. 23–24
  • Targeting a fixed effective sample size controls the prior update’s size in χ2 divergence, limiting excessively concentrated importance weights while shifting the prior toward higher-return trajectories.In the population limit, normalized ESS is related to the divergence between the updated prior and its proposal distribution.Source: §L.3 · p. 24
  • Langevin perturbations add support beyond the current prior and direct exploration toward high-scoring regions, helping mitigate mode collapse and preserve opportunities to find improved solutions.Without added support, the idealized update cannot recover optimal regions lost by the prior, violating the positivity assumption required for convergence.Source: §L.3 · p. 24

NeurIPS Paper Checklist

Related papers

Loading 2610.18677v1…