PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Hengrui Zhang, Yuhu Cheng, C. L. Philip Chen, Xuesong Wang

arXiv:2609.34575v1cs.AI

TL;DR

Offline GCRL can have unreliable value guidance for distant goals, and hierarchy does not remove that high-level dependence. DSP uses classifier-free-guided diffusion to generate goal-directed subgoals, and the paper reports strong performance across challenging navigation and manipulation benchmarks. Its analysis removes one explicit value-noise path, but does not establish horizon-independent error.

  • Problem

    In offline GCRL, distant goals and limited data coverage can make value estimates noisy, while hierarchical methods still rely on value guidance for high-level subgoals.

    Source: §1 · p. 1

  • Method

    DSP learns conditional and unconditional velocity fields and uses classifier-free guidance to generate goal-directed high-level subgoals without explicit value-based guidance.

    Source: §1 · p. 2

  • Results

    DSP achieves strong performance across challenging OGBench locomotion and manipulation tasks, and experiments report gains across navigation and manipulation, particularly long-horizon mazes.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The results suggest guided generative subgoal modeling is effective across challenging offline GCRL tasks, particularly when navigation requires stable multi-step subgoal generation.

    Source: §6.2 · p. 9, §7 · p. 10

    DSP's error is not guaranteed to be horizon-independent: generated subgoal distributions may deteriorate with longer horizons or reduced offline data coverage.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Offline goal-conditioned reinforcement learning (GCRL) learns goal-directed policies from reward-free data, but in long-horizon tasks, goal-conditioned value functions often provide unstable guidance due to sparse rewards and discounting. Hierarchical methods partially mitigate this issue via subgoal decomposition; however, high-level decision-making still relies on noise-sensitive value estimates, leading to unstable behavior in complex environments. We address this limitation by proposing \textbf{D}iffusion \textbf{S}ubgoal \textbf{P}lanning (\textbf{DSP}), a diffusion-based framework for high-level subgoal generation. DSP casts high-level planning as guided generative inference over goal-conditioned subgoals and learns both conditional and unconditional flows, enabling classifier-free guidance to introduce a goal-directed bias at inference time. By removing explicit value-based guidance from high-level planning, DSP generates reachable and goal-directed subgoals through a generative model while retaining hierarchical execution. Experiments on offline GCRL benchmarks demonstrate that DSP outperforms prior methods on a range of navigation and manipulation tasks, with particularly strong performance in maze environments that require multi-step subgoal planning.

1 Introduction

Offline GCRL learns from reward-free trajectories, but distant goals and limited data coverage make value estimates noisy and unreliable in long-horizon tasks. DSP replaces explicit value-based high-level guidance with guided diffusion subgoal generation, and the paper reports gains across navigation and manipulation benchmarks, especially long-horizon mazes.

1 Introduction

  • Offline GCRL uses reward-free trajectories, yet distant goals and limited data coverage can make value estimates noisy and unreliable in long-horizon tasks.The paper frames this as a challenge for goal-directed learning from offline data.Source: §1 · p. 1
  • HIQL adds high- and low-level policies, but noisy value guidance can still impair complex navigation, spatial search, and high-dimensional manipulation [47] [1] [44].Related analyses attribute some failures to imprecise value guidance and high-level planning errors [48] [1].Source: §1 · pp. 1–2
  • DSP casts high-level planning as guided generative inference over goal-conditioned subgoals, using conditional and unconditional velocity fields with classifier-free guidance at inference.The paper reports gains across offline GCRL navigation and manipulation benchmarks, particularly in long-horizon mazes requiring multi-step subgoal planning.Source: §1 · p. 2

2 Related Works

Prior offline GCRL methods commonly optimize high-level subgoals with value-derived signals, which can be noisy. DSP instead uses guided generation for high-level subgoal decisions while retaining a separately trained low-level executor.

Offline GCRL

  • Offline GCRL methods use future-state goals and self-supervised relabeling, but commonly derive high-level subgoal signals from goal-conditioned value functions.The paper identifies noisy value estimates as a bottleneck for reliable long-horizon planning in offline settings.Source: §2 · p. 2

Hierarchical RL

  • Hierarchical RL uses reachable subgoals to simplify long-horizon planning, but graph methods can be costly to scale and value- or distance-based waypoint methods are sensitive to estimation accuracy.The sensitivity is especially relevant under sparse data coverage and offline settings.Source: §2 · p. 2

Diffusion Policies for RL

  • Unlike prior diffusion planning approaches that often combine generation with MPC, return guidance, or multi-step procedures, DSP uses guided generation for high-level subgoal decisions and retains a separate low-level executor.The paper positions DSP within diffusion applications to action generation, trajectory modeling, and planning.Source: §2 · p. 2

3 Preliminaries

The paper formulates offline GCRL as a discounted goal-conditioned decision process learned from a fixed trajectory dataset. It uses HIQL as a hierarchical baseline with value learning and advantage-weighted policy training.

Problem Setting

  • Offline GCRL is formulated as a finite-horizon discounted MDP over state, action, and goal spaces, with goals drawn at episode start and the objective of maximizing expected discounted return.The formulation assumes the goal space equals the state space.Source: §3 · p. 3
  • In the offline setting, policies are learned solely from a fixed trajectory dataset without further environment interaction.Source: §3 · p. 3

Hierarchical Implicit Q-Learning

  • HIQL learns goal-conditioned values from offline data using expectile regression and decomposes its policy into a high-level subgoal predictor and a low-level executor.Both levels are trained with advantage-weighted regression; their advantages use differences between the corresponding value estimates.Source: §3 · p. 3

4 High-Level Planning with Reduced Horizon-Dependent Value Noise

The section examines how noisy high-level values affect hierarchical planning, then analyzes and evaluates DSP as an alternative. Its central distinction is that DSP removes an explicit horizon-scaled value-noise path, not every possible source of horizon-dependent error.

4.1 Why Value-Function Noise Limits Long-Horizon Subgoal Planning

  • In the toy setting, learned value errors are modeled as growing with distance to the goal, and the analysis compares flat and hierarchical policy error.The example is one-dimensional, has left-or-right actions, a rightmost goal, and sparse reward.Source: §4.1 · pp. 3–4
  • Hierarchy reduces the effective decision horizon from T to T/k, but HIQL still uses value estimates for high-level subgoals, so horizon-dependent noise can remain a bottleneck.Source: §4.1 · p. 4

4.2 Diffusion Subgoal Planning Reduces Horizon-Dependent Value Noise

  • The matched-distance comparison controls for low-level execution difficulty; as T/k grows, HIQL's wrong-direction probability approaches 1/2, while DSP's ordering depends on its wrong-direction probability relative to HIQL's.The comparison isolates generative versus value-based high-level direction selection.Source: §4.2 · pp. 4–5

4.3 Empirical Evidence: DSP Generates More Reliable Subgoals

  • In antmaze-giant-navigate-v0, HIQLw/o trajectories can become trapped at local optima or intersect obstacles, whereas DSP paths are smoother, respect maze topology, and maintain progress.The visualization projects state space onto x-y coordinates over a goal-conditioned value heatmap, with brighter regions indicating higher values.Source: Figure 2 · p. 5, §4.3 · pp. 5–6

4.2 Diffusion Subgoal Planning Reduces Horizon-Dependent Value Noise

  • Proposition 4.2 isolates removal of HIQL's explicit T/k-scaled high-level value-noise path, but does not establish horizon-independent DSP error.The generated subgoal distribution can deteriorate as task horizon increases or offline data coverage decreases.Source: §4.2 · p. 5
  • The general error derivation allows arbitrary generated subgoal distributions, including non-Gaussian and multimodal ones; the Gaussian expression is only a closed-form special case.Source: §4.2 · p. 5

4.3 Empirical Evidence: DSP Generates More Reliable Subgoals

  • HIQL and HIQLw/o have comparable antmaze-giant-navigate-v0 performance, enabling the section's qualitative comparison of their subgoal trajectories.Source: §4.3 · p. 5

5 Diffusion Subgoal Planning

DSP replaces explicit value-guided high-level planning with controllable diffusion subgoal generation while retaining hierarchical execution. Conditional and unconditional generative flows enable goal-directed guidance at inference.

5.1 Subgoal Generation as Probabilistic Inference

  • DSP frames high-level planning as sampling subgoals from a posterior associated with reaching the final goal, rather than deriving the planner from a value function.The paper identifies value-estimation errors in that alternative as a source of horizon-dependent noise amplification and brittle planning.Source: §5.1 · p. 6

5.2 Learning the Goal-Directed Velocity Field

  • A deterministic flow maps prior noise toward future-state subgoals, using flow matching and conditional dropout to learn goal-conditioned and unconditional velocity fields.Training supervises the velocity along an interpolation path from prior noise to a dataset future state.Source: §5.2 · pp. 6–7

5.3 Guided Sampling as Implicit Policy Extraction

  • At inference, classifier-free guidance combines unconditional and goal-conditioned velocity fields, with ω controlling the strength of goal-directed guidance.The conditional-unconditional difference is interpreted as a goal-conditioning direction under the score approximation.Source: §5.3 · p. 7
  • Under future-state relabeling, the guidance density-ratio tilt corresponds to implicit relative-advantage-weighted subgoal extraction.The proposition relates the tilt to the behavior-policy value and advantage; the resulting value transformation preserves subgoal ordering without explicit high-level value estimation.Source: §5.3 · p. 7

5.4 Practical Algorithm

  • DSP pairs its guided-diffusion high-level planner with HIQL’s value function and low-level policy learning scheme.The high-level planner generates subgoals, while the low-level policy executes actions using value-based advantage-weighted updates.Source: §5.4 · p. 8

6 Experiments

DSP is evaluated on OGBench offline GCRL tasks spanning locomotion and manipulation, with additional analysis of guidance, data support, and executor reachability.

6.1 Experimental Setup

  • The OGBench evaluation covers locomotion and manipulation, and compares DSP with flat and hierarchical baselines using reported original hyperparameters and rerun released code.The locomotion suite includes point, ant, and humanoid embodiments across maze sizes and antsoccer; manipulation includes robotic-arm object interactions.Source: §6.1 · p. 8

6.2 Locomotion Results

  • DSP performs strongly across locomotion tasks, outperforming prior methods in point medium and large mazes, remaining competitive in giant mazes, and reaching near-optimal success in antmaze-medium and antmaze-large.It is also among the strongest methods on several humanoidmaze tasks and best or near-best on both antsoccer configurations; Table 1 reports success rates across test-time goals and seeds.Source: §6.2 · p. 9, Table 1 · p. 9

6.3 Manipulation Results

  • In manipulation, DSP shows clearer advantages on multi-step scene-play and puzzle tasks, while remaining competitive across the suite; GCIVL performs strongly on local cube tasks.The results characterize DSP as most useful when high-level planning requires stable multi-step subgoal generation.Source: §6.3 · p. 10

6.4 Guidance, Data Support, and Executor Reachability

  • Relative to ω = 1, family-default scales improve success by 23.2 and 27.0 percentage points on Ant-Giant and Scene-Play, respectively, while reducing valid-and-reached rates by 6.7 and 5.8 points.At ω = 10, support distance rises and validity, reachability, and success deteriorate; Table 2 reports the support, validity, valid-and-reached, and success measures.Source: §6.4 · p. 10, Table 2 · p. 10

7 Conclusion

DSP replaces value-based high-level subgoal guidance with conditional and unconditional generative modeling and classifier-free guidance, while retaining hierarchical execution. The conclusion reports strong OGBench locomotion and manipulation performance, alongside stated setting and sampling-cost boundaries.

7 Conclusion

  • DSP uses conditional and unconditional velocity fields with classifier-free guidance to bias generated subgoals toward goals, while a low-level policy uses implicit value learning.The conclusion presents guided generative subgoal modeling as a promising alternative to explicit value-based high-level planning.Source: §7 · p. 10

A Limitations and Future Work

  • DSP targets offline, state-based GCRL with meaningful local reachability; limited data coverage or goals far from behavior data may affect performance.Iterative diffusion sampling adds inference cost, although a modest number of steps suffices for strong performance in the experiments.Source: A Limitations and Future Work · p. 15
  • Future work may extend DSP to partially observable domains, speed subgoal sampling, or reduce dependence on task-specific data coverage through pretrained representations or transfer.Source: A Limitations and Future Work · p. 15

B.1 Proof of Proposition 4.1

  • The flat-policy error E(π) is the probability that noisy value at s + 1 is no greater than at s − 1; hierarchical-policy error is bounded by the sum of two noisy comparison probabilities.Source: §B.1 · p. 15

B.2 Proof of Proposition 4.2

The proof analyzes one-step errors under generated subgoals and connects classifier-free guidance to behavior-policy goal-reaching values. The appendices also describe goal sampling, architectures, hyperparameters, and computational resources.

B.2 Proof of Proposition 4.2

  • For arbitrary generated subgoals, Equation (42) gives exact one-step error without Gaussianity, symmetry, or unimodality; matched-distance analysis separates direction selection from low-level execution difficulty.Source: §B.2 · p. 16

B.3 Proof of Proposition 5.1

  • Under the score-approximation interpretation, classifier-free guidance links conditional and unconditional velocity fields to a subgoal distribution weighted by behavior-policy goal-reaching value.The transformed value is a strictly increasing transformation of the behavior-policy advantage and preserves its ordering over subgoals.Source: §B.3 · pp. 18–19

C.1 Goal Mixed Sampling

  • Training mixes goals at the current state, future states from the same trajectory, and arbitrary dataset states, with task-specific mixing ratios.Source: §C.1 · p. 19

C.2 Target Value Network

  • DSP retains target value networks to stabilize training, a technique the paper says prior studies found supports faster convergence with negligible computational overhead.Source: §C.2 · p. 19

C.3 Architectures

  • DSP operates directly in the original subgoal space without an explicit subgoal encoder, unlike most hierarchical baselines; HIQL without representation encoding is also evaluated.Source: §C.3 · p. 19
  • HIQL without subgoal representation encoding performs worse in high-dimensional settings, especially humanoidmaze, where encoding can noticeably improve performance.The authors note that computational constraints prevented a systematic ablation of architectural choices.Source: §C.3 · p. 19

D Hyperparameters

  • DSP uses 20 flow-integration steps globally, while guidance scale, policy temperature, and subgoal interval settings are shared within environment families.Tables 3 and 4 are identified as listing common and task-specific hyperparameters, respectively.Source: D Hyperparameters · p. 20, Table 3 · p. 20, Table 4 · p. 20

E Computational Resources

  • Training DSP on the largest locomotion environment takes approximately one hour on a single GPU; manipulation training takes around 40 minutes.Source: E Computational Resources · p. 21

F Environments and Datasets

The OGBench suite spans sparse-reward maze navigation, multi-goal ball manipulation, object rearrangement, sequential scene tasks, and combinatorial button puzzles. DSP is evaluated alongside offline GCRL and diffusion-planning baselines, with analyses of subgoal support, guidance, diffusion steps, subgoal horizons, and inference cost.

G Offline GCRL Baseline Algorithms

  • Baselines span hindsight-cloned behavior (GCBC), diffusion-guided action generation (CFGRL), value learners (GCIVL, OTA, Pi-HIQL), and HIQL’s latent subgoal hierarchy.Source: G Offline GCRL Baseline Algorithms · p. 22

H Subgoal Support and Executor Reachability

  • Family-default guidance improves success over ω=1 while moderately reducing valid-and-reached rate; ω=10 worsens support distance, validity, reachability, and success.Source: H Subgoal Support and Executor… · p. 23

I.1 Guidance Scale

  • In pointmaze-giant-navigate-v0, success rose from 11.0% to 43.4% as ω increased from 1.0 to 3.0–5.0, but excessively large guidance degraded performance.Guidance sensitivity shows an inverted-U trend, with task-dependent preferred strengths.Source: Figure 5 · p. 23, §I.1 · p. 23

I.2 Diffusion Step

  • Performance rose across tasks from N=1 to N=10 or N=20, then saturated or mildly declined at N=50, including antmaze-giant and scene-play.Inference time rose from about 1 second at N=1 to over 35 seconds at N=50; N=20 was near-optimal or optimal on most tasks at reasonable cost.Source: Figure 5 · p. 23, §I.2 · pp. 23–24

I.3 Subgoal Steps

  • Performance improves as k grows from very small values, plateaus around k=20–25, and may decline when distant subgoals exceed executor reach and dataset support.Source: §I.3 · p. 24

J.1 Controlled Comparison of High-Level Mechanisms

  • CFG-guided DSP achieved the strongest performance across evaluated tasks, outperforming standard conditional diffusion sampling and MLE subgoal imitation.The ablation compares high-level mechanisms while using the same type of low-level AWR executor.Source: §J.1 · pp. 24–25, Table 5 · p. 24

J.2 Comparison with Diffusion-Based Hierarchical Planners

  • DSP achieved the highest mean success in all six matched task–representation settings against Merlin, HDMI, and HD.The comparison uses the matched OGBench protocol.Source: Table 6 · p. 25, §J.2 · p. 25

K Inference-Time Analysis

L Stitching Experiments

Stitching experiments test whether algorithms compose shorter offline trajectory fragments into long-horizon behavior. DSP improves over flat baselines on most environments and is best or near-best in several settings, but does not uniformly outperform Bellman-backup-based hierarchical methods.

L Stitching Experiments

  • DSP was best or near-best on pointmaze-large, pointmaze-teleport, antmaze-giant, humanoidmaze-giant, and both antsoccer stitching tasks.Table 8 reports average binary success across five test-time goals, averaged over five seeds.Source: Table 8 · p. 26, L Stitching Experiments · p. 26
  • In larger or dynamically complex environments, task-specific guidance often improves performance over DSP1, suggesting CFG helps select more useful subgoals.The passage relates CFG to amplifying the goal-conditioned component of the learned subgoal distribution.Source: L Stitching Experiments · p. 26
  • DSP remains weaker than HIQL variants on some antmaze and humanoidmaze stitching tasks, where Bellman backups may propagate goal information across trajectory fragments.Source: L Stitching Experiments · p. 27
  • DSPmix results indicate OGBench actor goal mixing is not always suitable: excessive random goals may shift generation toward broad connectivity over directional, goal-relevant subgoals.Source: L Stitching Experiments · p. 27

M.1 Reduced-Data Coverage

Across reduced-data, Explore, visual-observation, and maze-trajectory evaluations, DSP is competitive but has clear data-support and transition-stochasticity boundaries. Maze visualizations show increasingly inefficient or unreliable HIQLw/o planning at larger scales, while DSP maintains topologically coherent, goal-directed paths.

M.1 Reduced-Data Coverage

  • The reduced-data evaluation uses nested subsets of complete trajectories and measures relative 0.5-unit XY-bin occupancy, not full-state coverage.Data fraction also affects visitation density, local transitions, and same-trajectory future-state pairs.Source: §M.1 · p. 27
  • DSP has the highest mean at every evaluated data fraction, but falls from 92.0 to 52.8 on AntMaze-Large and from 66.0 to 5.6 on HumanoidMaze-Giant.The severe data-support loss remains a failure condition, so the result supports relative competitiveness in these reduced-data settings rather than coverage independence.Source: §M.1 · p. 27, Table 9 · p. 27

M.2 Official Explore Datasets

  • DSP has the highest mean on Explore Medium and Large, but not Teleport, where stochastic, action-uncontrollable transitions limit uniform robustness.Explore trajectories retain locally executable transitions despite poor global trajectory quality.Source: §M.2 · pp. 27–28

N Evaluation with Visual Observations

O Visualization of Subgoal Trajectories in Maze Environments

  • The visualization section compares HIQLw/o and DSP subgoal trajectories across five maze evaluation tasks.The comparisons examine planning behavior as maze scale increases.Source: O Visualization of Subgoal… · p. 28

O.1 Medium Maze

  • In the medium maze, DSP produces smooth, topology-following paths, while HIQLw/o sometimes detours or proposes subgoals in unreachable areas.Both methods generally reach the goals in this shorter-horizon setting.Source: §O.1 · p. 28, Figure 7 · p. 29

O.2 Large Maze

  • As maze scale increases, HIQLw/o develops detours, while DSP maintains efficient, topologically coherent trajectories.The large-maze comparison describes noisier baseline guidance despite eventual directional correction.Source: §O.2 · p. 28, Figure 8 · p. 29

O.3 Giant Maze

  • In the giant maze, HIQLw/o often fails to produce valid subgoals, while DSP more consistently reaches the target and reissues local corrections.DSP’s clustered subgoals remain topologically valid and goal-directed, compensating for short-horizon execution errors.Source: Figure 9 · p. 29, §O.3 · pp. 29–30

P Comparison Experiments on D4RL Antmaze

On D4RL AntMaze, DSP is competitive with and often outperforms prior diffusion-based hierarchical planning methods. The comparison also distinguishes DSP’s goal-conditioned subgoal generation from trajectory- or sequence-level diffusion planning.

P Comparison Experiments on D4RL Antmaze

  • D4RL AntMaze offers a direct comparison with prior diffusion-based planners, but is narrower than OGBench for offline GCRL because it mainly uses one fixed goal.The prior methods were originally reported on this benchmark.Source: P Comparison Experiments on D4RL Antmaze · p. 30
  • DSP’s averages over overlapping tasks are 90.9 vs. 70.3 for DTAMP, 96.9 vs. 77.1 for HDMI, 94.9 vs. 88.8 for HD, and 94.9 vs. 92.7 for SIHD.The comparison reports mean and standard deviation over five seeds, with 100 episodes per seed.Source: Table 12 · p. 30, P Comparison Experiments on D4RL Antmaze · p. 30
  • DSP models high-level decisions as a goal-conditioned generative subgoal policy, rather than using diffusion as a trajectory- or sequence-level planner.Classifier-free guidance provides controllable goal-directed subgoal selection at inference time.Source: P Comparison Experiments on D4RL Antmaze · p. 30

Q Dexterous Manipulation Evaluation

On two bimanual dexterous-manipulation tasks, DSP outperforms HIQL across both task-space and higher-dimensional configuration-space waypoint settings. The results also show that direct DSP generation faces a waypoint-dimensionality challenge, within a limited evaluation using datasets of executable local behaviors.

Q Dexterous Manipulation Evaluation

  • The evaluation uses fixed datasets of 106 transitions per task, collected with PPO, and trains DSP and HIQL on identical data.PPO is a data-collection reference, not an offline or compute-matched baseline.Source: Q Dexterous Manipulation Evaluation · pp. 30–31
  • With 7D task-space waypoints, DSP exceeds HIQL by 17.6 and 8.6 percentage points on ShadowHandOver and ShadowHandCatchUnderarm, respectively; with 55/67D configuration-space waypoints, leads remain 13.4/5.2 points.Source: Q Dexterous Manipulation Evaluation · p. 31
  • Increasing waypoint dimensionality reduces DSP success by 6.6/8.2 points, versus 2.4/4.8 for HIQL, indicating a challenge for direct DSP generation.The paired values correspond to ShadowHandOver and ShadowHandCatchUnderarm, respectively.Source: Q Dexterous Manipulation Evaluation · p. 31
  • Because the datasets contain executable local behaviors, the comparison tests planning and execution under shared data support, not acquisition of absent skills.These two tasks provide initial evidence beyond parallel-jaw manipulation, not a comprehensive dexterous-manipulation evaluation.Source: Q Dexterous Manipulation Evaluation · p. 31

Related papers

Loading 2609.34575v1…