PaperlayerView exact revision on arXiv

Source-linked AI summary

PathBridger: Subgoal Bridges for Offline Goal-Conditioned Reinforcement Learning

Soohyun Choi, Seonvin Cho, Songnam Hong

arXiv:2608.29061v1cs.LGcs.RO

TL;DR

Offline GCRL must reach diverse goals from fixed trajectories despite sparse long-horizon signals and execution errors that cannot be corrected online. PathBridger explicitly bridges selected intermediate endpoints to short-horizon actions through state-space bridge construction and inverse dynamics. Across evaluated OGBench tasks, PBF achieves the highest aggregate performance, with clearest gains on multi-object Cube manipulation, while Puzzle remains limited by learned subgoal selection.

  • Problem

    Offline GCRL must propagate sparse goal-reaching signals over long horizons and execute reliably without correcting errors through additional environment interaction.

  • Method

    PathBridger selects intermediate endpoints, constructs endpoint-pinned state-space bridges, decodes them into short action chunks with inverse dynamics, and replans after execution.

  • Results

    PBF achieves the highest aggregate performance across eight state-based OGBench tasks, with clearest gains on multi-object Cube manipulation.

  • Takeaways & Limitations

    Bridge-based execution contributes beyond endpoint-conditioned low-level control, but suitable learned subgoals remain a bottleneck on Puzzle.

  • Takeaways & Limitations

    Learned subgoal proposal and selection remain a key limitation on Puzzle, where oracle waypoints yield nearly perfect success while learned subgoals perform substantially worse.

Abstract

from arXiv · show

Offline goal-conditioned reinforcement learning (GCRL) aims to learn policies for reaching diverse goals entirely from fixed trajectory data. Long-horizon offline GCRL remains challenging because sparse goal-reaching signals must be propagated over many steps, while execution errors cannot be corrected through additional environment interaction. Existing methods address these challenges by improving long-range value estimation or reducing the effective decision horizon through subgoals, options, and action chunks. In several hierarchical methods, however, a selected subgoal specifies where to go, while the intervening state-space path remains implicit in an endpoint-conditioned low-level policy. To address this interface, we propose PathBridger, a hierarchical offline GCRL method that explicitly connects subgoal selection to short-horizon execution. PathBridger constructs a state-space bridge toward the selected intermediate endpoint and decodes it into a short executable action chunk using an inverse dynamics model. Experiments across the evaluated OGBench tasks demonstrate strong aggregate performance, with particularly large gains on the multi-object Cube manipulation tasks. Code: https://github.com/SChoish/PathBridger

1 Introduction

Offline GCRL must propagate sparse goal-reaching information over long horizons while executing reliably without online correction. PathBridger addresses an underexplored gap by making the state-space path between selected subgoals explicit, achieving the strongest aggregate performance on evaluated OGBench tasks.

  • Offline GCRL is difficult because sparse goal signals must support long-range decisions, while value, policy, and coverage errors cannot be corrected through further data collection.
  • Hierarchical methods reduce long-horizon difficulty through subgoals, but endpoint-conditioned low-level policies can leave the intervening state-space path implicit.
  • PathBridger selects an intermediate subgoal, constructs a state-space bridge toward it, and decodes the bridge into executable actions with inverse dynamics.
  • Controlled analyses support bridge-based execution while indicating that learned subgoal proposal and selection remain a limitation on Puzzle.
  • PathBridger connects distributional endpoint proposal and transitive value-based selection to endpoint-pinned bridges, inverse-dynamics decoding, and receding-horizon execution.
  • PathBridger achieves the highest aggregate performance on evaluated OGBench tasks, with particularly large gains on multi-object Cube manipulation.

2 Related Work

Related work addresses offline GCRL through direct goal-conditioned imitation, hierarchical subgoal planning, long-horizon value learning, action chunking, and trajectory generation. These approaches differ in how they represent reachability and reduce planning or decision complexity.

  • Offline goal-conditioned reinforcement learning: Offline GCRL methods mainly differ in how they represent long-horizon reachability and extract policies from fixed trajectory data.
  • Offline goal-conditioned reinforcement learning: Goal-conditioned behavioral cloning imitates actions conditioned on hindsight-relabeled future goals, constructing successful state–goal pairs from trajectories.
  • Hierarchical reinforcement learning and subgoal planning: Hierarchical reinforcement learning represents temporally extended decisions with options, skills, macro-actions, or intermediate subgoals.
  • Hierarchical reinforcement learning and subgoal planning: HIQL extracts high-level subgoal and low-level action policies from a shared goal-conditioned value function, while OTA improves subgoal extraction with option-aware backups.
  • Long-horizon value learning and action chunking: Long-horizon value-learning methods reduce recursive propagation, while action-chunking and trajectory models reduce the number or complexity of planning decisions.

3 Background

Offline GCRL models goal-reaching as hitting a target region in a fixed-data Markov process and uses sparse rewards and discounted values to represent success. Intermediate states provide a compositional structure for long-horizon reachability and value learning.

  • Offline Goal-Conditioned Reinforcement Learning: The formal setting uses a Markov process with state space S, action space A, dynamics p(s′ | s, a), and goals represented as target states in G ⊆ S.
  • Offline Goal-Conditioned Reinforcement Learning: Offline GCRL assumes a fixed dataset of trajectories collected by one or more unknown behavior policies.
  • Offline Goal-Conditioned Reinforcement Learning: A task-dependent representation map and metric define the goal region, sparse binary reward, and first hitting time.
  • Offline Goal-Conditioned Reinforcement Learning: The process terminates upon first entry into the goal region, so the goal reward is received at most once.
  • Offline Goal-Conditioned Reinforcement Learning: Hindsight relabeling forms successful state–goal pairs from future states in the same trajectory, but distant goals still require long-range propagation or temporal abstraction.
  • Transitive Structure in Goal Reaching: For exact state goals, directed temporal distance satisfies d⋆(s, g) ≤ d⋆(s, z) + d⋆(z, g), with equality when z lies on a shortest path.
  • Transitive Structure in Goal Reaching: Under binary hitting rewards, discounted values inherit this compositional structure, motivating transitive value composition and divide-and-conquer value learning.

4 PathBridger

PathBridger connects transitive endpoint selection to short-horizon execution by constructing an endpoint-pinned state-space bridge and decoding its prefix into actions. The policy executes the chunk, observes the resulting state, and replans.

  • Bridge policy: PathBridger maps a current state and final goal to a short executable action chunk through endpoint proposal, transitive selection, bridge construction, and inverse-dynamics decoding.Transitive values determine where to bridge, while the generated bridge specifies how to execute it.
  • Endpoint selection and bridge construction: The policy proposes K-step endpoints, selects one using a transitive value score, and expands it into an endpoint-pinned state-space bridge.The bridge is structurally pinned to both the current state and selected endpoint.
  • Bridge representation: The method represents endpoints and paths as displacements from the current state to improve conditioning for short trajectory windows.Predicted displacements are mapped back to absolute states by adding them to the current state.
  • Action decoding and replanning: The bridge horizon K controls endpoint and path prediction, while the execution horizon h_a ≤ K controls decoded actions and replanning frequency.The decoded prefix contains h_a transitions.
  • Transitive value learning: PathBridger learns a bounded transitive value using in-sample lower-bound targets for short temporal distances and transitive composition for longer distances.The value is defined on a discounted hitting-time scale, with binary cross-entropy and expectile-weighted transitive losses.
  • Endpoint proposal: Endpoint proposals are value-weighted from offline trajectory endpoints, with Gaussian and rectified-flow variants modeling endpoint displacement distributions.PBF can capture multimodal endpoint distributions, whereas PBG uses a diagonal Gaussian proposal.
  • Action decoding and replanning: Inverse dynamics decodes the first h_a bridge transitions into an h_a-step action chunk, which is executed before replanning.Only the execution-relevant bridge prefix is decoded into actions.

5 Experimental Results

Across eight state-based OGBench tasks, PathBridger is evaluated against multiple offline GCRL baselines and ablations, achieving its strongest results on Cube manipulation while retaining performance across dataset sizes.

  • Main benchmark results: PBF achieves the highest overall average, 82.7, outperforming the strongest prior method by 10.4 points.PBG also reaches 60.7, slightly exceeding TRL and substantially surpassing HIQL and OTA.
  • Main benchmark results: PathBridger achieves the best result on all three Cube tasks, with larger margins on double- and triple-cube variants.It matches the best result on antmaze-medium and remains competitive on antmaze-large, while staying below DQC on Puzzle and Scene.
  • Bridge-removal ablation: 14.7 points separate PBG from HTVL in average performance, supporting bridge construction and inverse-dynamics decoding beyond endpoint-conditioned execution.HTVL improves HIQL’s average from 33.8 to 46.0, whereas PBG reaches 60.7.
  • Oracle-waypoint diagnostic: 98.7 success is achieved on both Puzzle tasks when oracle waypoints are supplied as selected subgoals, versus 40.0 and 52.0 when supplied as task goals.The gap indicates that learned subgoal proposal and selection remain challenging even when the task is externally decomposed.
  • Dataset-size sensitivity: PBF attains the highest aggregate success rate at every tested dataset size, including 100k offline transitions.The ablation evaluates 100k, 300k, 500k, and 1M transitions while fixing training to 1M gradient steps.

6 Conclusion

PathBridger connects transitive value-based endpoint selection to state-space bridges and inverse-dynamics action decoding, achieving its clearest aggregate gains on multi-object Cube manipulation. Controlled analyses support bridge-based execution while identifying learned subgoal proposal and selection as a Puzzle bottleneck.

  • Across eight state-based OGBench tasks, PBF achieves the highest aggregate performance, with its clearest gains on multi-object Cube manipulation.
  • Removing the bridge lowers average performance, supporting bridge-based execution beyond endpoint-conditioned low-level control.On Puzzle, directly supplied oracle waypoints yield nearly perfect success, whereas retaining learned subgoal proposal and selection produces substantially lower performance.

A Implementation Details

The implementation combines value-based weighting, bridge reconstruction, flow matching, endpoint sampling, and endpoint-pinned interpolation, with shared hyperparameters summarized in Table 4.

  • The value objective uses binary cross-entropy and down-weights long state–goal pairs through a value-implied temporal distance.
  • Base and transitive losses are multiplied by wλ, with gradients stopped through target values and distance weights.
  • PBG minimizes a weighted endpoint-proposal objective while retaining a centered form in unnormalized empirical weights for numerical conditioning.
  • The weighted flow-matching objective samples interpolation states using K, a uniform u, and xu = (1−u)x0 + ux1.
  • The bridge path is pinned to both endpoints because the endpoint masks satisfy m0 = mK = 0.
  • At evaluation, PBG samples endpoints with temperature T, while T = 0 evaluates the Gaussian mean and PBF integrates eight forward-Euler steps.

B.1 Evaluation Protocol

Evaluation uses OGBench’s predefined goals and reports performance across seeds and checkpoints, with comparisons against established offline GCRL baselines and controlled component analyses.

  • Evaluation protocol: Each OGBench environment provides five predefined evaluation goals, evaluated with 50 episodes per goal at the 800k, 900k, and 1M checkpoints.Results average the three checkpoints within each seed and then report mean and standard deviation across seeds.
  • Baselines: The study compares PathBridger with GCIVL, CRL, HIQL, OTA, TRL, DQC, and HTVL using official, released, or specified implementations.HTVL is included as a controlled baseline, while baseline-specific settings and implementations are documented separately.
  • Baseline objectives: Baseline objectives include value-based, contrastive, hierarchical, transitive, option-aware, and chunk-based policy-learning procedures.The supplied protocol describes objectives for GCIVL, CRL, HIQL, OTA, TRL, and DQC, including their policy-extraction or chunk-selection mechanisms.
  • Ablation protocol: The evaluation includes controlled analyses that separate endpoint proposal, endpoint selection, and bridge-based execution using retrained or checkpoint-reused ablations.Inference-only ablations reuse trained checkpoints and evaluation seeds, matching candidate samples when applicable.

D.1 Bridge-Removal Ablation

The bridge-removal ablation holds PathBridger’s high-level endpoint pipeline fixed while replacing bridge execution with an endpoint-conditioned actor, isolating the contribution of bridge-based control.

  • Bridge-removal comparison: PBG improves from 33.8 for HIQL to 46.0 with HTVL, while PBG reaches 60.7 under the same high-level endpoint pipeline.HTVL retains PBG’s transitive value learner, Gaussian endpoint proposer, and endpoint-selection procedure, but removes bridge-based execution.
  • Bridge-removal comparison: The 14.7-point gap between HTVL and PBG supports the contribution of bridge construction and inverse-dynamics decoding.The comparison changes the low-level execution mechanism while preserving the shared high-level endpoint components.
  • Oracle-waypoint diagnostic: Oracle-waypoint evaluation distinguishes subgoal proposal and selection from execution by supplying waypoints either as task goals or directly as selected subgoals.Directly supplying a selected subgoal bypasses learned proposal and selection, leaving execution as the evaluated component.
  • Oracle-waypoint diagnostic: PBG executes the oracle sequence with near-perfect success on both Puzzle-3x3 and Puzzle-4x4 when waypoints are supplied as selected subgoals.The gap between waypoint roles suggests learned subgoal proposal and selection remain a bottleneck on Puzzle.

E.1 Endpoint-Sampling Sweep

Endpoint-sampling breadth is task-dependent: concentrated sampling is often sufficient, whereas Puzzle benefits strongly from broad candidate coverage.

  • Sweep definition: The (1, 0) setting produces one deterministic endpoint, whereas (32, 1) uses full-temperature sampling with 32 candidates.N denotes endpoint candidates and T controls sampling dispersion.
  • Endpoint-sampling breadth: PBF performance on Puzzle rises from 17.0/19.5 under (1, 0) to 69.3/78.1 under (32, 1) on p3/p4.The joint sweep varies endpoint-candidate count and sampling dispersion, so these settings characterize overall candidate-set breadth.
  • Endpoint-sampling breadth: PBG generally favors narrow sampling, with (1, 0) performing best on five of the eight environments.PBF also favors concentrated sampling on AntMaze-medium and the Cube tasks, while broader sampling is important on Puzzle.
  • Bridge-horizon sweep: The bridge horizon is task-dependent: PBF benefits from K = 40 on all Cube tasks, while K = 25 performs better on AntMaze and Puzzle.Averaged across tasks, K = 25 is slightly better for both PBF (78.3 vs. 76.0) and PBG (55.9 vs. 54.0).
  • Endpoint-weighting sweep: For PBG, csg = 10 improves seven of eight environments and raises the average from 51.6 to 60.4.PBF is less sensitive in aggregate, with averages of 80.2 versus 79.9 across the two scales.

F.1 Full Data-Scaling Results

The data-scaling results report per-environment performance at four transition budgets, using fixed training duration and two-seed mean-standard-deviation estimates.

  • Data budgets: Per-environment results are provided at 100k, 300k, 500k, and 1M transitions.All methods are trained for 1M gradient steps at each data budget.
  • Reporting: Values are reported as mean ± one standard deviation over two seeds.Table 15 marks means at or above 95% of the best mean in each setting in bold.
Loading 2608.29061v1…