PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning

Pedro Robles Dutenhefner, Dikshant Shehmar, Wagner Meira Jr., Marlos C. Machado

arXiv:2610.00849v1cs.AIstat.ML

TL;DR

Offline GCRL struggles to rank states over long horizons because discounting can erase distant value differences, while no fixed temporal scale preserves both distant and nearby distinctions. GITA learns a shared value function across scales and trains one policy from their aggregated advantages. On OGBench, it improves average success rate over HIQL.

  • Problem

    In long-horizon offline GCRL, discounting can shrink value differences between distant states below function-approximation error, while fixed temporal scales trade distant-state signal against nearby-state resolution.

    Source: §1 · p. 1

  • Method

    GITA conditions one shared value function on k and trains a single policy from advantage-weighted supervision aggregated across scales.

    Source: §1 · p. 1

  • Results

    Across OGBench tasks, GITA improves average success rate by 25 percentage points (73% relative) over HIQL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GITA addresses the trade-off between long-range value propagation and local value discrimination without committing to one fixed abstraction scale.

    Source: §6 · p. 9

    GITA is less effective on Cube and Scene, which are short-horizon and compositional rather than long-horizon.

    Source: §6 · p. 9

Abstract

from arXiv · show

Existing approaches to offline goal-conditioned reinforcement learning (GCRL) struggle with long-horizon tasks. Discounting shrinks value differences between distant states until they fall below the function approximation error, leaving the agent with no signal for ranking states. Temporal abstraction, which treats k environment steps as a single transition, restores this signal at long range, but no single fixed k suits all state-goal distances: large k preserves value differences across long temporal distances while collapsing distinctions between nearby states, and small k does the reverse. We make this trade-off explicit and introduce Generalized Implicit Temporal Abstraction (GITA), which conditions a single value function on k. GITA trains one policy by aggregating advantage-weighted supervision across multiple k values, so scales assigning larger positive advantages to a state-goal pair contribute more strongly to its update. GITA does not need to choose between local resolution and long-range signal; it retains both without committing to a single k. On OGBench, GITA outperforms a broad range of offline GCRL baselines, raising average success rate across all tasks by 25 percentage points (73% relative improvement) over HIQL. It also improves over the strongest fixed-k method, OTA, by 7 percentage points (14% relative).

1 INTRODUCTION

Offline GCRL faces a value-discrimination trade-off across state-goal distances: fixed temporal abstraction can preserve distant-state signal or nearby-state distinctions, but not both. GITA conditions one value function on multiple scales and aggregates their advantages to train one policy.

1 INTRODUCTION

  • A large k preserves value distinctions for distant goals but collapses distinctions between nearby states, while a small k has the reverse trade-off.No fixed k serves every state-goal pair.Source: §1 · p. 1
  • GITA learns a single value function conditioned on k and aggregates advantages across scales to train one policy, with larger positive advantages contributing more strongly.The scale-specific advantages induce an implicit soft assignment rather than an explicit scale-selection mechanism.Source: §1 · p. 1, Figure 1 · p. 2
  • Nearby state-goal pairs can be resolved at small k and distant pairs at large k within one value function.This lifts the single-scale constraint inherited by fixed-scale methods.Source: §1 · p. 2
  • 25 percentage points of success-rate improvement over HIQL (73% relative) and 7 percentage points over OTA (14% relative) on OGBench.GITA also outperforms a broad range of other GCRL methods across multiple morphologies, datasets, and environment types.Source: §1 · p. 2

2 PRELIMINARIES

Offline GCRL learns goal-reaching policies from previously collected trajectories without further environment interaction. The section introduces hierarchical value-based methods and explains how discounting can erase the signal needed to rank subgoals over long horizons.

2 PRELIMINARIES

  • HIQL predicts intermediate subgoals with a high-level policy and uses a low-level policy to select primitive actions toward them.Both policies rely on a goal-conditioned value function trained with an expectile temporal-difference objective.Source: §2 · p. 2
  • As goals move farther away, discounting can shrink value differences between candidate subgoals until estimation error dominates, degrading high-level policy learning.The high-level advantage should be positive when a candidate subgoal represents progress toward the goal.Source: §2 · p. 3
  • OTA propagates value across k primitive steps in one update with a single discount, reducing the backups needed to connect a state to its goal to roughly ⌈d(st, g)/k⌉.OTA uses the state reached k steps along the dataset trajectory as the successor state.Source: §2 · p. 3
  • OTA fixes one k for all state-goal pairs, requiring environment-specific tuning and making states within k steps indistinguishable under its temporally abstracted value function.A k suited to distant goals is larger than needed for nearby ones.Source: §2 · p. 3

3 NO FIXED ABSTRACTION SUITS ALL STATE-GOAL DISTANCES

No fixed abstraction factor preserves value discrimination at every state-goal distance: the best factor grows with distance, while the optimal range differs across environments. The analysis explains this trade-off and finds it empirically in OGBench environments.

Theoretical analysis

  • Coarser abstraction improves discrimination of distant states but blurs nearby ones, so a fixed-stride subgoal advantage vanishes when k is too small or too large.The analysis models k primitive transitions as one abstract transition with reward −1 and a single discount γ.Source: §3 · pp. 3–4
  • When ∆≪d, the optimal abstraction factor k⋆opt(d, ∆) grows with state-goal distance; as γ→1, its distance-dependent scale is approximately d/(1−γ), keeping remaining distance within one effective discount horizon.Source: §3 · p. 4

Empirical characterization

  • The empirical analysis measures Spearman correlation between the subgoal advantage and progress toward the goal, calling it Advantage-Progress Correlation (APC).It uses geodesic shortest paths as a proxy for minimum distance.Source: §3 · p. 4
  • No factor dominates across distances; in pointmaze, k = 1 falls to APC ≈0 by d ≈0.4, while k = 8 retains APC ≈0.5 at twice that distance.The best-performing factor increases with distance in every environment, peaking at k = 3 in antmaze and k = 13 in humanoidmaze.Source: §3 · pp. 4–5, Figure 2 · p. 4
  • As k increases, the value-sensitive region expands over longer distances while local resolution progressively degrades in pointmaze-large-navigate.The figure evaluates normalized values at every navigable cell for k ∈ {1, 2, 3, 5, 13, 21}, relative to a fixed reference state.Source: Figure 3 · p. 5

4 GENERALIZED IMPLICIT TEMPORAL ABSTRACTION

GITA replaces a fixed abstraction scale with one value function conditioned on k and trains a single high-level policy using advantages across scales. The policy weights let scales better suited to each state-goal pair contribute more strongly, without estimating that pair’s distance.

4 GENERALIZED IMPLICIT TEMPORAL ABSTRACTION

  • GITA uses one k-conditioned value function to represent multiple temporal abstraction scales and trains a single high-level policy from the advantages it induces.The low-level policy remains unchanged from HIQL, predicting primitive actions toward the proposed subgoal.Source: §4 · p. 5, §4.1 · p. 5, §4.2 · p. 6

4.1 ONE NETWORK, MULTIPLE ABSTRACTION SCALES

  • During training, GITA samples k geometrically, favoring shorter scales while retaining a long tail over larger ones.The value network can be queried at any scale after training.Source: §4.1 · p. 5
  • The shared value network is trained with a temporal-difference objective whose successor is evaluated at the same k, allowing information to generalize across scales.Scales are therefore not learned independently.Source: §4.1 · p. 5

4.2 HIGH-LEVEL POLICY TRAINING WITH MULTI-SCALE ADVANTAGES

  • GITA evaluates each subgoal across multiple scales but trains one high-level policy that is not conditioned on k.The candidate subgoal offset m is held fixed independently of k.Source: §4.2 · pp. 5–6
  • A soft aggregation of scale-specific AWR weights lets the scale with the clearest positive advantage drive the update, without estimating state-goal distance.In antmaze-giant-navigate, smaller scales receive the largest weights at short distances, while larger scales become dominant at longer distances.Source: §4.2 · p. 6, Figure 4 · p. 6

5 EXPERIMENTS

The experiments evaluate GITA against offline GCRL baselines on OGBench and test which components make its multi-scale approach reliable across regimes.

5.1 EVALUATION ON OGBENCH

  • The evaluation covers locomotion mazes and manipulation tasks in state- and pixel-based variants, reporting average success across five goal-reaching tasks per environment.State-based results average 8 seeds and pixel-based results average 4 seeds; locomotion probes long-horizon navigation, while manipulation probes compositional generalization.Source: §5.1 · p. 7
  • GITA outperforms every baseline across the evaluated tasks, with the locomotion results progressing from HIQL to OTA to GITA.The aggregate comparison is statistically significant under a two-sided paired Wilcoxon signed-rank test with Holm-Bonferroni correction (all adjusted p < 0.05).Source: §5.1 · p. 7, Table 1 · p. 8
  • On state-based Cube and Scene, GCIQL outperforms GITA, HIQL, and OTA; GITA nevertheless improves over OTA on every pixel-based manipulation task.The paper attributes the state-based results to open-loop, non-Markovian play data with temporally correlated noise, while pixel-based datasets are Markovian and provide broader state coverage.Source: §5.1 · p. 7

5.2 ABLATION STUDIES

  • Unconditioned Value has the largest performance drop, and selecting only the highest-advantage scale also underperforms GITA; the two averaging variants are not significantly different overall.Mean Advantage Before Exponentiation and GITA with Geometric k each collapse on one of the widest-distance maze settings, while GITA does not.Source: §5.2 · pp. 7–8
  • Every ablation variant degrades sharply on at least one maze setting, whereas GITA does not.The ablations test value-function scale conditioning, scale-specific advantage aggregation, and scale selection.Source: §5.2 · pp. 7–8, Table 2 · p. 9

6 CONCLUSION

The conclusion presents GITA as a shared value function across temporal scales, trained through aggregated advantages, to address the trade-off between long-range propagation and local discrimination. The accompanying related work and analysis place this approach among prior methods and explain why no fixed abstraction factor serves every state-goal pair.

6 CONCLUSION

  • GITA addresses the trade-off between long-range value propagation and local discrimination by learning a shared multi-scale value function and training one policy from aggregated advantages.The conclusion describes this as implicitly emphasizing the most informative scales for each state-goal pair.Source: §6 · p. 9

A RELATED WORK

  • Prior offline GCRL methods address long-range estimation through contrastive objectives, temporal-distance structure, or recursive value learning; hierarchical methods also vary subgoal generation and execution.GITA retains HIQL’s two-level policy structure and instead studies temporal abstraction in the high-level value function.Source: A RELATED WORK · p. 12
  • Prior temporal-abstraction work uses a fixed stride in offline GCRL, while GITA represents multiple abstraction scales directly in one k-conditioned goal-value function.Other cited multiscale approaches apply horizons to predictive models or state-goal representation learning.Source: A RELATED WORK · p. 12

B PROOFS FOR THE FIXED-SCALE RESOLUTION–HORIZON TRADE-OFF

  • The analysis uses an idealized shortest-path model and is not intended to exactly characterize dataset-induced transitions.It follows OTA’s abstracted-update convention, which shifts reward indexing by one step at k = 1 without changing the maximizing abstraction factor in the analysis.Source: B PROOFS FOR THE FIXED-SCALE… · p. 12

B.1 PROOF OF PROPOSITION 1

  • The subgoal advantage factors into reachability and resolution terms, which pull in opposite directions as the abstraction factor grows.A coarser abstraction improves distant-state discrimination but blurs nearby-state distinctions, so no k maximizes both.Source: §B.1 · p. 13, §B.2 · p. 14

B.2 PROOF OF THE VANISHING-ADVANTAGE COROLLARY

  • The vanishing-advantage corollary formalizes the opposing limits: coarser abstraction raises the reachability factor toward 1 while driving resolution toward 0.Thus, no single k simultaneously maximizes both factors.Source: §B.2 · p. 14

B.3 PROOF OF COROLLARY 1.2

The relaxed advantage has a unique global maximizer, and the maximizing abstraction factor grows with state-goal distance. Consequently, no fixed factor preserves both long-range signal and local resolution.

B.3 PROOF OF COROLLARY 1.2

  • The relaxed advantage has a unique global maximizer k⋆opt(d, ∆).The derivative is positive before this solution and negative after it.Source: §B.3 · p. 14
  • For fixed ∆ and γ, the advantage-maximizing abstraction factor increases with state-goal distance.The local-progress approximation gives k⋆opt(d, ∆) ≈ −1/log γ ≈ 1/(1−γ) when ∆≪d and γ→1.Source: §B.3 · pp. 14–15
  • No fixed abstraction factor preserves informative advantages at all distances: small k loses distant signal, while large k sacrifices local resolution.The optimal scale therefore grows with distance.Source: §B.3 · p. 15

C FIXED-SCALE TRADE-OFF EXPERIMENT DETAILS

The fixed-scale experiment measures how well advantages rank geodesic progress across goal distances. The OGBench evaluation spans diverse locomotion, manipulation, observation, layout, and offline-dataset settings.

C FIXED-SCALE TRADE-OFF EXPERIMENT DETAILS

  • The experiment bins sampled state-goal triples by geodesic distance and measures rank agreement between scale-conditioned advantages and geodesic progress.Geodesic distance is a progress proxy, not necessarily expert-policy temporal distance; APC near 1 indicates aligned rankings and negative APC indicates inversion.Source: C FIXED-SCALE TRADE-OFF EXPERIMENT… · pp. 15–16

D ENVIRONMENTS

  • OGBench evaluation covers PointMaze, AntMaze, HumanoidMaze, Cube, and Scene, using state observations and pixel variants of AntMaze, Cube, and Scene.The locomotion tasks combine maze navigation with increasingly complex control; Cube and Scene are manipulation environments.Source: D ENVIRONMENTS · p. 16

D.1 OGBENCH ENVIRONMENTS

  • PointMaze directly controls a point mass, while AntMaze and HumanoidMaze add quadruped and humanoid locomotion; medium, large, and giant layouts provide increasingly long paths.Cube and Scene evaluate block arrangements and composed manipulation behaviors, respectively.Source: §D.1 · p. 16

D.2 OFFLINE DATASET VARIANTS

  • Maze datasets vary from coherent goal-directed navigate trajectories to local stitch segments and high-coverage, suboptimal explore behavior; manipulation uses play and noisy datasets.State-based Cube and Scene use play, while their visual variants use noisy.Source: §D.2 · pp. 16–17, Figure 6 · p. 17

D.3 EVALUATION PROTOCOL

The evaluation follows OGBench’s multi-task protocol, aggregates performance across checkpoints and seeds, and reports ablations at the final checkpoint.

D.3 EVALUATION PROTOCOL

  • Each environment has five predefined evaluation tasks, with 50 episodes per task per checkpoint and 250 episodes per checkpoint and seed.State-based experiments use eight seeds and pixel-based experiments use four, reporting mean success rate and standard deviation across seeds.Source: §D.3 · p. 17
  • Main-table scores average the final three checkpoints: state-based models are evaluated at 800K, 900K, and 1M steps, while pixel models use 300K, 400K, and 500K steps.Each per-seed score averages 750 episodes before aggregation across seeds.Source: §D.3 · pp. 17–18
  • Ablation-study performance is reported from only the final training checkpoint, at 1M steps for state-based tasks.The accompanying specifications tables summarize the evaluated environments and offline datasets.Source: §D.3 · p. 18, Table 3 · p. 18, Table 4 · p. 18

E IMPLEMENTATION DETAILS

GITA’s implementation conditions the high-level value function on abstraction scale while retaining the HIQL policy architecture and most settings. Statistical tests compare its aggregate results with OGBench baselines, including fixed-scale OTA.

E IMPLEMENTATION DETAILS

  • The main architectural change is conditioning the high-level value function on k; the high- and low-level policies retain HIQL architectures.A logarithmic k embedding is injected into hidden layers through FiLM modulation.Source: E IMPLEMENTATION DETAILS · p. 19
  • Because k enters only through FiLM, the state-goal representation is reused across abstraction factors, with modest additional training cost.The state-goal representation is independent of k.Source: E IMPLEMENTATION DETAILS · p. 19
  • GITA re-tunes γ per task because it controls both temporal-difference discounting and the geometric distribution over abstraction factors; candidate factors are {3, 5, 8, 13, 21}.Other HIQL hyperparameters are kept unchanged.Source: E IMPLEMENTATION DETAILS · p. 20

F STATISTICAL TESTING

  • Holm-corrected paired tests find significant aggregate improvement over every main-evaluation baseline and significant gains over removing scale conditioning or using hard scale selection.Alternative aggregation and sampling variants are not significantly different after correction.Source: F STATISTICAL TESTING · p. 21

G.1 OGBENCH BASELINES

  • The comparison includes six OGBench offline GCRL algorithms and OTA, the closest fixed-scale baseline, which uses one task-selected abstraction factor.OTA shortens the value-propagation horizon with temporally extended transitions.Source: §G.1 · pp. 21–22

H ADDITIONAL ABLATION DETAILS

The ablations vary GITA’s value conditioning, scale aggregation, and candidate abstraction factors. Across maze tasks, larger abstraction factors retain informative policy weights over longer state–goal distances, while smaller factors lose their advantage signal sooner.

H ADDITIONAL ABLATION DETAILS

  • The unconditioned-value ablation removes k from the value function while retaining multi-scale value training, testing whether explicit scale conditioning is necessary.Without scale-specific values, the policy uses a single advantage with the standard AWR weight.Source: H ADDITIONAL ABLATION DETAILS · p. 22
  • The highest-advantage-factor ablation replaces soft aggregation across scales with hard selection of one scale’s advantage per training sample.It tests whether selecting only the scale with the largest advantage suffices instead of combining supervision across temporal resolutions.Source: H ADDITIONAL ABLATION DETAILS · p. 22
  • The aggregate-before-exponentiation ablation averages raw scale-specific advantages before applying the AWR exponential, removing scale-specific amplification from the weights.This isolates whether preserving differences in individual scale-conditioned advantages through nonlinear AWR weighting matters for policy supervision.Source: H ADDITIONAL ABLATION DETAILS · p. 23
  • The geometric-k ablation replaces GITA’s fixed candidate set with newly sampled abstraction factors at each high-level policy update.It uses ζ = 0.9, giving a mean abstraction factor of 10, while leaving the value-learning procedure unchanged.Source: H ADDITIONAL ABLATION DETAILS · p. 23

I.1 SCALE-DEPENDENT POLICY WEIGHTS ACROSS STATE–GOAL DISTANCES

  • Larger abstraction factors retain informative AWR weights over longer state–goal distances, especially in giant layouts.The plotted weights are clipped exponentiated advantages, averaged by geodesic distance; the transition occurs progressively later as k increases.Source: §I.1 · pp. 23–24, Figure 7 · p. 24

I.2 RESULTS ACROSS DISCOUNT FACTORS

The discount-factor sweep reports GITA’s maze-task success rates using final checkpoints. Because α is set to γ, the sweep also changes the abstraction-factor distribution, complicating interpretation of performance drops at larger γ.

I.2 RESULTS ACROSS DISCOUNT FACTORS

  • Because α = γ, the sweep also changes the abstraction-factor distribution, so the performance drop at larger γ may partly reflect trajectory-boundary truncation rather than discounting alone.At γ = α = 0.995, E[k] = 200, matching the data episode length for PointMaze and AntMaze stitch.Source: §I.2 · p. 24
  • The sweep varies γ over {0.98, 0.99, 0.995}; its best global discount factor is γ = 0.99, selected by average performance across all tasks.The table reports success rates from the final checkpoint after 10^6 gradient steps, averaged over 8 seeds.Source: §I.2 · p. 24, Table 8 · p. 25

Related papers

Loading 2610.00849v1…