Source-linked AI summary
HIQL: Offline Goal-Conditioned RL with Latent States as Actions
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, Sergey Levine
TL;DR
Offline goal-conditioned RL seeks to use diverse unlabeled data, but distant goals make value estimation and flat policy extraction difficult. HIQL extracts high- and low-level policies from one value function, using latent subgoals to improve long-horizon control. It outperforms prior methods across challenging benchmarks, scales to image observations, and uses action-free data, while its action-free value objective is limited in stochastic environments.
Problem
Offline goal-conditioned RL could leverage large reward-free or action-free datasets, but accurately estimating values for distant goals is difficult and can make flat policies unreliable.
Method
HIQL extracts a high-level latent-subgoal policy and a low-level reaching policy from a single goal-conditioned value function learned from offline data.
Results
Across six state-based and pixel-based benchmark types, HIQL significantly outperforms prior offline goal-conditioned methods, solves complex long-horizon tasks, scales to image observations, and incorporates action-free data.
Takeaways & Limitations
Hierarchical policy extraction provides a practical way to improve offline goal-conditioned control while combining value learning, subgoal representations, and policy extraction in one framework.
Takeaways & Limitations
The action-free value objective is unbiased only for deterministic dynamics and may overestimate values in stochastic or partially observed settings.
Abstract
from arXiv · showhide
Unsupervised pre-training has recently become the bedrock for computer vision and natural language processing. In reinforcement learning (RL), goal-conditioned RL can potentially provide an analogous self-supervised approach for making use of large quantities of unlabeled (reward-free) data. However, building effective algorithms for goal-conditioned RL that can learn directly from diverse offline data is challenging, because it is hard to accurately estimate the exact value function for faraway goals. Nonetheless, goal-reaching problems exhibit structure, such that reaching distant goals entails first passing through closer subgoals. This structure can be very useful, as assessing the quality of actions for nearby goals is typically easier than for more distant goals. Based on this idea, we propose a hierarchical algorithm for goal-conditioned RL from offline data. Using one action-free value function, we learn two policies that allow us to exploit this structure: a high-level policy that treats states as actions and predicts (a latent representation of) a subgoal and a low-level policy that predicts the action for reaching this subgoal. Through analysis and didactic examples, we show how this hierarchical decomposition makes our method robust to noise in the estimated value function. We then apply our method to offline goal-reaching benchmarks, showing that our method can solve long-horizon tasks that stymie prior methods, can scale to high-dimensional image observations, and can readily make use of action-free data. Our code is available at https://seohong.me/projects/hiql/
1 Introduction
Offline goal-conditioned RL could exploit large unlabeled datasets, but distant goals make value estimates noisy and flat policy extraction unreliable. HIQL addresses this by extracting hierarchical policies from one goal-conditioned value function, improving long-horizon offline performance.
- HIQL is designed to use unlabeled or action-free data as a self-supervised source for goal-conditioned policy learning.
- Distant goals produce weak value differences that can be overwhelmed by estimation errors, especially in offline RL where erroneous predictions are not corrected through interaction.
- HIQL extracts a high-level subgoal policy and a low-level action policy from the same goal-conditioned value function.The high-level policy predicts a subgoal representation, while the low-level policy predicts actions to reach it.
- Across six state-based and pixel-based benchmark types, HIQL significantly outperforms prior offline goal-conditioned methods, particularly on complex long-horizon tasks.The method also scales to high-dimensional observations and incorporates action-free data.
2 Related work
Prior work addresses goal-conditioned learning through relabeling, contrastive, occupancy-based, hierarchical, graph-based, and offline skill-extraction methods. HIQL shares the use of subgoals and offline value learning while targeting offline goal-reaching directly.
- Goal-conditioned RL has used hindsight relabeling, contrastive learning, and state-occupancy matching to learn from data.
- Hierarchical and graph-based methods address the difficulty of direct goal-reaching in complex, long-horizon environments by introducing subgoals or intermediate planning structure.
- HIQL focuses on offline goal-conditioned RL, using subgoals and an offline-learned goal-conditioned value function rather than extracting a predefined skill hierarchy.
- Offline skill-extraction methods learn latent skills from trajectory segments, but must decide how trajectories should be hierarchically decomposed.
3 Preliminaries
The paper formulates offline goal-conditioned RL over transition and optional state-only trajectory datasets, then adapts IQL to learn values without action labels. Policy extraction still requires action-labeled data, and the action-free objective is limited in stochastic environments.
- The problem uses an MDP, a trajectory dataset with transitions and actions, and optionally a state-only dataset for action-free learning.Goals share the state space in the stated setting.
- IQL avoids out-of-distribution action queries by replacing the Bellman maximum with expectile regression and extracting policies through advantage-weighted regression.
- The action-free value objective backs up values from next states, while action-labeled data is needed only for policy extraction.
- The action-free IQL variant is unbiased for deterministic dynamics but may overestimate values in stochastic environments.State-only trajectories cannot distinguish good actions from favorable environmental noise.
4 Hierarchical policy structure for offline goal-conditioned RL
Flat policies struggle when distant-goal value differences are smaller than estimation noise. HIQL separates action selection into high-level subgoal and low-level reaching policies, and analysis shows that suitable temporal abstraction can reduce policy errors.
- Motivation: why non-hierarchical policies might struggle: For distant goals, small differences between next-state values are easily overshadowed by value-function noise, weakening flat policy learning signals.
- Motivation: why non-hierarchical policies might struggle: In the gridworld illustration, hierarchical selection uses the same noisy value function to predict an intermediate subgoal before choosing the action toward it.
- Hierarchical policy structure: HIQL uses a high-level policy to select a k-step waypoint and a low-level policy to select actions that reach that waypoint.
- Didactic example: hierarchical policies mitigate the signal-to-noise ratio challenge: The one-dimensional analysis models value noise proportional to the optimal value, so noise generally increases as the goal becomes more distant.
- Hierarchical policy extraction can significantly reduce flat-policy error when the subgoal step k is chosen appropriately.The analysis finds that each hierarchical component has an error bound no larger than the flat policy’s error.
5 Hierarchical Implicit Q-Learning (HIQL)
HIQL extracts hierarchical goal-conditioned policies from a single value function, using subgoals to improve robustness and learned representations to support image-based settings.
- Hierarchical policy extraction: HIQL extracts a high-level subgoal policy and a low-level action policy from one action-free goal-conditioned value function.The high-level policy predicts intermediate subgoals, while the low-level policy predicts primitive actions for reaching them.
- Hierarchical policy extraction: Splitting policy learning across subgoals and primitive actions provides clearer value signals and reduces total policy error.The high-level objective regresses toward useful subgoals, while the low-level objective regresses toward actions that reach local subgoals.
- Action-free data: Only the low-level objective requires action labels, allowing action-free data to train the value function and high-level policy.The low-level policy remains easier to learn because it reaches local subgoals rather than learning the full long-horizon task.
- Goal representations: HIQL uses a value-function representation of goals so the high-level policy can predict compact subgoal representations instead of high-dimensional states.The representation is learned end-to-end from an intermediate layer of the value function, and concatenating the current state was empirically preferred.
- Goal representations: The value-function representation is sufficient for optimal action selection when the value function can be expressed through that representation.The proposition establishes this sufficiency for deterministic MDPs under the stated representability condition.
6 Experiments
The experiments evaluate HIQL across six offline goal-conditioned tasks, comparing its performance with prior methods and testing image-based learning and goal representations.
- Experimental setup: The experimental study uses six offline goal-conditioned tasks to assess HIQL against prior methods.The first evaluation question concerns performance across a variety of goal-conditioned tasks.
- Experimental questions: The experiments test whether HIQL can solve image-based tasks and whether goal representations are important for good performance.These questions directly evaluate the method's scalability to visual observations and the role of its representation design.
4. Does HIQL mitigate policy errors caused by noisy and imperfect value functions in practice?
HIQL’s hierarchical policy extraction is evaluated across state- and pixel-based offline goal-conditioned benchmarks, including tests designed to expose value-function noise and action-data limitations. It achieves strong benchmark performance and the best policy accuracy for distant goals, where learned values are noisier.
- Results on state-based environments: 88% success on AntMaze-Large and 53% on AntMaze-Ultra exceed previously reported results on these datasets.HIQL mostly achieves the best performance across nine state-based offline datasets and six baselines.
- Results on pixel-based environments: 62% average success on unseen Roboverse manipulation tasks shows that HIQL generalizes from pixel observations to new robotic tasks.Across Procgen Maze, Visual AntMaze, and Roboverse, representation learning combined with hierarchical extraction improves image-based performance.
- Results with action-free data: With actions labeled for only 25% of trajectories, HIQL mostly maintains its original performance and outperforms baselines trained on fully labeled data.The action-limited setting uses state-only trajectories for the remaining 75%.
- Policy errors and value noise: HIQL achieves the best policy accuracy for distant Procgen Maze goals, especially when dist(s, g) ≥50.The learned value function becomes noisier as state-goal distance increases, while hierarchical extraction provides clearer learning signals.
7 Conclusion
HIQL uses one value function to drive representation learning and both policy levels, while supporting strong performance, action-free data, and image-based tasks. Its main limitations concern stochastic or partially observed environments and an independence assumption in the theoretical analysis.
- A single value function drives HIQL’s representation, high-level policy, and low-level policy in a simple, easy-to-train framework.
- HIQL shows strong performance on challenging goal-conditioned tasks, leverages action-free data, and provides representation learning for image-based tasks.
- HIQL’s action-free value objective is unbiased only under deterministic dynamics, while stochastic or partially observed settings may induce value overestimation.
- The theoretical analysis assumes independent value-function noise, which may not hold for smooth approximators in continuous state spaces.
A Training details
The training procedure specifies goal sampling, advantage estimation, latent state representations, and evaluation diagnostics for HIQL. It uses contextualized representations and reports training curves, reliability plots, and ablations across environment types.
- HIQL samples value-function goals from random states, future states, and the current state with probabilities 0.3, 0.5, and 0.2.
- The practical advantage estimates simplify reward and value-difference terms that are mostly constant under the paper’s reward function.
- The representation uses a normalized 10-dimensional latent vector, and conditioning it on both goal and current state improves empirical performance.
- At test time, HIQL queries both policies at every step without temporal abstraction because fixing subgoals longer does not significantly affect performance.
- Training curves cover state-based, pixel-based, Roboverse, and action-free-data experiments, with shaded regions showing 95% confidence intervals across 8 seeds.
C Ablation Study
The ablations examine subgoal horizon, representation parameterization, and auxiliary gradient flow, while the experiments use diverse state-based, pixel-based, and task-agnostic offline datasets. Performance is generally best with moderate subgoal horizons, contextualized representations, and additional representation gradients for pixel tasks.
- Subgoal steps: HIQL generally performs best with subgoal steps k between 25 and 50, while remaining reasonably effective outside this range unless k is too small.
- Representation parameterizations: Passing the goal and current state together to the representation function generally improves performance over alternative parameterizations.
- Auxiliary gradient flows for representations: In pixel-based environments, allowing low-level policy gradients to reach the representation improves performance and helps preserve control-relevant information.
- Datasets: The evaluation spans AntMaze, Kitchen, CALVIN, Procgen Maze, Visual AntMaze, and Roboverse datasets with state, image, and task-agnostic demonstrations.
- Experimental setup: Experiments use shared architectures for HIQL and four baselines, with environment-specific choices for discounting, temperature, expectile, and subgoal steps.
E.1 Proof of Proposition 4.1
The analysis compares flat and hierarchical action selection under value-function noise and establishes a representation-based equivalence under deterministic dynamics. Visualizations further show that HIQL learns intermediate subgoals leading toward target goals.
- Proposition 4.1: The hierarchical policy’s incorrect-action probability is bounded by errors in selecting both the k-step subgoal and the next action toward it.
- Assumptions: The proof assumes deterministic dynamics, independent Gaussian value noise, and k dividing T with k ≤ T.
- Proposition 5.1: The parameterized and unparameterized optimal policies have the same arg max action sets for every state and goal.
- Subgoal visualizations: Subgoal visualizations in AntMaze-Large and Procgen Maze show learned k-step subgoals that lead to the target goal.