Q-Shaped Options for Hierarchical Reinforcement Learning

1 INTRODUCTION

• Existing abstractions either discard option distinctions needed for optimal control or retain irrelevant distinctions, forfeiting coarser state abstraction.The former can undermine hierarchy; the latter preserves horizon reduction but leaves functionally equivalent behaviours distinct.Source: §1 · p. 2
• QSO learns an action abstraction shaped by the Q functions at both hierarchy levels to address three desiderata for effective options.Its architecture uses separate value functions, Q functions, and policies at each level.Source: §1 · p. 2
• The low-level Q function encourages distinctions needed for optimal control, while the high-level Q function encourages discarding distinctions between functionally equivalent behaviours.The low-level Q function treats the option as a goal, and the high-level Q function treats it as an action.Source: §1 · p. 2
• QSO encodes a trajectory segment’s outcome rather than the exact state-action sequence that produced it.It encodes the current and target states.Source: §1 · p. 2
• QSO learns semantically meaningful option spaces and outperforms baselines in offline goal-conditioned locomotion and manipulation, with non-zero performance where other evaluated algorithms fail.The formal contributions include a definition of control sufficiency, a proof that exact low-level Q-shaping satisfies it, and a PAC bound motivating the removal of unnecessary option distinctions.Source: §1 · p. 2

2 RELATED WORKS

• Table 1 compares methods by what their action abstractions encode, how they are learned, and whether they preserve needed or discard unnecessary distinctions.The related-work discussion notes that approaches range from encoding a target state to encoding a full trajectory segment.Source: §2 · pp. 2–3, Table 1 · p. 3
• HIQL learns options through an action-free value function, but equal value does not ensure that state-goal pairs require the same immediate actions.Its abstraction can therefore discard information needed for optimal control.Source: §2 · p. 3
• SHARSA and DSHARSA use target states directly as options, retaining low-level control information but distinguishing equivalent behaviours at different states.This limits coarser state abstraction and data aggregation.Source: §2 · p. 3
• OPAL reconstructs action sequences with a β-VAE, so its options encode execution details rather than outcomes.Segments with the same outcome can be represented differently when their executions differ, hindering coarser state abstraction.Source: §2 · p. 3
• QC provides temporal abstraction through action chunks, but its single-level policy prevents coarser state abstraction.One policy handles the entire decision process.Source: §2 · p. 3

3 PRELIMINARIES

• The setting is a fully observed deterministic goal-conditioned MDP, with the goal fixed for each episode and a policy learned from a fixed dataset of trajectories.The dataset is collected by an arbitrary policy.Source: §3 · p. 3
• The reward is −1 per step until the goal is reached, after which the return terminates; maximizing return is equivalent to minimizing discounted time to the goal.The discount is γ, and the return is bounded in [−1/(1 − γ), 0].Source: §3 · p. 3
• The high-level policy selects an option from state and goal, and the low-level policy uses that option and state to select a primitive action.Their composition defines the overall policy.Source: §3 · p. 4
• During training, fixed-length options encode trajectory segments for high-level action selection and low-level conditioning; deployment communicates within option space without the encoder.The paper focuses on fixed-length options to avoid degenerate solutions and simplify separation of the two decision processes.Source: §3 · p. 4

4.1 DESIDERATA

• The abstraction represents a trajectory segment’s outcome rather than its exact state-action execution.QSO encodes the current state and achieved state, allowing different executions with the same outcome to share an option.Source: §4.1 · p. 4
• Low-level Q-shaping pressures target-state pairs requiring different actions from the same current state to map to different options.Compatible action-value sets can allow segments with different dataset executions to share an option, including better and poorer executions.Source: §4.1 · p. 4
• Q-shaping is stronger than control sufficiency because it distinguishes some options whose Q values differ only by an additive offset or suboptimal-action ranking.Such differences need not affect the optimal action.Source: §4.1 · p. 5
• Proposition 1 states that low-level Q-shaping implies control sufficiency.The property ensures a low-level policy can choose optimal actions using the option representation.Source: §4.1 · p. 5
• Discarding unnecessary distinctions can aggregate experience across states, support conservative extrapolation, and reduce the effective complexity of the hierarchical decision problem.The paper motivates this desideratum with a PAC bound on offline error for hierarchical policies with joint state and action abstraction.Source: §4.1 · p. 5
• High-level Q-shaping provides an inductive bias for merging functionally equivalent state-target pairs, but does not explicitly enforce that invariance.The paper reports that shared function approximation and gradient-based optimization empirically produce coherent option clusters.Source: §4.1 · p. 5

4.2 PRACTICAL IMPLEMENTATION

• QSO uses a continuous, length-normalized option space; high-level rejection sampling with advantage-weighted behavior cloning; and low-level behavior cloning.The normalization keeps proposed options in-distribution for the low-level policy, while resampling options each timestep lets intent shift across the space.Source: §4.2 · pp. 6–7
• Training jointly learns the option representation, high- and low-level values and Q functions, and policies from sampled states, goals, and targets.The provided pseudocode describes encoding options and target options, training values and Q functions, and updating target networks.Source: §4.2 · pp. 6–7

• The experiments isolate policy learning on fixed offline data, evaluate on challenging OGBench tasks, and include ablations of option inputs, dimension, goal sampling, reward labeling, extraction, and horizon.Baselines use their original hyperparameters; QSO adds no hyperparameters beyond option dimension.Source: §5 · pp. 7, 10
• 54% average success across 11 environments gives QSO the highest rate, versus 49% for DSHARSA.QSO also achieves at least 65% of the best success rate in every environment, while each other algorithm falls below 35% in at least one.Source: Table 2 · p. 7, §5 · pp. 7–8
• QSO’s option space groups functionally equivalent behaviors while separating behavior groups, unlike the inconsistent structure of the compared alternatives.The analysis compares PCA projections and whitened distances for sampled trajectory sequences from antmaze-giant, cube-double, and cube-quadruple.Source: §5 · p. 8, Figure 2 · p. 8
• Both high- and low-level Q-function gradient flows are required for clean semantic grouping and separation in QSO’s learned option space.Neither Q function alone recovers the structure obtained with both; high-level-only shaping also degrades significantly in cube-double.Source: §5 · p. 8, Figure 3 · p. 9
• The high-level state-value function depends mainly on slow-moving dimensions, while the low-level policy depends more on fast-moving dimensions.The reported shares are 88% and 74% for the high-level value function in antmaze-giant and cube-double, and 59% and 77% for the low-level policy.Source: §5 · p. 9
• Consecutive sampled options are much closer than random options during deployment, consistent with gradual changes in intent rather than a collapsed policy.Their distances are roughly 30% of random-option distance in antmaze-giant and 25% in cube-double; the random-trajectory-embedding reference is approximately 1.Source: §5 · p. 10, Figure 4 · p. 10

• QSO’s contribution is an action abstraction that is control-sufficient while discarding unnecessary distinctions between options.Source: §6 · p. 10
• In stochastic environments, retrospective option representation can make the high-level critic evaluate an option conditioned on its realised outcome, leading to optimistic bias.The paper says this issue is shared by hindsight-based hierarchical methods including HIQL, OPAL, SHARSA, and DSHARSA.Source: §6 · p. 10
• Future work could address the stochastic-environment issue through progress-adaptive high-level control.Source: §6 · p. 10

• The work aims to advance machine learning, specifically hierarchical reinforcement learning.Source: ETHICS STATEMENT · p. 10
• Evaluation is limited to simulated environments using public datasets.Source: ETHICS STATEMENT · p. 10
• The statement notes possible broader societal consequences but identifies none that must be specifically highlighted.Source: ETHICS STATEMENT · p. 10

AI DISCLOSURE

• The authors used AI to render figures, debug code, and check spelling and grammar, while taking responsibility for the final submission.Source: AI DISCLOSURE · p. 11

A EXTENDED PRELIMINARIES

• Implicit Q-Learning evaluates state values under dataset actions selected for maximum value, while AWR and rejection sampling extract policies constrained to high-quality dataset actions.These conservative methods avoid querying out-of-distribution actions or directly propagating Q-function errors into the policy.Source: A EXTENDED PRELIMINARIES · p. 14

B LOW-LEVEL Q-SHAPING IMPLIES CONTROL SUFFICIENCY

• The proof shows that when low-level Q-shaping gives two targets the same option, their action-value functions and maximizing actions agree at that state.Source: B LOW-LEVEL Q-SHAPING IMPLIES CONTROL… · p. 15

C CONSERVATIVE EXTRAPOLATION

• In the locomotion example, shared coordinates let the high-level abstraction select a good relative option, and shared pose lets the low-level abstraction select a good action.Together, these transfers let the hierarchical policy take an action never observed for the first state-goal pair; the example depends on abstract relative options.Source: C CONSERVATIVE EXTRAPOLATION · p. 16

D OPTION SPACE COMPRESSION BOUNDS OFFLINE ERROR

• With probability at least 1 − δ, Proposition 2 bounds expected-return suboptimality for a finite-state, finite-action MDP with at most two possible next-state transitions and a dataset of N transitions.The bound concerns a goal-conditioned hierarchical policy.Source: D OPTION SPACE COMPRESSION BOUNDS… · p. 17
• The bound uses ϕh to aggregate state-goal pairs requiring the same option and ϕl to aggregate state-option pairs requiring the same action.The resulting abstract spaces are Ch and Cl, respectively.Source: D OPTION SPACE COMPRESSION BOUNDS… · p. 17
• Discarding unnecessary option distinctions reduces the bound by shrinking the high-level action space, tightening the low-level abstract-state bound, and aggregating occupancy mass to reduce both concentrability coefficients.In continuous spaces, the aim is to map functionally equivalent behaviours to similar option-space points, reducing effective complexity in both error terms.Source: D OPTION SPACE COMPRESSION BOUNDS… · p. 17

E STATE AND ACTION ABSTRACTION SAMPLE COMPLEXITY IN ONLINE GOAL-CONDITIONED RL

• The section presents a PAC sample-complexity bound for reaching a unique goal-state, then extends the setting to finding a policy that reaches any given goal among |G| unique goal-states.The unique-goal setup assumes a finite discounted MDP and at most two possible next states per state-action pair.Source: E STATE AND ACTION ABSTRACTION SAMPLE… · p. 18
• The hierarchical policy’s overall sample complexity includes the low-level contribution across the |Ω| options that can be executed.The passage attributes the overall expression to Robert et al. (2023).Source: E STATE AND ACTION ABSTRACTION SAMPLE… · p. 18
• When k = 1, the section says the hierarchical complexity approximately recovers that of a non-hierarchical policy, under long-horizon and goal-space-size assumptions.The approximation relies on γ → 1 and |G| ≥ |A|; the text notes that G = S is commonly used when the goal-space is unknown.Source: E STATE AND ACTION ABSTRACTION SAMPLE… · pp. 18–19
• Options need not reproduce the full low-level action sequence, but must preserve enough information to approximate high-level value and select the low-level action.The section expects |Ω| ≪ |A|^k in most real-world settings.Source: E STATE AND ACTION ABSTRACTION SAMPLE… · p. 19
• Joint state and action abstraction reduces the effective state-space complexity required by the high- and low-level decision processes.The respective abstract spaces are bounded by |Ch| ≤ |S||G| and |Cl| ≤ |S||Ω|.Source: E STATE AND ACTION ABSTRACTION SAMPLE… · p. 19

F STATE AND ACTION ABSTRACTION ERROR IN OFFLINE GOAL-CONDITIONED RL

• Concentrability coefficients account for distribution shift between the behavior dataset and the optimal policy; a narrow or incomplete dataset can make these coefficients large.The coefficients compare occupancy under the behavior policy with occupancy under the optimal policy.Source: F STATE AND ACTION ABSTRACTION ERROR… · p. 20
• The offline bound for a hierarchical policy is tightened by ignoring irrelevant low-level information at the high level and irrelevant high-level information at the low level.This yields joint state and action abstraction in the error bound.Source: F STATE AND ACTION ABSTRACTION ERROR… · p. 20
• Under state abstraction, offline dataset mass is aggregated across equivalent state-goal and state-option pairs.The pairs are equivalent under the information loss represented by ϕh and ϕl.Source: F STATE AND ACTION ABSTRACTION ERROR… · p. 20
• Aggregating equivalent pairs reduces the likelihood of support mismatch, where the optimal policy requires a transition assigned zero mass by the dataset.The section applies this reasoning to the offline error bound for hierarchical policies.Source: F STATE AND ACTION ABSTRACTION ERROR… · p. 20

G.1 ACTION ABSTRACTION

• QSO learns options jointly with both Q functions, using each option as a low-level goal and a high-level action, with normalized rewards and continuous, length-normalized option embeddings.A full-length option is calibrated to cost the same as one low-level action; the embedding lies on a hypersphere to limit out-of-distribution options for the low-level policy.Source: §G.1 · p. 21

G.2 STATE ABSTRACTIONS

• State abstractions use different goal distributions and reward criteria at each level: environment success for high-level training and stricter exact-match success for low-level training.High-level sampling mixes local, trajectory, and dataset-wide goals; low-level sampling emphasizes nearby trajectory targets.Source: §G.2 · p. 22

G.3 POLICY EXTRACTION

• The high-level policy uses advantage-weighted flow matching and rejection sampling, while the low-level policy uses behavior-cloning flow matching.At deployment, both policies use rejection sampling in some baselines, while SHARSA and ASO use it only for the high-level policy.Source: §G.3 · p. 23, §H.2.1 · p. 25

H.1 ENVIRONMENTS

• Table 4 catalogs OGBench environment and dataset characteristics.Source: Table 4 · p. 24

H.2.1 ALGORITHMS

• The compared hierarchical baselines include absolute-subgoal, abstract-option, and action-chunking methods, with their option representations and policy extraction procedures differing.OPAL uses a β-VAE to encode trajectories; DSHARSA uses absolute subgoals, while HIQL shapes abstract subgoals with a state-value function.Source: §H.2.1 · pp. 25–27

H.2.2 DATASET SAMPLING FOR NON-HIERARCHICAL AND ACTION CHUNKING BASELINES

• Non-hierarchical and action-chunking baseline datasets are sampled to resemble hierarchical datasets, helping isolate the effect of option structure from dataset sampling.The action-chunking baseline uses the hierarchical high-level goal distribution and masks actions after target attainment.Source: §H.2.2 · p. 27

H.3 HYPERPARAMETERS

• Table 5 lists the hyperparameters used in the experiments.Source: Table 5 · p. 28

H.4 FURTHER RESULTS

• Figure 5 reports success rates on selected OGBench environments using default datasets, averaged over five evaluation tasks and four seeds.The caption specifies 95% bootstrap confidence intervals.Source: Figure 5 · p. 29

H.5 ANALYSIS AND ABLATIONS

• In cube-double and cube-quadruple, QSO alone consistently groups functionally equivalent behaviors while separating behavior groups.Source: Figure 6 · p. 31, Figure 7 · p. 32
• Relative achievement encoding substantially improves antmaze-giant performance but has little effect in cube-double.The authors hypothesize that relative encoding helps most when many absolute state-goal pairs can map to the same context-dependent achievement.Source: §H.5 · p. 35
• Both QSO and ASO converge to similar performance across three random high-level goal-sampling ratios in antmaze-giant and cube-double.Source: §H.5 · p. 35, Figure 11 · p. 35
• Environment reward labeling at the high level substantially improves QSO and ASO performance in cube-double, while its effect is small in antmaze-giant; for ASO, the environment-labeled policy with the strict-labeled critic yields 70% rather than 20%.Source: §H.5 · p. 36, Figure 12 · p. 36
• Low-level environment reward labeling reduces the option encoder’s fast-moving-dimension variance share from 79% to 8% in cube-double and 45% to 23% in antmaze-giant, without downstream performance loss.Source: §H.5 · pp. 33, 36
• Advantage weighting and rejection sampling yield more consistent performance for QSO and ASO; advantage weighting also increases the high-level policy’s goal dependence relative to behavior cloning.Source: §H.5 · p. 39, Figure 17 · p. 39
• QSO performance is largely insensitive to option dimensions from 5 to 20, but QSO and ASO degrade with 100-step options in antmaze-giant, whose trajectories are 200 steps long.Source: §H.5 · p. 40, Figure 18 · p. 40, Figure 19 · p. 40