Source-linked AI summary
A note on goal-based hierarchical RL
Kevin Murphy
TL;DR
ACGVF gives agents control over goal choice and termination but assumes full observation, while recurrent-agent designs use internal history state yet take goals as external inputs. This note combines those ideas in an HHMM, making goals internal and persistent between completion boundaries. The resulting framing supports goal, action, and termination decisions based on filtered state, with explicit call–return structure and distinct segment-level and continual objectives.
Problem
ACGVF assumes full observation, whereas recurrent-agent policies use internal history state but take goals as external inputs.
Source: §1 · p. 1Method
Results
The HHMM framing yields persistent goal intentions, filtered-state decisions, explicit call–return semantics, and segment-level and continual values distinguished by an upper-level continuation term.
Source: §6 · p. 10Takeaways & Limitations
Goal choice, action choice, and termination can depend on filtered state rather than requiring observation to stand in for world state.
Source: §6 · p. 10Boundary states differ in distribution from arbitrary states, so a critic fit across all time steps may be least accurate where the upper-level advantage is evaluated.
Source: §5 · p. 9
Abstract
from arXiv · showhide
The agent-centric general value function (ACGVF) construction of \citet{tasse2026goal} lets the agent make two decisions that are normally imposed by the environment or agent designer: which goal to pursue and when to declare a goal as finished (in addition to choosing the action). This is a very general framework that subsumes almost all prior work on reinforcement learning, control and planning, as well as more general formalisms proposed in the cognitive sciences. However, it assumes the environment is fully observed, i.e., that the observation is a sufficient statistic. In \citet{murphy2025rl}, a general agent design was proposed where the policy is based on an internal belief state $z_t$ and an internal goal; however, the goals were assumed to be externally provided. In this note, we unify and extend these two approaches using the formalism of hierarchical hidden Markov models (HHMM) \citep{murphy2001hhmm}.
1 Introduction
The introduction contrasts agent-chosen goals and termination under full observation with recurrent policies that use internal history state but take goals as external inputs. It proposes unifying these approaches by making the goal part of the agent’s dynamical state, with the done action linking goal-level and within-goal timescales.
1 Introduction
- The ACGVF lets agents choose goals and when to finish them, but assumes the world state is also the controller’s sufficient information state.Its policy is defined directly on S_t.Source: §1 · p. 1
- The recurrent-agent design acts on an internal history summary, but treats the goal as an external input.This separates controller information from observations without making goal choice internal.Source: §1 · p. 1
- The proposed unification makes the goal another component of the agent’s dynamical state, alongside a piecewise-constant goal state and an observation-updated within-goal state.The done action couples the two timescales as the boundary event in HHMM call–return semantics.Source: §1 · p. 1
2 A controlled two-level HHMM
The controlled two-level HHMM represents an agent whose actions are generated from an observation-conditioned internal state, while goals persist between completion boundaries. Its call–return structure selects a new goal at those boundaries and preserves the internal state across goal changes.
2 A controlled two-level HHMM
- The goal selector µη is evaluated only at a boundary, so the selected goal persists between boundaries rather than being resampled each step.Between boundaries, the goal transition is the identity map.Source: §2 · p. 2
- The paper represents the distribution as a generic two-level HHMM drawn as a DBN, labeling its lower (concrete) level with superscript 1 and upper (abstract) level with superscript 2.Source: §2 · p. 2
- Dt indicates whether the transition produced by At completes the current goal; completion returns control to the upper-level selector.Without completion, the current goal is held fixed.Source: §2 · p. 2
- The agent’s actions are the process outputs, while the process is conditioned on observations and includes the environment’s hidden physical state.The observation is distinct from the hidden state.Source: §2 · p. 2
- A lower-level sub-process boundary is not directly observed and must be inferred because its finish variable does not affect the output.The lower-level finish variable marks when the sub-model has just finished.Source: §2 · p. 3
- Unlike the original HHMM formulation, changing goals does not reset the lower-level state; the internal state can depend on its prior value across boundaries.The state persists because it is driven by incoming observations, rather than serving as a sub-model index reset on entry.Source: §2 · p. 3
3 Goal satisfaction and agent-centric reward in latent state
The framework replaces inaccessible transition-based goal satisfaction with an internal satisfaction model under partial observability. It separates learning to achieve and recognize a fixed goal from choosing among goals.
3 Goal satisfaction and agent-centric reward in latent state
- Under partial observability, goal satisfaction cannot generally be a predicate of the inaccessible world transition, so the framework uses an internal satisfaction model.Its targets may describe a latent-space set, an observation-inferred proposition, or a distributional target.Source: §3 · p. 3
- Continuing and correct termination receive environmental reward, while incorrect termination receives the lower-bound penalty rmin.This defines an internal reward for evaluating goal completion in latent state.Source: §3 · p. 4
- The within-goal value formulation recovers the ACGVF Bellman equation by replacing the world state s with the agent’s information state z.The fixed-goal value accounts for the next value only when the done action is false.Source: §3 · p. 4
- For a fixed goal, the greedy lower controller learns both to reach it and recognize its completion; the upper controller selects among mastered goals.Goal selection via µ*(g | z) and goal achievement via π*g(a, d | z) are distinct problems.Source: §3 · p. 4
4 Segment values versus continual values
Segment values assign no continuation value after a goal ends, whereas continual values account for future goals. This distinction supports an agent-delimited stream of goals in both HHMM and semi-MDP views.
4 Segment values versus continual values
- Segment value Vg assigns zero future value after d = 1, while continual value Wg includes H at the next upper-level decision point.H(z) is the value after a goal finishes; Vg(z) assumes achieving g ends the process.Source: §4 · pp. 4–5
- Continual value Wg includes the upper-level continuation H after goal completion, optimizing successive agent-delimited goals rather than one segment.Removing the continuation term and setting γ = 1 recovers the segment-level Vg formulation.Source: §4 · pp. 4–5
- When two goal-completing actions earn the same immediate reward, Wg prefers the one leading to valuable future goals, unlike Vg.The contrast is between accounting for post-completion prospects and valuing only the current segment.Source: §4 · p. 5
- The construction is an HHMM in primitive time and an option-like semi-MDP at goal boundaries, with goal labels, termination, and latent information state internal to the agent.At a boundary, the induced return follows the semi-Markov upper-level structure.Source: §4 · p. 5
5 Learning and inference
The section describes learning and inference for the hierarchical agent, including latent-goal inference, satisfaction and value learning, and actor–critic optimization. It also states conditions and regularizers that constrain these procedures.
5 Learning and inference
- When goals and done decisions are latent, the controlled HHMM likelihood can be estimated with EM; its discrete-state exact E step is linear in sequence length at fixed depth and cardinality.Amortized filtering, particle methods, or structured variational inference are suggested for neural and continuous internal states.Source: §5 · p. 5
- A single primitive-time replay stream can train both hierarchy levels, with boundaries supplying segmentation and the goal selector trained on boundary transitions.The upper-level return accounts for segment duration.Source: §5 · pp. 5–6
- The objective combines environmental-return optimization with satisfaction, temporal-difference, and actor–critic regression losses, whose relative weights are controlled separately.The actor objective J has implicit coefficient 1; λcrit controls critic regression strength.Source: §5 · pp. 6–7
- Supervised satisfaction learning requires an independent target; without one, satisfaction must be learned as latent or weakly supervised, since using its own prediction supplies no signal.A target may come from an environment label, observable certificate, hindsight, or belief-state model.Source: §5 · p. 6
- The TD loss learns optimal internal-reward goal values, while J grounds goal selection in environmental reward; the actor objective can instead use internal reward.If the actor uses internal reward, its critics and advantages must use that same reward.Source: §5 · pp. 6, 7, 9
- Duration regularization can prevent one-step goals or no boundaries, while learned goal vocabularies may need diversity or information constraints.These regularizers define the abstraction; environmental reward determines which abstractions are useful.Source: §5 · p. 7
- The state-goal pair must be sufficient for control; this is exact for a belief state and an operative approximation for a learned state.The pair must preserve future observation and reward distributions under candidate actions for histories mapped to the same state and goal.Source: §5 · p. 7
- An upper critic can be dropped because its value equals the lower critic’s value, but a critic trained on all states may be least accurate at boundary states.Boundary states are not distributed like arbitrary states, which can affect the upper-level advantage estimate.Source: §5 · pp. 8–9
6 Conclusion
The HHMM framing unifies agent-selected goals and completion with an internal information state, giving the controller a temporal grammar for goal pursuit and return. It supports persistent intentions, partial observability, continual objectives, structured credit assignment, and deeper hierarchies.
6 Conclusion
- The HHMM gate makes goal selection a persistent intention rather than an independent per-step label.The controller returns to the goal selector when it declares completion.Source: §6 · p. 10
- Goal choice, action choice, and termination can depend on filtered state Zt without identifying observation with world state.Source: §6 · p. 10
- The done action explicitly returns control from the within-goal controller to the goal selector.Source: §6 · p. 10
- Segment-level ACGVFs and continual hierarchical values differ by the upper-level continuation term.Source: §6 · p. 10
- Primitive rewards train the lower controller, while boundary-to-boundary returns train the goal selector.Source: §6 · p. 10
- Additional hierarchical states represent nested intentions, with each level changing only after all lower descendants return.Source: §6 · p. 10