Source-linked AI summary
Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic
Olivier Serris, Stéphane Doncieux, Olivier Sigaud
TL;DR
Long-horizon goal-conditioned reinforcement learning is difficult with sparse rewards, while inference-time planners introduce deployment complications and existing guidance methods have failure cases. The paper proposes RS and LG-AC, with LG-AC explicitly decomposing a subgoal-aware critic for intermediate-goal guidance while keeping the policy conditioned on the final goal. Across the tested environments, LG-AC achieves the best average mean and IQM performance.
Problem
Goal-conditioned reinforcement learning struggles with long horizons and sparse rewards, while planner-guided methods can require difficult or costly inference-time subgoal management.
Method
LG-AC conditions a value estimator on intermediate goals, decomposes it into per-goal values, and trains an actor conditioned only on the final goal.
Results
LG-AC achieves the best average IQM and mean performance across the tested environments.
Takeaways & Limitations
Action regularization and PBRS each exhibit task-specific failures, whereas LG-AC avoids the failures seen in the other baselines and performs best overall.
Takeaways & Limitations
The approach has increased computational cost that scales with the number of intermediate goals used to train the actor and critic.
Abstract
from arXiv · showhide
Goal-conditioned reinforcement learning struggles with long horizons when rewards are sparse. While a planner can provide subgoals to guide a low-level policy, its use at test time may introduce practical subgoal management difficulties. An alternative paradigm utilizes a high-level planner to assist learning, while the policy remains conditioned only on the final goal, enabling planner-free deployment. Among these methods, Reinforcement Learning with Imagined Subgoals (RIS) introduces a regularization term that encourages the policy to take the same actions for the final goal as it does for an intermediate goal. This regularization, however, may lead to goal-chaining issues when intermediate goals are low-dimensional. Potential-based reward shaping (PBRS) translates plans into an additional reward while ensuring that the optimal policy remains unchanged. Yet, it can generate deceptive rewards in terminal states. We study these failure cases and first propose an alternative reward shaping method (RS) that removes these deceptive rewards at the expense of theoretical guarantees of PBRS. Similar to this RS variant, we then propose another method named Locally-Guided Actor Critic (LG-AC) that rewards the agent for reaching intermediate goals. Unlike RS, where intermediate rewards are implicit in the shaping signal, we explicitly condition a value estimator on the full sequence of intermediate goals but represent the value function as a sum of subgoal-conditioned value functions, enabling dense hindsight relabeling. We evaluate all these methods in tasks with challenging goal-chaining requirements and empirically highlight specific cases in which either action regularization or reward shaping yield low performance, while LG-AC achieves the best overall performance across tasks.
1 Introduction
Goal-conditioned policies struggle with long horizons and sparse rewards, motivating planner-guided methods that can avoid planner use during deployment. The paper compares RIS, PBRS, RS, and LG-AC, finding LG-AC has the strongest aggregate performance across five environments.
- Motivation: Long horizons and sparse rewards make distant goals difficult for standard goal-conditioned policies.Hierarchical methods address this by decomposing tasks into subtasks.
- Planner-guided learning: Planner-guided approaches use options, intermediate goals, or graph search, but typically require a high-level planner during inference.Inference-time planning can increase computation cost or create subgoal-management difficulties.
- Failure modes: RIS regularizes actions for distant goals toward actions for midway goals, but low-dimensional intermediate goals can create goal-chaining failures.A reached intermediate goal may leave the agent in a state incompatible with completing the remaining trajectory.
- Proposed methods: RS removes deceptive terminal-state rewards from PBRS, while LG-AC explicitly conditions a decomposed value estimator on intermediate goals and supports hindsight relabeling.LG-AC keeps the actor conditioned on the final goal while using subgoal-aware critic guidance during training.
- Evaluation: LG-AC achieves the best overall mean and interquartile mean across five goal-conditioned environments.RIS fails in GC-Hopper and GC-Walker, PBRS struggles with terminal states, and RS performs poorly in AntMaze.
2 Preliminaries
The preliminaries formalize reinforcement learning, partial observability, goal-conditioned tasks, actor-critic learning, and hindsight experience replay. These frameworks explain both sparse-reward difficulty and how past trajectories can be relabeled for learning.
- Markov Decision Process: An MDP models states, actions, rewards, transitions, initial-state sampling, terminal states, horizon, and discounting.The agent chooses actions over time and seeks to maximize discounted cumulative reward.
- Partial observability: A POMDP extends an MDP with observations when the agent cannot directly observe the underlying state.Actions are selected from observations produced by an observation function.
- Goal-conditioned RL: Goal-conditioned reinforcement learning learns a universal policy that reaches multiple objectives represented in a goal space.The paper uses low-dimensional goals and a sparse reward for states satisfying the goal.
- Actor-critic methods: Actor-critic methods update a goal-conditioned actor using a critic that estimates expected action values for pursuing a goal.Off-policy learning is useful because past experiences can support learning across goals.
- Hindsight experience replay: HER densifies sparse-reward learning by relabeling failed goals with goals actually achieved at future trajectory states.Relabeling can be applied independently to transitions in off-policy algorithms.
3 Related Work
Related methods use intermediate goals either to regularize actions or to shape rewards for flat goal-conditioned policies. Their benefits are accompanied by goal-chaining risks for low-dimensional goals and terminal-state issues for PBRS.
- Plan-guided learning: The paper frames its approach as using high-level plans during training to distill planner-based rewards into a low-level policy.The deployed policy is intended to remain goal-conditioned rather than planner-conditioned.
- Policy regularization: RIS constrains action distributions for final and intermediate goals using KL-divergence regularization.The intended mechanism is to reuse knowledge of reaching nearby goals for distant-goal progress.
- Goal chaining: Low-dimensional goals may reward only one state element, allowing an intermediate goal to be reached in a state incompatible with later trajectory requirements.This mismatch creates goal-chaining issues when valid actions for intermediate and final goals differ.
- Reward shaping: PBRS adds a potential difference to rewards while theoretically preserving the original MDP’s optimal policy.The paper studies a sequence-of-goals shaping formulation as an alternative to action regularization.
4 Methods
The methods hide intermediate goals from the actor while giving them to a privileged critic, then decompose the critic into fixed-size per-goal values. LG-AC optimizes the actor across the plan’s intermediate goals and uses hindsight relabeling to learn the required critic values.
- 4.1 Intermediate goals as privileged information: The training planner supplies intermediate goals, but deployment uses a policy conditioned only on the final goal.The actor-critic formulation gives the critic access to the full goal sequence as privileged information.
- 4.1 Intermediate goals as privileged information: An SG-POMDP treats intermediate goals as hidden state variables and provides rewards for reaching them.The actor observes the environment state and final goal, while the intermediate sequence is updated as goals are achieved.
- 4.1 Intermediate goals as privileged information: The SG-POMDP yields denser learning signals but requires a critic to handle variable-length goal sequences and combinatorial goal subsets.These requirements increase architectural complexity and generalization demands.
- 4.2 Value Decomposition over individual goals: Per-goal value estimation gives the critic fixed-size inputs, supports arbitrary plan lengths, and can be learned with standard temporal-difference methods.This estimator is named the subgoal-aware critic.
- 4.2 Value Decomposition over individual goals: The total return decomposes exactly into a sum of per-goal contributions because rewards are additive and goals can be reached independently.Each contribution accumulates rewards only until its goal or a terminal state is reached.
- 4.3 Locally-Guided Actor Critic: LG-AC optimizes the actor for final and intermediate goals, using nearer-goal values to provide more local guidance when distant-goal estimates are noisy.The planner-generated sequence is used during training, and the actor objective sums critic values over the sequence.
- 4.3 Locally-Guided Actor Critic: Hindsight relabeling trains the critic on planned intermediate goals and currently achieved goals to densify learning signals.The relabeling strategy targets the critic values that appear in the actor loss.
5 Experiments
The experiments compare planner-assisted and unassisted goal-conditioned methods across five environments, testing distant goals, goal chaining, and terminal-state failures. LG-AC avoids the major failures of the baselines and achieves the strongest aggregate performance beyond median success.
- Experimental setup: All methods share SAC+HER, while planner-informed baselines add auxiliary objectives using intermediate goals; SAC+HER alone uses no planning information.A handcrafted planner supplies intermediate goals, and the study isolates the low-level policy’s contribution to success or failure.
- Experimental setup: Agents are evaluated on five continuous-control environments using success rates for increasingly difficult individual goals and mean success during training.The benchmark includes Dubins Hallway, SnakeMaze5, AntMaze, GC-Hopper, and GC-Walker; evaluation uses fixed goals of increasing difficulty.
- Baseline performance: SAC+HER reaches nearby goals but fails on distant goals in environments where the farthest goals require more than 100 time steps.In Dubins Hallway, where the farthest goal takes approximately 40 time steps, SAC+HER achieves high success across goals.
- Baseline performance: PBRS performs poorly because discounted shaping gives positive rewards without progress and terminal-state handling can reward early failure.When plan cost is unchanged, the PBRS bonus becomes C(s, g)(1 −γ); terminal failures receive a reward equivalent to the remaining plan length in affected locomotion tasks.
- Baseline performance: RIS performs well in Dubins Hallway, SnakeMaze5, and AntMaze but drops significantly in GC-Hopper and GC-Walker, where intermediate-goal regularization can induce myopic jumps.The reported GC-Hopper trajectories show RIS failing while LG-AC and RS complete the task.
- LG-AC performance: LG-AC is comparable to the best method in each environment, avoids the failures of other baselines, and significantly improves aggregate mean, IQM, and optimality-gap metrics.RIS, RS, and LG-AC have similar median performance, but LG-AC outperforms the baselines on the other aggregate metrics.
6 Conclusion
The study distills planning guidance into final-goal-conditioned policies for planner-free deployment and proposes LG-AC as an asymmetric actor-critic approach. Across five benchmarks, LG-AC achieves the best average performance, while its main limitation is computational cost scaling with the number of intermediate goals.
- LG-AC uses a per-goal critic decomposition to reward the actor for completing plans and provide local guidance for each individual goal.
- Across five environments, RIS fails in GC-Hopper and GC-Walker, PBRS has terminal-state limitations, and RS performs strongly except in AntMaze.
- LG-AC achieves the best average IQM and mean performance across the tested environments.
- LG-AC’s computational cost scales with the number of intermediate goals used to train both the actor and critic.
A Proof of Value Decomposition
The proof establishes that the subgoal-aware value function decomposes into independent per-goal contributions. Each contribution measures discounted reward until its goal is reached or a terminal state occurs.
- The proof uses induction to relate whether a goal remains in the active sequence to indicators that it has not previously been reached.
- The derivation applies a lemma about active goals before obtaining the value decomposition.
- The value function for a goal set is the sum of independent per-goal value functions.
- Each per-goal value function measures expected discounted reward from that goal until it is reached or a terminal state is encountered.
B Relabeling Strategies
The relabeling study compares three strategies for training the subgoal-aware critic. Sampling goals from the current plan is important, and the full LG-AC strategy performs at least as well as plan-sampling while costing more computationally.
- The critic always uses HER for the episode’s final goal and evaluates three approaches for the reward-conditioning goal.
- LG-AC-future randomly selects a goal achieved later in the same trajectory, whereas LG-AC-sample samples from the current plan.
- LG-AC trains the critic on all goals in the current plan to reduce out-of-distribution values during actor optimization.
- LG-AC-future achieves the lowest performance, while LG-AC performs similarly to or better than LG-AC-sample at higher computational cost.
C Environment details
The experiments use five continuous-control environments with handcrafted intermediate-goal planners. Navigation tasks use expert-graph shortest paths, while locomotion tasks use evenly spaced goals between the agent and final target.
- Figure 5 shows handcrafted planning graphs for Dubins Hallway, SnakeMaze5, and AntMaze, while locomotion planners output uniformly spaced points.
- Dubins Hallway: Dubins Hallway controls a car in a 2D maze, with position goals reached inside a radius-0.1 ball.
- SnakeMaze5: SnakeMaze5 controls a ball in a 2D maze, and the harder setup raises the velocity limit to [−10, 10] m/s to require anticipating turns and decelerating.
- GC-Hopper and GC-Walker: GC-Hopper and GC-Walker are locomotion tasks sharing a goal defined by desired x-position displacement, with GC-Walker using a two-legged version of Hopper.
- AntMaze: AntMaze uses a quadruped navigating toward an (x, y) goal in a maze, with termination triggered by reaching the goal or an unhealthy posture.
- Handcrafted Planner: For Dubins Hallway, SnakeMaze5, and AntMaze, the planner computes shortest paths through an expert graph; for Hopper and Walker, it spaces goals evenly.
D Hyperparameters
The hyperparameter tables specify common settings, environment-specific exceptions, and batch-size configurations; an empirical upper bound is used to avoid unstable maze collisions.
- Common hyperparameters are listed in Table 1, with environment-specific exceptions recorded separately.
- The maze-related upper bound was determined empirically because larger values make collisions unstable and can allow occasional passage through walls.
- Table 2 reports batch sizes by environment and method, using 256 for LG-AC and tuned values for other methods.
E Impact of batch size on performance
LG-AC processes all intermediate goals for each transition, increasing its effective batch size and computational cost; baseline batch sizes are tuned to control this comparison.
- LG-AC’s effective batch size is calculated as base batch size 256 multiplied by average plan length per transition.
- LG-AC’s transition loss incorporates all intermediate goals, increasing the number of targets processed per update.
- The resulting effective batch size and computational cost could bias comparisons if gains came from additional computation alone.
- Baseline batch sizes were independently tuned to control for LG-AC’s larger effective computation.
F Impact of the KL penalty factor on RIS performance
RIS performance is sensitive to the KL penalty factor and reward specification, whereas LG-AC remains stable across both reward types without additional hyperparameters.
- RIS performance varies smoothly with KL penalty factor α and remains near-optimal over a specific range for both reward specifications.
- The effective α range differs between positive and negative rewards, indicating reward-dependent tuning requirements for RIS.
- LG-AC introduces no additional hyperparameters and maintains stable performance across positive and negative reward specifications.
- Table 5 aggregates success rates over SnakeMaze5 and Dubins Hallway across 10 independent runs for each reward specification.