Source-linked AI summary
FeUdal Networks for Hierarchical Reinforcement Learning
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu
TL;DR
Long-term credit assignment and memorisation remain difficult in sparse-reward and non-Markovian environments. FuN addresses these challenges with a decoupled Manager–Worker hierarchy that operates across temporal resolutions, and experiments show improved performance across ATARI and 3D DeepMind Lab tasks.
Problem
Sparse rewards and non-Markovian environments make long-term credit assignment and memory especially challenging for deep reinforcement learning.
Method
FuN decouples a low-resolution Manager that sets latent goals from a high-resolution Worker that produces primitive actions and learns through intrinsic reward.
Results
FuN significantly improves long-term credit assignment and memorisation across selected ATARI games and 3D DeepMind Lab memory tasks.
Takeaways & Limitations
FuN’s modular hierarchy supports sub-goals and behavioral primitives associated with different Manager goals.
Takeaways & Limitations
The Worker’s transition-policy approximation depends on its trajectories following the assumed distribution around the Manager’s direction.
Abstract
from arXiv · showhide
We introduce FeUdal Networks (FuNs): a novel architecture for hierarchical reinforcement learning. Our approach is inspired by the feudal reinforcement learning proposal of Dayan and Hinton, and gains power and efficacy by decoupling end-to-end learning across multiple levels -- allowing it to utilise different resolutions of time. Our framework employs a Manager module and a Worker module. The Manager operates at a lower temporal resolution and sets abstract goals which are conveyed to and enacted by the Worker. The Worker generates primitive actions at every tick of the environment. The decoupled structure of FuN conveys several benefits -- in addition to facilitating very long timescale credit assignment it also encourages the emergence of sub-policies associated with different goals set by the Manager. These properties allow FuN to dramatically outperform a strong baseline agent on tasks that involve long-term credit assignment or memorisation. We demonstrate the performance of our proposed system on a range of tasks from the ATARI suite and also from a 3D DeepMind Lab environment.
1. Introduction
FuN addresses long-term credit assignment and memory challenges with a hierarchical Manager–Worker architecture that separates goal setting from action generation across temporal resolutions. Experiments on ATARI and 3D DeepMind Lab show improved long-term credit assignment and memorisation.
- Sparse rewards and non-Markovian environments make long-term credit assignment and selective memory especially challenging for deep reinforcement learning.
- The Manager sets latent goals at a lower temporal resolution, while the Worker produces primitive actions at higher temporal resolution.
- Manager learning is decoupled from Worker learning: the Worker follows goals through intrinsic reward, while the Manager learns from extrinsic environmental reward.
- FuN combines end-to-end differentiability, directional goals, a transition policy gradient, and a dilated LSTM for long-horizon learning.The dilated LSTM supports gradient flow through large temporal hops and hundreds of steps.
- Experiments on ATARI and 3D DeepMind Lab show that FuN significantly improves long-term credit assignment and memorisation.Ablations identify transition policy gradient and directional goals as crucial for best performance.
2. Related Work
Related hierarchical reinforcement learning work includes options and end-to-end option learning, while FuN distinguishes itself through explicit latent goals and naturally diverse sub-goals. Prior end-to-end options can collapse into trivial behaviors requiring regularization.
- The options framework uses temporally extended sub-policies selected by a policy-over-options until termination.
- End-to-end option learning can degenerate into one active option or option changes at every step, motivating regularizers for extended temporal abstractions.
- FuN’s top level produces explicit goals for the bottom level, with sub-goals emerging as diverse directions in latent state space.
- FuN achieves significantly better ATARI scores than Option-Critic.
- Non-hierarchical methods also improve reinforcement learning through auxiliary rewards and unsupervised auxiliary tasks.
3. The model
FuN is a recurrent two-module architecture in which the Manager emits latent directional goals and the Worker converts them into primitive actions. Its goal embedding, intrinsic reward, and transition-policy training align Worker behavior with Manager-selected directions.
- Architecture: FuN shares a perceptual representation while the recurrent Manager produces goals and the recurrent Worker produces actions conditioned on those goals.
- Goal embedding: The Worker’s policy uses a multiplicative interaction between action embeddings and a pooled goal embedding in a low-dimensional space.Goal pooling makes Manager conditioning vary smoothly, while the bias-free projection ensures the goal influences the final policy.
- Manager learning: The Manager’s goals represent advantageous directions in latent state space over horizon c, which sets its temporal resolution.
- Worker learning: The Worker maximizes a weighted combination of environmental and intrinsic rewards, with α controlling the intrinsic-reward influence.
- Temporal resolution: Manager and Worker may use different discount factors, allowing the Worker to focus on immediate rewards while the Manager considers longer-term outcomes.
- Transition policy gradients: Transition policy gradients model the Worker’s behavior through predicted end states, allowing Manager gradients to skip over complex Worker trajectories.FuN assumes the state-space displacement follows a von Mises–Fisher distribution, while intrinsic reward encourages that transition model.
4. Architecture details
FuN combines a convolutional perceptual module with standard Worker memory and a dilated-LSTM Manager that updates hierarchical memory at lower temporal resolution. Its Manager preserves long-term information through separately updated core groups while processing every input and producing pooled outputs.
- Perceptual module: The perceptual module uses a CNN followed by a 256-unit fully connected layer, with rectifier nonlinearities after each layer.The CNN has 16 8x8 stride-4 filters followed by 32 4x4 stride-2 filters.
- Recurrent modules: The Worker uses a standard LSTM, while the Manager uses a novel dilated LSTM; both recurrent networks have 256 hidden units.The dilated LSTM is designed for the Manager’s lower temporal resolution.
- Dilated LSTM: The dilated LSTM contains r separate groups of sub-states, with the modulo operation selecting which group is updated at each time step.The same LSTM parameters govern updates across all r groups.
- Dilated LSTM: Only the corresponding state group is updated at each step, while the output is pooled across the previous c outputs.This design lets core groups preserve memories for long periods while the network processes every input and updates its output at every step.
- Dilated LSTM: In experiments, the dilated LSTM uses dilation radius r = 10.Unlike clockwork RNNs, it observes all available training data rather than operating only at a fixed slow pace.
5. Experiments
FuN is evaluated across Atari, Montezuma’s Revenge, and DeepMind Lab memory tasks, with ablations testing its hierarchical design and temporal resolution. Across these settings, FuN generally improves learning speed, final performance, long-term credit assignment, and memorisation relative to recurrent LSTM baselines.
- Montezuma’s Revenge: FuN solves Montezuma’s Revenge’s first room in less than 200 epochs and later reaches up to 2600 points.The LSTM baseline takes more than 300 epochs to reach 400 points and remains there until about 900 epochs.
- ATARI: On Atari, FuN strongly outperforms other methods on Ms. Pacman, Amidar, and Gravitar when using a 0.99 Manager discount.On Frostbite, the best FuN agent with a 0.95 Manager discount outperforms the rest by a factor of 7, while some reactive games favor LSTM.
- ATARI: FuN performs well with BPTT of 400, whereas an unroll of 100 significantly hurts the LSTM baseline.The authors attribute FuN’s longer-sequence performance to its dilated LSTM and connect it to memory-related tasks.
- Memory in Labyrinth: FuN consistently outperforms the LSTM baseline on Labyrinth memory tasks, learning faster and reaching higher final reward.In the water maze, FuN combines meaningful sub-policies with memory, whereas the LSTM agent repeatedly circles at roughly the same radius.
- Ablative analysis: Ablations show that transitional policy gradient and directional goals are crucial for best performance, while removing Manager dilation causes catastrophic failure.Reducing the prediction horizon to c = 1 also hurts performance, though less than other ablations.
- ATARI action repeat transfer: FuN’s transition policy is invariant to the agent’s primitive-action embodiment, suggesting potential transfer across agents with different physical designs.The transferred FuN agent significantly outperforms every other method and shows positive transfer on each tested environment without action repeat.
6. Discussion and future work
FuN separates sub-goal discovery from primitive-action generation, creating a stable hierarchy that makes long-term credit assignment and memorisation more tractable. Its modular structure also suggests reuse across skills, tasks, and embodiments.
- FuN formulates sub-goals as directions in latent state space that translate into meaningful behavioural primitives.
- FuN separates sub-goal discovery and setting from primitive-action generation, allowing the two modules to learn in complementary ways.
- The experiments demonstrate that FuN makes long-term credit assignment and memorisation more tractable.
- FuN’s modular structure supports potential transfer and multitask learning through reusable behavioural primitives or transferable Manager policies.