Source-linked AI summary
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Tejas D. Kulkarni, Karthik R. Narasimhan, Ardavan Saeedi, Joshua B. Tenenbaum
TL;DR
Sparse-feedback reinforcement learning needs better exploration because agents may fail to learn robust value functions. The paper introduces h-DQN, which combines hierarchical value functions, intrinsic goals, and entity-relation goal specifications across temporal scales. It demonstrates the approach on a discrete stochastic process and Montezuma’s Revenge, while identifying missing object disentanglement and memory components.
Problem
Sparse feedback creates an exploration challenge that can prevent reinforcement learning agents from learning robust value functions.
Method
h-DQN uses a meta-controller to select intrinsic goals and a controller to choose atomic actions that satisfy them, with hierarchical value functions operating at different temporal scales.
Results
The approach learns hierarchical goal sequences in a discrete stochastic process and learns to reach the key and open the door in Montezuma’s Revenge, obtaining around +400 reward per episode.
Takeaways & Limitations
Entity- and relation-based intrinsic goals provide a promising space for temporally extended exploration in environments with delayed rewards.
Takeaways & Limitations
The framework lacks automatic object disentanglement from raw pixels and short-term memory for storing histories of goals, actions, and representations.
Abstract
from arXiv · showhide
Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. The primary difficulty arises due to insufficient exploration, resulting in an agent being unable to learn robust value functions. Intrinsically motivated agents can explore new behavior for its own sake rather than to directly solve problems. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical value functions, operating at different temporal scales, with intrinsically motivated deep reinforcement learning. A top-level value function learns a policy over intrinsic goals, and a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse, delayed feedback: (1) a complex discrete stochastic decision process, and (2) the classic ATARI game `Montezuma's Revenge'.
1 Introduction
Sparse-feedback environments remain difficult because effective exploration and multi-level abstractions are needed. The proposed h-DQN framework combines hierarchical value functions with intrinsic goals across temporal scales.
- Sparse feedback makes goal-directed learning in complex environments a fundamental artificial-intelligence challenge.
- Existing exploration methods improve on ϵ-greedy but remain limited because their models operate at the level of basic actions.
- h-DQN integrates deep reinforcement learning with hierarchical value functions and intrinsically motivated goals to aid exploration.
- The framework uses a top-level module to choose goals and a lower-level module to select actions until goals are reached or episodes terminate.
- Training occurs at different temporal scales to optimize intrinsic controller rewards and extrinsic meta-controller rewards.
2 Literature Review
Prior work develops temporal abstractions, hierarchical decompositions, intrinsic motivation, and object-based representations for reinforcement learning. The paper combines these directions by simultaneously learning options and a policy to compose them in deep reinforcement learning.
- The options framework extends action spaces with multi-step policies that can be selected and terminated stochastically, yielding a semi-Markov decision process.
- Option discovery remains an open problem when nonlinear state approximations are required.
- Earlier hierarchical approaches decompose decisions across granularity levels or combine value functions for constituent subtasks.
- This work simultaneously learns options and a control policy to compose them in a deep reinforcement learning setting.
- Treating the option as an input enables shared learning between options and potential scalability to many options.
- Intrinsic reward research includes salient events, evolved reward functions, predictive-model improvements, information maximization, and policy-structure-based subgoal discovery.
- Object-oriented representations encode states through objects and their relations, while deep networks have struggled with sparse, delayed rewards in high-dimensional tasks.
- Cognitive research motivates intrinsic goals based on entities, agents, actions, quantities, space, and social structures.
3 Model
The model uses a two-level hierarchy in which a meta-controller selects goals and a controller executes actions to achieve them. Separate deep-Q networks learn goal selection from extrinsic rewards and goal-directed action policies from intrinsic rewards at different temporal scales.
- Temporal Abstractions: Goal-directed option policies provide intrinsic motivation for exploring sequences of goals that can maximize cumulative extrinsic reward.The controller learns policies for individual goals while the meta-controller learns how to sequence them.
- Temporal Abstractions: The meta-controller observes the state and selects a goal, while the controller uses the state and goal to choose actions until the goal is reached or the episode terminates.An internal critic evaluates goal achievement and supplies intrinsic rewards to the controller.
- Deep Q-Learning: Separate deep-Q networks represent the controller and meta-controller, with the controller estimating action values conditioned on the current goal.The controller’s Q-function evaluates actions given state and goal, while the meta-controller selects goals from state.
- Learning Algorithm: Meta-controller transitions occur only when a controller terminates, so goal-level learning operates at a slower time-scale than action-level learning.The two networks store distinct experience types: goal transitions for Q2 and state-action-goal transitions for Q1.
- Deep Q-Learning: The controller is trained with a deep-Q loss using replayed transitions and a target formed from immediate intrinsic reward plus discounted next-action value.Previous-iteration parameters are held fixed during optimization, and the corresponding meta-controller loss is derived similarly.
4 Experiments
The experiments evaluate h-DQN on a stochastic decision process and Montezuma’s Revenge, both involving delayed rewards. Across these settings, hierarchical goal selection and intrinsic motivation support exploration and reward acquisition.
- 4.1 Discrete stochastic decision process: The experiments cover a stochastic decision process with delayed rewards and the ATARI game Montezuma’s Revenge.The decision process uses stochastic transitions and history-dependent extrinsic rewards, while Montezuma’s Revenge requires long action sequences before receiving rewards.
- 4.1 Discrete stochastic decision process: h-DQN alternates between meta-controller goal selection and controller actions until the selected goal is reached or the episode terminates.The algorithm stores controller transitions at each action and meta-controller transitions when a goal terminates, then anneals exploration parameters.
- 4.1 Discrete stochastic decision process: Intrinsic rewards are positive when the controller reaches the corresponding goal, encouraging visits to otherwise difficult states such as s6.The model treats states as possible goals and uses goal achievement to provide intrinsic feedback.
- 4.1 Discrete stochastic decision process: Around 0.13 average reward was achieved by the hierarchical approach, whereas Q-learning converged to a sub-optimal policy obtaining 0.01 by reaching s1 directly.The comparison averages reward over 10 runs; the hierarchical policy selects goals s4, s5, or s6 that statistically lead to visiting s6 before s1.
- 4.1 Discrete stochastic decision process: Visits to states s3, s4, s5, and s6 increased over training, indicating that the model increasingly selected goals leading to the critical state s6.Figure 4 averages visits over the last 1000 episodes, with s2 as the initial state and s1 as the terminal state.
- 4.2 ATARI game with delayed rewards: In Montezuma’s Revenge, entity- and relation-based intrinsic goals guide exploration toward objects such as the key and doors.The internal critic evaluates goals in ⟨entity1, relation, entity2⟩ space, rewarding completion when the agent entity reaches another entity.
- 4.2 ATARI game with delayed rewards: The architecture uses convolutional DQN modules for the controller and meta-controller, with the controller receiving a binary mask encoding the selected goal location.Training first pre-trains the controller for about 2.3 million steps and then jointly trains both modules.
- 4.2 ATARI game with delayed rewards: After an additional 2 million joint-training steps, the model learned to obtain around +400 reward per episode by reaching the key and opening the door.The agent increasingly selected and successfully reached the key, while learning simpler goals before harder goals such as the key and bottom ladders.
5 Conclusion
h-DQN combines hierarchical value functions across time scales with intrinsic motivation to support exploration under delayed rewards. The framework also identifies limitations in representation learning and memory for harder settings.
- 5 Conclusion: h-DQN uses hierarchical value functions operating at different time scales to decompose behavior and support intrinsically motivated exploration.Temporal decomposition lets the agent pursue intrinsic behavior while addressing environments with delayed rewards.
- 5 Conclusion: Parameterizing intrinsic motivation over entities and relations provides a promising space for temporally extended exploration.The authors identify alternative goal parameterizations as a direction for future work.
- 5 Conclusion: The framework lacks automatic object disentanglement from raw pixels and memory for storing histories needed in longer-range, non-Markovian settings.The authors suggest combining h-DQN with deep generative image models and flexible episodic memory.