Source-linked AI summary
Deep Successor Reinforcement Learning
Tejas D. Kulkarni, Ardavan Saeedi, Simanta Gautam, Samuel J. Gershman
TL;DR
Deep reinforcement learning needs robust value functions from raw observations and rewards, while model-free and model-based approaches offer different trade-offs. DSR generalizes successor representations with end-to-end deep networks that factor rewards from future feature occupancy. It demonstrates goal-directed behavior across MazeBase and Doom, adapts quickly to distal reward changes, and extracts plausible subgoals, although discriminative feature learning remains a major issue for scaling.
Problem
Reinforcement learning lacks a single approach combining model-free efficiency with model-based flexibility while learning value functions from raw observations and rewards.
Method
DSR uses separate deep networks to learn reward features and successor features from raw observations, estimating values through their dot product.
Results
DSR shows efficacy in MazeBase and Doom, adapts quickly to distal reward changes, and extracts plausible subgoals under a random policy.
Takeaways & Limitations
Factoring rewards from successor dynamics supports distal reward sensitivity and subgoal extraction for hierarchical reinforcement learning.
Takeaways & Limitations
Scaling DSR to more expressive environments requires better discriminative features, potentially combining deep generative and self-supervised models.
Abstract
from arXiv · showhide
Learning robust value functions given raw observations and rewards is now possible with model-free and model-based deep reinforcement learning algorithms. There is a third alternative, called Successor Representations (SR), which decomposes the value function into two components -- a reward predictor and a successor map. The successor map represents the expected future state occupancy from any given state and the reward predictor maps states to scalar rewards. The value function of a state can be computed as the inner product between the successor map and the reward weights. In this paper, we present DSR, which generalizes SR within an end-to-end deep reinforcement learning framework. DSR has several appealing properties including: increased sensitivity to distal reward changes due to factorization of reward and world dynamics, and the ability to extract bottleneck states (subgoals) given successor maps trained under a random policy. We show the efficacy of our approach on two diverse environments given raw pixel observations -- simple grid-world domains (MazeBase) and the Doom game engine.
1 Introduction
Successor representations offer a third reinforcement-learning approach by factoring value into future state occupancy and rewards. DSR extends this factorization to raw observations with deep networks and evaluates it in grid-world and Doom environments.
- Successor representations factor the value function into expected discounted future state occupancies and immediate rewards.
- SR combines model-free-like computational efficiency with model-based-like flexibility, including rapid adaptation to distal reward changes.
- The framework also extracts candidate subgoals from successor representations under a random policy for hierarchical reinforcement learning.
- DSR learns reward features and successor features with separate deep networks from raw observations, estimating value through their dot product.
- DSR experiments cover goal solving in MazeBase grid worlds and resource navigation in the Doom game engine.
2 Related work
Prior work connects successor representations to neuroscience, game evaluation, option modeling, and deep value-function approximation. These studies motivate DSR's relationship to occupancy-based representations and deep reinforcement learning.
- Successor representations have been used to model cognitive phenomena including episodic memory, hippocampal preplay, and rapid path planning.
- Related occupancy-based models evaluate Go positions or construct general option models that can recover traditional option models given rewards.
- Deep-Q learning and related methods learn Q-value functions from high-dimensional complex input states, forming a neighboring deep value-approximation literature.
3 Model
DSR uses deep networks to factor value estimation into learned reward features and successor feature occupancy. Its losses jointly train reward prediction, successor estimation, and shared state representations for raw-observation reinforcement learning.
- 3.3 Deep successor representation: Large state spaces motivate nonlinear function approximation, representing each state with a D-dimensional feature vector from a deep neural network.The feature map is fθ: S → R^D.
- 3.3 Deep successor representation: DSR estimates feature-based successor occupancy with a separate network and approximates immediate rewards as a linear function of the shared features.The successor network uα approximates m_s^a, while R(s) ≈ φ_s · w.
- 3.3 Deep successor representation: The resulting Q-value combines successor features with reward weights, while the optimal-policy successor representation follows a Bellman equation.The maximizing next action is selected using the successor estimate and reward weights.
- 3.4 Learning: Training uses separate reward and successor losses, with a cached successor-network parameter for stable Q-learning with function approximation.The cached parameter α_prev is updated periodically to α.
- 3.4 Learning: The shared representation is trained to predict immediate rewards and discriminate states, using an autoencoder reconstruction loss as an intrinsic reward signal.The composite objective sums the three loss functions, and parameters are optimized online with stochastic gradient descent.
- 3.4 Learning: Optimization alternates feature and reward learning with successor learning, using replay memory and annealed exploration during training.The implementation uses a replay memory of size 1e6, learning rate 2.5e−4, momentum 0.95, discount factor 0.99, and ε annealed from 1 to 0.1.
4 Automatic Subgoal Extraction
The method extracts subgoals from successor representations learned under a random policy by identifying bottleneck states through normalized cuts. DSR training and subgoal selection are implemented with replay-based gradient updates and repeated cut participation statistics.
- 4 Automatic Subgoal Extraction: Sparse or delayed rewards make policy learning difficult, motivating exploration across different temporal scales through hierarchical reinforcement learning.Options can target subgoals and provide intrinsic motivation.
- 4 Automatic Subgoal Extraction: DSR training initializes replay memory and parameters, explores with ε-greedy actions, stores transitions, and alternates gradient updates for reward-feature and successor losses.The algorithm begins with ε = 1 and samples minibatches from replay memory during episodes.
- 4 Automatic Subgoal Extraction: Under a random policy, the learned successor representations are collected across many states and converted into an affinity matrix using pairwise radial-basis-function similarities.The sampled collection is T = {m_s1,a1, m_s2,a2, ..., m_sn,an}.
- 4 Automatic Subgoal Extraction: The second-largest eigenvalue of the normalized-cut matrix approximates the minimum normalized cut, whose endpoint states become plausible subgoal candidates.States are ranked by how often they lie along the cut, and the top-k are selected as subgoals.
5 Experiments
Experiments evaluate DSR on raw-pixel MazeBase and Doom tasks, test distal-reward adaptation, and extract subgoals from successor representations. DSR successfully navigates the environments, adapts by updating reward weights, and produces useful environment partitions and subgoal candidates.
- 5.1 Goal-directed Behavior: Experiments compare DSR with DQN on raw-pixel goal-reaching tasks in MazeBase and Doom.MazeBase uses grid-world navigation; Doom requires navigating a 3D maze to collect a resource.
- 5.1 Goal-directed Behavior: DSR successfully navigates the Doom environment to obtain the reward and is competitive with DQN.The task imposes a -0.01 per-step penalty and gives a positive reward after collecting an item.
- 5.2 Value function sensitivity to distal reward changes: Changing the distal goal reward from 1.0 to 3.0 lets DSR quickly adapt by updating reward weights w while leaving the SR unchanged.The experiment changes only the reward scalar while keeping the goal location fixed.
- 5.3 Extracting subgoals from the DSR: Under a random policy, normalized cuts on DSR successor representations identify useful subgoals and cluster environments into reasonable segments.The procedure collects transitions and successor representations after training, then extracts bottleneck states and partitions.
- 5.3 Extracting subgoals from the DSR: Subgoal candidates are often noisy because the successor-representation samples come from a random policy.The authors propose statistically robust extraction methods as future work.
6 Conclusion
The paper presents DSR as a deep reinforcement-learning framework that factorizes value estimation into successor representations and immediate reward predictions. It reports sensitivity to distal reward changes and subgoal extraction, while identifying discriminative feature learning and hierarchical integration as important future challenges.
- Conclusion: DSR learns goal-directed behavior from raw sensory observations by estimating value as the inner product of successor representations and immediate reward predictions.The factorization separates successor structure from reward prediction.
- Conclusion: The factorized value function increases sensitivity to distal reward changes and permits subgoal extraction from successor representations under a random policy.The extracted subgoals are intended to support hierarchical reinforcement learning.
- Conclusion: A major issue is learning discriminative features, especially for scaling DSR to more expressive environments.The authors suggest combining deep generative and self-supervised models in future work.
- Conclusion: Future work plans to combine DSR with hierarchical reinforcement learning for exploration through periodically extracted subgoals and skills.The proposed direction uses an options framework to organize policies over subgoals.