Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement

André Barreto; Diana Borsa; John Quan; Tom Schaul; David Silver; Matteo Hessel; Daniel Mankowitz; Augustin Žídek; Rémi Munos

Source-linked AI summary

Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement

André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Žídek, Rémi Munos

arXiv:1901.10964v1Jan 30, 2019cs.LGcs.AI

TL;DR

The paper addresses how to transfer knowledge among MDPs that differ in reward functions, without restricting rewards to a fixed feature span. It extends SF&GPI guarantees to arbitrary reward functions and uses reward functions as features, enabling stable deep-learning integration and near-instantaneous transfer to unseen tasks. The paper also describes specialising transferred policies so they can be added to the reusable skill set.

Problem
The central problem is how to leverage knowledge from some reward-different MDPs to solve new tasks faster while making transfer practical for deep learning.
Method
The paper extends SF&GPI guarantees beyond feature-span tasks and replaces predefined features with reward functions as features for transfer.
Results
In a complex first-person-image 3D environment, SF&GPI transferred good policies to unseen tasks almost instantaneously, while SF&GPI-continual outperformed other methods in almost all tasks.
Takeaways & Limitations
SF&GPI can support rapid transfer and continued skill accumulation in challenging visual environments.
Takeaways & Limitations
Good performance from GPI is not guaranteed merely by spanning the reward-function space; constructing a behavioural basis that yields good policies remains open.

Abstract

from arXiv · show

The ability to transfer skills across tasks has the potential to scale up reinforcement learning (RL) agents to environments currently out of reach. Recently, a framework based on two ideas, successor features (SFs) and generalised policy improvement (GPI), has been introduced as a principled way of transferring skills. In this paper we extend the SFs & GPI framework in two ways. One of the basic assumptions underlying the original formulation of SFs & GPI is that rewards for all tasks of interest can be computed as linear combinations of a fixed set of features. We relax this constraint and show that the theoretical guarantees supporting the framework can be extended to any set of tasks that only differ in the reward function. Our second contribution is to show that one can use the reward functions themselves as features for future tasks, without any loss of expressiveness, thus removing the need to specify a set of features beforehand. This makes it possible to combine SFs & GPI with deep learning in a more stable way. We empirically verify this claim on a complex 3D environment where observations are images from a first-person perspective. We show that the transfer promoted by SFs & GPI leads to very good policies on unseen tasks almost instantaneously. We also describe how to learn policies specialised to the new tasks in a way that allows them to be added to the agent's set of skills, and thus be reused in the future.

1. Introduction

Deep RL has largely treated problems as isolated challenges, motivating transfer methods that reuse skills across tasks. This paper extends SF&GPI to broader reward settings and demonstrates rapid transfer in a challenging visual 3D environment.

Motivation: Deep RL agents have typically approached each problem as an isolated monolithic challenge rather than reusing solutions across tasks.Explicit skill transfer could help agents adapt quickly to new tasks and scale RL applications.
Prior framework: SF&GPI combines successor features, which evaluate policies across tasks, with generalised policy improvement, which supports transfer between tasks regardless of temporal order.The framework integrates closely with reinforcement learning.
Contributions: The paper extends SF&GPI guarantees beyond tasks whose rewards are linear combinations of a fixed feature set.The guarantees are extended to tasks outside the span of the features.
Contributions: The authors show that reward functions themselves can replace predefined features, enabling SF&GPI to combine with deep learning and operate online at scale.This reframes feature construction as a task-level approximation problem.
Empirical validation: In a first-person-image 3D environment, SF&GPI produced good policies on unseen tasks almost instantaneously and supported adding specialised policies to the skill set.The specialised policies can be reused for future tasks and support continual learning.

2. Background

The paper builds on MDP-based reinforcement learning, successor features, and generalised policy improvement. SFs separate environment dynamics from rewards, allowing previously learned policies to be evaluated and combined on new tasks.

2.1. Reinforcement learning: Reinforcement learning models an agent that selects actions in an MDP to maximise expected cumulative reward.An MDP specifies state and action spaces, transition dynamics, rewards, and a discount factor.
2.1. Reinforcement learning: Policy evaluation computes Qπ, while policy improvement selects a greedy policy whose value is guaranteed to be at least as high as the evaluated policy.Many dynamic-programming-based RL methods approximate these alternating steps.
2.2. SF&GPI: The original SF&GPI setting represents expected one-step rewards as φ(s, a, s′)^T w, defining tasks through different weight vectors over shared features.The environment Mφ contains all MDPs induced by the same features and varying weights.
2.2. SF&GPI: For a new task, SFs quickly compute each stored policy’s value, after which GPI combines those evaluations into a transferred policy.This turns much of new-task transfer into determining the new task representation and applying GPI.
2.2. SF&GPI: Successor features represent discounted future feature occupancy and decouple MDP dynamics from rewards, so replacing task weights evaluates a policy on another task.SFs satisfy a Bellman equation and can be learned with conventional RL methods.
2.2. SF&GPI: GPI selects actions using the maximum of multiple policy value functions, producing a policy at least as good as that maximum when values are exact.With approximate values, the guarantee weakens according to approximation error.

3. Extending the notion of environment

The paper enlarges the environment class from feature-linear tasks to all MDPs sharing dynamics but potentially differing in rewards. It proves transfer guarantees for this broader class and uses reward functions as features to simplify deep-learning integration.

3. Extending the notion of environment: The extended environment M contains all MDPs sharing state and action spaces, dynamics, and discount factor, regardless of reward-feature linearity.Thus M contains the original feature-induced environment Mφ.
3. Extending the notion of environment: The proposed framework seeks both theoretical guarantees for transferred policies and practical methods compatible with deep learning.The paper obtains both by reinterpreting SF&GPI.
3.1. Guarantees on the extended environment: The main result provides performance guarantees for GPI on any MDP with an arbitrary reward function, although efficient policy evaluation still requires an available transfer mechanism such as SFs.When tasks lie in Mφ, SFs provide that efficient evaluation mechanism.
3.1. Guarantees on the extended environment: The bound implies graceful degradation as the target environment moves away from Mφ; when the target remains in Mφ, the additional mismatch term vanishes.The original SF&GPI theorem is recovered in that special case.
3.2. Uncovering the structure of the environment: When the needed feature structure is unavailable, the paper formulates approximate feature construction as a multi-task learning problem over base-task rewards.Each regression approximates a base task’s reward using shared learned features.
3.2. Uncovering the structure of the environment: Because rewards are linearly related to the original features under independent base tasks, the rewards themselves can serve as features without losing expressiveness.This replaces the original feature approximation with a simpler direct reward approximation.
3.2. Uncovering the structure of the environment: Direct reward approximation removes the flexibility to distinguish the environment dimension d from the number D of base tasks, although using D > d may improve data-based approximation.The paper argues that other advantages can outweigh this loss of flexibility in relevant settings.
3.2. Uncovering the structure of the environment: Using reward approximations as features makes the resulting successor features ordinary action-value functions and supports a stable deep-learning approach.The paper identifies this reinterpretation as the basis for its practical method.

4. Transfer in deep reinforcement learning

This section develops an online SF&GPI approach for deep RL that learns transferable representations while solving base tasks. It addresses instability in jointly learning features and successor features by using observable rewards as features and retaining reusable policies.

Deep learning motivates an online SF&GPI design because storing the large datasets commonly required by deep learning is impractical.
Algorithm 1 uses GPI to select behavior instead of acting greedily with the newly learned task-specific value function.The improvement can be significant when the available successor features are useful.
The method can either rely exclusively on GPI, or learn a policy specialized to the current task through least-squares regression.In the exclusive-GPI option, no specialized policy is learned.
Challenges involved in building features: Learning features and successor features concurrently is difficult because each depends on the other, creating a circular dependency that can make training unstable.The paper reports observing this instability in practice.
Learning features online while retaining transferable knowledge: The proposed solution directly approximates observable reward functions and adopts those approximations as features for SF&GPI.Using rewards as features makes the resulting successor features collections of value functions.
Learning features online while retaining transferable knowledge: While solving D base tasks, the agent computes reward approximations and evaluates each resulting policy across tasks to build transferable knowledge.The construction is intended to learn the feature representation and initial successor-feature set concurrently without storing transitions.

5. Experiments

The experiments test SF&GPI transfer in a challenging image-based 3D navigation environment. SF&GPI-transfer provides near-instantaneous policies on unseen tasks, while SF&GPI-continual further improves performance and generally outperforms the baselines.

5.1. Environment: The environment uses first-person 84 × 84 image observations that do not uniquely identify the underlying MDP state.The agent navigates a 3D room, collecting desirable objects and avoiding undesirable ones.
5.1. Environment: Tasks are defined by four rewards assigned to the four object types, with base tasks used to construct the learned features and successor representations.Transfer is evaluated on different, unseen test tasks.
5.2. Agents: The SF&GPI agent uses a recurrent deep network to compute the learned state signal, features, and successor representations, trained end-to-end on base tasks.Q(λ) with eligibility traces and IMPALA distributed processing support learning and evaluation.
5.3. Results and discussion: SF&GPI-transfer learns very good policies for test tasks almost instantaneously by treating transfer as a simple supervised learning problem.Baseline methods solve the full reinforcement-learning problem and can eventually match or surpass transferred policies on some tasks.
5.3. Results and discussion: SF&GPI-continual combines instantaneous transfer with continued learning of a task-specialised policy and outperforms the other methods in almost all tasks.The specialised policies can be learned from the transferred policy rather than from scratch.
5.3. Results and discussion: SF&GPI performs well on test tasks with negative rewards despite exposure only to positive rewards in the base tasks.The authors interpret this as evidence that GPI combines the base-task policies non-trivially.
5.3. Results and discussion: Using linearly dependent base tasks can hinder transfer in some cases but generally has no strong impact, producing the smooth degradation predicted by Proposition 1.The comparison uses base-task sets ˆ M and ˆ M′ on tasks where the third and fourth object rewards differ.
5.3. Results and discussion: Representing the reward-function space does not guarantee a useful policy space: a basis with negative rewards can make standing still optimal for every base task.The authors leave defining a behavioural basis that yields good policies across tasks as an open question.

6. Related work

The paper is situated within deep RL transfer, multi-task learning, continual learning, modular architectures, and hierarchical RL. Related approaches pursue shared-policy regularisation, rapid adaptation, specialised network modules, or skill management.

6. Related work: Recent deep RL transfer work includes shared policies as regularisers for specialised policies and methods that optimise adaptability for rapid transfer.The cited approaches address multi-task learning and fast transfer through different training objectives.
6. Related work: Other work introduces neural-network architectures designed for continual learning and combines successor features with deep learning.These approaches are related to the paper’s focus on reusable skills in deep RL.
6. Related work: Modular deep-RL architectures commonly decompose problems into specialised sub-networks managed by another network.The paper connects this recurring design pattern with hierarchical reinforcement learning.

7. Conclusion

The paper extends SF&GPI beyond fixed feature-span assumptions and shows how reward functions can serve as features for stable deep-learning integration. Experiments demonstrate rapid transfer and continual construction of reusable specialised skills.

7. Conclusion: Theoretical guarantees extend to MDP sets differing only in reward functions, even when rewards are not linear combinations of fixed features.This removes the original requirement that all tasks lie in the feature span.
7. Conclusion: Reward functions themselves can be used as features, eliminating the need to specify a feature set beforehand without loss of expressiveness.The reinterpretation supports combining SF&GPI with deep learning in a stable way.
7. Conclusion: In a complex 3D environment requiring hundreds of millions of transitions, SF&GPI-transfer produced skilful, non-trivial policies almost instantaneously.The approach turns the transfer step into a supervised learning problem.
7. Conclusion: SF&GPI-continual uses transferred policies to learn specialised policies that can be added to the agent’s reusable skill set.The combined framework supports building, refining, and using skills during interaction with the environment.

Supplementary Material

The supplementary material identifies the paper’s authors and their Google affiliations. No additional scientific content is supplied in these passages.

Supplementary Material: The paper lists André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Žídek, and Rémi Munos as authors.The supplied passage also provides their Google email addresses.

Abstract

The supplement provides omitted theoretical proofs and additional empirical analyses, reproducing theoretical statements before their proofs.

The supplement expands the paper with theoretical details and experiments excluded because of space limits.
Theoretical results are restated before their corresponding proofs for reader convenience.
Supplement citations refer to the references listed in the main paper.

A. Proof of theoretical results

The section develops bounds for transferring policies between tasks with shared dynamics but differing rewards, combining GPI guarantees with reward and value-function discrepancies.

A. Proof of theoretical results: Generalized Policy Improvement constructs a policy from approximated action-value functions of multiple decision policies.
A. Proof of theoretical results: The value-function discrepancy between tasks is bounded by their maximum reward difference scaled by 1/(1−γ).
A. Proof of theoretical results: The theoretical argument applies to MDPs sharing transition dynamics while potentially differing in rewards.
A. Proof of theoretical results: The derivation obtains the task-discrepancy bound by solving the recursive inequality Δij ≤ δij + γΔij.
A. Proof of theoretical results: The transferred-policy performance bound combines task reward mismatch with twice the action-value approximation error, both scaled by 1/(1−γ).

B.2. Agents’s training

The agents use RMSProp and IMPALA training, while continual learning adds a standard Q(λ) loss to stabilize successor-feature learning and verifies its semantics empirically.

B.2. Agents’s training: RMSProp minimized the algorithmic losses, using decay rate 0.99, ϵ = 0.01, and tested learning rates including 0.01 and 0.001.
B.2. Agents’s training: IMPALA separates parallel actors collecting trajectories from a learner that updates parameters from queued trajectories.
B.2. Agents’s training: SF&GPI-continual combines the successor-feature loss with a standard Q(λ) loss using weights 0.1 and 1, respectively.
B.2. Agents’s training: The monitored loss is minimized, supporting preservation of successor-feature semantics when the specialized policy is added to ˜Ψ.
B.2. Agents’s training: Figure 5 tracks the approximation loss for ˜ψπn+1 across 10 runs and test tasks, with shading indicating one standard deviation.

C. Additional results

Additional experiments evaluate transfer across nine unseen tasks and continual learning across reward regimes, showing satisfactory but base-task-dependent transfer and eventual specialization when negative rewards dominate.

C. Additional results: Nine test tasks cover positive-only, negative-only, and mixed reward combinations for evaluating SF&GPI-transfer and baselines.
C. Additional results: For positive-reward test tasks, SF&GPI-transfer and SF&GPI-continual perform virtually the same, while learned base-task policies largely dominate selection.
C. Additional results: For negative-reward test tasks, SF&GPI-continual eventually outperforms SF&GPI-transfer and increasingly selects the specialized policy πtest.
C. Additional results: All three base-task sets produce satisfactory transfer, but transferred-policy performance varies significantly with the relation between base and test tasks.
C. Additional results: Figure 10 reports reward distributions per episode between 50 and 100 million learning steps for three base-task sets across nine test tasks.