PaperlayerView exact revision on arXiv

Source-linked AI summary

Successor Features for Transfer in Reinforcement Learning

André Barreto, Will Dabney, Rémi Munos, Jonathan J. Hunt, Tom Schaul, Hado van Hasselt, David Silver

arXiv:1606.05312v2cs.AI

TL;DR

The paper addresses transfer when tasks change rewards but retain the same environment dynamics. It combines successor features with generalized policy improvement, achieving transfer with theoretical guarantees and strong empirical performance across navigation and robotic-arm tasks.

  • Problem

    The paper asks how reinforcement learning can transfer knowledge across tasks defined by different reward functions in the same environment.

  • Method

    The method represents values with successor features that decouple dynamics from rewards and applies generalized policy improvement over multiple policies.

  • Results

    The combined approach promotes transfer in navigation and simulated robotic-arm experiments, with SFQL improving average return by more than 100% over PRQL.

  • Takeaways & Limitations

    The framework supports exchanging information across tasks within reinforcement learning and provides performance guarantees before new-task learning begins.

  • Takeaways & Limitations

    The framework assumes tasks are reward-function instantiations induced by shared features and an MDP model of agent–environment interaction.

Abstract

from arXiv · show

Transfer in reinforcement learning refers to the notion that generalization should occur not only within a task but also across tasks. We propose a transfer framework for the scenario where the reward function changes between tasks but the environment's dynamics remain the same. Our approach rests on two key ideas: "successor features", a value function representation that decouples the dynamics of the environment from the rewards, and "generalized policy improvement", a generalization of dynamic programming's policy improvement operation that considers a set of policies rather than a single one. Put together, the two ideas lead to an approach that integrates seamlessly within the reinforcement learning framework and allows the free exchange of information across tasks. The proposed method also provides performance guarantees for the transferred policy even before any learning has taken place. We derive two theorems that set our approach in firm theoretical ground and present experiments that show that it successfully promotes transfer in practice, significantly outperforming alternative methods in a sequence of navigation tasks and in the control of a simulated robotic arm.

1 Introduction

The paper studies transfer across tasks that differ in reward functions while sharing an environment, aiming to make knowledge exchange flexible and integrated with reinforcement learning. It combines successor features with generalized policy improvement and evaluates the approach theoretically and empirically.

  • Motivation: Transfer reuses knowledge acquired in one task to improve performance on related tasks.The paper motivates decomposing complex tasks into subtasks whose knowledge can accelerate learning elsewhere.
  • Problem setting: The target setting changes reward functions while keeping the environment dynamics fixed.Different reward delivery patterns can induce different task decompositions, including hierarchical, independent, or temporally ordered subtasks.
  • Design goals: The framework seeks flexible information flow between tasks rather than a rigid hierarchy or temporal task diagram.Transfer is also intended to remain nearly transparent within the reinforcement learning framework.
  • Approach: Successor features decouple environment dynamics from rewards, while generalized policy improvement transfers knowledge from multiple policies to a new task.Together, these ideas support theoretical guarantees before learning and motivate reusable skill libraries.
  • Evaluation: The method is evaluated on navigation tasks and control of a simulated two-joint robotic arm.The introduction presents these experiments as tests of transfer across distinct environments and tasks.

2 Background and problem formulation

The paper formulates reinforcement learning transfer in Markov decision processes where tasks are different reward-function instantiations. It connects the objective of maximizing discounted return to dynamic-programming policy evaluation and improvement.

  • MDP formulation: An MDP models the agent–environment interaction through states, actions, transition dynamics, rewards, and discounting.The transition function specifies next-state distributions, while rewards may depend on the current state, action, and next state.
  • RL objective: RL seeks a policy that maximizes the expected discounted sum of rewards, called the return.The discount factor assigns smaller weights to rewards received farther in the future.
  • Policy improvement: Dynamic-programming methods evaluate a policy through its action-value function and improve it by acting greedily with respect to that function.The improved policy is guaranteed to be at least as good as the evaluated policy, and repeated application can lead to an optimal policy.
  • Transfer definition: Transfer means that training on a larger task set yields performance at least as good on a target task as training on a subset.Here, each task is a specific reward-function instantiation for a given MDP, and the subset may be empty.

3 Successor features

Successor features represent expected discounted future feature occupancy under a policy and express action values as a feature-dependent component combined with reward weights. This separates dynamics from rewards and extends successor representations to approximation and continuous spaces.

  • Reward representation: The method models one-step expected rewards as an inner product between transition features and task-specific weights.The feature representation can recover arbitrary reward functions when reward components are included among the features.
  • Successor features: Successor features are the expected discounted sums of feature vectors under a policy, starting from a state–action pair.Their action-value decomposition combines successor features with the reward weights.
  • Generalization: Successor features generalize successor representations to function approximation and continuous state and action spaces.Their feature-vector formulation makes it possible to learn the representation from data.
  • Bellman structure: Successor features satisfy a Bellman equation in which feature vectors play the role of rewards.This means standard reinforcement-learning methods can, in principle, compute them.
  • Modularity: The representation decouples MDP dynamics from rewards, with rewards captured by the weights.Consequently, only the relevant module needs relearning when either dynamics or rewards change.

4 Transfer via successor features

The framework transfers across tasks that share dynamics but differ in reward by combining successor features with generalized policy improvement. It provides performance guarantees for new tasks, including bounds related to task similarity and approximation error.

  • Transfer setup: Tasks share environment dynamics while differing through reward weights w, allowing reward changes to define a family of related MDPs.The task family Mφ fixes the dynamics and varies rewards represented as φ(s,a,s′)^⊤w.
  • Generalized policy improvement: Generalized policy improvement constructs a policy by acting greedily with respect to the maximum value across multiple policies.The theorem permits approximate action-value functions, with approximation error represented by ϵ.
  • Generalized policy improvement: With exact value functions, the GPI policy performs no worse than any policy in the supplied set and can be strictly better when no single policy dominates.Approximation introduces a penalty term in the performance lower bound.
  • Guarantees: Applying GPI to value functions recomputed under a new reward performs at least as well as using any subset of the previously learned policies, including the empty set.This establishes successful transfer relative to learning without prior task knowledge.
  • Successor-feature transfer: Successor features let previously learned policies’ value functions be recomputed under a new reward by combining successor features with the new reward weights.This avoids relearning each policy’s value function from scratch when the reward changes.
  • Guarantees: Theorem 2 bounds transfer loss using approximation error and the distance between the new task vector wi and the closest previously solved task vector wj.The bound formalizes why performance should improve when a previously solved task is similar to the new task.

5 Experiments

Experiments evaluate successor-feature transfer in navigation and simulated robotic-arm control. The method outperforms comparison methods in the four-room domain and improves performance on experienced and unexperienced reacher tasks.

  • Four-room navigation: The four-room experiment cycles through 250 navigation tasks whose object-class rewards are resampled every 20 000 transitions.The agent must collect favorable objects, avoid unfavorable ones, and reach the goal.
  • Four-room navigation: SFQL-φ uses reward-predictive features, whereas SFQL-h learns an approximate feature representation from data collected during the first 20 tasks.The true feature dimension in this environment is 4, while h may differ.
  • Four-room navigation: SFQL variants outperform QL and PRQL in average return, with improvement over PRQL exceeding 100%.PRQL itself improves over QL by around 100%.
  • Experimental presentation: Figure 2 reports average and cumulative return per task with one-standard-error bands over 30 runs, while SFQL-h receives no reward during its first 20 tasks.The initial period is used to learn the approximate feature representation.
  • Reacher control: The reacher experiment trains on 4 of 12 target-location tasks, requiring performance on tasks absent from training.SFDQN combines the successor-feature framework with DQN and task-conditioned inputs.
  • Reacher control: When SFDQN trains on one task, return improves on that task and also on other tasks, including test tasks without specialized policies.SFDQN uses GPI policies to select actions across the learned successor-feature representations.

6 Related work

The paper relates successor features and GPI to transfer, predictive representations, general value functions, universal value function approximators, and hierarchical reinforcement learning. These connections distinguish the framework’s task representation and multi-policy selection mechanisms.

  • Transfer methods: Related transfer methods differ in their assumptions and reuse strategy, including methods that observe φ and w or relearn successor-feature functions for each new task.The paper compares these alternatives with its own setting and reuse mechanism.
  • Representations: Unlike predictive state representations, successor features summarize dynamics under a single policy rather than the entire environment.The comparison concerns the scope of the represented predictions.
  • Representations: Successor features can be viewed as a special case of general value functions when feature components serve as pseudo-rewards.The connection links the representation to predictions of multiple quantities.
  • Goal-conditioned transfer: The GPI expression maxj ˜ψj(s,a)^⊤˜w resembles a universal value function approximator because ˜w represents the task or goal.This interpretation permits task descriptions to be functions of observations in principle.
  • Hierarchical reinforcement learning: The framework’s reuse of successor-feature policies is related to temporal abstraction and hierarchical RL, with each policy representation associated with an option terminating after one step.Greedy selection over their values corresponds to higher-level planning over these components.

7 Conclusion

The paper combines successor features and generalized policy improvement to create a transfer framework for reinforcement learning. Theorems formalize guarantees for similar new tasks, and experiments demonstrate transfer in practice.

  • Conclusion: Successor features generalize successor representations to continuous spaces and facilitate function approximation.They are one of the paper’s two foundational concepts.
  • Conclusion: Generalized policy improvement extends Bellman’s policy improvement theorem from one policy to multiple policies.The paper formalizes this extension in Theorem 1.
  • Conclusion: Combining successor features with GPI provides a framework for transfer across tasks with shared dynamics and changing rewards.The paper presents this combination as its central transfer contribution.
  • Conclusion: Theorem 2 formalizes the intuition that performance on a novel task should be better when a similar task has previously been seen.The conclusion describes the theorem as a complement to the transfer approach.
  • Conclusion: Experiments show that the combined framework promotes transfer in practice across the evaluated task settings.The conclusion characterizes the experiments as comprehensive.

Supplementary Material

The supplement provides omitted theoretical, experimental, algorithmic, and empirical details supporting the main paper.

  • The supplement reproduces theoretical results before presenting their proofs.
  • It gives a detailed description of the experimental protocol and algorithms, including pseudo-code.
  • It reports additional empirical analyses excluded from the main paper because of space limitations.

A Proofs of theoretical results

The paper develops theoretical foundations for transferring between reward-defined tasks and details the experimental framework used to evaluate the SFs–GPI approach. Its formal results relate generalized policy improvement and task similarity to policy performance, while the experiments instantiate shared dynamics with changing rewards.

  • Theoretical results: Generalized Policy Improvement constructs a policy by acting greedily with respect to the maximum value across multiple decision policies.
  • Theoretical results: Theorem 2 bounds transferred-policy error using the distance between the new task’s weight vector and the closest previously solved task, plus approximation error.
  • Experimental framework: The four-room environment keeps movement dynamics fixed while task-specific object rewards vary, with twelve objects, three classes, and goal reward 1.
  • Experimental framework: SFQL uses reward-predictive features when available or learns an approximate feature representation from data collected on the first 20 tasks.
  • Framework: The framework combines successor features and generalized policy improvement rather than proposing one particular algorithm.

B.1.3 Experimental setup

The experiments configure and compare QL, PRQL, SFQL, and SFDQN across navigation and reacher environments with shared dynamics and task variation.

  • Algorithm configuration: QL, PRQL, and SFQL were tuned by testing three values for each algorithm-specific parameter and selecting the configuration with the highest average return.The resulting search covered 3 QL, 27 PRQL, and 9 SFQL configurations.
  • Reacher environment: The reacher domain is a two-joint torque-controlled MuJoCo arm whose tasks differ by target location while sharing dynamics.Its continuous two-dimensional action space was discretized into 9 actions.
  • Baseline and function approximation: DQN generalizes across reacher tasks by receiving target locations in the state and uses an MLP that outputs values for all 9 actions.The network has two hidden layers of 256 units.
  • Successor-feature implementation: SFDQN represents successor features with a separate MLP for each training task, producing a 9×12 matrix of action-conditioned feature predictions.The 12 features are the negated distances to target locations.

B.2.3 Experimental setup

The experiments train agents on selected tasks, evaluate them across training and unseen tasks, and normalize returns for comparison.

  • Training protocol: Agents trained for 200 000 transitions on each of four training tasks using an ϵ-greedy policy with ϵ = 0.1.Terminal transitions were excluded during training.
  • Evaluation protocol: Performance was monitored on all 12 tasks with an ϵ-greedy policy using ϵ = 0.03.Returns were summed over two episodes from fixed start states.
  • Evaluation protocol: Evaluation returns were normalized per task using standard DQN performance from separate experiments.This normalization was applied to both training and test tasks.
  • Reported results: Figure 4 averages results over 30 runs and displays standard errors as shadowed regions, with PRQL included for reference.The standard-error regions are described as almost imperceptible at the plotted scale.

C.1 Understanding the types of transfer promoted by SFs

SFQL promotes transfer through both its decoupled value representation and generalized policy improvement, with GPI providing the stronger observed benefit.

  • Transfer mechanisms: SFQL decouples environment dynamics from rewards, allowing its representation to support faster value learning and adaptation when rewards decompose into features.The experiments examine this mechanism in the four-room environment.
  • Transfer mechanisms: SFQL uses GPI to select actions from a set of stored successor features, while retaining weaker transfer even with only one successor-feature estimate.The single-estimate case evaluates the current task’s reward weights using the retained representation.
  • Experimental isolation: The comparison isolates transfer sources by running SFQL with and without GPI alongside PRQL.GPI is disabled by replacing the current-task choice with the training-task choice in Algorithm 3.
  • Results: Without GPI, SFQL initially outperforms PRQL, but both methods eventually reach the same performance level.The authors attribute the initial advantage to the decoupled representation and weaker transfer.
  • Results: SFQL with GPI consistently outperforms the other two methods, supporting GPI as a crucial component of the proposed approach.After 200 transitions into a new task, previously computed successor features already provide an informative value function.

C.2 Analyzing the robustness of SFs

SFQL relies on an approximate reward-feature decomposition but remains robust to spurious features and noise, with performance degrading gracefully under either perturbation.

  • Approximate representations: SF benefits depend on approximating rewards as ˜φ(s, a, s′)⊤w, whether ˜φ is handcrafted or learned from data.The paper tests robustness because exact decompositions are not generally expected.
  • Spurious features: The spurious-feature experiment adds objects that never contribute to the reward, creating predictions unnecessary for reward approximation.The modified environment contains 15 added objects from 6 classes.
  • Robustness results: Both spurious features and noise reduce SFQL performance, while their effects appear to combine approximately additively.The experiment compares performance across different distortions of ˜φ.
  • Robustness results: SFQL performance degrades gracefully under either intervention, corroborating robustness to approximation errors in ˜φ.The reported conclusion concerns both added spurious features and injected noise.
Loading 1606.05312v2…