PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Successor Features Approximators

Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, Tom Schaul

arXiv:1812.07626v1cs.LGcs.AIstat.ML

TL;DR

The paper asks how reinforcement-learning agents can generalise from multiple training tasks to unseen tasks. It proposes USFAs, combining UVFAs with successor features and GPI, and reports strong zero-learning transfer in a three-dimensional navigation environment, with performance regimes depending on policy-space coverage.

  • Problem

    The central problem is how an agent can leverage knowledge from training tasks to solve an unseen task w′ with little or no additional learning.

  • Method

    USFAs extend successor features with task-conditioned approximation and combine them with GPI to exploit both value-function structure and shared environment dynamics.

  • Results

    USFAs generalise well without learning on test tasks, outperform unstructured UVFAs, and structured UVFAs nearly match GPI when policy-space coverage is broadened.

  • Takeaways & Limitations

    USFAs provide a spectrum between UVFAs and SF&GPI, supporting different generalisation regimes according to policy-space coverage.

  • Takeaways & Limitations

    UVFA-style generalisation depends on structural assumptions such as smoothness in task space, whereas SF&GPI requires a learned policy whose successor features transfer effectively to the new task.

Abstract

from arXiv · show

The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential benefits, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills. We focus on one aspect in particular, namely the ability to generalise to unseen tasks. Parametric generalisation relies on the interpolation power of a function approximator that is given the task description as input; one of its most common form are universal value function approximators (UVFAs). Another way to generalise to new tasks is to exploit structure in the RL problem itself. Generalised policy improvement (GPI) combines solutions of previous tasks into a policy for the unseen task; this relies on instantaneous policy evaluation of old policies under the new reward function, which is made possible through successor features (SFs). Our proposed universal successor features approximators (USFAs) combine the advantages of all of these, namely the scalability of UVFAs, the instant inference of SFs, and the strong generalisation of GPI. We discuss the challenges involved in training a USFA, its generalisation properties and demonstrate its practical benefits and transfer abilities on a large-scale domain in which the agent has to navigate in a first-person perspective three-dimensional environment.

1 INTRODUCTION

The paper targets generalisation to unseen tasks in multitask reinforcement learning by combining task-space and problem-structure approaches. It proposes USFAs, which unify UVFAs with successor features and GPI for transfer in a three-dimensional navigation domain.

  • 1 INTRODUCTION: Generalising to unseen tasks can let an agent perform with little or no additional learning by leveraging previously learned tasks.
  • 1 INTRODUCTION: The approach uses both similarities among task solutions and shared environment dynamics as sources of transferable structure.
  • 1 INTRODUCTION: UVFAs encode task descriptions in value functions, whereas SFs and GPI evaluate and combine previously learned policies for new tasks.
  • 1 INTRODUCTION: USFAs combine UVFA-style task generalisation with SF&GPI's exploitation of reinforcement-learning problem structure.The model extends successor features as universal value functions and can recover UVFAs and SF&GPI as special cases when combined with GPI.

2 BACKGROUND

The background formalises multitask RL with shared dynamics and linearly parameterised rewards, then introduces UVFAs, successor features, and GPI as transfer mechanisms. These methods respectively interpolate across task descriptions, evaluate policies across rewards, and improve policies by selecting among learned value functions.

  • 2 BACKGROUND: Multitask RL uses multiple MDPs sharing environment structure while differing in reward functions represented by feature-weight products.The expected one-step reward is represented as φ(s, a, s′)^⊤w, with observable features assumed in the paper.
  • 2 BACKGROUND: The paper asks how knowledge from training tasks M can speed solution of a new task w′ not in M.Generalisation is evaluated using a task distribution, a training set sampled from it, and a separate test set.
  • 2 BACKGROUND: UVFAs extend optimal value functions to take a task description w as an additional input, producing Q∗(s, a, w).
  • 2 BACKGROUND: Successor features allow immediate policy evaluation on any task through Qπ_w(s, a) = ψπ(s, a)^⊤w.Successor features satisfy a Bellman equation in which the feature vector φ acts as the reward signal.
  • 2 BACKGROUND: GPI computes a new policy by taking the action maximizing the largest task-specific value among learned policies, with the theorem guaranteeing Qπ(s, a) ≥ Qmax(s, a).The result also extends to approximate value functions.

3 UNIVERSAL SUCCESSOR FEATURES APPROXIMATORS

USFAs combine UVFA-style task generalisation with SF&GPI's policy reuse by learning successor features that depend on an encoded policy. This representation supports value estimation across tasks and policies, while allowing GPI over flexible task sets.

  • 3 UNIVERSAL SUCCESSOR FEATURES APPROXIMATORS: USFAs bridge UVFAs and SF&GPI by combining functional generalisation across tasks with rapid policy evaluation and GPI.UVFAs exploit structure in the value-function space, whereas SF&GPI exploits structure in the RL problem itself.
  • 3.1 UNIVERSAL SUCCESSOR FEATURES: USFAs extend successor features into a policy-conditioned function, enabling value generalisation over both tasks and policies.The paper defines universal successor features as ψ(s, a, π) ≡ ψπ(s, a), with an approximator ˜ψ(s, a, π).
  • 3.1 UNIVERSAL SUCCESSOR FEATURES: Encoding a policy through e(π) determines the structure of the learned USF, with task-induced policies offering the simple encoding e(πz) = z.Under this encoding, Q(s, a, w, z) = ψ(s, a, z)ᵀw, and UVFs arise as the special case w = z.
  • 3.2 USFA GENERALISATION: For an unseen task, the GPI policy can range from UVFA-like inference to SF&GPI by choosing the candidate task set C.C = {w′} recovers UVFA-style generalisation, C = M recovers SF&GPI, and intermediate choices such as M ∪ {w′} are also available.
  • 3.2 USFA GENERALISATION: USFA generalisation depends on both the distance between the target task and the task inducing each policy and the quality of its successor-feature approximation.The theoretical bound separates these contributions as δd(z) and δψ(z).
  • 3.3 HOW TO TRAIN A USFA: Training a USFA requires choices about task sampling and policy sampling, which affect performance and the trade-offs in the GPI set C.Because the TD error depends on z and w, the learned representation can evaluate any encoded policy on any task by plugging in the corresponding vectors.

4 EXPERIMENTS

The experiments evaluate USFAs on illustrative and large-scale multitask RL environments, testing zero-shot generalisation to unseen reward tasks. Across these settings, USFAs combine policy-space and task-space generalisation and outperform precursor approaches.

  • 4.1 ILLUSTRATIVE EXAMPLE: TRIP MDP: USFAs quickly achieve near-optimal performance in the Trip MDP, outperforming UVFAs and SF&GPI by combining off-policy and off-task training with diverse GPI candidates.The reported explanation emphasizes both richer policy-space structure and candidate-set diversity.
  • 4.2 LARGE SCALE EXPERIMENTS: The large-scale environment uses four object types as features, with task weights assigning positive or negative rewards to collected objects.Negative weights require avoiding the corresponding object type.
  • 4.2 LARGE SCALE EXPERIMENTS: The large-scale USFA combines convolutional-LSTM state processing, policy-conditioned successor-feature prediction, and task-weighted policy evaluation.The evaluation module computes ˜Q(s, a, w, z) = ˜ψ(s, a, z)⊤w.
  • 4.3 RESULTS AND DISCUSSION: Including training-task policies in the GPI candidate set improves performance, while adding the test task can sometimes hurt, especially for negative-reward tasks.The paper attributes this behavior partly to unreliable successor-feature approximations and the absence of a strict monotonicity guarantee.
  • 4.3 RESULTS AND DISCUSSION: With broader policy-space sampling, structured-UVFA generalisation improves considerably and almost matches GPI, revealing complementary USFA operating regimes.The broader distribution uses Dz = N(w, 0.5 I), compared with the narrower sampling regime.

5 RELATED WORK

Multitask RL research includes task-conditioned value functions, hierarchical skill reuse, meta-learning, and successor-feature methods for generalising across tasks.

  • Multitask RL methods target training performance, data efficiency, and generalisation to new tasks.
  • UVFAs incorporate task descriptions into value functions and support zero-shot generalisation to task combinations or previously encountered goals.
  • Hierarchical and modular approaches generalise by learning, composing, reusing, or fine-tuning skills and subnetworks across tasks or configurations.
  • USFAs estimate successor features over multiple policies so GPI can provide zero-shot policies for unseen tasks, unlike prior neural SF approaches that did not use GPI.

6 CONCLUSION

The conclusion presents USFAs as a model combining value-function structure, shared dynamics, and GPI, with applicability to large-scale training and varied generalisation regimes.

  • USFAs combine UVFA-style value-function generalisation with SF&GPI structure and can recover their precursor methods while spanning intermediate models.
  • The paper illustrates USFA trade-offs through examples where UVFAs struggle with many policies and vanilla SF&GPI requires too many policies for effective generalisation.
  • USFAs are presented as readily applicable to large-scale training despite its associated complexities.
  • The authors identify three exploitable structures: shared dynamics through SFs, policy-space structure through UVFAs, and RL-problem structure through GPI.

A TWO TYPES OF GENERALISATION: INTUITION

The intuition section contrasts UVFA interpolation in task space with GPI’s evaluation and selection of previously learned policies, showing how USFAs can combine both mechanisms through candidate-set choice.

  • Illustrative task space: The scalar-task example defines rewards by scaling one feature with w, producing a simple optimal-value space used to contrast the two generalisation mechanisms.
  • UVFA-like generalisation: UVFA-like generalisation estimates an unseen task’s value function by fitting a parametric function over training tasks, but approximation error can distort zero-shot decisions.
  • GPI generalisation: GPI evaluates previously learned policies on the test task using successor features, then selects the highest-valued behaviour.
  • GPI generalisation: In the illustrative example, GPI selects the policy associated with w3, which matches the test task’s optimal policy; perfect successor features would recover the optimal value function.
  • USFA candidate-set choice: USFAs can recover UVFA-like or GPI generalisation by choosing the candidate set C, and GPI can generalise perfectly across the illustrated task space when trained points include positive and negative w values.

B.1 ADDITIONAL RESULTS

Additional evaluations show that USFAs generalize across broad and adversarial regions of task space, recovering near-optimal policies where precursor methods can struggle. The candidate set used during GPI affects this generalization.

  • B.1 ADDITIONAL RESULTS: A more diverse candidate set C changes the optimality-gap profile of USFA evaluation across task space.The reported plots identify a subtle evaluation effect induced by the choice of C.
  • B.1 ADDITIONAL RESULTS: USFAs recover near-optimal performance on the adversarial diagonal task set where UVFA and SF&GPI generalization can be poor.The diagonal is maximally distant from the canonical training tasks in this evaluation.

C.1 AGENT’S ARCHITECTURE

The USFA agent separates visual processing, policy-conditioned successor-feature estimation, and task evaluation. This late policy conditioning enables multiple policy embeddings to be evaluated efficiently and combined through GPI.

  • C.1 AGENT’S ARCHITECTURE: The policy conditioning module samples policy embeddings z and combines each with the shared state representation to estimate successor features.In the detailed architecture, 30 policy embeddings are sampled and processed through multilayer perceptrons.
  • C.1 AGENT’S ARCHITECTURE: The task evaluation module computes Q(s, a, z, w) by combining successor features with task description w, producing candidate value functions for GPI.This module has no learned parameters because the value functions are composed from the successor features and task vector.
  • C.1 AGENT’S ARCHITECTURE: Late policy conditioning lets the network reuse the recurrent state representation while computing multiple successor-feature estimates at relatively low computational cost.The design also avoids policy-conditioning the LSTM unroll, reducing the cost of successor-feature computation and off-policy learning.
  • C.1 AGENT’S ARCHITECTURE: Training uses a simplified IMPALA learner with Q(λ), while evaluation freezes periodic snapshots and measures average undiscounted reward on held-out tasks.All agents use λ = 0.9, and each test task is evaluated for 20 episodes.

D ADDITIONAL RESULTS

The additional zero-shot results evaluate canonical USFA training on easy and harder unseen tasks, including positive, negative, and mixed reward profiles. The figures provide the complete held-out-task results beyond the main-paper selection.

  • D ADDITIONAL RESULTS: The easy test set assesses generalization to tasks close to the canonical training set, including tasks such as w′ = [0, 0.9, 0, 0.1].The appendix presents the full results for this evaluation rather than only the illustrative subset in the main text.
  • D ADDITIONAL RESULTS: The harder-task evaluation covers unseen reward profiles with only positive rewards, only negative rewards, and mixed rewards.These tasks are intended to test generalization beyond tasks similar to the training set.
  • D ADDITIONAL RESULTS: The complete evaluation compares a canonical USFA against on-policy and off-policy UVFAs across easy and harder unseen tasks.The canonical USFA is trained on M = {1000, 0100, 0010, 0001} with Dz = N(w, 0.1I).

D.2 CANONICAL BASIS: USFAS IN DIFFERENT TRAINING REGIMES.

A second experiment compares USFAs trained with low- and high-variance policy-sampling distributions. The higher-variance regime samples policies farther from the canonical training tasks to test its effect on generalization.

  • D.2 CANONICAL BASIS: USFAS IN DIFFERENT TRAINING REGIMES.: The experiment compares two USFAs trained on the same canonical tasks but with Dz = N(w, 0.1I) versus Dz = N(w, 0.5I).The low-variance distribution focuses learning near the training set, whereas the higher-variance distribution covers more policies away from it.
  • D.2 CANONICAL BASIS: USFAS IN DIFFERENT TRAINING REGIMES.: The higher-variance policy distribution is intended to facilitate UVFA-style generalization by learning about more policies away from the training set.The passage frames this as a potential benefit rather than a guaranteed outcome.
  • D.2 CANONICAL BASIS: USFAS IN DIFFERENT TRAINING REGIMES.: Figures 16 and 17 report zero-shot performance on harder tasks for the two policy-sampling regimes.The comparison is split across two figure parts.

D.3 LARGER COLLECTION OF TRAINING TASKS

The USFA agent was trained on a larger collection combining canonical tasks with four tasks containing both positive and negative rewards, with selected training and generalisation results shown in Figure 18.

  • The expanded training set combines four canonical tasks with four additional tasks containing both positive and negative rewards.
  • The figure compares USFA evaluation using GPI over the training tasks plus the test task, over the training tasks alone, or over the test task alone.
  • Figure 18 reports learning curves for training task [1000] and generalisation performance on sampled test tasks after training on all tasks in M.
Loading 1812.07626v1…