PaperlayerView exact revision on arXiv

Source-linked AI summary

Sharing Knowledge in Multi-Task Deep Reinforcement Learning

Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, Jan Peters

arXiv:2401.09561v1cs.LG

TL;DR

The paper studies when sharing representations across related tasks benefits deep Multi-Task Reinforcement Learning. It extends AVI/API guarantees to MTRL and develops shared-network algorithm variants, finding significant improvements over single-task methods while identifying task-count, representation-size, and sampling assumptions.

  • Problem

    Deep MTRL needs effective shared feature extraction, but unrelated tasks can degrade learning and the benefits of shared representations require theoretical justification.

  • Method

    The paper extends AVI/API finite-time and approximation-error bounds to MTRL, then proposes shared-network MFQI, MDQN, and MDDPG variants.

  • Results

    Theoretical analysis shows reduced AVI/API error propagation with more tasks, while experiments report significant performance improvements over single-task counterparts across RL problems, including MuJoCo domains.

  • Takeaways & Limitations

    Shared representations can provide theoretical and empirical benefits for learning multiple related RL tasks with a single model.

  • Takeaways & Limitations

    The bounds involve assumptions, including sufficient representation capacity and, for the practical DQN and DDPG evaluations, i.i.d. sampling; increasing tasks can also increase representation-size costs.

Abstract

from arXiv · show

We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We leverage the assumption that learning from different tasks, sharing common properties, is helpful to generalize the knowledge of them resulting in a more effective feature extraction compared to learning a single task. Intuitively, the resulting set of features offers performance benefits when used by Reinforcement Learning algorithms. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the well-known finite-time bounds of Approximate Value-Iteration to the multi-task setting. In addition, we complement our analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing significant improvements over the single-task counterparts in terms of sample efficiency and performance.

1 INTRODUCTION

Multi-Task Reinforcement Learning shares representations across related tasks to extract common features, but unrelated tasks can degrade learning. This paper develops theoretical guarantees and practical single-network algorithms for exploiting shared representations.

  • Multi-Task Learning jointly trains tasks with common properties so learned hidden-layer features represent structures shared across them.
  • Deep models can extract common features in MTRL, but feature learning remains challenging and performance may degrade when unrelated tasks are combined.
  • A single deep neural network is desirable because it can reduce training time and architectural complexity compared with multiple networks.
  • The paper extends AVI/API finite-time bounds to MTRL and extends approximation-error bounds to multiple tasks with different dimensionalities.
  • The proposed MFQI, MDQN, and MDDPG methods use shared representations and show significant improvements over single-task counterparts across RL problems, including MuJoCo domains.

2 PRELIMINARIES

The preliminaries define MDPs, policies, value functions, and the multi-task setting, then introduce Gaussian complexity for analyzing representation complexity. They also define the Gaussian average of Lipschitz quotients used in the bounds.

  • An MDP is represented by state and action spaces, a transition distribution, a reward function, and a discount factor.
  • A policy maps states to actions, while its action-value function measures expected discounted cumulative reward under that policy.
  • The optimal action values satisfy the Bellman optimality equation and are characterized as the fixed point of the optimal Bellman operator.
  • MTRL contains multiple MDPs, each with its own state space, action space, transition distribution, reward function, and discount factor.
  • Gaussian complexity measures representation-class complexity using Gaussian random variables over task-specific input samples.
  • The analysis also uses a Gaussian average of Lipschitz quotients and an upper bound on the Lipschitz constants of functions in a class.

3 THEORETICAL ANALYSIS

The theoretical analysis extends Approximate Value- and Policy-Iteration guarantees to multi-task reinforcement learning and bounds task-averaged approximation error. These results identify when shared representations reduce learning complexity while exposing a tradeoff between task count and representation capacity.

  • Multi-task AVI/API bounds: The analysis extends finite-time AVI/API guarantees to MTRL by averaging approximation losses across tasks.The AVI extension uses task-averaged regression error and combines it with approximation-error bounds for shared representations.
  • Multi-task AVI/API bounds: The multi-task AVI bound depends on task-averaged approximation errors across iterations and the maximum discount factor and reward scale.The bound includes E_avg formed from ε_avg,k and a term involving γ, K, and R_max,avg.
  • Approximation error: Under the stated assumptions, learning the shared representation contributes a term that decreases with the number of tasks, while task-specific mappings retain O(1/√n) costs.The shared-representation cost is described as O(1/√nT), and another term can vanish or become arbitrarily small for relevant classes such as neural networks.
  • Implications: The composed MTRL bound shows that reducing approximation error improves convergence toward the optimal action-value function, while multiple tasks mitigate the cost of learning complex shared features.The analysis emphasizes this benefit particularly for complex representations such as deep neural networks.
  • Implications: Adding tasks creates a tradeoff: the shared representation must grow enough to cover them, increasing a bound term tied to representation magnitude.For a reasonable number of tasks, the single-task feature count may still suffice, as reported in the experiments.

4 SHARING REPRESENTATIONS

The proposed architecture shares hidden representations across tasks while retaining task-specific input and output layers. This shared structure regularizes learning, supports more general features, and applies to both value-based and policy-search methods.

  • Architecture: The architecture maps each task’s input through task-specific layers, shared layers h, and task-specific output layers f_t.It computes y_t = (f_t ◦ h ◦ w_t).
  • Representation sharing: Shared representations regularize joint learning by encouraging general features and reducing the variance of the learned function.The paper contrasts this with potential overfitting when learning a single task.
  • Applications: The architecture can serve as a regressor in value-based methods and policy-search methods, with task-specific targets learned in each output block.The paper states that extending RL algorithms requires no major algorithmic changes beyond using the proposed architecture.

5 EXPERIMENTAL RESULTS

The experiments evaluate MFQI, MDQN, and MDDPG against single-task counterparts across discrete and continuous-control benchmarks. Results generally favor shared representations, while the paper notes an i.i.d.-sample assumption for the theoretical bounds used with DQN and DDPG.

  • Experimental design: MFQI extends FQI to evaluate the theoretical AVI benefit, while MDQN and MDDPG extend DQN and DDPG to multi-task settings.The three methods use the proposed shared-representation network structure.
  • Evaluation protocol: Figure 2 reports discounted cumulative reward averaged over 100 experiments, with each epoch followed by greedy-policy evaluation and 95% confidence intervals.The Acrobot plots include both per-task learning and transfer-learning evaluations.
  • Experimental design: The experiments compare shared-representation methods only with their single-task counterparts to isolate the benefit of learning multiple tasks jointly.The evaluation spans challenging RL problems, including MuJoCo tasks.
  • Scope and assumptions: For DQN and DDPG, the AVI/API bounds rely on the simplifying assumption that samples are i.i.d., although the experiments target complex scenarios such as MuJoCo.The authors use these comparisons to assess shared versus single-task representations.
  • MDQN: MDQN uses separate replay memories but one model, lowering time complexity relative to vanilla DQN while increasing sample-storage memory complexity.Each training step samples equally from the task memories and performs one step in every task.
  • MDQN: MDQN is higher and more stable than DQN in the first three plotted tasks, while Car-On-Hill is initially slower but eventually more stable.The Inverted-Pendulum task is easy for both approaches but remains useful for shared feature extraction.
  • Transfer learning: Pretrained shared weights improve Acrobot transfer, with delayed adjustment after epoch 10 achieving the best reported results among the tested initialization settings.Fixed weights help earliest learning, whereas immediate adjustment helps intermediate stages.
  • MDDPG: MDDPG improves over DDPG substantially on the harder pendulum tasks, is slightly better on Inverted-Pendulum, and is better on Hopper while matching the other two tasks.Figure 3 averages discounted cumulative reward over 40 experiments for the continuous-control evaluations.

6 RELATED WORKS

The work builds on theoretical multi-task learning, Bayesian and common-generative-model analyses in MTRL, and deep-RL approaches that combine or distill task-specific policies. Its emphasis is theoretical guarantees for shared representations in deep MTRL.

  • Theoretical MTL: Prior theoretical MTL work establishes advantages of shared representations with linear approximators and derives corresponding multi-task bounds.The paper situates its analysis alongside results by Cavallanti et al. and Maurer.
  • Theoretical MTRL: MTRL analyses commonly assume that task properties or value functions arise from a common generative model or prior distribution.The cited examples include hierarchical Bayesian and common-prior formulations.
  • Deep MTRL: Deep-RL studies have used separately trained DQN experts to construct multi-task policies and have proposed DQN variants similar to MDQN.These works provide empirical context for the paper’s shared-network approach.

7 CONCLUSION

The paper combines MTRL bounds with a shared deep representation and evaluates corresponding extensions of FQI, DQN, and DDPG. Its theoretical and empirical results support advantages over single-task learning.

  • Conclusion: The analysis extends AVI/API finite-time bounds and MTL approximation-error bounds to multiple RL tasks with shared representations.The resulting error propagation during AVI iterations is reduced according to the number of tasks.
  • Conclusion: MFQI, MDQN, and MDDPG operationalize the shared representation approach across value-based and policy-search reinforcement learning.The paper reports favorable empirical results on challenging RL problems.
  • Conclusion: The reported experiments confirm the theoretical benefit of learning multiple tasks with a shared representation rather than learning each task separately.This conclusion is stated across the paper’s empirical evaluations.

A PROOFS

The proofs derive multi-task AVI and API bounds by averaging expected losses across tasks and applying inequalities from prior results. They also bound the shared-function class using Lipschitz, contraction, diameter, and concentration arguments.

  • Multi-task bounds: The proofs compute expected loss averaged across tasks as the starting point for the multi-task AVI and API analyses.The AVI and API derivations explicitly begin by computing average expected loss across tasks.
  • Multi-task bounds: Jensen’s inequality moves task averaging inside the bounds, yielding API bounds averaged over multiple tasks.The derivation applies Jensen’s inequality to the single-task result before averaging across tasks.
  • AVI bound: The averaged AVI error uses Eavg = Σ α_k^2r εavg,k with γ defined as the maximum task discount factor.The coefficient construction also includes task reward magnitudes and the sequence α_k.
  • Concentration argument: Hoeffding’s inequality bounds one component with probability 1 − δ/2, while the empirical-risk minimizers make another component zero.The remaining component is identified as requiring a separate, less straightforward bound.
  • Function-class bounds: The function-class proof bounds generalization terms through Lipschitz constants, contraction, Euclidean diameters, and auxiliary function spaces.It invokes Maurer et al. results and bounds the diameters using supremum norms.
  • Concentration argument: A union bound over components A, B, and C completes the proof.The proof combines the separate component bounds after controlling each one individually.

B.1 MULTI FITTED Q-ITERATION

The MFQI evaluation uses the Car-On-Hill problem with eight tasks formed by varying car mass and discrete action values. Figures evaluate selected task combinations and compare multi-task settings with single-task results.

  • Experimental setup: MFQI is evaluated on Car-On-Hill with discount factor 0.95 and horizon 100.The experiment trains a neural network with two shared 30-neuron layers, sigmoid activations, Adam, and mean squared loss.
  • Task construction: Eight tasks are selected by changing the car mass m and the value of the discrete actions a.The task configurations are listed in Table 1.
  • Evaluation design: Figure 1(b) uses the first four tasks, while Figure 1(c) compares results using one, two, and four selected tasks.The passage specifies the task subsets used for the plotted multi-task settings.

B.2 MULTI DEEP Q-NETWORK

The multi-task DQN evaluation spans five benchmark problems, while the architecture combines task-specific blocks with a shared block and task-specific action-value outputs. The shared sigmoid layer is intended to produce bounded common features and reduce task interference.

  • Benchmarks: The evaluation covers Cart-Pole, Acrobot, Mountain-Car, Car-On-Hill, and Inverted-Pendulum with task-specific discount factors and horizons.The listed discount factors are 0.99 for the first three problems and 0.95 for the last two.
  • Architecture: The MDQN architecture uses five task-specific blocks with 80 ReLU units, followed by a shared block containing 80 ReLU and 80 sigmoid units.Each task-specific output layer has one linear unit per available discrete action.
  • Benchmark sources: The benchmark task identifiers include OpenAI Gym environments for Cart-Pole, Acrobot, and Mountain-Car, plus PyBullet and DeepMind Control Suite environments for later tasks.The passages provide the corresponding environment names and identifiers.
  • Shared representation: The sigmoid layer extracts shared features bounded between 0 and 1 for the final linear layer.The paper reports that bounded sigmoid features help reduce task interference relative to linear ReLU responses.
  • Continuous-control setup: A second continuous-control experiment groups three inverted-pendulum tasks and three locomotion tasks, using discount factor 0.99 and horizon 1,000.Its actor network has three task-specific 600-unit ReLU blocks, a shared 500-unit ReLU block, and task-specific tanh outputs.
Loading 2401.09561v1…