PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Multi-Task by Active Sampling

Sahil Sharma, Ashutosh Jha, Parikshit Hegde, Balaraman Ravindran

arXiv:1702.06053v4cs.NEcs.LG

TL;DR

Multi-task reinforcement-learning agents often require costly task-specific experts or scale poorly across tasks. This paper uses active task sampling to train online multi-task agents, achieving more than double the baseline performance while remaining competitive across proposed methods.

  • Problem

    Existing multi-task reinforcement-learning methods require computation-intensive expert networks or grow parameters substantially with each new task.

  • Method

    The framework actively selects harder tasks for online training through adaptive sampling, a UCB bandit meta-learner, and an actor-critic reinforcement-learning meta-learner.

  • Results

    More than double the performance obtained by the BA3C baseline, with A5C best on MT1 and UA4C generalizing better on larger multi-task instances.

  • Takeaways & Limitations

    Choosing training tasks based on current weaknesses provides an important first step toward online multi-task reinforcement learning.

  • Takeaways & Limitations

    The authors do not claim that the framework definitively solves online multi-task reinforcement learning.

Abstract

from arXiv · show

One of the long-standing challenges in Artificial Intelligence for learning goal-directed behavior is to build a single agent which can solve multiple tasks. Recent progress in multi-task learning for goal-directed sequential problems has been in the form of distillation based learning wherein a student network learns from multiple task-specific expert networks by mimicking the task-specific policies of the expert networks. While such approaches offer a promising solution to the multi-task learning problem, they require supervision from large expert networks which require extensive data and computation time for training. In this work, we propose an efficient multi-task learning framework which solves multiple goal-directed tasks in an on-line setup without the need for expert supervision. Our work uses active learning principles to achieve multi-task learning by sampling the harder tasks more than the easier ones. We propose three distinct models under our active sampling framework. An adaptive method with extremely competitive multi-tasking performance. A UCB-based meta-learner which casts the problem of picking the next task to train on as a multi-armed bandit problem. A meta-learning method that casts the next-task picking problem as a full Reinforcement Learning problem and uses actor critic methods for optimizing the multi-tasking performance directly. We demonstrate results in the Atari 2600 domain on seven multi-tasking instances: three 6-task instances, one 8-task instance, two 12-task instances and one 21-task instance.

1 Introduction

Deep reinforcement learning combines deep-learning representation power with reinforcement-learning objectives to solve complex visual-control tasks from raw pixels. Existing multi-tasking approaches primarily distill multiple task-specific experts or transfer knowledge across successive tasks, but progressive networks are not truly online [Parisotto et al., 2016; Rusu et al., 2016b].

  • Motivation: DRL combines deep-learning representation power with reinforcement-learning objectives and enables agents to solve complex visual-control tasks directly from raw pixels.The introduction cites DL and RL foundations alongside multiple DRL demonstrations.
  • Related Work: Goal-directed multi-tasking approaches fall into expert-distillation and transfer-learning categories.The first category extracts multiple task-specific expert networks into one student, while the second transfers knowledge across tasks.
  • Related Work: Policy Distillation and Actor-Mimic Networks train k task-specific DQNs before distilling their individual task policies into a single student network [Parisotto et al., 2016].Here, k denotes the number of tasks.
  • Related Work: Progressive networks iteratively learn successive tasks and can be adapted to multi-task learning, but they are not truly online [Rusu et al., 2016b].The cited transfer-learning approaches include Progressive Networks and related DRL work [Parisotto et al., 2016; Rusu et al., 2016b; Fernando et al., 2017].

2 Background

The background introduces UCB methods for bandit task selection, including Discounted UCB for non-stationary problems, and actor–critic reinforcement learning with A3C for deep-network training.

  • UCB algorithms select actions by tracking uncertainty and adding exploration bonuses to less-explored arms.
  • Discounted UCB methods address non-stationary bandits, and the experiments use Discounted UCB1-Tuned+.
  • Parametric actor–critic algorithms combine an actor that maps states to policies with a critic-based, low-variance policy-gradient estimate.The actor uses πθa(at|st), and the gradient estimate includes Q(st, at)−b(st).
  • A3C addresses correlated consecutive updates in on-policy deep-network reinforcement learning by asynchronously running multiple actor–critic network versions [Mnih et al., 2016a].

3 Model Definition

The framework trains a single A3C-based multi-tasking network online by interleaving observations across tasks and actively deciding which task to train next. It includes random, adaptive, UCB bandit, and episodic reinforcement-learning task-selection methods.

  • Framework: The online framework trains one A3C multi-tasking network on interleaved task observations and selects the next training task at regular decision steps.The network receives no expert-policy predictions and optimizes a reinforcement-learning objective across the tasks.
  • A4C: A4C uses active-learning principles to choose training data from tasks, allocating future training toward tasks where the multi-tasking agent needs more improvement.Its active-sampling framework is instantiated through alternative meta-decision methods.
  • BA3C: BA3C trains a single A3C network by uniformly random task selection at the end of each training episode.It serves as the simple online multi-task learning algorithm, with the task set interleaved throughout training.
  • A5C: A5C estimates each task’s performance lag m_i and samples tasks from a softmax distribution, assigning higher probability to tasks with larger deficits.The method is the only proposed variant that does not learn the sampling distribution directly; τ controls the softmax temperature.
  • UA4C: UA4C formulates next-task selection as a multi-armed bandit, treating tasks as arms and using the latest task’s performance lag m_i as the meta-learner reward.Task decisions occur at the end of each training episode.
  • EA4C: EA4C formulates task selection as a full reinforcement-learning problem that optimizes future rewards while emphasizing improvement on the three worst-performing tasks.Its state includes task identity and the previous sampling policy, enabling conditioning on task-sampling counts.

4 Experimental Setup and Results

Experiments evaluate the proposed multi-task agents across seven Atari multi-tasking instances containing 6–21 tasks, using several aggregate performance metrics. On MT1, A5C performs best, while UA4C and EA4C are only slightly lower.

  • Experimental Setup: The multi-task agents use a shared 18-action output layer across tasks rather than separate task-specific output heads.The A3C multi-task networks generally match the size and architecture of a single-task network, except when solving the 21-task MT7 instance.
  • Evaluation: Performance is evaluated with pam, qam, qgm, and qhm, with qam results reported in Table 1 and the other metric evaluations provided in Appendix E.The authors argue that the arithmetic-mean metric pam is not robust, motivating additional aggregate metrics.
  • Experimental Setup: The framework is tested on seven multi-tasking instances ranging from 6 to 21 constituent tasks, with hyperparameters tuned on the six-task MT1.The 21-task MT7 instance contains more than twice as many tasks as any previous work cited by the authors.
  • Results: A5C performs best on MT1, while UA4C and EA4C achieve only slightly lower performance.MT1 was the instance used for hyperparameter tuning; the authors attribute the difference partly to the greater number of hyperparameters in UA4C and EA4C.

5 Analysis of Trained Agents

The analysis attributes A4C’s stronger multitasking performance than BA3C to task-agnostic abstract features learned by the framework, while examining neuron specialization and causal importance through activation analyses.

  • Interpretation: A4C’s advantage over BA3C is attributed to task-agnostic abstract features, although deep networks may also multitask through task-specific representations.The analysis frames task-agnostic representation learning as the key explanation for strong performance across tasks.
  • Neuron-firing analysis: Neuron firing is measured across ten episodes per constituent task, identifying neurons active on a task when their firing fraction reaches 0.01.A neuron fires at a timestep when its output’s absolute value is at least 0.3; the low threshold captures potentially rare but important neurons.
  • Neuron-firing analysis: The firing analysis orders neurons from broadly shared to task-specific, with the figure’s piecewise-constant line counting how many tasks activate each neuron.The leftmost neurons fire across six tasks, whereas rightward neurons become increasingly task-specific; Appendix H extends the analysis across all instances and methods.
  • Turnoff-analysis: Figure 5 presents turnoff-analysis heatmaps for all agents, with BA3C normalized across five games because it scored zero on one game.The normalization excludes the zero-scoring game for BA3C.
  • Turnoff-analysis: Turnoff-analysis estimates each neuron’s importance without thresholds by zeroing one LSTM output activation and measuring absolute percentage performance changes on individual tasks.The altered scores are compared with the original agent scores when no neurons are switched off.

6 Conclusions and Future Work

The paper proposes an active-learning framework for training MTNs to perform online multi-task learning by allocating training to tasks on which performance is currently poor. The authors present it as an important first step rather than a definitive solution to online multi-task reinforcement learning.

  • The framework trains MTNs for online multi-task learning through active task selection.The method uses an MTA to choose which task to train on.
  • By selecting the next task, the MTA concentrates resources on tasks where the agent currently performs poorly.
  • The authors characterize the method as an important first step, not a definitive solution to online multi-task reinforcement learning.

Appendix A: Fine-Grained Meta-learner Active Sampling Method

FA4C refines EA4C by choosing tasks every 20 training steps rather than at episode boundaries, using fine-grained performance-based rewards and target scores. It outperforms random sampling on some multi-tasking instances but performs poorly on others, motivating improved fine-grained meta-controllers.

  • Fine-grained task selection: FA4C makes task decisions every 20 multi-tasking-network training steps instead of at episode ends, using EA4C’s architecture and training procedure.The choice N = 20 matches the n-step-return value used in the work and by [Sharma et al., 2017].
  • Reward and target construction: FA4C rewards task-selection performance over each 20-step interval by comparing multi-tasking scores with task-specific averaged 20-step target scores.The targets are computed from trained single-task agents by averaging their scores across episode segments.
  • Reward and state design: The fine-grained meta-learner uses the same state function as the episodic meta-learner, while its reward uses targets defined for each task.The reward modification defines task-specific target and multi-tasking scores over twenty time steps.
  • Limitations: FA4C performs better than random on some multi-tasking instances but poorly on others, indicating a need for better fine-grained meta-controller designs.Its target-score approach also requires training single-task networks because fine-grained targets are difficult to obtain.

Appendix B: Experimental Details … FA4C

The appendix defines seven Atari multi-tasking instances and specifies the shared training, evaluation, architecture, and meta-task-decider settings for BA3C, A5C, UA4C, EA4C, and FA4C. It also reports architectural findings for EA4C and distinguishes FA4C’s time-driven task decisions from EA4C’s event-driven decisions.

  • Multi-tasking instances: The experiments cover seven multi-tasking instances: three 6-task instances, one 8-task instance, two 12-task instances, and one 21-task instance.MT4 reproduces the 8-task set studied by Parisotto et al. [Parisotto et al., 2016] to enable comparison with prior results.
  • Simple hyper-parameters: Training lasted 300 million steps for MT1–MT3, 400 million for MT4, 600 million for MT5–MT6, and 1.05 billion for the 21-task instance.These durations were chosen so each method would be at least 50% more data efficient than competing off-line policy-distillation methods.
  • Simple hyper-parameters: Training and evaluation were interleaved, but evaluation ran every task for equal time: five episodes capped at 30,000 steps each.Active sampling was used during training, whereas testing used equal-duration execution across tasks.
  • Architecture details: The multi-tasking agent uses an LSTM architecture based on prior A3C designs, with convolutional layers, two fully connected layers, and separate policy and value-function outputs.Because task identity must be inferred from visual input, constituent tasks need visually distinct observation spaces; the actor and critic share all but their final layers.
  • Simple hyper-parameters: A5C selects tasks with a tuned softmax temperature of 0.05, while UA4C uses discounted UCB1-tuned+ with γ = 0.99 and β = 0.25.A5C also sets n = 10 and l = 4 million; UA4C’s decider is based on Kocsis & Szepesvári.
  • Simple hyper-parameters: EA4C assigns one A3C-based meta-learner to each multi-task learner thread, samples tasks from its policy, and trains the meta-learner with 1-step returns.The meta-learner uses async-rms-prop and therefore differs from the usual 20-step A3C returns.
  • FA4C: FA4C tunes hyper-parameters as in EA4C but makes task decisions at regular training-time intervals rather than at episode ends, using n = 20 for both decision intervals and returns.This contrasts FA4C’s time-driven decisions with EA4C’s event-driven decisions.

Appendix C: Training algorithms for our proposed methods … Training Fine-grained Meta-learner based active sampling agent (FA4C)

Appendix C gives pseudocode for training the proposed active-sampling methods and the BA3C baseline. It specifies uniform task sampling for BA3C and increasingly adaptive task-selection procedures for A5C, UA4C, EA4C, and FA4C.

  • Appendix C: Training algorithms for our proposed methods: Appendix C instantiates the generic active-sampling training algorithm for every proposed method and provides a separate training algorithm for the MTA baseline.
  • Training baseline multi-tasking networks (BA3C): BA3C samples each of k tasks with probability 1/k throughout training and updates the baseline agent after each selected task episode.
  • Training adaptive active-sampling agents (A5C): A5C selects tasks using probabilities based on target scores, recent per-task performance, and a softmax temperature, while ensuring every task receives at least n episodes initially.
  • Training UCB-based meta-learner (UA4C): UA4C treats task selection as discounted UCB, tracking discounted rewards, mean rewards, exploration bonuses, and discounted episode counts for each task.
  • Training Episodic Meta-learner based active sampling agent (EA4C): EA4C trains an episodic meta-learner whose task-selection state includes recent scores and episode counts, with rewards combining two components over selected tasks.
  • Training Fine-grained Meta-learner based active sampling agent (FA4C): FA4C extends episodic meta-learning with a fine-grained episode length N and fine-grained target scores, while tracking recent scores, task counts, and meta-learner rewards.
  • Training Fine-grained Meta-learner based active sampling agent (FA4C): FA4C computes one reward component from the l smallest normalized scores and combines it with a second component as r = λr1 + (1 − λ)r2 before sampling the next task.

Appendix D: Training Curves for all methods and multitasking instances … Multi-tasking instance 6 (MT6)

Appendix D plots raw gameplay performance over training for BA3C, A5C, UA4C, EA4C, and FA4C against task-specific A3C agents across multitasking instances ranging from 6 to 21 tasks. The experiments use half the data and computation of training task-specific agents for all tasks, with training budgets increasing from 300 million to 1.05 billion time steps.

  • Appendix D: Training Curves for all methods and multitasking instances: The appendix tracks raw gameplay performance versus training progress, with STA3C denoting a task-specific A3C agent trained on one task.Performance uses the evaluation procedure described in Appendix B.
  • Multi-tasking instance 1 (MT1): For MT1, the five multitasking agents are compared with task-specific A3C agents on 6 tasks after 300 million time steps using half the data and computation.The compared agents are BA3C, A5C, UA4C, EA4C, and FA4C.
  • Multi-tasking instance 2 (MT2): MT2 likewise evaluates BA3C, A5C, UA4C, EA4C, and FA4C against task-specific A3C agents on 6 tasks, with 300 million training steps and half the data and computation.The figure reports training-curve comparisons for all listed agents.
  • Multi-tasking instance 3 (MT3): MT3 compares the same five multitasking methods with task-specific A3C agents on another 6-task instance trained for 300 million time steps using half the data and computation.The results are presented as performance curves over training.
  • Multi-tasking instance 4 (MT4): MT4 evaluates the same methods on 8 tasks, including the task set used in [Parisotto et al., 2016], with 400 million training steps and half the data and computation.Performance is compared with task-specific A3C agents.
  • Multi-tasking instance 5 (MT5): MT5 scales to 12 tasks while keeping the multitasking network size equal to those used for MT1–MT3 and a single-task network, using 600 million steps and half the data and computation.The figure compares BA3C, A5C, UA4C, EA4C, and FA4C with task-specific A3C agents.
  • Multi-tasking instance 6 (MT6): MT6 presents a 12-task comparison of BA3C, A5C, UA4C, EA4C, and FA4C with task-specific A3C agents after 600 million training steps using half the data and computation.The supplied passage identifies this instance as having 12 tasks.

Appendix E: Performance of Our Methods on All the Evaluation Metrics · Appendix F: Demonstrative visualization of our methods

Appendix E compares the methods across four proposed metrics, showing that EA4C generalizes best to the largest instance while adaptive sampling remains a strong baseline. Appendix F provides visual depictions of the methods A5C, UA4C, EA4C, and FA4C.

  • Appendix E: Performance of Our Methods on All the Evaluation Metrics: qam is designed to reward agents that perform well across all tasks, making it a robust multi-tasking metric.The appendix reports method performance on pam, qam, qgm, and qhm.
  • Appendix E: Performance of Our Methods on All the Evaluation Metrics: EA4C generalizes best to the largest multi-tasking instance, MT7, while the adaptive method remains a hard baseline to beat and UCB generalizes only to medium-sized instances.These comparisons are reported for qam, which rewards agents that perform well across all tasks.
  • Appendix E: Performance of Our Methods on All the Evaluation Metrics: On MT4, BA3C achieves the best non-clipped average performance under pam but is not a good multi-tasking algorithm because uniform sampling overtrains Enduro.High performance on that single task causes pam to identify BA3C as the best multi-tasking method in this case.
  • Appendix E: Performance of Our Methods on All the Evaluation Metrics: qgm and qhm were introduced because qam can remain high when an agent performs well on only a few tasks, with qhm described as the strongest measure of multi-tasking performance.For MT4, A5C is slightly better than EA4C under qam, whereas their qhm performance is much closer.
  • Appendix F: Demonstrative visualization of our methods: The appendix presents visualizations specific to each proposed method as a general visual depiction of the overall approach.These visualizations complement the metric-based comparisons in Appendix E.

Appendix G: Robustness of our proposed framework (A4C) to target scores … Appendix H: Firing Analyses for all multitasking instances

Appendix G examines whether target-score requirements compromise the framework’s online setting or robustness, while Appendix H analyzes neuron-firing patterns across multitasking instances. The results show that target scores can be estimated without single-task agents, performance remains strong under alternative targets despite task-specific weaknesses, and active-sampling methods use more broadly shared neurons than BA3C.

  • Appendix G: Robustness of our proposed framework (A4C) to target scores: The appendix evaluates two concerns about target scores: whether obtaining them requires trained single-task agents and whether changing them can cause dramatic under-performance.These concerns motivate separate analyses of target-score availability and robustness.
  • On the availability of target scores: Published target scores do not require training single-task agents, but when no solved tasks exist, the doubling-target paradigm provides rough estimates instead.The doubling scheme doubles a task’s estimated target whenever the agent reaches or exceeds it, allowing continued improvement toward progressively higher thresholds.
  • On the availability of target scores: DUA4C achieved impressive performance in the unfavorable doubling-target setup, although it failed to solve two tasks and might improve with paradigm-specific hyperparameter tuning.The experiment was conducted on MT1 using doubling target estimates rather than single-task network scores.
  • On the robustness of A4C to target scores: The robustness experiments test whether the framework’s performance depends on using single-task performance as target scores.They replace single-task scores with human scores or with twice the single-task scores.
  • Use of human scores: HUA4C remained impressive without retuning hyperparameters, but learned nothing on two tasks and achieved at best mediocre performance on three others.The experiment used human tester scores from as targets for UA4C on MT1.
  • Use of twice the single-task scores as targets: Using twice the single-task performance as an optimistic target was evaluated with A5C on MT2 and MT3, where performance was expected to differ from MT1 because hyperparameters were tuned on MT1.Table 9 compares agents using usual versus doubled target scores.
  • Appendix H: Firing Analyses for all multitasking instances: Across the firing analyses, A5C, UA4C, and EA4C had many neurons firing for much of the time across at least half the tasks, whereas BA3C had relatively more task-specific neurons.The analyses cover all multitasking instances presented in the work, with figures for MT1 through MT7.

Appendix I: Different output heads for different tasks · Appendix J: Target scores for all the tasks

Appendix I evaluates task-specific output heads against shared-head agents and finds that shared-head A5C performs better, while Appendix J documents the target scores used for all experimental tasks.

  • Appendix I: Different output heads for different tasks: Different output heads were motivated by prior multi-task and policy-distillation methods that shared most layers while using task-specific final layers.The appendix cites Parisotto et al. (2016) and Rusu et al. (2016a) in this context.
  • Appendix I: Different output heads for different tasks: DA5C is the different-head version of A5C, retaining the same architecture except for task-specific output heads.A5C was selected because it is the simplest method with impressive performance on smaller multi-task instances.
  • Appendix I: Different output heads for different tasks: Each DA5C head outputs only the valid actions for its task, with task-specific projection matrices and an added parameter count based on the action-set sizes.The appendix defines πij as the probability of selecting action j in task i and Wi as task i’s projection matrix.
  • Appendix I: Different output heads for different tasks: Hyperparameter tuning on MT1 selected β = 0.02 and τ = 0.05 for DA5C, while DBA3C used β = 0.1.The tuning procedure was similar to that used for A5C.
  • Appendix I: Different output heads for different tasks: DA5C outperforms DBA3C on MT1, MT2, and MT3, especially when optimizing performance across all tasks.Table 10 reports the comparison between the different-head adaptive and baseline agents.
  • Appendix I: Different output heads for different tasks: Shared-head A5C strictly outperforms different-head DA5C, arguing against task-specific output heads when constituent tasks share an action space.This conclusion is supported by comparisons across the small multi-tasking instances MT1, MT2, and MT3.
  • Appendix J: Target scores for all the tasks: Appendix J lists the target scores for every task used in the experiments.These targets were taken from the single-task A3C scores in Table 4 of Sharma et al. (2017), and are presented in Table 11.
Loading 1702.06053v4…