PaperlayerView exact revision on arXiv

Source-linked AI summary

Many Task Learning with Task Routing

Gjorgji Strezoski, Nanne van Noord, Marcel Worring

arXiv:1903.12117v1cs.CV

TL;DR

Multi-task learning methods can require increasingly complex architectural adjustments and resources as task counts rise. Task Routing applies task-specific masks to convolutional activations, providing routes through a model without prior domain knowledge. The authors report competitive performance in regular MTL and state-of-the-art performance on all possible UCSD-Birds tasks in an MaTL setting.

  • Problem

    As task counts increase, typical MTL methods face greater architectural-adjustment complexity and resource requirements, while searching for task-sharing structures also becomes less efficient.

    Source: §1 · p. 1

  • Method

    Task Routing uses task-specific masks in a Task Routing Layer to conditionally transform convolutional activations and route task data flow without prior domain knowledge.

    Source: §1 · p. 2, §3.2 · p. 4, §6 · p. 8

  • Results

    The authors report competitive performance at small task counts and state-of-the-art performance on all possible UCSD-Birds tasks in an MaTL setting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method offers a way to explore sharing possibilities in regular MTL and MaTL without changing the underlying architecture.

    Source: §6 · p. 8

    The routing masks are immutable during training, so the sharing strategy is fixed from the start.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Typical multi-task learning (MTL) methods rely on architectural adjustments and a large trainable parameter set to jointly optimize over several tasks. However, when the number of tasks increases so do the complexity of the architectural adjustments and resource requirements. In this paper, we introduce a method which applies a conditional feature-wise transformation over the convolutional activations that enables a model to successfully perform a large number of tasks. To distinguish from regular MTL, we introduce Many Task Learning (MaTL) as a special case of MTL where more than 20 tasks are performed by a single model. Our method dubbed Task Routing (TR) is encapsulated in a layer we call the Task Routing Layer (TRL), which applied in an MaTL scenario successfully fits hundreds of classification tasks in one model. We evaluate our method on 5 datasets against strong baselines and state-of-the-art approaches.

1. Introduction

The introduction frames multi-task learning as a search for useful task-sharing structures that becomes less efficient as models and task sets grow. Task Routing regulates per-task data flow to form specialized subnetworks without prior domain knowledge, and the paper reports competitive small-task performance and state-of-the-art performance across all possible UCSD-Birds tasks in a Many Task Learning setting.

1. Introduction

  • Task Routing uses structured random per-task data flow to form specialized subnetworks and provide alternate routes through the parameter space.The authors report that alternate routes increase feature robustness and improve scalability while boosting or maintaining predictive performance.Source: §1 · pp. 1–2
  • The method aims to mitigate destructive interference by partially sharing structures, distributing negative task interactions, and allowing tasks to adapt if interference is present.The authors describe destructive interference as the negative influence between tasks that share a representation [47].Source: §1 · p. 2
  • Task Routing offers a way to forge beneficial sharing without prior domain knowledge, which existing statistical, architecture, and branch-design approaches often require.The introduction notes that such prior knowledge may be unavailable or expensive to obtain.Source: §1 · p. 2
  • The authors define Many Task Learning (MaTL) as MTL with more than 20 tasks and report scaling experiments from four Zappos-50K tasks to 312 UCSD-Birds tasks.They report competitive performance at small task counts and state-of-the-art performance on all possible tasks in UCSD-Birds in an MaTL context.Source: §1 · p. 2
  • The introduction identifies scalable MTL without prior domain knowledge, structured deterministic sampling of subarchitectures, and non-parametric task-relationship formation during training as contributions.The paper also reports evaluation on five classification and retrieval datasets against strong baselines and state-of-the-art approaches.Source: §1 · p. 2

2. Related Work

The related-work discussion situates Task Routing among symmetric multi-task learning, task-sharing approaches, and regularization or architecture-search methods. It contrasts methods that depend on prior knowledge or structural changes with controlling data flow in an existing model, and describes routing as a way to use many subnetworks with a small computational and memory footprint.

2. Related Work

  • Symmetric MTL seeks to improve all tasks together through shared representations, but prior approaches can be vulnerable to noisy or outlier tasks.The paper says this vulnerability is associated with low feature robustness and an assumption that tasks positively influence one another [44].Source: §2 · pp. 2–3
  • Task-sharing and grouping methods often rely on prior knowledge, while Task Routing controls data flow without structural adjustments and is presented as applicable to deep MTL models.The paper notes that domain knowledge can be unavailable or require expert analysis.Source: §2 · p. 3
  • The authors report that Task Routing has a small memory and computational footprint and scales to 300+ tasks, compared with modulation work reported at 20 and 40 tasks [47].The cited modulation approach reports 15% overhead in feedforward passes [47].Source: §2 · p. 3
  • Convolutional neural fabrics search connected response maps across layers, whereas Task Routing is described as pooling from an exponentially large set of subnetworks [26].Source: §2 · pp. 3–4
  • Unlike Dropout's changing Bernoulli-sampled units, Task Routing is described as deterministic inter-task regularization that can be combined with Dropout.The cited related approaches also use dropout-style regularization in single-task learning [29] [6] [39].Source: §2 · p. 4

3. Task Routing

Task Routing reuses convolutional units across tasks by masking activations for the active task, without changing the underlying model structure. Fixed task masks and a sharing-ratio setting govern which units are shared or task-specific.

3. Task Routing

  • Task Routing applies task-specific binary masks to convolutional activations, rerouting each task through shared and specialized units within one model.The active task's mask is applied channel-wise to activations before they reach the next layer.Source: §3 · p. 4, §3.2 · p. 5

3.1. Task Routing Mask Creation

  • Masks are randomly initialized, remain fixed during training and testing, and use sharing ratio σ to set the balance between shared and task-specific units.σ = 0 gives distinct subnetworks; σ = 1 gives a fully shared MTL architecture.Source: §3.1 · p. 4, §3 · p. 4

3.2. Task Routing Layer

  • The TRL applies the active task's mask across convolutional channels and passes the masked output onward; it is placed after batch normalization when present.Changing the active task selects the mask applied across the model's TRLs.Source: §3.2 · pp. 4–5

3.3. Complexity

  • Without separate task embeddings, increasing task count adds a negligible number of parameters, while task-specific branches account for most added parameters.TRL parameters scale with the number of convolutional layers, units, and channels.Source: §3.3 · p. 5

4. Experimental Design

The experiments test Task Routing across classification datasets and task counts, using baselines, multiple sharing ratios, and accuracy, precision, and recall. They also examine implementation settings and compare performance across sharing ratios.

4.1. Datasets

  • FashionMNIST and CIFAR-10 provide proof-of-concept tests, while UCSD Birds, CelebA, and UT-Zappos50K extend evaluation to attribute-rich or real-world datasets.UCSD Birds supplies 312 binary attributes; CelebA includes 40 face attributes, and Zappos50K tests four shoe attributes.Source: §4.1 · p. 5

4.2. Multi-task Setup

  • The setup tests small-task and Many Task Learning regimes, including up to 312 UCSD-Birds tasks and up to 40 CelebA tasks.UCSD-Birds uses VGG-11, while CelebA uses VGG-16 with task-specific classification branches.Source: §4.2 · p. 6

4.3. Implementation Details

  • Experiments use VGG-11, VGG-16, or ResNet50 with TRLs after convolutional layers, batch size 64, and SGD at learning rate 0.01.UCSD-Birds also uses horizontal flipping and includes pretrained and scratch-trained VGG-11 settings.Source: §4.3 · p. 6

4.4. Evaluation criteria

  • The evaluation tracks accuracy, precision, and recall, and Table 1 summarizes average scores across datasets and sharing ratios.The reported comparison includes FashionMNIST, CIFAR-10, Zappos50K, and CelebA at 10 and 40 tasks.Source: §4.4 · p. 6, Table 1 · p. 7
  • Partial sharing ratios σ = 0.2–0.6 are reported as beneficial to accuracy, precision, and recall compared with σ = 0 or σ = 1.Figure 5 covers Fashion-MNIST, CIFAR10, CelebA, and Zappos50K.Source: Figure 5 · p. 7

5. Results

Across five datasets, Task Routing outperforms modulation for MTL and cross-stitch networks. Results also show how task count and sharing ratio relate to performance, including a drop for CelebA’s expanded task pool.

5. Results

  • Experiments span five datasets, from proof-of-concept tasks to task-count scaling, with σ ranging from no sharing to fully shared structure.σ = 0 denotes a distinct specialized subnet per task, whereas σ = 1 denotes a completely shared structure.Source: §5 · p. 7
  • Task Routing surpasses modulation for MTL and cross-stitch networks across five datasets in MTL/MaTL settings.The comparison is reported in Table 1 and Figure 4.Source: §5 · p. 7
  • For σ = 0.2 and 0.4, accuracy, precision, and recall improve as task count increases; σ = 0.8 drops significantly, while σ = 1 is hard-shared MTL.Table 2 covers UCSD-Birds task counts and sharing ratios from σ = 0 to 1.Source: §5 · p. 7, Table 2 · p. 8
  • On CelebA, performance drops across applicable approaches when the pool expands from 10 to 40 tasks; the authors suspect fewer positive training instances for the additional tasks.The suspected explanation is specific to the additional tasks in the classification setting.Source: §5 · p. 7
  • As task count rises, cross-stitch networks cannot fit beyond 12 tasks, while routing gains slightly and modulation declines.Cross-stitch requires a complete model per task; routing and modulation are more parameter-savvy.Source: §5 · p. 8

6. Conclusion

The conclusion presents Task Routing as a way to perform many classification tasks in one model through conditional transformations of convolutional outputs. Its adjustable sharing does not require prior knowledge of the domain or intertask relationships.

6. Conclusion

  • Task Routing uses conditional feature-wise transformations of convolutional-layer outputs to perform many classification tasks with one model.The approach is reported to perform well in both regular MTL and MaTL without prior domain or intertask-relationship knowledge.Source: §6 · p. 8
  • The Task Routing Layer inserts after convolutional layers with minimal effort and computational overhead, using task-specific masks to fit multiple tasks in one model.The masks train specialized subnetworks for each task.Source: §6 · p. 8
  • A task-specific subnet’s dimensionality depends on σ, and the subnet can be extracted for use instead of the complete model.This provides a task-specific alternative to deploying the full model.Source: §6 · p. 8
  • The sharing-ratio hyperparameter σ adjusts routing without changing the underlying architecture, offering flexibility across MTL and MaTL.The paper presents it as a way to explore the model’s sharing possibilities.Source: §6 · p. 8

Related papers

Loading 1903.12117v1…