An Overview of Multi-Task Learning in Deep Neural Networks

Sebastian Ruder

Source-linked AI summary

An Overview of Multi-Task Learning in Deep Neural Networks

Sebastian Ruder

arXiv:1706.05098v1Jun 15, 2017cs.LGcs.AIstat.ML

TL;DR

Multi-task learning can improve a target task by leveraging training signals from related tasks, but choosing what to share and which auxiliary tasks to use remains challenging. This overview synthesizes MTL methods and literature, highlighting promising learned-sharing approaches while noting that task relationships and auxiliary-task benefits remain insufficiently understood.

Problem
Single-task models ignore potentially useful training signals from related tasks that could improve generalization on the target metric.
Method
The article surveys MTL literature, explains hard and soft parameter sharing, reviews recent deep-learning methods, and discusses auxiliary-task selection.
Results
Recent approaches that learn what to share generally outperform hard parameter sharing, while evidence on helpful auxiliary-task properties remains limited in scope.
Takeaways & Limitations
Practitioners should consider learned sharing and auxiliary tasks with compact, uniform labels or non-plateauing signals, while treating current guidance as preliminary.
Takeaways & Limitations
The field still lacks a principled understanding of task similarity, relationships, hierarchy, and benefits for MTL with deep neural networks.

Abstract

from arXiv · show

Multi-task learning (MTL) has led to successes in many applications of machine learning, from natural language processing and speech recognition to computer vision and drug discovery. This article aims to give a general overview of MTL, particularly in deep neural networks. It introduces the two most common methods for MTL in Deep Learning, gives an overview of the literature, and discusses recent advances. In particular, it seeks to help ML practitioners apply MTL by shedding light on how MTL works and providing guidelines for choosing appropriate auxiliary tasks.

1 Introduction

Multi-task learning (MTL) improves generalization by leveraging domain-specific information from related training signals and has succeeded across diverse machine-learning applications. This overview focuses on MTL in deep neural networks, covering core methods, mechanisms, literature, recent advances, and auxiliary-task selection.

1 Introduction: MTL has been applied successfully in natural language processing, speech recognition, computer vision, and drug discovery.It is also described using terms such as joint learning, learning to learn, and learning with auxiliary tasks.
1 Introduction: MTL leverages information from related training signals to improve generalization beyond optimizing a single task alone.The paper contrasts MTL with single-task optimization, which may ignore useful information.
1 Introduction: The article provides a general overview of MTL in deep neural networks, including its motivation, two most frequently employed deep-learning methods, mechanisms, literature, and recent advances.The sections progress from motivation and core methods to mechanisms, prior literature, and newer neural-network-based approaches.
1 Introduction: The article concludes by examining common auxiliary-task types and discussing what makes an auxiliary task effective for MTL.This guidance is intended to help practitioners choose appropriate auxiliary tasks.

2 Motivation

Multi-task learning is motivated by how humans transfer knowledge and acquire foundational skills for more complex tasks. In machine learning, auxiliary tasks provide an inductive bias that guides models toward preferred hypotheses.

Motivations: Human learning motivates MTL because knowledge acquired on related tasks can transfer to new tasks, such as recognizing objects after faces.The passage uses a baby’s learning sequence as an example of biologically inspired transfer.
Motivations: Pedagogically, learners first develop foundational skills that enable mastery of more complex techniques.Examples include learning how to fall before advanced Judo and programming skills.
Motivations: Seemingly unrelated tasks can equip learners with skills relevant to a target task, as illustrated by Mr Miyagi’s training in The Karate Kid.Sanding the floor and waxing a car ultimately provide skills useful for learning karate.
Motivations: From a machine-learning perspective, MTL is a form of inductive transfer in which auxiliary tasks provide an inductive bias favoring some hypotheses over others.The passage contrasts this with ℓ1 regularization, which induces a preference for sparse solutions.

3 Two MTL methods for Deep Learning

Deep neural network MTL typically uses hard or soft parameter sharing. Hard sharing shares hidden layers while retaining task-specific outputs and reduces overfitting risk; soft sharing keeps separate models and regularizes parameter distance.

Deep neural network MTL typically uses hard or soft parameter sharing of hidden layers.
Hard parameter sharing: Hard parameter sharing shares hidden layers across tasks while retaining several task-specific output layers.It is the most commonly used neural-network MTL approach and dates to Caruana (1993).
Hard parameter sharing: Hard parameter sharing reduces overfitting risk because shared parameters are harder to overfit as the model learns representations across more tasks.Baxter (1997) showed the risk of overfitting shared parameters is an order N smaller than for task-specific parameters, where N is the number of tasks.
Soft parameter sharing: Soft parameter sharing gives each task its own model and regularizes distances between task parameters to encourage similarity.Duong et al. (2015) use ℓ2-distance regularization, whereas Yang and Hospedales (2017b) use the trace norm.

4 Why does MTL work?

MTL works by using related tasks to learn representations that are less sensitive to task-specific noise and more useful across tasks. It also focuses learning on relevant features, transfers features between tasks, and regularizes models against overfitting.

4 Why does MTL work?: MTL effectively increases the training sample size, helping models learn more general representations from tasks with different noise patterns.Joint learning reduces the risk of overfitting to one task’s data-dependent noise.
4 Why does MTL work?: MTL helps models identify relevant features when data are limited, high-dimensional, or noisy by providing additional evidence across tasks.Other tasks can indicate which features matter and which are irrelevant.
4 Why does MTL work?: MTL enables a model to learn features that are easy for one task but difficult for another through cross-task transfer or direct feature-prediction hints.Hints directly train the model to predict important features.
4 Why does MTL work?: MTL biases models toward representations preferred by multiple tasks, which can support learning novel tasks from the same environment.A hypothesis space performing well across sufficiently many training tasks is expected to perform well on future tasks in that environment.
4 Why does MTL work?: MTL acts as a regularizer by introducing an inductive bias that reduces overfitting risk and the model’s Rademacher complexity.Rademacher complexity refers to the ability to fit random noise.

5 MTL in non-neural models

MTL methods in linear, kernel, and Bayesian models primarily enforce sparsity across tasks or model relationships among tasks. Sparsity methods exploit shared features, whereas relationship-based methods accommodate clustering and structured task dependencies to reduce negative transfer.

Overview: MTL literature centers on shared-feature sparsity and modeling relationships between tasks.These ideas recur across linear models, kernel methods, and Bayesian algorithms.
Sparsity across tasks: Mixed ℓ1/ℓq regularization produces block sparsity by setting entire feature rows of the task parameter matrix A to zero.The choice of ℓq determines the constraint across tasks; ℓ1/ℓ∞ and mixed ℓ1/ℓ2 group lasso are examples.
Sparsity across tasks: Trace-norm regularization offers a convex alternative to non-convex group lasso by forcing A to be low-rank.This constrains task parameter vectors to lie in a low-dimensional subspace.
Sparsity across tasks: When tasks share few features, block-sparse regularization can underperform element-wise ℓ1 regularization.Combining block-sparse and element-wise sparse penalties addresses this limitation by decomposing A into block-sparse B and element-wise sparse S.
Modeling task relationships: Because unrelated tasks can cause negative transfer, relationship-based methods cluster task parameters toward shared means or model tree and graph structures.Cluster regularization penalizes task norms and variance, while later formulations can learn clusters jointly when their number is not fixed.

6 Recent work on MTL for Deep Learning · 6.1 Deep Relationship Networks · 6.2 Fully-Adaptive Feature Sharing

Recent deep-learning MTL work largely continues to use hard or soft parameter sharing, while a smaller body of research develops more flexible task-sharing mechanisms. Deep Relationship Networks learn task relationships through matrix priors, whereas fully-adaptive feature sharing dynamically widens networks to group similar tasks during training.

6 Recent work on MTL for Deep Learning: Most recent deep-learning MTL approaches use hard or soft parameter sharing, with comparatively few developing new mechanisms for MTL.These approaches may use MTL explicitly or implicitly as part of their models.
6.1 Deep Relationship Networks: Deep Relationship Networks share convolutional layers while retaining task-specific fully connected layers.This is a common structure in computer-vision MTL.
6.1 Deep Relationship Networks: Matrix priors on fully connected layers enable Deep Relationship Networks to learn relationships between tasks.The mechanism is described as similar to Bayesian models discussed earlier.
6.1 Deep Relationship Networks: Deep Relationship Networks combine shared convolutional and task-specific fully connected layers with matrix priors.Figure 3 depicts this architecture.
6.2 Fully-Adaptive Feature Sharing: Fully-adaptive feature sharing starts with a thin network and dynamically widens it greedily during training.The widening procedure dynamically creates branches.
6.2 Fully-Adaptive Feature Sharing: The widening criterion promotes grouping similar tasks, but the greedy method may fail to find a globally optimal model.Assigning each branch to exactly one task also limits the model’s ability to learn more complex interactions.
6.2 Fully-Adaptive Feature Sharing: Figure 4 illustrates the widening procedure for fully-adaptive feature sharing.The procedure is associated with Lu et al., 2016.

6.3 Cross-stitch Networks · 6.4 Low supervision · 6.5 A Joint Many-Task Model

The paper describes cross-stitch units that learn how task-specific networks combine information, then turns to hierarchical supervision and jointly structured NLP task architectures. These approaches place auxiliary-task supervision at appropriate depths and pre-define task hierarchies for multi-task learning.

6.3 Cross-stitch Networks: Cross-stitch networks begin with separate architectures and learn linear combinations of previous-layer outputs to control cross-task knowledge sharing.Cross-stitch units are placed after pooling and fully-connected layers.
6.3 Cross-stitch Networks: The two-task cross-stitch architecture places cross-stitch units after pooling and fully-connected layers, as illustrated in Figure 5.
6.4 Low supervision: NLP research focuses on task hierarchies in which low-level auxiliary tasks are supervised at lower network layers.The cited low-level tasks include part-of-speech tagging and named entity recognition, typically used for preprocessing.
6.4 Low supervision: Part-of-speech tagging and named entity recognition exemplify low-level NLP auxiliary tasks that should receive supervision at lower layers.
6.5 A Joint Many-Task Model: Building on hierarchical supervision, Hashimoto et al. pre-define an architecture containing several NLP tasks as a joint multi-task learning model.The architecture is presented in Figure 6.
6.5 A Joint Many-Task Model: The Joint Many-Task Model organizes several NLP tasks within a pre-defined hierarchical architecture, as illustrated in Figure 6.

6.6 Weighting losses with uncertainty · 6.7 Tensor factorisation for MTL

Sections 6.6 and 6.7 describe uncertainty-based loss weighting and tensor factorisation as approaches for improving multi-task learning in deep networks. The former adjusts task weights using task-dependent uncertainty, while the latter separates shared and task-specific parameters across layers.

6.6 Weighting losses with uncertainty: Uncertainty-based MTL adjusts each task’s relative cost-function weight according to its task-dependent uncertainty.Kendall et al. derive the weighting through a multi-task loss based on maximizing Gaussian likelihood.
6.6 Weighting losses with uncertainty: The uncertainty-weighting approach differs from methods that learn the structure of parameter sharing.It instead considers the uncertainty associated with each task when setting relative loss weights.
6.6 Weighting losses with uncertainty: The resulting architecture jointly addresses per-pixel depth regression, semantic segmentation, and instance segmentation.The paper presents this architecture in Figure 7.
6.6 Weighting losses with uncertainty: Figure 7 illustrates uncertainty-based loss-function weighting for multi-task learning.The figure is attributed to Kendall et al. (2017).
6.7 Tensor factorisation for MTL: Tensor factorisation generalizes existing multi-task learning approaches to deep learning.Yang and Hospedales extend previously discussed matrix factorisation approaches.
6.7 Tensor factorisation for MTL: Tensor factorisation splits model parameters into shared and task-specific components for every layer.This provides a layer-wise parameterization of sharing across tasks.

6.8 Sluice Networks · 6.9 What should I share in my model?

Sluice Networks learn which layers and subspaces to share, extending several deep MTL approaches. The paper argues that robust MTL should learn task interactions rather than rely on fixed sharing, especially for unrelated tasks.

6.8 Sluice Networks: Sluice Networks generalize hard parameter sharing, cross-stitch networks, block-sparse regularization, and task-hierarchy approaches.They learn which layers and subspaces should be shared and where the network has learned useful input-sequence representations.
6.8 Sluice Networks: The model learns both the layers and subspaces to share across tasks.This makes sharing decisions part of the learned architecture rather than fully specifying them beforehand.
6.9 What should I share in my model?: MTL must accommodate unrelated or only loosely related tasks because tasks are not always drawn from the same distribution.Although same-distribution tasks can benefit from sharing, this assumption does not always hold.
6.9 What should I share in my model?: Pre-specifying shared layers for each task pairing does not scale and heavily biases MTL architectures.Hard parameter sharing remains common but quickly breaks down when tasks are not closely related or require reasoning at different levels.
6.9 What should I share in my model?: Optimizing more than one loss function already constitutes MTL.The authors frame MTL broadly, without requiring a particular sharing architecture.
6.9 What should I share in my model?: Rather than compressing all task knowledge into one parameter space, models should learn how tasks interact.This recommendation draws on the MTL advances surveyed in the paper.

7 Auxiliary tasks

When only one task matters, auxiliary tasks can improve its representations by being related, adversarial, feature-focused, smoother, or explicitly transferable. However, selecting useful auxiliaries remains difficult because task relatedness lacks a principled definition, although early empirical clues identify favorable label distributions and learning dynamics.

Related tasks: Related tasks are the classical auxiliary-task choice, including road-characteristic prediction for steering and head-pose estimation for facial landmark detection.These examples illustrate using related predictions to support the main objective.
Adversarial tasks: Adversarial tasks can help when related labels are unavailable by maximizing training error through gradient reversal, as in domain adaptation.Domain prediction serves as the adversarial task while gradient reversal discourages domain-specific representations.
Feature-focused tasks: Auxiliary tasks can force models to learn otherwise neglected features, such as sentiment cues or lane markings, by predicting them directly.These tasks focus representation learning on information that the main task may otherwise ignore.
Objective design: Less quantized auxiliary objectives may help discrete-label tasks because smoother objectives can be easier to learn.This is relevant to human-assessed labels such as disease risk levels and sentiment categories.
Representation learning: Auxiliary tasks are useful when they enable representations that are shared with or helpful for the main task.This benefit may arise implicitly from task relatedness or explicitly from objectives known to produce transferable representations.
Task selection: Useful auxiliary-task selection remains uncertain because task relatedness has no satisfactory principled definition, while similarity lies on a spectrum rather than being binary.Early experiments suggest compact, uniform auxiliary-label distributions and non-plateauing auxiliaries can benefit some sequence-tagging tasks, but findings remain limited.

8 Conclusion

The overview finds that hard parameter sharing remains pervasive in neural-network MTL, while learning what to share is promising and task relationships remain poorly understood.

8 Conclusion: Hard parameter sharing, a 20-year-old paradigm, remains pervasive in neural-network-based MTL despite its increasing use.The conclusion contrasts continued prevalence with the broader growth of MTL applications.
8 Conclusion: Recent advances in learning what to share are promising for multi-task learning.These advances address how parameters should be shared across tasks.
8 Conclusion: Understanding of task similarity, relationships, hierarchies, and benefits for MTL remains limited and requires further study.The conclusion identifies task understanding as an unresolved research need.