PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Task Learning with Deep Neural Networks: A Survey

Michael Crawshaw

arXiv:2009.09796v1cs.LGcs.CVstat.ML

TL;DR

MTL seeks to learn multiple tasks together while addressing conflicting task needs and the challenge of deciding which tasks and representations should be shared. This survey synthesizes deep MTL methods across architectures, optimization, and task relationship learning, together with benchmark coverage. It concludes that the field has developed diverse sharing, optimization, and relationship-learning techniques but still requires continued work to achieve robust, human-like generalization.

  • Problem

    MTL promises data-efficient shared learning, but joint training introduces design challenges, negative transfer, and difficulty deciding which tasks to learn together.

  • Method

    The survey organizes deep MTL research into architecture design, optimization techniques, and task relationship learning, while defining its scope around fixed sets of simultaneously learned tasks.

  • Results

    The review identifies shared extractors with task-specific decoders, flexible parameter sharing, module recombination, learned sharing, loss weighting, gradient methods, multi-objective optimization, and task relationship methods as major techniques.

  • Takeaways & Limitations

    Developing MTL methods remains a promising route toward learning general concepts that transfer between tasks, but realizing that potential will require approaches from many directions.

  • Takeaways & Limitations

    The survey’s task-relationship discussion notes that gradient similarity becomes less representative of task similarity as training progresses in a sufficiently non-convex loss landscape.

Abstract

from arXiv · show

Multi-task learning (MTL) is a subfield of machine learning in which multiple tasks are simultaneously learned by a shared model. Such approaches offer advantages like improved data efficiency, reduced overfitting through shared representations, and fast learning by leveraging auxiliary information. However, the simultaneous learning of multiple tasks presents new design and optimization challenges, and choosing which tasks should be learned jointly is in itself a non-trivial problem. In this survey, we give an overview of multi-task learning methods for deep neural networks, with the aim of summarizing both the well-established and most recent directions within the field. Our discussion is structured according to a partition of the existing deep MTL techniques into three groups: architectures, optimization methods, and task relationship learning. We also provide a summary of common multi-task benchmarks.

1 Introduction

MTL learns multiple tasks simultaneously through shared representations, aiming to improve data efficiency and learning speed while reducing deep learning’s data and computational demands. The survey frames its central challenges as negative transfer, system design, scope definition, and organizing a rapidly diversifying field.

  • Motivation: MTL trains models on multiple tasks simultaneously, using shared representations to learn common ideas across related tasks.These representations can increase data efficiency and potentially speed learning for related or downstream tasks.
  • Motivation: Shared knowledge can reduce the need for each task to be learned entirely from scratch.The survey relates this benefit to how previously learned general skills can support later, more complex tasks.
  • Challenges: Conflicting task needs can cause negative transfer, where improving one task harms another.Reducing this destructive interference is a key goal of MTL methods.
  • Survey scope: The survey broadens hard and soft parameter-sharing categories into multi-task architectures and multi-task optimization.It argues that the traditional dichotomy is not broad enough for the field’s growing diversity.
  • Survey scope: The review defines MTL as simultaneous learning of a fixed, equally treated set of tasks, excluding a single main task with auxiliary tasks.Settings whose task set changes over time are also outside this convention, though related models may be discussed when applicable.
  • Survey structure: The survey organizes its discussion around architectures, optimization strategies, task relationship learning, and multi-task benchmarks.The sections primarily present methods in publication order.

2 Multi-Task Architectures

The survey treats multi-task architecture design as the problem of deciding what to share and how to combine shared and task-specific components. It groups architectures by domain, modality, whether structure is learned, and whether computation is conditional.

  • Architecture design: MTL architecture design varies in how much of the model is shared and how shared and task-specific modules are parameterized and combined.Domain-specific designs add choices such as partitioning convolutional filters into shared and task-specific groups.
  • Architecture taxonomy: The survey partitions MTL architectures into domain-specific, multi-modal, learned, and conditional groups.Its single-domain coverage includes computer vision, natural language processing, and reinforcement learning.

2.1 Architectures for Computer Vision

Computer-vision MTL architectures balance shared information flow with task-specific processing through shared trunks, cross-talk, prediction distillation, and fine-grained routing. These designs differ in where and how task information is exchanged, from shared features to predictions and masked subnetworks.

  • Shared Trunk: Multi-task network cascades pass each task’s output forward as input to the next task.This creates sequential task-to-task information flow rather than only sharing a common feature extractor.
  • Shared Trunk: Shared-trunk architectures use a global convolutional feature extractor followed by task-specific output branches.TCDCN applies this template while jointly learning facial landmark detection, head pose estimation, and facial attribute inference.
  • Shared Trunk: Task-specific modules can be inserted into shared architectures so features combine shared extractor parameters with task-specific parameters before output branches.One example uses task-specific channel-wise linear projections of feature maps.
  • Shared Trunk: Multi-gate Mixture-of-Experts uses multiple shared trunks whose outputs are linearly mixed for each task by shared or task-specific gating functions.Task-specific gates allow different output heads to receive different mixtures of shared-trunk outputs.
  • Cross-Talk: Cross-talk architectures maintain separate task networks while exchanging information between parallel layers.Cross-Stitch units use learned task-specific linear combinations of outputs from the previous layer in every task network.
  • Cross-Talk: NDDR-CNN fuses features from separate task networks by concatenation followed by a 1x1 convolution instead of Cross-Stitch linear combinations.This changes the feature-fusion operation while retaining cross-network information flow.
  • Prediction Distillation: Prediction-distillation architectures recombine preliminary predictions from multiple tasks to refine final predictions.PAD-Net combines predictions for depth, scene parsing, surface normals, and contours to produce refined depth and scene-parsing outputs.
  • Prediction Distillation: Multi-scale prediction distillation addresses conflicting local and global task interactions, and the described model improves more over single-task counterparts than high-performing baselines.The adjacent-cars example shows why a local depth discontinuity can conflict with global semantic consistency.

2.2 Architectures for Natural Language Processing

NLP MTL architectures evolve from shared feed-forward representations to recurrent, hierarchical, adversarial, and BERT-based designs. These methods vary how information and supervision are shared across tasks while addressing task interference and output differences.

  • Traditional Feed-Forward: Traditional NLP architectures share a global word representation or feature extractor before task-specific output branches.Inputs may be represented as lookup-table features or hashed letter 3-grams, followed by shared transformations and task-specific heads.
  • Recurrence: Sequence-to-sequence MTL uses one-to-many, many-to-one, or many-to-many sharing schemes to match encoder and decoder sharing with task output formats.One-to-many shares encoders for differently formatted outputs, while many-to-one shares decoders when outputs use the same format.
  • Recurrence: Recurrent architectures also partition embeddings, layers, and memory into task-specific and shared components to control information flow between tasks.Uniform-Layer, Coupled-Layer, and Shared-Layer architectures provide increasingly fine-grained sharing, while shared memory can be jointly read and written by tasks.
  • Cascaded Information: Cascaded architectures supervise lower-level tasks at earlier layers so their learned features can inform higher-level tasks.Examples organize POS tagging before syntactic chunking and CCG supertagging, or use five supervised tasks in a hierarchy.
  • Adversarial Feature Separation: Adversarial feature separation combines shared and task-specific LSTM layers with a task discriminator and orthogonality penalty.The discriminator encourages shared features to avoid task-specific information, while the penalty encourages shared and task-specific features to encode separate information.
  • BERT for MTL: BERT-based MT-DNN achieved state-of-the-art performance on eight of nine GLUE tasks at publication.The architecture adds shared BERT contextual embedding layers to a multi-task NLP model.

2.3 Architectures for Reinforcement Learning

Reinforcement-learning MTL improves task performance through joint training and shared or modular policies. Architectures range from shared trunks to robot-task modules and subpolicy compositions that promote reusable computation and transfer.

  • Joint Task Training: Joint training can improve reinforcement-learning task performance with or without parameter sharing.A shared-trunk network jointly learns robotic grasping, pushing, and poking from pixels.
  • Joint Task Training: The shared-trunk robotic architecture uses three shared convolutional layers followed by task-specific output branches.Its supervised loss averages cross-entropy and squared Euclidean losses across tasks.
  • Modular Policies: Modular policies compose smaller subnetworks differently across tasks, providing shared neural building blocks rather than a single shared trunk.This design aims to learn computation that is general enough to serve multiple tasks.
  • Modular Policies: Heess et al. divide locomotion policies into a low-level spinal network for actions and a high-level cortical network that modulates its inputs.The spinal network receives task-independent proprioceptive information, while the cortical network receives all observations; their input frequencies differ.
  • Modular Policies: Devin et al. combine separate task and robot modules so each task-robot policy can share information across both domains.Modules can be paired in combinations unseen during training, enabling zero-shot generalization, while each module receives domain-specific observations.
  • Modular Policies: Modularity can restrict information to designated modules, forcing missing-information modules to learn representations that generalize across tasks.Policy sketches provide another modular template by composing task policies from neural subpolicies whose parameters are shared between tasks.
  • Joint Task Training: Auxiliary tasks provide additional supervision in a shared-trunk CNN-LSTM actor-critic and can also reduce sparsity of rewards in the original task.The auxiliary-task branches are learned alongside the main reinforcement-learning task.

2.4 Multi-Modal Architectures

Multi-modal MTL shares representations across tasks and modalities, extending single-modality sharing to settings where inputs span visual, linguistic, audio, and text domains. The survey covers fixed-modality architectures and universal models that handle varying input domains, including a 12-task model with broad gains.

  • Multi-modal motivation: Multi-task multi-modal learning shares representations across both tasks and modalities, adding another layer of abstraction beyond single-modality MTL.The survey associates this setup with reduced overfitting and increased data efficiency.
  • Fixed-modality architectures: Dense co-attention architectures organize tasks hierarchically and supervise lower-level tasks at earlier network layers.The approach integrates visual and linguistic information for shared vision-language tasks.
  • Fixed-modality architectures: Other fixed-modality models combine visual, audio, and text inputs using bi-directional GRUs and pairwise attention to learn shared representations.The cited architecture targets emotion and sentiment classification in videos of human speakers.
  • Universal models: Universal multi-modal multi-task models are designed to handle multiple tasks with varying input domains through shared encoder, mixer, and decoder components.The surveyed examples include models from Kaiser et al. and Pramanik et al.
  • Universal models: 12-in-1 handles 12 datasets simultaneously, outperforming corresponding single-task models on 11 tasks and reaching SOTA after multi-task pre-training on 7.Its architecture is based on ViLBERT and combines dynamic task scheduling, curriculum learning, and hyperparameter heuristics.

2.5 Learned Architectures

Learned architectures adapt parameter sharing rather than fixing it by hand. The survey groups these methods into architecture search, branched sharing, modular sharing, and fine-grained sharing, spanning task-specific networks, evolving pathways, learned module combinations, and weight-level masks.

  • Overview: Learned architectures provide adaptive parameter sharing, which can be more precise than hand-designed sharing schemes.The survey organizes them into four broad groups whose boundaries can overlap.
  • Branched sharing: Branched sharing starts with all tasks sharing layers, then progressively separates less-related tasks into clusters as training proceeds.The learned branching process decouples parameters layer by layer, beginning near the output heads.
  • Architecture search: Architecture-search methods generate task-specific architectures, evolve reusable modules and their orderings, or search feature-fusion operations between task backbones.MNMS uses a controller, Liang et al. use evolutionary strategies, and MTL-NAS uses gradient-based search.
  • Modular sharing: Modular sharing assigns tasks different pathways or combinations within a shared network, including genetically evolved subnetworks and task-specific module compositions.PathNet evolves task pathways through tournament selection, while modular methods vary how modules are combined.
  • Modular sharing: Soft Layer Ordering combines module outputs continuously and supports gradient-based joint optimization, whereas Modular Meta-Learning learns discrete computation graphs with simulated annealing.The contrasting parameterizations lead to different optimization strategies and computational costs.
  • Fine-grained sharing: AdaShare learns whether each shared layer is included for each task, while fine-grained methods learn masks over individual weights or convolutional feature channels.These schemes permit task subnetworks with overlapping but more detailed parameter sharing.

2.6 Conditional Architectures

Conditional architectures dynamically select or compose shared network components according to the input, task, or question structure. The survey contrasts discrete routing and parser-driven module assembly with soft, end-to-end alternatives.

  • Overview: Conditional computation selects parts of a neural network based on the input, making multi-task architectures dynamic across both inputs and tasks.Shared components are intended to generalize across varied inputs and tasks.
  • Module composition: Neural Module Networks use a question’s semantic structure to instantiate modules corresponding to its compositional elements for visual question answering.A semantic parser determines the question structure and module relationships.
  • Routing: Routing Networks use a router to iteratively choose modules for each input, learning the module composition alongside module weights.The router dynamically instantiates a network for each input.
  • Routing: Some conditional architectures route locally within layers and use variational methods to treat module choice as a latent variable rather than relying on reinforcement learning.The cited design is intended to reduce module collapse.
  • Soft conditional architectures: Soft Modularization relaxes discrete routing by using learned linear combinations between modules, allowing end-to-end training of the entire network.Its router outputs layer-specific combination weights conditioned on an observation and task index.

3 Optimization for Multi-Task Learning

The survey frames MTL optimization as a broader alternative to soft parameter sharing and groups methods into six overlapping strategies for balancing task objectives and addressing negative transfer.

  • Optimization strategies: MTL optimization is partitioned into loss weighting, regularization, gradient modulation, task scheduling, multi-objective optimization, and knowledge distillation.The survey notes that methods may fit more than one group, but the partition supports conceptualizing research directions.
  • Loss weighting: Loss weighting combines task-specific losses into an aggregated objective whose weights determine each task’s contribution during training.The survey identifies balancing individual task losses as a common approach to multi-task optimization.
  • Loss weighting: Uncertainty weighting assigns less uncertain tasks more influence while logarithmic regularization prevents indefinitely increasing uncertainty estimates.The method was reported to outperform identical models using the best-performing constant loss weights.
  • Loss weighting: Learning-speed methods generally increase a task’s loss weight when that task is learning more slowly than the others.Dynamic Weight Averaging uses successive loss ratios, while GradNorm adjusts weights through an auxiliary gradient-based objective and adds computation.
  • Weighting by performance: Performance-based weighting emphasizes tasks with poorer performance, including methods that use task metrics or example-level focal weighting.Dynamic Task Prioritization assigns weights at both task and example levels, using metrics such as classification accuracy for tasks.
  • Weighting by reward magnitude: PopArt normalizes task contributions so their relative importance is agnostic to reward scale and significantly improves IMPALA agents on DeepMind Lab tasks.The reported improvement concerns multi-task deep reinforcement learning agents trained with IMPALA.

3.2 Regularization

Regularization methods encourage useful relationships among task-specific parameters, representations, or routing decisions without requiring identical parameterization across tasks.

  • Soft parameter sharing: Soft parameter sharing adds the L2 distance between task-model parameters to the training objective to encourage similar parameters.Unlike hard sharing, task models retain separate weights.
  • Soft parameter sharing: Separate language-specific networks can softly share only selected layers that transform embedded part-of-speech tags and arc labels.This illustrates fine-grained soft sharing rather than sharing the entire network.
  • Soft parameter sharing: Replacing parameter-distance penalties with tensor trace norms provides a convex relaxation of the rank of stacked task-parameter vectors.The trace norm is described as the sum of singular values of the resulting matrix.
  • Prior-based regularization: Multilinear Relationship Networks impose a tensor normal prior on parameters in task-specific layers.The tensor normal distribution extends the multivariate normal with a covariance decomposition based on Kronecker products.
  • Auxiliary-task regularization: Deep-AMTFL adds an autoencoder objective that reconstructs penultimate-layer features from network outputs across tasks.Each task’s predictions are used to construct features for the other tasks.
  • Adaptive sharing: AdaShare regularizes sharing parameters that encode which shared blocks each task uses, encouraging sparsity and similarity across tasks.Its auxiliary terms include sparsity and sharing objectives, with sharing importance decreasing in deeper layers.
  • Adaptive sharing: Maximum Roaming randomly varies task-parameter partitioning during training while requiring each parameter to serve a maximal number of tasks.The survey presents it as an MTL-specific variant of Dropout.

3.3 Task Scheduling

Task scheduling determines which tasks receive updates at each training step and is closely tied to loss weighting because both control task influence during optimization.

  • Scheduling basics: Task scheduling selects the task or tasks trained at each step, commonly through all-task updates or random subset sampling.The survey notes that simple schedulers still allow variation in how tasks are sampled.
  • Scheduling basics: Scheduling and loss weighting can represent similar training decisions because changing sampling frequency changes the relative exposure of tasks to updates.The survey explicitly describes the two problems as strongly tied.
  • Adaptive scheduling: The meta task-decider is trained to sample tasks with a signal that more frequently selects tasks with worse relative performance.This figure summarizes the scheduling principle visually.
  • Adaptive scheduling: Sharma et al. assign scheduling probabilities from each task’s distance to a target performance, favoring tasks farther from that target.The method is based on active learning and has three variants for multi-task reinforcement learning.
  • Adaptive scheduling: Agents using Sharma et al.’s three scheduling variants vastly outperform identical agents with uniform sampling across Atari collections containing 6 to 21 games.The reported comparison spans collections of different sizes.
  • Adaptive scheduling: Jean et al. normalize task scores into sampling probabilities and introduce α and ϵ to control over- and under-sampling and numerical stability.The survey describes this as closely related to implicit task scheduling through loss weighting.

3.4 Gradient Modulation

Gradient modulation methods address negative transfer by modifying conflicting task gradients rather than allowing opposing updates to interfere. The survey describes gradient replacement approaches and notes empirical success alongside unresolved theoretical questions for adversarial gradient alignment.

  • Negative transfer arises when opposing task gradients make improving one task reduce performance on another.
  • Adversarial Gradient Modulation: GREAT encourages gradient distributions from different sources to become statistically indistinguishable through an adversarial loss term.
  • Adversarial Gradient Modulation: GREAT improves multi-task performance and outperforms baselines such as GradNorm, although its premise lacks rigorous justification.
  • Gradient modulation replaces conflicting task gradients with modified vectors that avoid conflicts during optimization.
  • Gradient Replacement: A-GEM relaxes GEM’s per-task non-conflict constraints to an average-gradient constraint, greatly reducing computation time while maintaining performance.
  • Gradient Replacement: PCGrad adapts the same update principle to simultaneous multi-task learning and completes 70% of tasks in Meta-World’s MT50 benchmark with Soft Actor-Critic.
  • The survey concludes that reducing conflicting gradients is an effective way to decrease negative transfer.

3.5 Knowledge Distillation

Knowledge distillation trains a multi-task student from task-specific teachers, transferring information into one model. In reinforcement learning, distilled students can match or exceed teachers, while transfer can occasionally slow learning on new tasks.

  • Knowledge distillation commonly transfers knowledge from multiple single-task teacher networks into one multi-task student network.
  • Policy Distillation and Actor-Mimic train converged task-specific policies before using supervised learning to imitate them.
  • Reinforcement Learning: Distral architectures combine KL-regularization of single-task policies with a multi-task policy, optionally adding a shared column to each task policy.
  • In Atari, distilled student networks match or outperform single-task teachers despite being trained to mimic behavior rather than maximize reward directly.
  • Student networks receive a richer training signal from dense teacher outputs than from one-hot ground-truth labels.
  • Most distillation methods use asymmetric information flow from teacher to student, raising whether teachers should also receive information from the student.
  • Natural Language Processing: BAM applies teacher annealing alongside the standard multi-task knowledge-distillation template for NLP.

3.6 Multi-Objective Optimization

Multi-objective optimization treats MTL as simultaneous optimization of several losses rather than collapsing them into a weighted average. The survey presents Pareto optimality and gradient-based methods as an alternative to information-losing, weight-sensitive scalarization.

  • MTL must optimize multiple potentially conflicting loss functions, unlike standard learning with a single loss.
  • Weighted averaging can lose information about the tuple of task losses and requires weights that are prone to error.
  • Multi-objective optimization jointly optimizes several objectives without requiring a single weighted loss function.
  • Pareto optimal solutions replace a universal global minimum when no solution simultaneously minimizes every objective.
  • Sener and Koltun extended MGDA with a gradient-based method that scales to deep learning’s high-dimensional problems by minimizing an upper bound.

4 Task Relationship Learning

Task relationship learning explicitly models which tasks should be grouped, transferred between, or embedded together. The survey covers empirical grouping, online gradient-based clustering, transfer-affinity estimation, and task embeddings, while emphasizing that joint-learning benefits depend on more than task relationships alone.

  • Task relationship learning represents task relationships to group similar tasks and leverage those relationships during learning.
  • Its directions include grouping tasks, learning transfer relationships, and learning task embeddings.
  • Grouping Tasks: Task grouping separates tasks exhibiting negative transfer, but identifying beneficial groups often requires costly trial-and-error training.
  • Grouping Tasks: Across 1440 NLP task combinations, auxiliary tasks with high-entropy and low-kurtosis label distributions improved main-task performance most.
  • Grouping Tasks: A vision study found joint training consistently improved four self-supervised tasks over single-task baselines, though the explanation remained uncertain.
  • Grouping Tasks: Selective Sharing clusters tasks online using gradient-vector similarity, avoiding trial-and-error grouping.
  • Grouping Tasks: Taskonomy grouping produced mixed baseline results, highly setting-dependent gains, and no correlation between multi-task and transfer affinity.
  • Transfer Relationships: Transfer relationships can be represented as directed graphs in which edges identify ideal source tasks for target tasks.

5 Multi-Task Benchmarks

The survey reviews multi-task benchmarks across computer vision, NLP, reinforcement learning, and multimodal problems. Dedicated MTL benchmarks are uncommon, so evaluation often uses datasets designed for broader task settings.

  • Benchmark scope: Only a few benchmarks are specifically designed for multi-task learning, notably Taskonomy and Meta-World.Most MTL methods are evaluated in multi-task settings using datasets that were not created specifically for MTL.
  • Computer vision: Computer vision benchmarks include NYU-v2, MS-COCO, CityScapes, and Taskonomy, spanning segmentation, classification, depth, and other visual tasks.Taskonomy is specifically designed for MTL, with 4 million indoor-scene images annotated for 26 visual tasks.
  • Natural language processing: NLP benchmarks range from annotated corpora such as Penn Treebank and OntoNotes to multi-task collections including GLUE and decaNLP.GLUE contains nine NLP tasks, while decaNLP poses ten tasks as question answering.
  • Reinforcement learning: Reinforcement-learning benchmarks include Atari games in ALE, customizable 3D environments in DeepMind Lab, and robotic manipulation tasks in Meta-World.Meta-World provides MT10 and MT50 for simultaneous learning of 10 and 50 tasks, respectively.
  • Multimodal learning: Multimodal benchmarks cover image captioning, visual question answering, visual grounding, and dialogue-based visual reasoning.Examples include Flickr30K Captions, MS-COCO Captions, Visual Genome, Flickr30K Entities, GuessWhat?!, VQA 2.0, and GQA.

6 Conclusion

The survey organizes deep multi-task learning around architecture design, optimization, and task relationship learning. It also identifies limited theoretical understanding as an important remaining gap and connects MTL's development to more human-like learning.

  • Architecture design: Architecture research includes shared feature extractors with task-specific decoders, varied parameter sharing, module recombination, learned sharing, and fine-grained sharing.These techniques address how neural-network components should be shared across tasks.
  • Optimization techniques: Optimization research includes per-task loss weighting, norm-based regularization, gradient modulation or replacement, and multi-objective optimization.The survey highlights uncertainty- or learning-speed-based weighting and methods intended to address conflicting task gradients.
  • Task relationship learning: Task relationship learning uses empirical learning-dynamics studies, representation comparisons, and task embeddings to characterize task similarity or compatibility.These methods examine which tasks show positive learning dynamics when trained simultaneously.
  • Open direction: Deep neural-network MTL still lacks substantial theoretical understanding compared with the progress in methods and applications.The survey identifies theory as an important direction for developing deeper understanding of the field.
  • Broader significance: The authors frame MTL as part of a broader effort to learn general concepts that transfer across tasks and support more human-like artificial intelligence.The conclusion links this goal with meta-learning, transfer learning, and continuous or lifelong learning.
Loading 2009.09796v1…