PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Multi-Task Learning

Yu Zhang, Qiang Yang

arXiv:1707.08114v3cs.LGcs.AI

TL;DR

MTL addresses how related tasks can share knowledge to improve all tasks, especially when individual tasks have limited labeled data. This survey organizes MTL models into five approaches and reviews combinations, scalable methods, applications, and theory. It concludes that MTL has broad usefulness while identifying outlier and noisy tasks as important unresolved limitations.

  • Problem

    MTL seeks to improve all related tasks by leveraging shared knowledge, while individual tasks may suffer from insufficient labeled data and costly manual labeling.

  • Method

    The survey defines MTL, classifies supervised models into five approaches, and reviews combinations with other paradigms, scalable methods, applications, and theoretical analyses.

  • Results

    The survey synthesizes MTL's algorithmic models, combinations with other learning paradigms, computational approaches, applications, and theoretical analyses.

  • Takeaways & Limitations

    MTL provides a broad framework for sharing knowledge across tasks, with reviewed methods addressing varied learning paradigms, task scales, data dimensionalities, and applications.

  • Takeaways & Limitations

    Outlier or noisy tasks can hamper joint learning, and principled methods and theoretical analyses of these negative effects remain lacking.

Abstract

from arXiv · show

Multi-Task Learning (MTL) is a learning paradigm in machine learning and its aim is to leverage useful information contained in multiple related tasks to help improve the generalization performance of all the tasks. In this paper, we give a survey for MTL from the perspective of algorithmic modeling, applications and theoretical analyses. For algorithmic modeling, we give a definition of MTL and then classify different MTL algorithms into five categories, including feature learning approach, low-rank approach, task clustering approach, task relation learning approach and decomposition approach as well as discussing the characteristics of each approach. In order to improve the performance of learning tasks further, MTL can be combined with other learning paradigms including semi-supervised learning, active learning, unsupervised learning, reinforcement learning, multi-view learning and graphical models. When the number of tasks is large or the data dimensionality is high, we review online, parallel and distributed MTL models as well as dimensionality reduction and feature hashing to reveal their computational and storage advantages. Many real-world applications use MTL to boost their performance and we review representative works in this paper. Finally, we present theoretical analyses and discuss several future directions for MTL.

1 INTRODUCTION

MTL jointly learns related tasks by sharing useful knowledge to improve the generalization performance of all tasks. The survey distinguishes MTL from related paradigms and reviews its motivations, scope, and algorithmic coverage.

  • MTL learns multiple related tasks jointly so knowledge from one task can help improve the generalization performance of all tasks.The motivation is inspired by humans transferring useful skills between related activities.
  • MTL can alleviate task-level data sparsity by aggregating labeled data across tasks, reusing knowledge, and reducing manual-labeling costs.Each task may lack enough labeled data to train an accurate learner independently.
  • Unlike transfer learning, MTL treats all tasks equally, whereas transfer learning prioritizes a target task assisted by source tasks.The two settings differ in task roles and the direction of knowledge flow.
  • Multi-view learning uses multiple feature sets to learn one task, whereas MTL addresses multiple tasks.This distinction is illustrated in Fig. 1(c).
  • The survey covers MTL algorithmic modeling, combinations with other learning paradigms, scalable and high-dimensional settings, applications, and theoretical analyses.Its algorithmic taxonomy includes feature learning, low-rank, task clustering, task relation learning, and decomposition approaches.

2 MTL MODELS

MTL models jointly learn related tasks by sharing knowledge, with design choices concerning when, what, and how to share. The survey organizes feature-based and parameter-based methods into five main approaches and describes representative formulations.

  • MTL definition: MTL jointly learns related tasks so knowledge from some or all tasks improves each task's model.The setting treats tasks jointly rather than prioritizing one target task.
  • MTL design issues: MTL design addresses when to share, what to share, and how to share knowledge among tasks.Choosing when to share can be formulated as model selection, although cross-validation may be computationally heavy and data-hungry.
  • What to share: Knowledge sharing can use features, instances, or parameters, with existing studies mainly focusing on feature-based and parameter-based methods.Instance-based MTL shares weighted training data using estimated density ratios, whereas parameter-based methods share model parameters through mechanisms such as regularization.
  • How to share: The survey groups feature-based and parameter-based MTL into five approaches: feature learning, low-rank, task clustering, task relation learning, and decomposition.Feature learning learns common representations, while decomposition separates task parameters into components penalized by different regularizers.
  • Feature learning approach: Feature learning includes transformation and selection: transformation maps original features to new representations, whereas selection retains a subset of original features.The survey describes multi-layer feedforward networks as transformation models that share hidden representations and use task-specific output units.
  • Representative formulations: MTFL learns a shared feature transformation and row-sparse task parameters, while related formulations impose covariance- or trace-based regularization.Its objective combines empirical loss with an ℓ2,1 penalty and an orthogonality constraint; sparse coding instead constrains task and transformation parameters.

2.2 Low-Rank Approach

The low-rank approach exploits task relatedness by constraining task-parameter matrices to share low-dimensional structure. Trace-norm and related regularizers, shared subspaces, and tensor extensions provide several ways to model this structure.

  • Greater task relatedness implies a lower-rank parameter matrix W, motivating low-rank regularization in MTL.
  • Shared low-rank subspaces decompose task parameters into task-specific components plus a common representation.The shared subspace can be constrained to be orthonormal and may have fewer rows than columns.
  • Convex relaxation can make a low-rank formulation easier to optimize globally than its original non-convex version.The cited formulation is described as having a global optimum that is easier to obtain.
  • Trace-norm regularization promotes low rank in W, while capped trace regularization penalizes only small singular values.With a sufficiently large threshold, capped trace regularization reduces to trace-norm regularization.
  • Tensor trace norms extend low-rank regularization to weights in deep multi-task neural networks.Weights across tasks can be organized as tensors, whose low-rank structure is then regularized.

2.3 Task Clustering Approach

Task clustering methods model related tasks as groups with shared structure, using statistical, regularized, and decomposition-based formulations. The survey highlights methods that jointly learn task parameters and clusters, automatically determine cluster counts, or allow overlapping memberships.

  • Task clustering assumes that tasks form several clusters of similar tasks, extending clustering from data points to tasks.
  • Early task clustering first learned cluster structure and then trained task models separately, which could make the learned clustering suboptimal for those models.Later methods instead learn model parameters and cluster structure together.
  • Bayesian approaches assign task-specific parameters to shared Gaussian or Dirichlet-process clusters.Dirichlet-process clustering uses α to control the probability of forming a new task cluster.
  • Regularized approaches use penalties, covariance constraints, or cluster indicators to model within-cluster similarity and between-cluster differences.Some formulations use cluster indicator matrices whose diagonal entries identify task membership.
  • When r equals 1, the cited multi-cluster method reduces to the MTFL method.
  • Structurally sparse regularization can automatically determine the number of task clusters by encouraging pairs of task-parameter columns to become identical.The resulting cluster structure is discovered by comparing columns in W.
  • Some methods relax single-cluster membership by representing each task as a sparse combination of latent bases.GO-MTL allows a task to belong to more than one cluster.
  • Only selected methods automatically determine cluster count, and among regularized models only the structurally sparse formulation is convex.The cluster-count mechanism differs between Dirichlet-process and structurally sparse approaches.

2.4 Task Relation Learning Approach

Task relation learning quantifies similarities, covariances, or other dependencies among tasks and can either use prior relations or learn them from data. The surveyed methods include Bayesian, regularized, local, and asymmetric relation models.

  • Task relations include quantitative notions such as task similarity, correlation, and covariance.
  • Earlier MTL methods often assumed task relations were known a priori and used them to guide parameter sharing.Some methods encouraged task parameters toward a shared average, while others used pairwise similarities in regularizers.
  • Multi-task Gaussian processes learn a positive-definite task covariance matrix from data through the marginal likelihood.The covariance matrix describes similarities between task function values.
  • Bayesian task-relation models can place an inverse-Wishart prior on the task covariance to address possible overfitting from point estimation.
  • Regularized relationship learning uses matrix-variate priors and covariance constraints to model structured dependencies among task parameters.The resulting objective penalizes model complexity and the covariance structure.
  • Relationship-learning models have also been extended to high-order task relations, multi-task boosting, multi-label learning, and multi-output regression.
  • Task relations can be learned locally through weighted voting among nearest neighbors drawn from multiple tasks.The learned similarity matrix controls each task’s contribution to another task’s prediction.
  • Asymmetric relation models represent each task parameter as approximately reconstructed from other tasks using sparse coefficients.The formulation permits directional information sharing from easier tasks to difficult ones.

2.5 Decomposition Approach

The decomposition approach represents the task-parameter matrix as multiple component matrices, each imposing distinct sparsity, low-rank, clustering, or robustness structures. Its parameter-level depth increases modeling capacity and can subsume several simpler MTL approaches.

  • Decomposition models write W as a sum of component matrices and commonly use decomposable regularizers with constraints linking those components.
  • A two-component model can combine row sparsity for shared feature selection with sparsity for task-specific outlier features.This yields a robust version of a feature-selection formulation.
  • Other two-component models combine low-rank structure in W1 with sparse or column-sparse W2 to accommodate outlier tasks.When the sparse component vanishes, one formulation behaves similarly to a low-rank model.
  • Component matrices can impose progressively different sparse and row-sparse structures while allowing the full parameter matrix W to remain non-sparse.This can reveal latent sparse structure among tasks.
  • Constraints and regularizers can directly relate component matrices, including by encouraging corresponding component parameters to become identical.
  • Hierarchical decomposition methods encode task relations through shared ancestor components and can reduce to multi-level task clustering when constraints are removed.
  • Compared with shallow approaches, decomposition is deep in model parameters and can have greater capacity; with one component matrix, it reduces to feature learning, low-rank, or task clustering approaches.

2.6 Comparisons among Different Approaches

The surveyed MTL approaches differ in what structure they learn across tasks and in how they respond to task relatedness, outliers, and model assumptions.

  • Feature learning learns common, generic, and task-invariant features, but unrelated outlier tasks can significantly influence those features and deteriorate performance.
  • Low-rank methods model the parameter matrix through an explicitly learned subspace or a convex or non-convex regularizer.The survey notes that this approach is powerful but appears mainly applicable to linear models, making nonlinear extensions non-trivial.
  • Task clustering identifies groups of similar tasks, but captures positive within-cluster correlations while ignoring negative correlations across clusters.Determining the number of clusters may also require model selection such as cross validation, adding computational cost.
  • Task relation learning jointly estimates task parameters and pairwise relations, improving interpretability by exposing relationships between tasks.

2.7 Benchmark Datasets and Performance Comparison

The survey reviews benchmark datasets spanning regression and classification tasks and compares representative MTL models with single-task learning baselines. MTL generally outperforms STL, while dataset structure and image-domain representation affect which approaches perform best.

  • The benchmark suite includes school, robot-arm, computer-purchase, Parkinson, sentiment, MHC-I, landmine, Office-Caltech, Office-Home, and ImageCLEF tasks.
  • MTL models perform better than single-task learning counterparts in most benchmark cases, supporting MTL’s effectiveness.Table 1 compares representative methods from five MTL approaches with STL using metrics including nMSE, RMSE, and AUC.
  • The datasets cover regression and classification settings, including binary classification for Sentiment, MHC-I, and Landmine and multi-class classification for the image datasets.
  • Task clustering, task relation learning, and decomposition perform better on datasets with identifiable plain or hierarchical cluster structure among tasks.
  • Deep MTL models outperform shallow models on Office-Caltech, Office-Home, and ImageCLEF, whereas shallow models perform well on datasets from diverse areas.

2.8 Another Taxonomy for Regularized MTL Methods

The taxonomy organizes regularized MTL methods by whether they learn feature covariance or task relations, while the survey also reviews extensions and optimization techniques for broader settings.

  • Regularized MTL methods are grouped into feature-covariance learning and task-relation learning, representing feature-based and parameter-based formulations.
  • Feature-covariance formulations use a regularizer or constraint on Θ, which models covariance between features shared across tasks.
  • Task-relation formulations use a regularizer or constraint on Σ, whose column covariance represents relationships among task parameters.
  • The taxonomy does not cover every regularized MTL method, although it provides insight for understanding existing methods and designing new models.
  • The survey extends MTL discussions to heterogeneous features, multi-class classification, tensor-valued data, and gradient, block-coordinate, and proximal optimization methods.MTDA addresses heterogeneous-feature face recognition; tensor methods use tensor trace norms, while proximal methods handle objectives combining smooth and nonsmooth terms.

3 MTL WITH OTHER LEARNING PARADIGMS

The survey examines combinations of MTL with unsupervised, semi-supervised, active, reinforcement, multi-view, and graphical-model learning. These combinations adapt MTL principles to exploit additional information or improve related learning paradigms.

  • Combining MTL with semi-supervised and active learning exploits unlabeled data through geometry or selective labeling, producing semi-supervised, active, and combined paradigms.The motivation is that labeled data are expensive while unlabeled data are abundant.
  • MTL is applied to clustering through multi-task Bregman clustering, which uses earth mover distance to reduce distances between task cluster centers.
  • Multi-task reinforcement learning adapts MTL ideas using Bayesian mixtures, regionalized policies, Gaussian processes, shared task bases, attention, and shared experience.
  • Multi-task multiview learning links views within tasks through consistency and links tasks through shared-view prediction agreement or relatedness.Multi-task multiview clustering additionally considers within-view-task clustering, view relation learning, and low-rank structure learning.
  • MTL supports graphical-model structure learning by encouraging similar Bayesian-network structures or joint sparsity among Gaussian graphical-model precision matrices.
  • Across these combinations, most reviewed studies retain the principles of the MTL approaches introduced earlier while adapting them to their specific settings.

4 HANDLING BIG DATA

The survey addresses two forms of scale in MTL: many tasks and high-dimensional data. It reviews online, parallel, and distributed learning, alongside feature-selection and hashing methods for computational efficiency.

  • Many tasks: A large task count increases total training data, motivating online, parallel, and distributed MTL models.These methods target faster learning when many tasks contribute data.
  • High-dimensional data: For high-dimensional data, MTL uses feature selection, dimensionality reduction, and feature hashing to reduce dimension while retaining useful information.Multiple hashing functions can accelerate joint learning of multiple tasks.
  • Parallel learning: Parallel MTL can accelerate learning on multi-CPU or multi-GPU devices, but task coupling differs across formulations.Feature-based formulations can decouple tasks given a feature covariance matrix, whereas parameter-based formulations remain coupled given a task covariance matrix.
  • Online learning: Online MTL handles many tasks by modeling task relatedness through global loss functions or hard constraints on task actions.The reviewed methods include absolute-norm global losses and constrained action tuples.
  • Distributed learning: Distributed MTL is important when training data reside on different devices, including communication-efficient and privacy-preserving settings.Reviewed examples jointly learn sparse features or extend MTRL to distributed optimization.

5 APPLICATIONS

MTL has been applied across computer vision, bioinformatics, health informatics, speech, NLP, and web applications. For new problems, the survey recommends assessing task relatedness and adapting models from similar application settings.

  • Application areas: MTL applications span computer vision, bioinformatics, health informatics, speech, NLP, web, and other areas.The survey categorizes application problems according to the MTL approaches they use.
  • Application areas: Application-dependent MTL models have been proposed for diverse problems across these areas.Table 2 organizes representative works by application area and MTL approach.
  • Model selection: For a new MTL problem, practitioners should judge task relatedness through low-level features or high-level concepts.The survey suggests using similar problems in Table 2 as references when tasks are related.
  • Model selection: When no similar application is available, popular MTL approaches in the relevant area can be tried.This is presented as an alternative to adapting a model from a related problem.

6 THEORETICAL ANALYSES

The survey reviews theoretical analyses of MTL, especially generalization bounds for different models and tools. It reports convergence-rate comparisons and also covers representer theorems and feature recovery.

  • Generalization bounds: Generalization bounds characterize MTL learnability and sample complexity using training loss, model complexity, and confidence.The survey compares bounds across analyzed models, tools, and convergence rates.
  • Generalization bounds: The best reported convergence rate is O(1/√(mn0)), where m is the number of tasks and n0 is the average data points per task.The comparison covers different MTL models and analysis tools.
  • Analysis tools: Local Rademacher complexity can derive tighter generalization bounds than other analyzed tools.The survey reports this advantage for several regularization settings, including graph regularization.
  • Other theory: Several studies analyze conditions for representer theorems and recovery of true features in multitask feature-selection models.These works address theoretical properties beyond generalization bounds.

7 CONCLUSIONS AND DISCUSSIONS

The survey synthesizes MTL methods, applications, computational strategies, and theory, then identifies open problems. Key concerns include outlier-task interference, robustness of deep MTL, and limited work beyond supervised learning.

  • Contributions: The survey covers MTL definitions, five supervised-learning approaches, combined paradigms, scalable methods, applications, and theoretical analyses.It presents these topics as the paper’s overall survey scope.
  • Future directions: Outlier tasks can hamper all tasks in joint learning, but principled methods and theoretical analyses of these effects remain lacking.The survey identifies this as an important issue for safer MTL use.
  • Future directions: Deep MTL models that share hidden layers are powerful when tasks are related but vulnerable to noisy and outlier tasks.The survey calls for flexible and robust deep multi-task models.
  • Future directions: Existing MTL studies mainly focus on supervised tasks, with relatively few studies addressing unsupervised, semi-supervised, active, multi-view, and reinforcement learning.The survey calls for appropriate adaptations and extensions to these settings.
Loading 1707.08114v3…