Source-linked AI summary
Learning Task Grouping and Overlap in Multi-task Learning
Abhishek Kumar, Hal Daume
TL;DR
Multi-task learning needs to share information among related tasks without allowing unrelated tasks to cause negative transfer. The paper learns sparse combinations of latent basis tasks so task groups can occupy low-dimensional subspaces while overlapping across groups, and reports considerable gains over competing subspace-regularized approaches on synthetic and real datasets.
Problem
Multi-task learning must selectively share information because assuming universal relatedness or disjoint groups can mishandle unrelated, partially related, or negatively correlated tasks.
Method
The model represents each task as a sparse linear combination of latent basis tasks, using shared sparsity patterns to define groups and partial overlap.
Results
The approach obtained considerable gains over competing subspace-regularized methods on two synthetic and four real datasets.
Takeaways & Limitations
Allowing different task groups to share latent bases supports learning when tasks are not similar enough for one group but still share exploitable information.
Takeaways & Limitations
The method’s latent-basis count controls the inductive bias: too few bases may shrink the hypothesis space excessively, while too many reduce enforced sharing.
Abstract
from arXiv · showhide
In the paradigm of multi-task learning, mul- tiple related prediction tasks are learned jointly, sharing information across the tasks. We propose a framework for multi-task learn- ing that enables one to selectively share the information across the tasks. We assume that each task parameter vector is a linear combi- nation of a finite number of underlying basis tasks. The coefficients of the linear combina- tion are sparse in nature and the overlap in the sparsity patterns of two tasks controls the amount of sharing across these. Our model is based on on the assumption that task pa- rameters within a group lie in a low dimen- sional subspace but allows the tasks in differ- ent groups to overlap with each other in one or more bases. Experimental results on four datasets show that our approach outperforms competing methods.
1. Introduction
The paper addresses negative transfer by learning task groups with selectively shared information rather than assuming all tasks are related or groups are disjoint. Its model represents tasks using sparse combinations of latent bases, allowing partial overlap between groups.
- Selective sharing is needed because unrelated tasks can negatively influence one another and worsen both tasks’ performance.
- Low-rank multi-task regularization can assume all tasks are related, while distance-based clustering may separate negatively correlated but related parameters.Parameters w and −w span a one-dimensional subspace despite being far apart in ℓ2 distance.
- Disjoint grouping methods restrict sharing to tasks within the same group, whereas this paper allows groups to overlap.
- Each observed task is modeled as a sparse linear combination of latent basis tasks, with shared nonzero bases determining the amount of information sharing.Tasks within a group lie in a low-dimensional subspace, while tasks from different groups may share one or more bases.
2. Related Work
Prior multi-task learning methods use assumptions ranging from universal relatedness and clustering to low-dimensional subspaces and learned covariance structures. Grouped subspace methods generally impose disjoint groups, unlike the proposed overlapping structure.
- Many multi-task learning methods assume all tasks are related, an assumption that can degrade performance when violated.
- Clustering and probabilistic-prior methods group tasks by similarity or shared distributions, while other models isolate outliers or learn full task covariance matrices.
- Low-dimensional-subspace methods capture shared predictive structure by assuming some features are inactive across all tasks.
- Existing grouped subspace approaches assign tasks to disjoint groups and optimize group-specific trace-norm objectives.The cited methods use stochastic gradient descent or mixed integer programming for their non-convex objectives.
- The proposed approach differs by allowing the low-dimensional subspace shared by one group to overlap with another group.
3. Learning Task Grouping and Overlap
GO-MTL models task parameters as sparse combinations of latent basis tasks, allowing within-group low-dimensional structure and overlap across groups. It learns the basis and task-specific coefficients through alternating optimization, with sparsity controlling sharing.
- Model: GO-MTL represents each task as a sparse linear combination of k latent basis tasks, with W = LS.L contains latent task vectors, while S contains task-specific combination weights.
- Model: Shared sparsity patterns identify task groups, while partial basis overlap represents graded relatedness and permits outlier tasks.Tasks with orthogonal sparsity patterns are treated as belonging to different groups; tasks sharing no bases are outliers.
- Objective: The objective combines empirical loss, an ℓ1 penalty on S, and Frobenius regularization on L; µ controls sparsity and the latter helps avoid overfitting.The Frobenius penalty regularizes predictor weights toward low ℓ2 norm.
- Optimization: Because the objective is convex in either L or S separately but not jointly, GO-MTL alternates updates and converges to a local minimum.For fixed L, coefficient problems decompose by task; for fixed S, the problem is convex in L.
- Optimization: For regression, the fixed-S subproblem has a closed-form solution under squared loss; classification uses logistic loss with Newton-Raphson or gradient updates.Newton-Raphson can require fewer iterations, whereas gradient updates can be faster for large problems.
- Algorithm: The algorithm initializes individual predictors, uses the top-k left singular vectors to initialize L, and alternates coefficient and basis updates until convergence.The initialization directions capture maximum variance of the initial task parameters; k controls the strength of the sharing bias.
4. Experiments
Experiments on two synthetic and four real-world datasets evaluate GO-MTL against single-task, no-group, and disjoint-group baselines. GO-MTL recovers task-sharing structure robustly and generally outperforms competing multi-task methods, especially when groups overlap.
- Experimental setup: The evaluation compares GO-MTL with no-group MTL, disjoint-group MTL, and single-task learning on two synthetic and four real-world datasets.The real-world datasets include two regression and two classification tasks.
- Synthetic data: GO-MTL achieves nearly the same RMSE for k ≥3 on the first synthetic dataset and k ≥4 on the second.The competing disjoint-group method is more sensitive to its number-of-groups parameter.
- Synthetic data: GO-MTL significantly outperforms disjoint-group MTL on both synthetic datasets, with a larger margin when groups overlap.The second synthetic dataset explicitly allows groups to overlap in one basis.
- Synthetic data: GO-MTL recovers grouping and overlap patterns for most synthetic tasks, with support recovery remaining stable for k ≥3 or k ≥4 depending on the dataset.Recovery is less precise for some tasks in the middle group of the overlapping synthetic dataset.
- Real datasets: Across the real-world datasets, all multi-task methods outperform single-task learning, but the School-data improvement is not statistically significant; GO-MTL outperforms both competing MTL methods.Table 1 reports RMSE for regression datasets and multi-class classification errors for MNIST and USPS.
5. Conclusion
The framework learns task grouping while allowing tasks in different groups to overlap through shared latent basis tasks. Across synthetic and real datasets, it achieved considerable gains over competing subspace-regularized multi-task methods.
- The framework assumes each task group lies in a low dimensional subspace while permitting overlap across groups through shared latent basis tasks.
- This non-disjoint structure accommodates tasks that are insufficiently related to share a group but still share exploitable information.
- The model was validated on two synthetic and four real datasets.
- The approach obtained considerable gains over competing subspace-regularized methods that either ignore grouping or prohibit interactions across groups.