Source-linked AI summary
A Convex Formulation for Learning Task Relationships in Multi-Task Learning
Yu Zhang, Dit-Yan Yeung
TL;DR
The paper addresses how to characterize relationships among tasks while using related tasks to improve generalization. It proposes MTRL, a convex regularization framework that learns task parameters and relationships jointly, and reports experiments on toy and benchmark data demonstrating effectiveness. The approach also covers symmetric and asymmetric multi-task learning settings.
Problem
Multi-task learning needs accurate task-relationship characterization, including positive, negative, and unrelated tasks, rather than relying only on fixed assumptions.
Method
MTRL represents task relationships with a task covariance matrix and jointly learns task parameters and relationships using a convex regularization objective and alternating optimization.
Results
Experiments on a toy problem and several benchmark data sets demonstrate MTRL’s effectiveness.
Takeaways & Limitations
MTRL provides one regularization framework for modeling positive and negative task correlation, identifying outlier tasks, and handling symmetric and asymmetric settings.
Takeaways & Limitations
Using a matrix-variate t distribution produces a non-convex optimization problem whose alternating solution is only a local optimum.
Abstract
from arXiv · showhide
Multi-task learning is a learning paradigm which seeks to improve the generalization performance of a learning task with the help of some other related tasks. In this paper, we propose a regularization formulation for learning the relationships between tasks in multi-task learning. This formulation can be viewed as a novel generalization of the regularization framework for single-task learning. Besides modeling positive task correlation, our method, called multi-task relationship learning (MTRL), can also describe negative task correlation and identify outlier tasks based on the same underlying principle. Under this regularization framework, the objective function of MTRL is convex. For efficiency, we use an alternating method to learn the optimal model parameters for each task as well as the relationships between tasks. We study MTRL in the symmetric multi-task learning setting and then generalize it to the asymmetric setting as well. We also study the relationships between MTRL and some existing multi-task learning methods. Experiments conducted on a toy problem as well as several benchmark data sets demonstrate the effectiveness of MTRL.
1 Introduction
Multi-task learning uses related tasks to improve generalization, making accurate task-relationship characterization central. MTRL learns task relationships from data while representing positive, negative, and unrelated tasks within a convex regularization framework.
- Multi-task learning seeks to improve one task’s generalization performance using other related tasks.
- MTRL is studied in symmetric and asymmetric settings, and existing multi-task methods are analyzed as special cases of it.
- Existing approaches may assume shared representations, use given relationships, or omit negative correlation and robustness to outlier tasks.
- Task relationships can represent positive correlation, negative correlation, or task unrelatedness corresponding to outlier tasks.
- Automatically learning task relationships from data is preferred because model assumptions may be incorrect and difficult to verify.
- MTRL models task relationships as a nonparametric task covariance matrix and jointly learns them with task parameters through a convex objective.
2 Multi-Task Relationship Learning
MTRL models task relationships through a task covariance matrix and jointly learns task parameters, biases, and relationships under a convex regularization framework. An alternating method solves the optimization efficiently, with extensions to kernelized and asymmetric settings.
- Probabilistic Framework: MTRL represents task relationships nonparametrically with a task covariance matrix Ω and generalizes single-task regularization to multiple tasks.The column covariance matrix Ω models relationships between task-specific parameter vectors, and the one-task case reduces to regularized least-squares models.
- Probabilistic Framework: The matrix-variate normal prior separately penalizes the complexity of each column of W while modeling the structure shared across task parameters.Its row covariance I_d models feature relationships, whereas Ω models relationships between tasks.
- Optimization Procedure: The optimization learns W, b, and Ω by minimizing empirical loss and regularization terms subject to positive-semidefinite and trace constraints on Ω.The trace constraint replaces a concave log-determinant term and restricts the complexity of Ω; task imbalance is addressed by modifying the formulation.
- Optimization Procedure: Problem (7) is jointly convex in W, b, and Ω because its objective and constraints are convex, including the matrix-fractional relationship term.The proof rewrites tr(WΩ^-1W^T) as a sum of convex matrix-fractional functions.
- Optimization Procedure: An alternating method fixes Ω while optimizing W and b, then fixes W and b while optimizing Ω, repeating until convergence.This procedure is used because simultaneous optimization is difficult despite joint convexity; the initial Ω corresponds to assuming tasks are unrelated.
- Extensions and Discussions: Kernel extension replaces the linear multitask kernel in the W,b step, while the Ω step changes the calculation of W^T W; alternative matrix-variate t priors make the problem non-convex.The t-prior formulation can still be optimized by alternating updates, but only to find a local optimum.
3 Relationships with Existing Methods
Existing regularization methods correspond to special cases of MTRL under particular choices of the task-relationship matrix or its inverse. Unlike these methods, MTRL learns all three relationship types nonparametrically, while avoiding several prior-knowledge, nonconvexity, and cluster-selection limitations.
- Common formulation: Existing methods share a regularization objective with MTRL but differ in their choices for the task-relationship function.The cited methods use objectives of a common form, with different formulations of f(·).
- Methods with prescribed similarity: Assuming all tasks are similar corresponds to a special case of MTRL and models only positive task correlation.The corresponding Laplacian is defined on a fully connected graph with equal edge weights.
- Methods with prescribed similarity: Methods that assume task clusters, pairwise similarities, or a task network require relationship information to be specified a priori.A task-network formulation and similarity-based formulation both reduce to MTRL for particular choices of Ω^-1.
- Task clustering: The cluster-learning method is more general than fixed-relationship methods but cannot model negative task correlation and relies on a nonconvex formulation before relaxation.Its convex relaxation is not guaranteed to recover the optimum of the original problem, and the number of clusters must also be determined.
- MTRL comparison: MTRL learns positive correlation, negative correlation, and task unrelatedness nonparametrically, helping identify useful tasks and tasks that should not be exploited.This distinguishes MTRL from the cited existing methods in the comparison.
4 Experiments
Experiments evaluate MTRL on a toy regression problem and benchmark tasks, examining learned relationships, prediction performance, and optimization convergence. The results show that MTRL recovers expected toy-task correlations, identifies varied SARCOS relationships, and learns sentiment-task patterns across training sizes.
- 4.1 Toy Problem: MTRL recovered the toy problem’s expected relationships, estimating strong negative correlation for the first two tasks and near-zero correlation for the other pairs.The learned regression functions closely matched the three generating functions.
- 4.2 Robot Inverse Dynamics: The SARCOS experiment evaluates seven robot inverse-dynamics tasks using nMSE, with Table 1 comparing different methods across tasks.The data use 600 training and 1400 test points per task.
- 4.2 Robot Inverse Dynamics: The learned SARCOS task correlations include positive, negative, and uncorrelated pairs, such as tasks three–six, two–three, and one–seven, respectively.These examples demonstrate that the learned matrix represents multiple relationship types.
- 4.2 Robot Inverse Dynamics: The SARCOS objective decreases rapidly before leveling off, and the alternating algorithm converges in no more than 15 iterations.Figure 2 tracks the objective function value during optimization.
- 4.3 Multi-Domain Sentiment Application: In multi-domain sentiment classification, MTRL is evaluated on books, DVDs, electronics, and kitchen-appliance reviews at 10%, 30%, and 50% training sizes.Table 3 reports classification errors and mean task-correlation matrices for each training fraction.
- 4.3 Multi-Domain Sentiment Application: Across sentiment training sizes, the electronics–kitchen correlation is consistently largest, while books–DVDs exceeds books’ correlations with electronics and kitchen.These recurring patterns are reported in the mean task-correlation matrices.
5 Conclusion
The paper presents a regularization formulation that learns global task relationships through a convex optimization problem in both symmetric and asymmetric settings. Future work will incorporate additional data sources such as unlabeled data.
- MTRL learns global task relationships through a regularization formulation that is directly convex.
- The proposed method is studied in both symmetric and asymmetric multi-task learning settings.
- Future work will incorporate additional data sources, such as unlabeled data, into the regularization formulation.
Appendix
The appendix reformulates the relevant optimization problem as a second-order cone programming problem. The resulting formulation has O(m) variables and O(m) constraints and can be solved with a standard solver.
- The appendix formulates problem (17) as a second-order cone programming problem.
- The reformulated problem is an SOCP problem with O(m) variables and O(m) constraints.
- A standard solver can solve the SOCP formulation efficiently.