Source-linked AI summary
Clustered Multi-Task Learning: A Convex Formulation
Laurent Jacob, Francis Bach, Jean-Philippe Vert
TL;DR
The paper addresses multi-task learning when related tasks form unknown clusters with similar weight vectors. It designs a cluster-encoding spectral norm and convex formulation that learns task relationships without a known partition. Experiments on synthetic data and IEDB MHC-I binding data report better performance than established convex and related nonconvex methods.
Problem
Multi-task learning needs a way to share information when tasks may form unknown groups with similar weight vectors rather than one global relationship.
Method
The paper jointly learns task weights and cluster structure using a dedicated norm, then applies a convex relaxation of the clustering optimization.
Results
The approach outperforms well-known convex multi-task methods and related nonconvex clustered-task methods on synthetic examples and the IEDB dataset.
Takeaways & Limitations
The convex clustered formulation can improve multi-task accuracy while providing cluster-structure information as a by-product.
Takeaways & Limitations
The formulation uses a selected simple spectral convex set, and the experimental comparison includes a method requiring the true clustering a priori.
Abstract
from arXiv · showhide
In multi-task learning several related tasks are considered simultaneously, with the hope that by an appropriate sharing of information across tasks, each task may benefit from the others. In the context of learning linear functions for supervised classification or regression, this can be achieved by including a priori information about the weight vectors associated with the tasks, and how they are expected to be related to each other. In this paper, we assume that tasks are clustered into groups, which are unknown beforehand, and that tasks within a group have similar weight vectors. We design a new spectral norm that encodes this a priori assumption, without the prior knowledge of the partition of tasks into groups, resulting in a new convex optimization formulation for multi-task learning. We show in simulations on synthetic examples and on the IEDB MHC-I binding dataset, that our approach outperforms well-known convex methods for multi-task learning, as well as related non convex methods dedicated to the same problem.
1 Introduction
The paper designs a norm from prior knowledge that tasks form unknown clusters with similar within-cluster weight vectors. It aims to use this structure for multi-task learning while identifying clusters during inference.
- The paper asks how to design a norm that enforces given prior knowledge about task relationships.
- It assumes tasks are partitioned into unknown groups whose members have similar weight vectors.
- Unlike approaches requiring known groups, the method seeks to identify clusters and use them for multi-task learning.
- Clustered regularization can avoid imposing similarity across all tasks when a few outliers differ substantially.
- The inferred cluster structure may also identify outliers or customer preference groups as a by-product.
- The algorithm constructs a penalty for known partitions and optimizes the inference objective over possible partitions.
2 Multi-task learning with clustered tasks
The formulation regularizes task weight vectors through global, between-cluster, and within-cluster components, then learns the cluster structure jointly with the weights. A convex relaxation makes the otherwise intractable partition optimization tractable.
- The model estimates one linear function per task, represented by the columns of a shared weight matrix W.
- Multi-task learning minimizes empirical risk plus a penalty designed to control information sharing among task weight vectors.
- Given clusters, the penalty separates average weight magnitude, between-cluster variance, and within-cluster variance.
- The parameters εM, εB, and εW balance the penalty components, with εW > εB > εM promoting compact within-cluster structure.
- The method jointly optimizes the weights and cluster matrix rather than assuming the task partition is known beforehand.
- The resulting objective is jointly convex in W and Σ, but the finite set of clustering-induced matrices is nonconvex and makes the direct problem intractable.
3 Convex relaxation
The paper relaxes the discrete optimization over task clusterings into a convex spectral formulation, yielding the cluster norm and an efficient optimization procedure. The formulation also connects to a convex relaxation of K-means.
- Convex relaxation: The relaxation centers task weights and replaces the discrete cluster-dependent matrix set with a convex spectral set constrained by positive semidefiniteness and trace.The centered cluster matrix satisfies 0 ⪯ f_M ⪯ I and tr(f_M) = r − 1.
- Convex relaxation: The resulting penalty defines the cluster norm (CN), a norm obtained by minimizing over the convex set of admissible positive matrices.The framework absorbs part of the penalty into the empirical risk and applies the norm to the centered weight matrix.
- Convex relaxation: The selected spectral set simplifies optimization, while additional constraints could encode point-wise positivity of the clustering matrix.The formulation recovers the endpoint cases with one cluster or one cluster per task.
- Convex relaxation: The cluster norm is a convex relaxation of K-means: K-means decomposes W into cluster centers and assignments, whereas this formulation relaxes the discrete assignment structure.The equivalence follows after optimizing the cluster centers and comparing the resulting expression with the proposed formulation.
- Convex relaxation: The norm and its gradient can be computed efficiently from the singular values of the centered weight matrix through a semi-closed-form spectral optimization.The optimal orthogonal matrix consists of eigenvectors of WᵀΠW, and the remaining coupling comes from a trace constraint.
4 Experiments
Experiments compare the convex cluster norm with independent, trace-norm, k-means, and related approaches on synthetic tasks and IEDB MHC-I binding data. The cluster norm performs especially well with limited data, while clustering recovery depends on sample size and dataset.
- Artificial data: All methods can run without prior clustering knowledge except the true-metric approach.The true metric uses the actual clustering as a golden-standard reference.
- Artificial data: CN penalization always gives lower testing error than trace-norm penalization, with the strongest advantage when few training points are available.With larger samples, method performances become increasingly similar.
- Artificial data: The true metric gives the best results, while CN outperforms k-means at 50 training points before k-means recovers the clustering with more data.At 28 points, neither method recovers the correct clustering; at larger sample sizes, k-means outperforms the relaxed approach.
- MHC-I binding data: Individual models outperform one pooled model on IEDB, even when few data points are available.The pooling baseline combines all molecules into one global prediction problem.
- MHC-I binding data: On IEDB, the cluster norm gives the best performance among multitask approaches for the 10 molecules with fewer than 200 known ligands.The learned structure did not recover the known supertypes, and reprojection methods did not improve performance.
5 Conclusion
The paper presents a convex clustered multitask-learning approach based on a dedicated norm, with promising results on synthetic data and IEDB. Future work targets refined relaxations, nonlinear extensions, and task-specific features.
- The paper presents a convex approach to clustered multitask learning based on a dedicated norm.
- Promising results are reported on synthetic examples and the IEDB dataset.
- Future work includes refined convex relaxations, nonlinear multitask learning, and specific task features.The paper notes that task-specific features have improved performance in other settings.