Source-linked AI summary
Deep Multi-task Representation Learning: A Tensor Factorisation Approach
Yongxin Yang, Timothy Hospedales
TL;DR
Contemporary MTL has largely relied on shallow linear models, while deep MTL commonly requires manually designed sharing structures. The paper generalises matrix factorisation to tensor factorisation for end-to-end, layer-wise knowledge sharing in deep networks. It reports consistently better performance than single-task learning and comparable or better performance than the best searched user-defined architectures.
Problem
Deep MTL typically requires users to manually specify which layers share or separate, while much contemporary MTL focuses on shallow linear models.
Method
The framework factorises fully connected and convolutional DNN parameters into shared and task-specific parts, learning layer-wise sharing for homogeneous and heterogeneous MTL.
Results
The method consistently outperforms single-task learning and achieves comparable or better performance than the best results from exhaustive searches of user-defined MTL architectures.
Takeaways & Limitations
The approach reduces the design choices and architectural search space required for deep MTL layer-sharing decisions.
Takeaways & Limitations
In heterogeneous MTL, the output layer is necessarily unshared because tasks have different dimensionality.
Abstract
from arXiv · showhide
Most contemporary multi-task learning methods assume linear models. This setting is considered shallow in the era of deep learning. In this paper, we present a new deep multi-task representation learning framework that learns cross-task sharing structure at every layer in a deep network. Our approach is based on generalising the matrix factorisation techniques explicitly or implicitly used by many conventional MTL algorithms to tensor factorisation, to realise automatic learning of end-to-end knowledge sharing in deep networks. This is in contrast to existing deep learning approaches that need a user-defined multi-task sharing strategy. Our approach applies to both homogeneous and heterogeneous MTL. Experiments demonstrate the efficacy of our deep multi-task representation learning in terms of both higher accuracy and fewer design choices.
1 INTRODUCTION
The paper revisits multi-task learning in deep networks because recent MTL work has emphasized shallow linear models, while deep MTL typically requires manually specifying layer sharing. It proposes tensor factorisation to learn sharing end to end across deep networks and task settings.
- Modern MTL research has largely focused on shallow kernel, sparse, and low-dimensional linear models despite the practical success of deep neural networks.
- Deep MTL commonly shares lower representation layers and separates task-specific layers, but users must manually choose the sharing structure.This creates architectural design choices about how many layers to share and where to separate tasks.
- The proposed framework generalises shallow MTL to learn how to share knowledge at every layer of a deep network.
- The method supports both homogeneous MTL, with single-output tasks, and heterogeneous MTL, with task-specific output sets.
- Tensor factorisation divides fully connected weights and convolutional kernels into shared and task-specific parts, learning layer-wise sharing directly from raw image data.For a no-hidden-layer, single-output network, the method reduces to matrix-based MTL methods.
2 RELATED WORK
Related work spans matrix-based MTL, tensor factorisation, heterogeneous-task networks, and parameterised DNNs. The paper positions its approach as combining factorised knowledge sharing with deep, task-conditioned models.
- Matrix-based MTL stacks D-dimensional task models into a D × T matrix and imposes structural constraints such as low rank or explicit factorisation.A common form is W = LS, where L contains shared latent basis tasks and S contains task-specific coefficients.
- Tensor factorisation has been used both to reduce parameters in deep models and to represent tasks described by multiple independent factors.
- Heterogeneous MTL allows tasks to have different numbers of outputs and commonly uses neural networks with multiple output sets and losses.
- Parameterised DNNs dynamically generate weights from side information, with task identity serving as the relevant side information in MTL.
- Related parameterised models can adapt to new data by retraining task-specific coefficients while keeping shared base models fixed.Other work also considers predicting those coefficients from abstract metadata to generate models for unseen tasks.
3 METHODOLOGY
The methodology uses tensor factorisation to construct task-specific DNN weights from shared and task-dependent factors, enabling soft sharing across fully connected and convolutional layers. It supports task-only or broader sharing across task and output dimensions, with trainable factorised parameters.
- 3.1 PRELIMINARIES: An N-way tensor is an N-dimensional array, and mode-n flattening rearranges its mode-n fibres into a matrix.
- 3.1 PRELIMINARIES: Matrix factorisation represents task weights as W = LS, combining shared latent basis tasks in L with task-specific coefficients in S.
- 3.1 PRELIMINARIES: Extending to multiple outputs forms a D1 × D2 × T weight tensor, while simple last-axis factorisation shares knowledge across tasks but not output dimensions within a task.
- 3.2 TENSOR FACTORISATION FOR KNOWLEDGE SHARING: Tucker decomposition represents a weight tensor with a core tensor and one factor matrix per mode, whereas Tensor Train uses two matrices and intermediate three-way tensors.
- 3.2 TENSOR FACTORISATION FOR KNOWLEDGE SHARING: When the final tensor axis indexes tasks, its last factor encodes task-specific knowledge and the remaining factors encode shared knowledge.
- 3.3 DEEP MULTI-TASK REPRESENTATION LEARNING: DMTRL learns one same-architecture DNN per task while generating corresponding layer weights from factorised knowledge-sharing structures.Weight tensors are synthesised right-to-left rather than decomposed as an input tensor, and the resulting model is trainable by standard backpropagation.
- 3.3 DEEP MULTI-TASK REPRESENTATION LEARNING: In heterogeneous MTL, the output layer is necessarily unshared because tasks have different dimensionality.
- 3.3 DEEP MULTI-TASK REPRESENTATION LEARNING: The framework extends naturally to convolutional layers and learns where, what, and how much to share instead of requiring manually tied or independent layers.
4 EXPERIMENTS
Experiments evaluate the proposed factorised deep MTL methods on homogeneous MNIST, heterogeneous face analysis, and multilingual Omniglot recognition. Across these settings, learned soft sharing generally outperforms single-task or user-defined sharing, especially with limited data, while adapting sharing by layer and task.
- Homogeneous MTL: All MTL approaches outperform STL on MNIST, with larger advantages when training data is small.
- Homogeneous MTL: DMTRL-TT and DMTRL-Tucker outperform the best user-defined MTL with very small MNIST training sets and are comparable when data is large.
- Homogeneous MTL: With 1000 training examples, the proposed methods achieve error rates below 6%, compared with reported shallow-MTL rates of 14.0% and 13.4%.
- Heterogeneous MTL: Face Analysis: On AdienceFaces, DMTRL-Tucker consistently outperforms STL and the best user-defined MTL for gender recognition, while proposed methods generally improve age-group recognition over STL.
- Heterogeneous MTL: Multi-Alphabet Recognition: On Omniglot, proposed MTL methods surpass STL across all 50 alphabet tasks, while user-defined MTL helps only when the training fraction exceeds 50%.
- Heterogeneous MTL: Multi-Alphabet Recognition: For 10% Omniglot training data, learned sharing is higher in Conv1–Conv3 and lower in FC1, smoothly varying with depth rather than switching hard between shared and separate layers.The sharing measure treats STL as ρ = 0 and completely shared user-defined MTL as ρ = 1.
5 CONCLUSION
The framework generalises matrix-factorisation MTL to tensor factorisation for flexible knowledge sharing in deep networks. It determines sharing data-dependently at each layer, reducing architectural design choices while achieving consistently strong performance.
- Tensor factorisation extends matrix-factorisation MTL to share knowledge flexibly in fully connected and convolutional DNN layers.
- The method consistently outperforms single-task learning and matches or exceeds the best exhaustively searched user-defined MTL architectures.
- Data-driven, layer-by-layer sharing reduces the architectural search space and removes the need to manually choose layer sharing or segregation.
- The learned sharing structure smoothly interpolates between sharing and non-sharing in progressively deeper layers.
A COMPARISON WITH CLASSIC (SHALLOW) MTL METHODS
The comparison evaluates classic shallow MTL methods against deep multi-task representation learning on MNIST. Deep methods benefit from soft feature sharing and end-to-end training, while classic matrix-based methods are limited to binary problems.
- The first experiment uses MNIST binary one-vs-rest classification with 1% training data and reports mean error rates from 10-fold cross-validation.
- Classic methods improve over STL and perform significantly better with CNN features than with PCA features.
- DMTRL methods outperform the best classic methods even when those baselines use CNN features.The paper attributes this to soft sharing of feature-extraction layers and end-to-end training of classifier and feature extractor.
- Classic matrix-based methods are unsuitable for multi-class tasks such as Omniglot and age-group classification because they are restricted to binary problems.
B MODEL CAPACITY AND PERFORMANCE
The capacity analysis compares deep models using error rate and parameter count. Soft-sharing models use fewer parameters than single-task learning but more than hard sharing, and matching capacity alone does not explain their performance.
- Soft-sharing methods significantly reduce parameters relative to STL but remain larger than UD-MTL hard sharing.
- UD-MTL shares all layers except the top layer and uses roughly 10% of STL’s parameters in the ten-task MNIST setting.
- UD-MTL-Large matches the proposed methods’ parameter count by adding hidden neurons, but its performance does not increase.
- The unchanged UD-MTL-Large performance indicates that the proposed model’s results are not simply due to greater capacity than UD-MTL.