Source-linked AI summary
Trace Norm Regularised Deep Multi-Task Learning
Yongxin Yang, Timothy M. Hospedales
TL;DR
Deep multi-task learning must choose which layers to share, but modern networks make that branching decision difficult. This paper regularizes layer-wise collections of parameters across task networks with tensor trace norms, learning sharing over all shareable layers. On Omniglot, multi-task regularisation alleviates over-fitting, with greater sharing in bottom layers and limited sensitivity to the LAF, Tucker, or TT choice.
Problem
Modern deep multi-task networks commonly rely on predefined sharing strategies, while deciding where to split many-layer architectures is extremely difficult.
Method
The framework collects corresponding parameters across same-architecture task networks and applies tensor trace-norm regularisation to all shareable layers.
Results
On Omniglot, multi-task regularisation alleviates over-fitting, with bottom layers sharing more than top layers and LAF, Tucker, and TT choices appearing not very sensitive.
Takeaways & Limitations
The technique provides a data-driven solution to deep multi-task branching architecture design and an alternative to explicit factorisation-based approaches.
Abstract
from arXiv · showhide
We propose a framework for training multiple neural networks simultaneously. The parameters from all models are regularised by the tensor trace norm, so that each neural network is encouraged to reuse others' parameters if possible -- this is the main motivation behind multi-task learning. In contrast to many deep multi-task learning models, we do not predefine a parameter sharing strategy by specifying which layers have tied parameters. Instead, our framework considers sharing for all shareable layers, and the sharing strategy is learned in a data-driven way.
1 INTRODUCTION AND RELATED WORK
Multi-task learning jointly trains related tasks so knowledge can be reused, extending from matrix and tensor formulations to deep networks. Deep architectures make predefined layer-sharing decisions difficult, motivating data-driven alternatives.
- Multi-task learning aims to learn multiple tasks jointly so knowledge obtained from one task can be reused by others.
- Matrix-based Multi-Task Learning: Matrix-based MTL couples task-specific weight vectors through regularisation, typically encouraging their collection into a low-rank matrix.
- Tensor-based Multi-Task Learning: Tensor-based MTL represents tasks indexed by multiple factors as a tensor and applies tensor regularisers such as matricisation-based trace norms.
- Multi-Task Learning for Neural Networks: Deep MTL studies commonly use predefined sharing strategies, such as shared bottom layers and task-specific top layers.
- Multi-Task Learning for Neural Networks: Modern networks contain many layers, making the choice of where to split one network into task-specific branches extremely difficult.
2 METHODOLOGY
The framework stacks corresponding parameters across neural networks and regularises shareable layers with tensor trace norms rather than prescribing tied layers. It offers LAF, Tucker, and TT variants optimized with gradient-based methods.
- Framework: The framework models each task with the same neural-network architecture and applies a tensor norm to each layer-wise collection of parameters.
- Framework: For tasks with different output dimensions, only shareable layers such as conv1, conv2, and fc1 are regularised, excluding the final layer.
- Framework: Unlike predefined sharing, the method retains separate parameters and regularises stacked corresponding layers, allowing sharing to be learned through optimization.
- Tensor Norm: The trace norm sums a matrix’s singular values and serves as a convex proxy for matrix rank; its tensor extension yields LAF, Tucker, and TT variants.
- Tensor Norm: Last Axis Flattening reshapes a tensor whose last axis indexes tasks into a matrix with task-specific columns.
- Optimisation: The method uses sub-gradient descent for trace-norm optimization and can also compute the sub-gradient using an SVD-based formulation.
3 EXPERIMENT
The experiments compare tensor-norm regularisation variants with single-task learning on Omniglot and show reduced over-fitting alongside learned, layer-dependent sharing.
- Experimental setup: The study evaluates LAF, Tucker, and TT against single-task learning on Omniglot using a convolutional neural network.Omniglot contains 1,623 characters from 50 alphabets, with each alphabet defining a multiclass recognition task.
- Results: STL achieves the lowest training loss but the worst testing performance, while multi-task regularisation alleviates over-fitting.
- Results: Parameter sharing is stronger in bottom layers than top layers, matching the design intuition that bottom-layer representations are more data/task independent.
- Results: The choice among LAF, Tucker, and TT appears relatively insensitive because optimising one norm also reduces the losses of the others.
- Implication: The technique provides a data-driven solution to deep multi-task learning’s branching-architecture design problem.It is presented as a norm-regulariser alternative to explicit factorisation-based approaches.
L. R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 1966.
This supplied passage set contains bibliographic references concerning matrix norms, tensor factorisation, and multi-task learning rather than substantive section content.
- Watson’s 1992 work concerns the subdifferential of some matrix norms.
- Wimalawarne, Sugiyama, and Tomioka’s 2014 NIPS paper concerns multitask learning and tensor factorisation.
- Yang and Hospedales’ cited works address unified multi-domain and multi-task learning and deep multi-task representation learning.
- Zhang and colleagues’ cited ECCV paper concerns facial landmark detection by deep multi-task learning.