Source-linked AI summary
Latent Multi-task Architecture Learning
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, Anders Søgaard
TL;DR
MTL requires choosing among an enormous space of sharing architectures, including which layers or subspaces to share, how strongly to share, and how to weight task losses. The paper introduces sluice networks, which learn these choices jointly with task models, and reports consistent improvements across four tasks and seven domains. The authors conclude that this learned architecture improves over strong single-task, architecture-learning, and multi-task baselines.
Problem
MTL architecture search spans an enormous space of layer, subspace, sharing-strength, and task-loss-weight choices, while prior work addresses these dimensions only in isolation.
Method
Sluice networks jointly learn task-specific sharing pathways and multi-task models through trainable sharing parameters, subspace interactions, and prediction-layer mixtures.
Results
Across four tasks and seven domains, the meta-architecture consistently improved over strong single-task learning, architecture-learning, and multi-task learning baselines.
Takeaways & Limitations
The learned architecture can adapt sharing to task and data relationships while recovering previously proposed multi-task and domain-adaptation architectures.
Takeaways & Limitations
The framework assumes task networks have the same hyperparameters at the outset and considers loosely related tasks, potentially non-overlapping datasets.
Abstract
from arXiv · showhide
Multi-task learning (MTL) allows deep neural networks to learn from related tasks by sharing parameters with other networks. In practice, however, MTL involves searching an enormous space of possible parameter sharing architectures to find (a) the layers or subspaces that benefit from sharing, (b) the appropriate amount of sharing, and (c) the appropriate relative weights of the different task losses. Recent work has addressed each of the above problems in isolation. In this work we present an approach that learns a latent multi-task architecture that jointly addresses (a)--(c). We present experiments on synthetic data and data from OntoNotes 5.0, including four different tasks and seven different domains. Our extension consistently outperforms previous approaches to learning latent architectures for multi-task problems and achieves up to 15% average error reductions over common approaches to MTL.
Introduction
The paper addresses the infeasible search over multi-task parameter-sharing architectures by learning a latent architecture that jointly determines sharing, subspace allocation, and task-loss weighting. Its meta-architecture generalizes prior approaches and consistently improves performance across four tasks and seven domains.
- Motivation: MTL architecture search becomes infeasible as private spaces and soft sharing create an enormous or infinite space of possible architectures.A three-layer example yields 19,073,486,328,125 architectures before considering soft sharing.
- Motivation: Previous methods address only subsets of the architectural choices or consider only a few sharing architectures.
- Approach: The proposed framework introduces trainable parameters for all components that differentiate multi-task learning approaches along these architectural dimensions.
- Approach: The meta-architecture learns which layers and subspaces to share, sharing strength, and an outer-layer mixture of skip connections.
- Evaluation: Across four tasks and seven OntoNotes 5.0 domains, it consistently outperforms prior architecture-learning methods and common multi-task baselines.
Multi-task Architecture Learning
Sluice networks jointly learn task-specific sharing pathways, subspace interactions, prediction-layer mixtures, and task weighting within an end-to-end differentiable architecture. The design adds relatively few parameters and increases training time by only 5–7% over hard parameter sharing.
- Sluice networks: Sluice networks are end-to-end differentiable unions of sharing architectures whose trainable sluices learn a task-specific MTL architecture during training.
- Subspaces and sharing: The architecture partitions recurrent layers into subspaces, allowing learned interactions between task-specific and shared representations.
- Learned pathways: The α parameters mediate cross-task information flow and sharing, while β parameters mix layer outputs for prediction.
- Regularization: Task losses are combined with learned task weights, while matrix regularization and orthogonality constraints encode sharing structure and discourage redundant subspaces.
- Generalization: The model generalizes several existing multi-task and transfer-learning architectures by recovering them through particular α, β, and regularization settings.
- Complexity: 5–7% is the reported training-time increase over hard parameter sharing, while additional α and β parameters scale with layers, tasks, and subspaces.
Prior Work as Instances of Sluice Networks
Sluice networks provide a common parameterization in which earlier multi-task, transfer-learning, and sharing methods appear as special cases. The paper also contrasts learned auxiliary-task reliance with the behavior of fixed or inappropriate sharing schemes.
- Existing architectures: Hard parameter sharing is recovered when all α values are set to the same constant, corresponding to a mean-constrained ℓ0 regularizer.
- Existing architectures: Frustratingly easy domain adaptation is encoded by disabling selected cross-task connections to create shared and private spaces.
- Existing architectures: Cross-stitch networks correspond to tying subspace-specific α values and retaining only the outer-layer β connection.
- Scope: The cited domain-adaptation method was not designed for the setting discussed here.
- Synthetic behavior: With more data, the model learns not to share when the auxiliary task is randomly relabeled.
Experiments
Experiments test whether sluice networks learn sensible sharing patterns and improve multi-task performance across synthetic settings, OntoNotes tasks, and domains. The model quickly distinguishes useful from noisy auxiliary signals and generally outperforms comparison architectures, especially out of domain.
- Synthetic experiment: The synthetic experiment contrasts a randomly relabeled auxiliary dataset with an identical copied dataset to test whether learned α weights adapt to signal usefulness.The Random setting represents noise injection, whereas Copy provides a perfect but redundant auxiliary signal.
- Synthetic experiment: The learned architecture quickly identifies the contrast between an all-noise auxiliary task and a perfect redundant signal.The authors report that the meta-architecture learns a good sharing architecture in these clear cases.
- OntoNotes experiments: The evaluation uses OntoNotes 5.0 English data across domains, with chunking, named entity recognition, and simplified semantic role labeling as main tasks and POS tagging as auxiliary task.The experiments assess in-domain and out-of-domain generalization using data from multiple domains.
- Chunking results: Sluice networks significantly outperform other architectures on chunking in both settings and perform best in every domain except telephone conversation, where cross-stitch networks win.The comparison includes single-task, low-supervision, hard-sharing, and cross-stitch baselines.
- Chunking results: Out-of-domain chunking accuracy exceeds hard parameter sharing by more than 1 point and the strongest baseline by almost .5 on average.The reported gains are accompanied by 12.8% in-domain and 8.9% out-of-domain error reductions versus single-task learning.
- NER, SRL, and joint model: For NER and simplified SRL, sluice networks outperform comparison models in-domain, generalize out of domain on average, and achieve the best results in 5 of 7 and 4 of 7 domains, respectively.In the four-task experiment, task-specific layers are needed beyond hard sharing for fine-grained semantic knowledge; sluice networks beat single-task models except on chunking and lead on 2 of 4 tasks.
Analysis
The analyses show how sluice networks adapt sharing to task properties and benefit from learnable sharing mechanisms, explicit subspaces, and task-specific layer aggregation.
- Task Properties and Performance: Sluice-network gains are higher with less training data, while learned sharing increases with greater training-data variance.
- Ablation Analysis: Learnable α parameters outperform constant α parameters, while learned β parameters are competitive with skip-connections depending on the sharing setting.
- Ablation Analysis: Explicitly modeling subspaces helps for almost all domains and supports partitioning LSTM weights into more structured representations.
- Analysis of α values: For simplified SRL, inner layers receive more sharing, whereas Chunking and NER also rely on outer-layer sharing.
- Analysis of β values: Semantic NER and Chunking aggregate representations across the second and third layers, while POS tagging and simplified SRL emphasize one later layer.
Related Work
Prior multi-task approaches provide hard, soft, selective, private/shared, or split forms of parameter sharing, but generally cover only subsets of the architecture space.
- Hard parameter sharing is simple and reduces overfitting, but is guaranteed to work only for certain closely related tasks.
- Soft parameter sharing keeps separate task parameters and regularizes their outer-layer loss using the distance between models.
- Selective-sharing methods let task predictors choose sparse parameter bases, while private/shared-space methods separate task-specific from cross-task features.
- Split architectures learn how many innermost layers two networks share, representing only a small subset of sluice-network sharing architectures.
- Unlike mixture-of-experts layers, the proposed method encourages beneficial cross-task sharing while distinguishing low-level from high-level representations.
Conclusion
The paper introduces sluice networks as a meta-architecture for multi-task architecture search. Across four tasks and seven domains, it consistently improves over strong single-task, architecture-learning, and multi-task baselines.
- Sluice networks consistently improve over strong single-task learning, architecture-learning, and multi-task baselines across four tasks and seven domains.