Source-linked AI summary
Identifying beneficial task relations for multi-task learning in deep neural networks
Joachim Bingel, Anders Søgaard
TL;DR
NLP MTL can improve performance, but its gains are inconsistent and the task relations that support them are poorly understood. This paper systematically analyzes 90 task configurations using shared-parameter sequence-labeling models and task and learning-curve features. It finds that MTL outcomes are often predictable, especially from single-task learning curves, while dataset balance is not a reliable predictor.
Problem
In NLP, MTL has produced mixed results, while the task relations that predict gains over single-task models remain unclear.
Method
The paper evaluates 90 directed task configurations with shared-parameter sequence-labeling models, fixed single-task hyperparameters, and 42 task and learning-curve features.
Results
In almost four in five cases, MTL outcomes can be predicted from dataset and single-task features, with learning-curve features the strongest predictors.
Takeaways & Limitations
MTL gains are more likely when target tasks plateau quickly while auxiliary tasks remain steep, and label entropy in the auxiliary task is also predictive.
Takeaways & Limitations
The conclusions apply to fixed hyperparameters, while individually optimized single- and multi-task settings remain future work.
Abstract
from arXiv · showhide
Multi-task learning (MTL) in deep neural networks for NLP has recently received increasing interest due to some compelling benefits, including its potential to efficiently regularize models and to reduce the need for labeled data. While it has brought significant improvements in a number of NLP tasks, mixed results have been reported, and little is known about the conditions under which MTL leads to gains in NLP. This paper sheds light on the specific task relations that can lead to gains from MTL models over single-task setups.
1 Introduction
MTL is attractive because it may reduce labeling needs and regularize models, but NLP results are mixed and the task relations that produce gains remain unclear.
- MTL may reduce the need for labeled data and induce more robust models.
- Theoretical guarantees for MTL rely on conditions such as shared optimal hypothesis classes or related sample-generating distributions.
- NLP tasks are typically heterogeneous, and reported MTL improvements coexist with mixed results.
- The task relations that guarantee or make MTL gains likely in NLP remain an open question.
- The paper systematically studies when and why MTL works for sequence labeling with deep recurrent neural networks.
2 Related work
Prior NLP research offers limited evidence about when MTL succeeds, pointing to data size, target-data scarcity, label distributions, and semantic similarity as possible factors.
- Structured-prediction research has rarely examined the conditions under which MTL works.
- Earlier studies suggest that auxiliary data should not outsize target data and that MTL can be effective with limited target data.
- Task-combination results suggest that success may depend on how uniformly auxiliary-task labels are distributed.
- For sentence classification, MTL success was reported to depend largely on semantic similarity between source and target datasets.
3 Multi-task Learning
The paper uses hard parameter sharing for sequence-labeling tasks, combining shared representations with task-specific predictions across ten NLP datasets.
- Hard parameter sharing shares hidden layers across tasks to learn a joint representation and acts as an efficient, easy-to-implement regularizer.
- The setup compares single-task and multi-task sequence-labeling models while reusing single-task hyperparameters in the MTL setting.
- The shared architecture uses pretrained, trainable word vectors, a shared 100-dimensional bi-LSTM, and task-specific dense projections.
- Each training step samples one task uniformly, uses a random batch of 32 examples, and optimizes shared and task-specific parameters with Adadelta.
- 3.2 Tasks: The experiments cover ten NLP tasks, including CCG tagging, chunking, sentence compression, semantic frames, POS tagging, hyperlink prediction, keyphrase detection, MWE detection, and two super-sense tasks.
4 Experiments
The experiments evaluate 90 directed main–auxiliary task configurations and examine whether task and single-task learning-curve features predict MTL gains. MTL improves 40 of 90 cases, with gains most likely when target tasks plateau early while auxiliary tasks continue learning.
- Experimental setup: 90 directed main–auxiliary task configurations compare multi-task models with single-task baselines.Single-task architectures’ hyperparameters are reused without additional tuning; multi-task models train for 50,000 batches versus 25,000 for single-task models.
- Results: 40 of 90 configurations improve with MTL, while chunking and high-level semantic tagging often help other tasks most.Hyperlink detection does not significantly improve other tasks, whereas multiword and hyperlink detection benefit from several auxiliary tasks.
- Predicting MTL benefit: The meta-learning classifier predicts MTL outcomes in almost four in five cases, whereas data-inherent features alone perform worse than the majority baseline on positive-class F1.The evaluation uses mean performance across 100 runs of randomized five-fold cross-validation.
- Predicting MTL benefit: Features from single-task learning curves are the strongest predictors of MTL benefit.Gradients around 20–30% of training are especially informative, after the curve typically flattens around 10%.
- Predicting MTL benefit: MTL is more likely to help target tasks that quickly plateau when auxiliary-task curves remain relatively steep.The authors speculate that auxiliary learning may help targets that become stuck early in local minima.
- Feature analysis: Main-task label count, auxiliary-task label entropy, and target-task OOV rate are predictive, while JSD and dataset-size differences are less predictive.The label-entropy association may partly reflect that more uniform label distributions are easier to learn; auxiliary data also updates embeddings.
5 Conclusion and Future Work
The study finds that MTL gains can be predicted from dataset characteristics and single-task learning features, especially learning curves and auxiliary-task label entropy. Future work will optimize hyperparameters separately and broaden the experimental settings.
- MTL gains are predictable from dataset characteristics and features extracted from single-task inductions.
- Features related to single-task learning curves were the most predictive, suggesting successful MTL often helps target tasks escape local minima.
- Auxiliary-task label entropy was a good predictor, supporting earlier hypotheses, whereas dataset balance showed little reliable predictive evidence.
- Future Work: Future experiments will optimize hyperparameters separately for single- and multi-task models to estimate MTL effects more reliably in practice.
- Future Work: Future work also proposes unequal task weighting and evaluation beyond sequence labeling, including language modeling and sequence-to-sequence problems.