Source-linked AI summary
Multi-Task Learning in Natural Language Processing: An Overview
Shijie Chen, Yu Zhang, Qiang Yang
TL;DR
NLP deep models face overfitting and data scarcity, motivating multi-task learning as a way to use related-task information. This overview organizes NLP MTL architectures, optimization techniques, applications, and benchmark datasets. It concludes that selecting task sets that benefit from MTL and understanding task relatedness remain immature research challenges.
Problem
NLP deep models commonly face overfitting and data scarcity, while selecting related tasks for effective multi-task learning remains insufficiently understood.
Method
The paper surveys NLP MTL architectures, optimization techniques, applications, and benchmark datasets, including parallel, hierarchical, modular, and generative adversarial architectures.
Results
The overview reports that dynamic task scheduling generally performs better than fixed mixing ratios, while uncertainty-based sampling outperforms annealed sampling.
Takeaways & Limitations
The survey frames task selection and task-relatedness understanding as important directions for extending effective MTL across NLP tasks.
Takeaways & Limitations
Identifying task sets that can benefit from MTL remains a challenge, and existing studies of task relatedness are far from mature.
Abstract
from arXiv · showhide
Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.
2.1 Parallel Architectures
Parallel architectures run task models concurrently while sharing selected representations, with task-specific branches or mechanisms for feature fusion. They range from shared trunks to dynamically controlled information flow and multi-level supervision.
- Parallel architecture: Parallel architectures share intermediate layers across concurrently running task models while retaining task-specific output layers.Tasks have no dependency beyond layer sharing, so their training samples need not follow a fixed order.
- Parallel feature sharing: Parallel feature sharing uses a shared trunk followed by task-specific encoders and output layers, with trunks ranging from word representations to nearly the entire model.This tree-like design can also share different trunk or decoder parts across task groups or processing stages.
- Parallel feature fusion: Feature fusion combines shared and task-specific representations through globally shared encoders, attention keys, feed-forward layers, gates, or shared memory units.These mechanisms actively integrate features rather than relying only on implicit parameter sharing.
- Parallel feature fusion: Task-relatedness models and routing networks dynamically control pairwise feature transfer or sample paths, allowing information flow to vary by task and input.LK-MTL uses leaky gates for pairwise transfer, while MCapsNet uses task routing between capsule layers.
- Parallel feature fusion: LK-MTL and MCapsNet outperform other feature-fusion methods and achieve state-of-the-art performance through fine-grained dynamic information-flow control.The cited passage attributes their advantage to dynamic control between tasks.
- Multi-level supervision: Parallel architectures also support supervision at different abstraction levels, including character, token, sentence, paragraph, and document levels.Auxiliary language modeling and other level-specific tasks can assist primary tasks such as sequence labeling, question generation, and dialogue generation.
3 OPTIMIZATION FOR MTL MODELS
MTL optimization combines task losses and adapts training emphasis through weighting, sampling, scheduling, and task-specific objectives. These techniques address imbalanced data, overfitting, negative transfer, and interference while supporting related-task learning.
- Loss Construction: Linear loss combinations train multiple tasks jointly, while adaptive losses and regularization terms can add task-specific constraints.Different task losses are combined into a global objective optimized with conventional back-propagation.
- Loss Construction: Task weights may be equal, tuned as hyperparameters, adjusted for dataset size, or dynamically changed according to uncertainty and training metrics.Dynamic weighting can emphasize different tasks at different training stages and reduce dominance by large datasets.
- Data Sampling: Increasing temperature during training encourages up-sampling of low-resource tasks and alleviates overfitting.The schedule increases task emphasis after a warm-up period or as training progresses.
- Data Sampling: Bandit-based sampling initially favors the primary task before switching more evenly among tasks using validation-loss rewards.The observed reward is defined as the negative validation loss of the primary task.
- Data Sampling: Uncertainty-based sampling selects the b most uncertain samples and outperforms annealed sampling while avoiding catastrophic forgetting and inter-task interference.Uncertainty is computed from prediction entropy, normalized for differences in task class counts.
- Task Scheduling: Sequential task scheduling orders tasks by dependency or difficulty, while regularization can stabilize alternating training with imbalanced dataset sizes.Auxiliary MTL may pre-train on easier or data-rich tasks before fine-tuning, whereas joint MTL benefits from selecting more compatible tasks.
5 DATA SOURCE AND BENCHMARKS FOR MULTI-TASK LEARNING
The paper describes how multi-task datasets are organized and introduces benchmark datasets for evaluating NLP models across tasks, languages, and capabilities. It also identifies open directions for extending MTL beyond established task types and for using unlabeled data and more advanced architectures.
- Dataset Organization: MTL datasets may combine distinct label spaces by alternating task-specific datasets during training.For tasks with different label spaces, the overall dataset is represented as D = {D1, . . . , D_M}, and training commonly alternates among tasks randomly or according to a schedule.
- Dataset Organization: Multi-label datasets share one feature space while providing multiple label sets for the same inputs.Such datasets can be created by adding annotations for different formalisms, demographic attributes, or relation types to existing data.
- Dataset Organization: Extra annotations can be generated automatically, including labels derived from rules or discourse connectives in unlabeled corpora.These procedures support self-supervised or auxiliary unsupervised tasks without requiring every label to be manually annotated.
- Benchmark Datasets: Public benchmarks include GLUE, SuperGLUE, MMMLU, Xtreme, and XGLUE, covering language understanding, world knowledge, cross-lingual generalization, and generation.GLUE and SuperGLUE contain diverse NLU tasks, MMMLU covers 57 subjects, Xtreme covers 9 tasks in 40 languages, and XGLUE includes 11 downstream tasks.
- Future Directions: Future work should extend MTL to challenging tasks such as dialogue systems and multimodal learning, while improving task-relatedness analysis.The paper also proposes leveraging abundant unlabeled data and developing more advanced MTL architectures for stronger pre-trained language models.