A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods

1 Introduction

• MTL seeks to improve generalization across related tasks by leveraging shared information, commonly sharing hidden layers while retaining task-specific output layers.The introduction also notes potential advantages including data efficiency, reduced overfitting, and faster learning.Source: §1 · p. 1
• Joint training fits tasks performed simultaneously, whereas multi-step training links decoders when a task depends on earlier-task outputs or representations.In the multi-step framework, task-specific decoders form a path from the encoder.Source: §1 · p. 2
• Unlike prior surveys on parameter sharing and optimization (Ruder, 2017; Zhang and Yang, 2018; Crawshaw, 2020), this survey organizes NLP MTL by task relatedness and maps prior categories into its two methods.The survey presents task relatedness as a basis for choosing a training method before considering detailed architecture or optimization designs.Source: §1 · pp. 1–2, Table 1 · p. 2
• The survey reviews MTL models and rationales, presents joint- and multi-step-training applications, analyzes task relatedness, and discusses future directions.Source: §1 · p. 2
• The survey does not distinguish parameter-sharing designs, which it identifies as outside its focus.It refers readers to Ruder (2017) for details.Source: §1 · p. 2

2.1 Encoder-Decoder Architecture and Two Multi-Task Training Frameworks

• The framework has encoder, decoder, and loss/optimization components; task inputs may be shared while desired outputs usually differ.Encoders produce hidden states, and decoders transform them into task outputs.Source: §2.1 · p. 2
• Joint training suits tasks independent at decoding, while multi-step training suits sequentially dependent tasks and can generalize to dependencies forming a directed acyclic graph.In the sequential case, a task decoder can use the preceding task’s output or hidden representation.Source: §2.1 · pp. 2–3
• A common optimization approach minimizes a weighted sum of task losses, with weights either predefined or dynamically adjusted.The survey notes alternative approaches, including optimizing a randomly selected task’s loss at each iteration.Source: §2.1 · p. 3

2.2 How does MTL Work

• MTL presupposes related tasks and data; positively correlated tasks with similar objectives or relevant data can benefit from shared lower-level representations.Combining features from different objectives can also support learning more complex features.Source: §2.2 · p. 3
• Different task noise patterns can act as implicit data augmentation, while high-resource tasks can provide additional training signals to low-resource tasks in similar domains.The survey links these mechanisms to more generalizable representations and reduced overfitting.Source: §2.2 · p. 3
• Auxiliary tasks can provide conditions or hints that reduce the main task’s output space, and explicit supervision can help a model learn otherwise difficult features.This setting usually falls under multi-step training.Source: §2.2 · p. 3

3.1.1 Information Extraction (IE)

• For information extraction, treating relation extraction as sequence labeling lets token-wise classifiers for NER and RE identify multiple relations in a sentence.The survey contrasts this approach with relation-type prediction and token-pair tagging, noting the latter’s training complexity with annotated data.Source: §3.1.1 · p. 4

3.1.2 Spoken Language Understanding

• Spoken language understanding jointly predicts slot labels for each word and an intent for the whole utterance, commonly using a shared utterance encoder.Some approaches also use intent-detection hidden states to assist slot filling, combining joint and multi-step training.Source: §3.1.2 · p. 4

3.1.3 Sentence/Document Classification

• Text classification uses joint training for tasks with multiple labels on one input or multiple datasets with their own labels and inputs.The survey notes that large-scale, high-quality dataset construction is labor-intensive and that related tasks can share features.Source: §3.1.3 · pp. 4–5

3.1.4 Multilinguality

• Multilingual training draws on lexical differences and semantic or syntactic relatedness across languages, with NMT applications including many-to-one and many-to-many setups.Other examples jointly train cross-lingual summarization with monolingual summarization and machine translation, or build multilingual models across 103 languages.Source: §3.1.4 · p. 5

3.1.5 Natural Language Generation (NLG)

• In natural language generation, related tasks include abstractive summarization with question generation, and question generation with question answering as dual tasks.The survey also describes semantic parsing and NLU tasks as complements to generation, and multi-view learning as an auxiliary objective for consistent predictions across altered input views.Source: §3.1.5 · p. 5

3.2 Multi-step Training: Applications

• Figure 3 presents four multi-step training applications in NLP, discussed in §3.2.1, the second and third subfigures of §3.2.2, and §3.2.3, spanning language understanding, multi-passage question answering, and natural language generation.Source: §3.2 · p. 5, Figure 3 · p. 6

3.2.1 Multi-level Language Understanding

• POS supervision at the innermost layer performed consistently better than supervision at the outermost layer for POS tagging and chunking.Other examples place increasingly complex predictions at deeper layers, from POS tagging through textual entailment, or use low-level entity tasks with higher-level coreference and relation extraction.Source: §3.2.1 · pp. 5–6

3.2.2 Multi-Passage Question Answering

• Multi-passage question answering commonly uses retrieval, reading comprehension, and answer reranking, with each phase supplying input to the next.Other approaches use two of these phases, while multi-choice systems can combine entity recognition, graph-based reasoning, and passage-based path scoring to support explicit-path reasoning.Source: §3.2.2 · p. 6

3.2.3 Retrieval-augmented Text Generation

• Retrieval-augmented generation retrieves relevant information as added context because the input alone may not provide enough knowledge for the desired output.The survey describes applications ranging from knowledge-aware dialogue to question generation, comment generation, image captioning, summarization, and long-form QA.Source: §3.2.3 · pp. 6–7

4.1 Joint Training

• Joint training commonly pairs similar tasks because they are more likely to benefit from shared generic representations, though similarity is often selected empirically.Examples include multilingual translation, sentiment and emotion classification, summarization with question generation, and question categorization with answer retrieval.Source: §4.1 · p. 7
• An adversarial auxiliary task can help separate shared and private features by predicting task identity; in domain adaptation, gradient reversal maximizes domain-prediction loss.This encourages representations that are indistinguishable between domains for the main task.Source: §4.1 · p. 7
• Self-supervised objectives and knowledge distillation provide auxiliary signals for representation learning without requiring expensive downstream labels or a larger deployed student.Examples include language modeling, auto-encoding, and transferring a teacher's hidden representations or output distribution to a smaller student.Source: §4.1 · pp. 7–8

4.2 Multi-step Training

• Retrieve-then-read methods narrow open-domain QA's output search space by conditioning the reader on documents selected by a retriever.ELECTRA instead uses masked-token reconstruction before a discriminator predicts whether each corrupted-input token was replaced, providing supervision on all tokens.Source: §4.2 · p. 8
• A preceding selection task can focus generation on salient input content, as in keyword-guided summarization and latent content selection for question generation.Source: §4.2 · p. 8
• Attribute classifiers can condition generation on a desired question type before decoding or guide dialogue generation toward a specified emotion afterward.Source: §4.2 · p. 8
• Knowledge-related auxiliary tasks can help models acquire and use external knowledge, including selecting candidate knowledge through attention and fusing it into a decoder.The cited example trains selection by minimizing KL-divergence between prior and posterior distributions queried by the input and output.Source: §4.2 · p. 8

5.1 Regarding Task Relatedness

• Downstream-oriented pre-training tasks jointly trained with masked language modeling can reduce the gap between pre-training and fine-tuning.The survey says these tasks should be self-supervised on pre-training data and share a learning objective with downstream tasks so relevant knowledge can transfer.Source: §5.1 · pp. 8–9
• Learning to multi-task is proposed as an alternative to human-designed training strategies and costly cross-validation over task groupings and loss weights.The survey identifies meta-learning for general-purpose MTL algorithms and task grouping as promising directions.Source: §5.1 · p. 9

5.2 Regarding Training Methods

• Adaptive parameter sharing remains an open need in NLP because existing layer-selection approaches suit architectures with repeated layers, unlike functionally and structurally discrepant NLP networks.Hard sharing requires finding an optimal configuration, while soft sharing does not reduce parameter count.Source: §5.2 · p. 9
• The role and timing of MTL in universal-model transfer learning remain unclear, including whether to use it in pre-training or fine-tuning.The survey also calls for theoretical analysis of MTL's benefits in transfer learning.Source: §5.2 · p. 9

6 Conclusions

• The survey organizes multi-task NLP approaches into joint training and multi-step training frameworks based on task relatedness.It presents framework designs across NLP applications and discusses future directions.Source: §6 · pp. 9–10

• The survey presents only prominent joint- and multi-step-training application scenarios, potentially leaving other multi-task NLP fields uncovered.Source: §7 · p. 10
• The work is theoretical and provides no software-level implementation of the mentioned framework.Source: §7 · p. 10
• Benchmark results for the mentioned models are omitted because of space limits.Source: §7 · p. 10