Source-linked AI summary
Multi-Task Deep Neural Networks for Natural Language Understanding
Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao
TL;DR
The paper addresses how to learn general text representations for multiple NLU tasks by combining multi-task learning with language-model pretraining. MT-DNN integrates BERT-based shared encoding with task-specific outputs and achieves state-of-the-art results across ten NLU tasks, while the authors identify further structural, training, linguistic-control, and adversarial-robustness questions.
Problem
NLU representation learning benefits from both supervised data across related tasks and unsupervised language-model pretraining, motivating a combined approach.
Method
MT-DNN incorporates BERT as shared text-encoding layers, combines them with task-specific outputs, and trains through pretraining followed by multi-task learning.
Results
MT-DNN achieves new state-of-the-art results on ten NLU tasks, including eight of nine GLUE tasks, reaching 82.7% on GLUE, 2.2% above BERT.
Takeaways & Limitations
The learned representations support domain adaptation with substantially fewer in-domain labels than pretrained BERT representations.
Takeaways & Limitations
Future work includes understanding MTL structure sharing, improving training across related tasks, controlling linguistic structure explicitly, and testing resilience against adversarial attacks.
Abstract
from arXiv · showhide
In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization effect that leads to more general representations in order to adapt to new tasks and domains. MT-DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirectional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI, SciTail, and eight out of nine GLUE tasks, pushing the GLUE benchmark to 82.7% (2.2% absolute improvement). We also demonstrate using the SNLI and SciTail datasets that the representations learned by MT-DNN allow domain adaptation with substantially fewer in-domain labels than the pre-trained BERT representations. The code and pre-trained models are publicly available at https://github.com/namisan/mt-dnn.
1 Introduction
The paper combines multi-task learning with language-model pretraining in MT-DNN to learn more general text representations and improve NLU performance. It uses shared encoding layers with task-specific outputs and reports strong results across GLUE, SNLI, and SciTail.
- Motivation and contribution: MT-DNN combines multi-task learning and language-model pretraining to improve text representations for NLU tasks.The approach is motivated by the complementary strengths of supervised cross-task data and unsupervised language-model pretraining.
- Motivation and contribution: Multi-task learning leverages labeled data from related tasks and regularizes representations against overfitting to one task.These effects support more universal representations across tasks.
- Model design: MT-DNN shares lower text-encoding layers across tasks while using task-specific top layers for sentence classification, pairwise classification, similarity, and ranking.The model can be fine-tuned for a specific task, while retaining multi-task learning in addition to language-model pretraining.
- Results: 82.7% GLUE score represents a 2.2% absolute improvement over BERT, with new state-of-the-art results on eight of nine GLUE tasks.The paper also reports stronger SNLI and SciTail results and domain adaptation with substantially fewer in-domain labels.
2 Tasks
The paper organizes MT-DNN around four NLU task types: single-sentence classification, pairwise text classification, text similarity, and relevance ranking. These tasks differ in whether they classify one sentence, compare sentence pairs, score similarity, or rank candidate answers for a query.
- Task taxonomy: MT-DNN combines single-sentence classification, pairwise text classification, text similarity scoring, and relevance ranking.The four task types are described using GLUE tasks as examples.
- Single-sentence classification: Single-sentence classification assigns a predefined class label to one sentence, such as grammatical plausibility or sentiment.CoLA and SST-2 illustrate this task type.
- Text similarity: Text similarity is a regression task that predicts a real-valued semantic-similarity score for a sentence pair.STS-B is the GLUE example identified for this task type.
- Pairwise text classification: Pairwise text classification predicts a predefined relationship between two sentences, including entailment, contradiction, neutrality, or semantic equivalence.RTE and MNLI represent inference tasks, while QQP and MRPC represent paraphrase datasets.
- Relevance ranking: Relevance ranking orders candidate answers by relevance to a query, with QNLI formulated as ranking rather than binary classification.The model ranks the candidate containing the correct answer above the candidate that does not.
3 The Proposed MT-DNN Model
MT-DNN combines shared Transformer-based representation learning with task-specific output layers across multiple NLU task types. It is pretrained with BERT objectives and then optimized through multi-task learning using task-specific objectives.
- Architecture: MT-DNN shares lower Transformer-based representation layers across tasks while using task-specific upper layers for different NLU outputs.The shared encoder produces contextual embeddings, followed by task-specific modules for classification, similarity scoring, or relevance ranking.
- Architecture: The lexicon encoder maps tokens into embeddings formed by summing word, segment, and positional embeddings.The first token is [CLS], and sentence pairs are separated with [SEP].
- Architecture: The bidirectional Transformer produces the shared contextual representation and learns it through multi-task objectives in addition to pretraining.This distinguishes MT-DNN from BERT’s representation-learning approach, which relies on pretraining.
- Task-specific outputs: MT-DNN supports single-sentence classification, pairwise text classification, text similarity scoring, and relevance ranking.Its task-specific outputs include softmax classification, similarity scoring, multi-step NLI reasoning, and candidate-answer ranking.
- Task-specific outputs: QNLI is formulated as pairwise ranking, ranking candidate answers containing the correct answer above candidates that do not.The relevance scores are computed for each candidate answer and used to order candidates for a given query.
- Training procedure: Training uses BERT-style masked language modeling and next sentence prediction followed by mini-batch stochastic gradient descent over task-specific objectives.Each task mini-batch updates shared and task-specific parameters using classification, regression, or ranking losses.
4 Experiments
The experiments evaluate MT-DNN against BERT and other systems across GLUE, SNLI, and SciTail, including fine-tuning, multi-task learning, and domain adaptation. MT-DNN achieves broad GLUE gains and stronger adaptation with limited in-domain data.
- Experimental Setup: The evaluation covers GLUE, SNLI, and SciTail, comparing MT-DNN with BERT and other state-of-the-art models.GLUE contains nine NLU tasks; SNLI and SciTail are used for domain adaptation.
- GLUE Main Results: MT-DNN uses BERTLARGE to initialize shared layers, refines them through multi-task learning, and fine-tunes on task-specific data.The experiments also compare single-task and no-fine-tuning variants.
- GLUE Main Results: 82.7% GLUE score represents a 2.2% absolute improvement over BERTLARGE, with new state-of-the-art results on eight GLUE tasks.WNLI is the exception; the paper notes dataset issues and a 65.1% majority-voting baseline.
- GLUE Main Results: MT-DNN without fine-tuning outperforms BERTLARGE on all GLUE tasks except CoLA, while CoLA accuracy rises from 58.9% to 62.5% after fine-tuning.Fine-tuned MT-DNN also exceeds fine-tuned BERTLARGE on CoLA, 62.5% versus 60.5%.
- Domain Adaptation Results: With 0.1% of SNLI training data, MT-DNN reaches 82.1% accuracy versus BERT’s 52.5%; with 1%, the results are 85.2% and 78.1%.The paper reports similar results on SciTail and larger MT-DNN improvements with fewer training examples.
- Domain Adaptation Results: MT-DNNLARGE reaches 91.6% on SNLI and 95.0% on SciTail, corresponding to absolute improvements of 1.5% and 6.7%.These are reported as new state-of-the-art results on both datasets.
5 Conclusion
MT-DNN combines multi-task learning with language-model pre-training for language representation learning, achieving strong results across ten NLU tasks and demonstrating domain-adaptation generalization. Future work includes understanding task sharing, improving training, incorporating linguistic structure, and testing adversarial robustness.
- MT-DNN combines multi-task learning and language-model pre-training for language representation learning.
- MT-DNN achieves new state-of-the-art results on ten NLU tasks across SNLI, SciTail, and GLUE.
- MT-DNN demonstrates generalization capability in domain-adaptation experiments.
- Future work includes analyzing MTL structure sharing, improving related-task training, incorporating linguistic structure explicitly, and testing resilience against adversarial attacks.