Source-linked AI summary
Unified Language Model Pre-training for Natural Language Understanding and Generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
TL;DR
Existing pre-trained language models use different context structures, with BERT’s bidirectionality limiting its application to generation. UNILM addresses this by sharing one Transformer across bidirectional, unidirectional, and sequence-to-sequence objectives with specialized attention masks. It compares favorably with BERT on understanding tasks and achieves state-of-the-art results on five natural language generation datasets.
Problem
BERT improves natural language understanding, but its bidirectional design makes it difficult to apply to natural language generation.
Method
UNILM jointly pre-trains a shared Transformer with bidirectional, unidirectional, and sequence-to-sequence language modeling objectives controlled by self-attention masks.
Results
UNILM compares favorably with BERT on GLUE and two question answering datasets and achieves new state-of-the-art results on five natural language generation datasets.
Takeaways & Limitations
A single jointly pre-trained model can be fine-tuned for both natural language understanding and generation tasks.
Takeaways & Limitations
The experiments focus on monolingual NLP tasks, while cross-lingual extension remains future work.
Abstract
from arXiv · showhide
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UniLM achieves new state-of-the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm.
1 Introduction
UNILM unifies bidirectional, unidirectional, and sequence-to-sequence language modeling in one pre-trained Transformer for both language understanding and generation. It compares favorably with BERT on understanding benchmarks and achieves new state-of-the-art results across five generation datasets.
- BERT’s bidirectional design improves language understanding but makes natural language generation difficult.
- UNILM jointly pre-trains one Transformer on three language modeling objectives and supports both NLU and NLG tasks.
- Shared parameters provide one architecture for multiple language models and jointly optimize representations across different context usage patterns.
- UNILM compares favorably with BERT on GLUE, SQuAD 2.0, and CoQA extractive question answering.
- UNILM achieves new state-of-the-art results on CNN/DailyMail and Gigaword summarization, SQuAD question generation, CoQA generative question answering, and DSTC7 response generation.
2 Unified Language Model Pre-training
UNILM uses a shared Transformer with task-specific self-attention masks to pre-train bidirectional, unidirectional, and sequence-to-sequence language models. The resulting model can be fine-tuned as an encoder for understanding or as an encoder-decoder for conditional generation.
- UNILM computes contextualized token representations with a shared multi-layer Transformer optimized for three unsupervised language modeling objectives.
- Different self-attention masks control which contextual tokens each prediction can access.The mask matrix determines whether token pairs can attend to one another.
- Cloze training masks randomly selected WordPiece tokens and learns to recover the original tokens using a shared prediction procedure.
- Bidirectional language modeling allows every token to attend across the input, while unidirectional modeling restricts attention to past or future context.
- Sequence-to-sequence modeling lets source tokens attend bidirectionally within the source and target tokens attend to the source plus earlier target tokens.
- The sequence-to-sequence objective jointly pre-trains a bidirectional encoder and unidirectional decoder, supporting conditional generation such as abstractive summarization.
- The combined objective samples bidirectional and sequence-to-sequence training one-third of the time each, with left-to-right and right-to-left objectives sampled one-sixth each.
- For downstream tasks, UNILM is fine-tuned as a bidirectional encoder for NLU and with sequence-to-sequence masks for NLG.
3 Experiments
UNILM is evaluated across natural language understanding and generation tasks, including summarization, question answering, question generation, and related benchmarks. It compares favorably with BERT on understanding tasks and achieves strong or state-of-the-art generation results across multiple datasets.
- Evaluation scope: UNILM is evaluated on both NLU and NLG tasks, spanning GLUE, extractive and generative QA, summarization, question generation, and dialog response generation.The experiments cover the benchmark families described across the paper’s evaluation sections.
- Abstractive summarization: UNILM outperforms previous abstractive systems on CNN/DailyMail and achieves better Gigaword performance than prior work.On CNN/DailyMail, it also exceeds the best reported extractive model by 0.88 point in ROUGE-L; in Gigaword’s 10K setting, it surpasses MASS by 7.08 points in ROUGE-L.
- Generative QA: 82.5 F1 is reported for UNILM on generative CoQA, substantially exceeding the Seq2Seq and PGNet baselines.The paper states that this wide margin significantly closes the gap between generative and extractive methods.
- Extractive QA: UNILM outperforms BERTLARGE on SQuAD 2.0 and CoQA extractive question answering under the reported evaluation settings.Both comparisons use the same fine-tuning setup as the corresponding BERTLARGE baseline.
- Question generation: UNILM achieves a new state-of-the-art result for question generation and its generated question-passage-answer examples improve a question answering model.The improvement from augmented data is reported with bidirectional masked language modeling as an auxiliary fine-tuning task.
- GLUE: UNILM obtains comparable performance to BERTLARGE across the GLUE tasks.The GLUE evaluation covers nine language understanding tasks, including sentiment analysis, paraphrase detection, and natural language inference.
4 Conclusion and Future Work
UNILM unifies multiple language-modeling objectives with shared parameters, enabling fine-tuning for both NLU and NLG tasks. The authors report favorable NLU comparisons and new state-of-the-art results across five NLG datasets, while identifying larger-scale, cross-lingual, and multitask extensions.
- UNILM jointly optimizes bidirectional, unidirectional, and sequence-to-sequence language-modeling objectives with shared parameters.
- The unified objectives enable straightforward fine-tuning of one pre-trained model for both NLU and NLG tasks.
- UNILM compares favorably with BERT on GLUE and two question-answering datasets.
- UNILM outperforms previous state-of-the-art models on five NLG datasets spanning summarization, question generation, generative question answering, and dialog response generation.
- Future work includes larger models and more training on web-scale corpora, cross-lingual tasks, and multitask fine-tuning across NLU and NLG.
Appendix A Long Text Generation: A Case Study
The case study examines left-to-right text generation with UNILM. Samples were selected from repeated top-40-truncated sampling, so they represent hand-picked outputs rather than average model generations.
- UNILM generates text samples using a left-to-right setting with top-40 truncating sampling and duplicate 4-gram blocking.
- The authors sampled each input ten times and hand-picked the best output, making the examples better than average model outputs.
Appendix B GLUE Benchmark
The appendix summarizes the GLUE benchmark and presents its test-result and benchmark-summary tables. The supplied passages provide the table descriptions but not the underlying scores or comparisons.
- One appendix table contains text samples generated by UNILM using left-to-right generation.
- Another appendix table summarizes the GLUE benchmark.