Source-linked AI summary
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
TL;DR
The paper addresses how to pre-train one language model for both autoencoding and partially autoregressive objectives. It introduces PMLM, which jointly trains bidirectional and sequence-to-sequence models while reusing context encodings, and reports improved performance across diverse understanding and generation benchmarks.
Problem
The paper seeks a unified pre-training approach that supports both autoencoding language understanding and partially autoregressive language generation.
Method
PMLM jointly trains AE and PAR objectives with shared parameters, pseudo masks, position embeddings, and self-attention masks that reuse context encodings in one forward pass.
Results
Unified PMLM pre-training improves performance on a wide range of natural language understanding and generation benchmarks.
Takeaways & Limitations
PMLM provides one computationally efficient framework for pre-training models that serve both language understanding and generation tasks.
Abstract
from arXiv · showhide
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM). Given an input text with masked tokens, we rely on conventional masks to learn inter-relations between corrupted tokens and context via autoencoding, and pseudo masks to learn intra-relations between masked spans via partially autoregressive modeling. With well-designed position embeddings and self-attention masks, the context encodings are reused to avoid redundant computation. Moreover, conventional masks used for autoencoding provide global masking information, so that all the position embeddings are accessible in partially autoregressive language modeling. In addition, the two tasks pre-train a unified language model as a bidirectional encoder and a sequence-to-sequence decoder, respectively. Our experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks across several widely used benchmarks.
1. Introduction
UniLMv2 introduces PMLM, a unified pre-training procedure that jointly learns autoencoding and partially autoregressive language models for understanding and generation. It reuses context encodings and reports improved performance across varied benchmarks.
- 1. Introduction: PMLM jointly pre-trains autoencoding and partially autoregressive language models within one unified framework.The model learns inter-relations between masked tokens and context through AE, and intra-relations between masked spans through PAR.
- 1. Introduction: The shared model supports both bidirectional encoding for language understanding and sequence-to-sequence decoding for language generation.AE pre-trains the bidirectional encoder, while PAR pre-trains the sequence-to-sequence decoder.
- 1. Introduction: Well-designed position embeddings and self-attention masks let AE and PAR run in one forward pass without redundant context computation.The framework reuses encoding results from the given context tokens across the two objectives.
- 1. Introduction: Conventional AE masks provide global masking information, allowing every PAR factorization step to access all position embeddings.This design supports the partially autoregressive objective while preserving access to positional information during fine-tuning.
- 1. Introduction: PMLM pre-training improves performance across a wide range of natural language understanding and generation benchmarks.The authors pre-train on large-scale text corpora and fine-tune on varied downstream tasks.
2. Preliminary
The preliminary model is a stacked Transformer that converts token embeddings into contextualized representations. Its self-attention mask controls which token pairs can attend to one another.
- 2. Preliminary: The model packs input embeddings into H0 and applies L stacked Transformer blocks to compute contextualized representations.The final hidden vectors HL represent the input tokens contextually.
- 2. Preliminary: Within each Transformer block, multiple self-attention heads aggregate previous-layer outputs before a feed-forward network processes them.The resulting hidden vectors form the contextualized token representations.
- 2. Preliminary: The self-attention mask matrix M controls whether pairs of tokens can attend to each other.Each attention head projects the previous layer into queries, keys, and values before applying the mask.
- 2. Preliminary: Input token representations combine token, absolute position, and segment embeddings in the format “[SOS] S1 [EOS] S2 [EOS]”.The two segments are contiguous texts, and each text ends with a special [EOS] token.
3. Unified Language Model Pre-Training
PMLM jointly pre-trains autoencoding and partially autoregressive language models by reusing shared encodings while controlling token context with pseudo masks and self-attention masks. This unified setup learns relations between masked tokens and context, as well as among masked spans, for both understanding and generation.
- Unified pre-training: PMLM jointly pre-trains autoencoding and partially autoregressive language models using the same input text and masked positions.The two objectives share model parameters and reuse encoding results from the same example.
- Pre-training tasks: Autoencoding independently predicts masked tokens from context, whereas partially autoregressive modeling factorizes predictions across masked tokens or spans.The masked positions remain the same, but their probability factorization differs between the two objectives.
- Pseudo masking: PMLM appends pseudo masks with the corresponding tokens’ position embeddings and uses their hidden states for partially autoregressive predictions.Original tokens remain in the input, while pseudo tokens serve as prediction placeholders.
- Attention control: Self-attention masks prevent explicit leakage from a pseudo token to its original token and implicit leakage through multi-step attention propagation.Tokens predicted in future factorization steps are masked from each token’s accessible context.
- Unified objective: The training loss combines the autoencoding and partially autoregressive objectives as L = LAE + LPAR.The two objectives capture inter-relations between context and masked tokens and intra-relations among masked tokens.
4. Experimental Results
UNILMv2 was evaluated across language understanding and generation tasks, including question answering, GLUE, summarization, question generation, and pre-training-objective ablations. It improved performance across several benchmark comparisons, with AE+PAR performing best among the tested objectives.
- Question Answering: UNILMv2BASE outperformed other BASE-size models on both SQuAD datasets.The reported metrics were F1 and exact match (EM).
- GLUE Benchmark: UNILMv2BASE outperformed BERTBASE and XLNetBASE across all 8 GLUE tasks and achieved the best performance on 6 of 8 tasks against RoBERTaBASE.On MNLI, it scored 88.4 accuracy versus 87.6 for RoBERTaBASE.
- Abstractive Summarization: Pre-training produced significant improvements over non-pretrained baselines on XSum and CNN/DailyMail summarization.The evaluation used F1 ROUGE scores.
- Abstractive Summarization: UNILMv2BASE outperformed other BASE-size pre-trained models on both summarization datasets despite having the smallest size among the compared models.The comparison included BERTSUMABS, MASS, and T5.
- Question Generation: UNILMv2BASE achieved better evaluation results than the compared methods on question generation.The reported metrics were BLEU, METEOR, and ROUGE across two data splits.
- Effect of Pre-Training Objectives: AE+PAR performed best among five pre-training objectives, supporting complementarity between autoencoding and partially autoregressive modeling.PAR-only performed favorably on MNLI and SST-2, while AE-only and PAR-only both outperformed AR-only on SQuAD.
5. Conclusion
The paper presents PMLM as an efficient unified pre-training procedure for language understanding and generation. It reports improved end-task results across several benchmarks.
- Conclusion: PMLM jointly pre-trains a bidirectional language model via AE and a sequence-to-sequence language model via PAR.The two modeling tasks target language understanding and generation, respectively.
- Conclusion: AE and PAR can be computed in one forward pass without redundant computation.PMLM reuses context encodings while conventional AE masks provide global masking information to PAR.
- Conclusion: PMLM learns inter-relations between masked tokens and context through AE and intra-relations between masked spans through PAR.The conclusion also states that PAR encourages long-distance dependencies by preventing local shortcuts.
- Conclusion: Experimental results show that PMLM improves end-task results on several language understanding and generation benchmarks.
A. Hyperparameters for Pre-Training
UNILMv2BASE uses the BERTBASE vocabulary and model size while following RoBERTaBASE optimization hyperparameters for comparison.
- A. Hyperparameters for Pre-Training: UNILMv2BASE uses the same WordPiece vocabulary and model size as BERTBASE.
- A. Hyperparameters for Pre-Training: The pre-training optimization hyperparameters follow RoBERTaBASE for comparisons.
B. GLUE Benchmark
The GLUE benchmark comprises multiple language understanding tasks spanning acceptability, sentiment, similarity, inference, question answering, and paraphrase detection.
- B. GLUE Benchmark: GLUE includes single-sentence acceptability and sentiment-classification tasks.These are CoLA and SST-2.
- B. GLUE Benchmark: GLUE includes a text-similarity regression task and pairwise classification tasks.The pairwise tasks include natural language inference, question answering, and paraphrase detection.
C. Hyperparameters for NLU Fine-Tuning
The section lists fine-tuning hyperparameters for UNILMv2BASE on SQuAD and GLUE, with settings selected on development sets using averaged five-run performance.
- UNILMv2BASE is fine-tuned on SQuAD v1.1, SQuAD v2.0, and the GLUE benchmark.
- Hyperparameters are searched on development sets according to the average performance of five runs.
- The same hyperparameters are used for both SQuAD question answering datasets.
- Table 9 lists the hyperparameters used for fine-tuning on SQuAD and GLUE datasets.
D. Hyperparameters for NLG Fine-Tuning
The section presents fine-tuning and decoding hyperparameters for CNN/DailyMail, XSum, and question generation, including sequence lengths, output limits, and label smoothing.
- Total length is 512 for question generation and 768 for CNN/DailyMail and XSum.
- Maximum output length is 160 for CNN/DailyMail and 48 for XSum and question generation.
- The label smoothing rate is 0.1, and decoding uses beam search.
- Table 10 lists fine-tuning and decoding hyperparameters for CNN/DailyMail, XSum, and question generation.