PaperlayerView exact revision on arXiv

Source-linked AI summary

MPNet: Masked and Permuted Pre-training for Language Understanding

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu

arXiv:2004.09297v2cs.CLcs.LG

TL;DR

The paper addresses limitations in BERT’s masked language modeling and XLNet’s permuted language modeling. MPNet combines predicted-token dependency modeling with auxiliary full-sentence position information, and outperforms these methods and prior pre-trained models across several language-understanding tasks.

  • Problem

    BERT’s MLM ignores dependencies among predicted tokens, while XLNet’s PLM lacks full-sentence position information during pre-training, creating a discrepancy with fine-tuning.

  • Method

    MPNet combines permuted language modeling with auxiliary position information so the model can use dependencies among predicted tokens and see the full sentence.

  • Results

    MPNet outperforms MLM and PLM and surpasses BERT, XLNet, and RoBERTa by 4.8, 3.4, and 1.5 points respectively on GLUE dev sets.

  • Takeaways & Limitations

    MPNet provides a pre-training method with less discrepancy from downstream language-understanding tasks by conditioning on more token and position information.

  • Takeaways & Limitations

    Pre-trained models generally have large model sizes and high training costs, making them expensive.

Abstract

from arXiv · show

BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models. Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for pre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and thus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel pre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the dependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position information as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in XLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of down-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large margin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g., BERT, XLNet, RoBERTa) under the same model setting. The code and the pre-trained models are available at: https://github.com/microsoft/MPNet.

1 Introduction

BERT’s masked language modeling overlooks dependencies among predicted tokens, while XLNet’s permuted language modeling lacks full-sentence position information. MPNet addresses both issues and outperforms these methods and earlier pre-trained models across evaluated benchmarks.

  • BERT’s masked language modeling efficiently uses bidirectional context but ignores dependencies among masked tokens.
  • XLNet introduces permuted language modeling to capture dependencies among predicted tokens, but lacks full-sentence position information during pre-training.
  • MPNet unifies MLM and PLM by modeling dependencies among predicted tokens and supplying position information for all tokens.
  • MPNet was pre-trained on over 160GB of text and fine-tuned on GLUE, SQuAD, RACE, and IMDB.
  • MPNet outperformed BERT, XLNet, and RoBERTa by 4.8, 3.4, and 1.5 points respectively on GLUE dev sets under the same model setting.

2 MPNet

MPNet unifies masked and permuted language modeling to model dependencies among predicted tokens while preserving full-sentence position information. Its two-stream attention and position compensation address the respective limitations of MLM and PLM.

  • Background: MLM predicts masked tokens independently, whereas PLM models dependency among predicted tokens but lacks full-sentence position information during pre-training.This creates a trade-off between output dependency modeling and consistency between pre-training and fine-tuning.
  • A Unified View of MLM and PLM: MPNet unifies MLM and PLM by splitting a permuted sequence into non-predicted and predicted parts.Under this view, MLM and PLM share a similar formulation but differ in their conditional information.
  • Our Proposed Method: MPNet conditions on preceding predicted tokens and additional mask and position information, combining dependency modeling with fuller sentence context.Its objective conditions on x_z<t and incorporates mask symbols for later predicted positions.
  • Our Proposed Method: Two-stream self-attention lets the query stream predict permuted tokens using prior tokens and positions without seeing the current token.The content stream can see previous and current tokens, while the query stream supports autoregressive prediction of the permuted predicted part.
  • Our Proposed Method: Position compensation designs query and content attention masks so each prediction can access n tokens and the original sequence's full position information.This reduces the input inconsistency between pre-training and downstream fine-tuning.
  • Discussions: MPNet conditions on more information than BERT and XLNet while minimizing discrepancy with downstream tasks that expose all token and position information.The paper identifies this fuller conditioning as MPNet's main advantage over the two baselines.

3 Experiments and Results

Experiments evaluate MPNet under BERTBASE settings across GLUE, SQuAD, RACE, and IMDB, including controlled ablations. MPNet consistently outperforms earlier pre-training methods, while ablations support the roles of position compensation and permutation.

  • GLUE Benchmark: MPNet outperforms BERT, XLNet, and RoBERTa by 4.8, 3.4, and 1.5 points on average on GLUE dev tasks.On GLUE test tasks, it also outperforms ELECTRA by 0.7 point on average.
  • Question Answering: MPNet outperforms BERT, XLNet, and RoBERTa by a large margin on both SQuAD v1.1 and v2.0.Results use BERTBASE single models without data augmentation for fair comparison.
  • RACE: MPNet greatly outperforms BERT and XLNet across the three RACE metrics under matched 16GB pre-training data.Using 160GB rather than 16GB yields an additional 5.7 points, from 70.4 to 76.1.
  • IMDB: MPNet trained on 16GB data outperforms BERT and XLNet by 0.6 and 0.1 point on IMDB, while 160GB data adds 0.4 point.IMDB evaluates binary sentiment classification on over 50,000 movie reviews.
  • Ablation Study: Removing position compensation reduces downstream accuracy by 0.6-2.3 points, causing MPNet to degenerate to PLM.This ablation supports the contribution of position compensation over PLM.
  • Ablation Study: Removing permutation while retaining output dependency reduces accuracy by 0.5-1.7 points, whereas removing both permutation and output dependency reduces it by 0.5-3.7 points.The latter configuration degenerates to MLM.

4 Conclusion

MPNet is presented as a pre-training method addressing limitations of MLM in BERT and PLM in XLNet. Experiments across varied tasks report that it outperforms both objectives and strong pre-trained baselines.

  • Conclusion: MPNet models dependencies among predicted tokens through permuted language modeling and adds auxiliary position information to reduce pre-training–fine-tuning discrepancy.The method is designed to address the problems of MLM in BERT and PLM in XLNet.
  • Conclusion: Experiments on various tasks show that MPNet outperforms MLM, PLM, BERT, XLNet, and RoBERTa by a large margin.The paper proposes extending MPNet to other advanced structures and more diverse language-understanding tasks.

Broader Impact

MPNet may inspire exploration of more advanced pre-training methods, while the large size and training cost of pre-trained models remain important negative impacts.

  • MPNet may encourage researchers to rethink MLM and PLM and explore more advanced pre-training methods.
  • Pre-trained models can be expensive for research and product use because of their large model size and training cost.
  • The paper identifies developing more lightweight and low-cost pre-trained models as future work.

A Pre-training Hyper-parameters

The section reports separate pre-training and fine-tuning hyper-parameter tables, with a RoBERTaLARGE initialization and a smaller continual-training learning rate for BERTLARGE.

  • Pre-training hyper-parameters for the BERTBASE and BERTLARGE settings are reported in Table 7.
  • For BERTLARGE, MPNet is initialized from RoBERTaLARGE to save computation, with relative position embedding and whole word masking disabled for consistency.
  • Continual training uses a smaller learning rate of 5e-5 because the model has already been well optimized.
  • Fine-tuning hyper-parameters for RACE, SQuAD, and GLUE are reported in Table 8.

C MPNet on Large Setting

On the BERTLARGE setting, MPNet is initialized from RoBERTaLARGE to reduce computation and outperforms RoBERTa by 0.5 points on average after 100K pre-training steps.

  • MPNet uses RoBERTaLARGE initialization on the large setting to save computation.
  • 0.5 points: MPNet outperforms RoBERTa on average on the GLUE development set in the BERTLARGE setting.
  • The reported model was pre-trained for only 100K steps, which the paper says cannot fully demonstrate MPNet’s advantages.
  • The authors were preparing a large-level MPNet model trained from scratch for a later update.

D Effect of MNLI initialization

The experiments examine whether MNLI initialization affects RTE, STS-B, and MRPC results, finding only slight performance costs on RTE and MRPC when it is removed.

  • The main fine-tuning setting uses MNLI initialization for RTE, STS-B, and MRPC to match RoBERTa’s setting.
  • Table 10 reports RTE, STS-B, and MRPC results without MNLI initialization, including the GLUE average score.
  • Removing MNLI initialization only slightly reduces performance on RTE and MRPC.
  • Without MNLI initialization, MPNet still outperforms ELECTRA on average score.

E Training Efficiency

MPNet’s training efficiency is compared with advanced approaches under the BERTBASE setting. Relative to XLNet and RoBERTa, MPNet achieves better performance with fewer computations.

  • MPNet achieves better performance with fewer computations than XLNet and RoBERTa.The comparison is conducted under the BERTBASE setting.
  • The efficiency analysis evaluates MPNet against other advanced approaches.
  • Table 11 compares training FLOPs across different methods under the BERTBASE setting.BERT uses 16GB of data, while the other methods use 160GB.

F More Ablation Studies

Additional ablations examine whole word masking and relative positional embedding in the BERTBASE setting. The study reports results on selected SQuAD and GLUE development tasks.

  • The ablation study analyzes whole word masking and relative positional embedding in the BERTBASE setting.
  • The experiments use Wikipedia and BooksCorpus data totaling 16GB for 1M pre-training steps.Training uses batches of 256 sentences, with each sentence containing up to 512 tokens.
  • Table 12 reports development-set results for SQuAD v1.1, SQuAD v2.0, MNLI, and SST-2.The listed tasks cover both SQuAD and GLUE evaluations.

G Training Speedup

MPNet speeds training by partitioning the attention matrix according to whether queries and key/value inputs come from predicted or non-predicted parts. Removing an unused matrix reduces content-stream computation.

  • The attention matrix is divided into four sub-matrices based on predicted and non-predicted query and key/value parts.Matrices A and B use queries from the non-predicted part, while C and D use queries from the predicted part; A and C use non-predicted key/value inputs, while B and D use predicted inputs.
  • Matrix B is unused during model training, so MPNet computes only matrices A, C, and D for the content stream.
  • Computing only matrices A, C, and D saves nearly 10% of total training computations.
  • Figure 4 shows the attention mask matrix for MPNet’s content stream and refers to Figure 2 for additional details.
Loading 2004.09297v2…