PaperlayerView exact revision on arXiv

Source-linked AI summary

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, Jie Tang

arXiv:2103.10360v2cs.CLcs.AIcs.LG

TL;DR

Existing pretraining frameworks do not perform best across NLU, unconditional generation, and conditional generation. GLM addresses this gap with autoregressive blank infilling enhanced by span shuffling and 2D positional encoding, and varying blank structure for different tasks. Across these task types, GLM outperforms BERT, T5, and GPT under matched model-size and data settings, with a single model achieving its best performance at 1.25× BERT Large’s parameters.

  • Problem

    Existing autoregressive, autoencoding, and encoder-decoder frameworks are not uniformly competitive across NLU, unconditional generation, and conditional generation.

  • Method

    GLM pretrains by autoregressively reconstructing randomly blanked spans, using span shuffling, 2D positional encoding, and task-specific blank numbers and lengths.

  • Results

    GLM outperforms BERT, T5, and GPT across NLU and generation tasks under matched model-size and data settings, with best single-model performance at 1.25× BERT Large’s parameters.

  • Takeaways & Limitations

    A single GLM can share parameters across NLU, conditional generation, and unconditional generation while outperforming standalone baselines.

Abstract

from arXiv · show

There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25x parameters of BERT Large , demonstrating its generalizability to different downstream tasks.

1 Introduction

Existing pretraining architectures each favor different NLP capabilities, leaving no single framework consistently competitive across understanding and generation. GLM combines autoregressive blank infilling with span shuffling and 2D positional encoding, improving performance across these task types.

  • Existing pretraining frameworks: Autoregressive models support long-text generation but their unidirectional attention limits context dependencies for NLU tasks.Autoencoding models provide bidirectional representations for NLU but are not directly applicable to text generation, while encoder-decoder models are typically used for conditional generation.
  • The pretraining gap: None of the existing pretraining frameworks is flexible enough to perform competitively across all NLP tasks, and simple objective combination cannot fully inherit autoregressive and autoencoding advantages.The two objectives differ by nature, limiting straightforward multi-task unification.
  • GLM framework: GLM randomly blanks continuous token spans and sequentially reconstructs them autoregressively, combining autoencoding-style corruption with autoregressive prediction.Its proposed improvements are span shuffling and 2D positional encoding.
  • Reported results: GLM significantly outperforms BERT on SuperGLUE by 4.6%–5.0% under matched parameters and computational cost, and outperforms T5 on NLU and generation with fewer parameters and data.The reported comparisons use a similarly sized pretraining corpus for RoBERTa and BART comparisons.
  • GLM framework: GLM naturally handles multi-token cloze answers through autoregressive blank filling, unlike BERT-based PET models.NLU tasks are reformulated as manually crafted cloze questions that mimic human language.
  • Multi-task generalization: Varying the number and lengths of missing spans lets one autoregressive blank-filling objective support conditional and unconditional generation.Multi-task pretraining allows a single GLM to share parameters across NLU, conditional generation, and language modeling tasks.

2 GLM Pretraining Framework

GLM unifies blank infilling and autoregressive generation by masking spans, reconstructing them sequentially in permuted order, and using 2D positional encodings. It extends this framework across NLU, conditional generation, and unconditional generation by varying span structure and reformulating classification as cloze completion.

  • 2.1.1 Autoregressive Blank Infilling: GLM masks consecutive token spans and autoregressively reconstructs their missing tokens from corrupted text and previously predicted spans.The spans are randomly permuted to capture interdependencies between them.
  • 2.1.1 Autoregressive Blank Infilling: Part A contains corrupted text, while Part B contains masked spans generated with attention to Part A and preceding tokens in Part B.Special [START] and [END] tokens support autoregressive span generation.
  • 2.1.2 Multi-Task Pretraining: GLM varies the number and lengths of missing spans to support both NLU and text generation in a multi-task pretraining setup.Document-level masking targets long text generation, while sentence-level masking targets sequence-to-sequence tasks.
  • 2.2.1 2D Positional Encoding: 2D positional encodings represent each token’s corrupted-text position and intra-span position using two learned embedding tables.This design avoids exposing masked-span length during reconstruction, fitting tasks whose output length is unknown beforehand.
  • 2.3 Finetuning GLM: NLU classification is reformulated as natural-language cloze completion, with candidate labels mapped to verbalizer answers and optimized using cross-entropy.This allows multi-token answers through autoregressive blank filling.
  • 2.3 Finetuning GLM: GLM applies the same pretrained framework to unconditional generation and downstream conditional generation by generating Part B autoregressively.The context forms Part A and a terminal mask initiates Part B generation.

3 Experiments

Experiments evaluate GLM across NLU, conditional and unconditional generation, text infilling, and ablations against similarly sized pretrained baselines. GLM generally improves over BERT and GPT, while performance varies across multitask objectives and model sizes.

  • SuperGLUE: GLMBase scores 4.6% higher than BERTBase on average, while GLMLarge scores 5.0% higher than BERTLarge on SuperGLUE.GLM consistently outperforms BERT on most tasks, with WiC as the only exception.
  • Multi-Task Pretraining: GLMDoc and GLMSent slightly trail GLMLarge on SuperGLUE but still outperform BERTLarge and UniLMLarge.Among multitask models, GLMSent outperforms GLMDoc by 1.1% on average.
  • Multi-Task Pretraining: Increasing GLMDoc to 410M parameters produces better SuperGLUE performance than GLMLarge, while the 515M model performs better still.These sizes correspond to 1.25× and 1.5× BERTLarge, respectively.
  • Sequence-to-Sequence: GLMSent can outperform GLMLarge on sequence-to-sequence tasks, whereas GLMDoc performs slightly worse; 410M GLMDoc achieves the best performance on both tasks.The document-level objective is described as less helpful for conditional generation because it teaches context extension rather than information extraction.
  • Text Infilling: GLM outperforms previous methods by 1.3 to 3.9 BLEU on Yahoo text infilling and achieves the state-of-the-art result.GLMDoc slightly underperforms GLMLarge, consistent with the sequence-to-sequence experiments.
  • Language Modeling: GLMDoc performs worse than GPTLarge at equal size in zero-shot language modeling, but GLM410M with bidirectional context encoding outperforms GPTLarge.GLM515M also outperforms GPTLarge, and removing 2D positional encoding lowers accuracy and raises perplexity.
  • Ablation Study: Removing span shuffling severely degrades GLM performance, while autoregressive pretraining particularly helps multi-token verbalizers on ReCoRD and WSC.Cloze-style finetuning improves large-model performance by 7 points.
  • Ablation Study: GLM uses a single encoder, shuffled masked spans, and a single [MASK], differing from T5’s encoder-decoder design, fixed span order, and sentinel tokens.The reported comparisons indicate GLM’s advantage over T5 despite differences in training data and parameter counts.

4 Related Work

Prior work organizes pretrained language models into autoencoding, autoregressive, and encoder-decoder families, each targeting different capabilities. Efforts to bridge these families include text-to-text modeling, unified masking, and blank language modeling for NLU and generation.

  • Autoencoding models learn bidirectional representations for NLU, autoregressive models learn left-to-right language modeling, and encoder-decoder models target sequence-to-sequence tasks.
  • BART uses encoder and decoder representations for NLU, whereas T5 formulates most language tasks in a text-to-text framework.Both require more parameters than autoencoding models such as RoBERTa to outperform them.
  • UniLM attempts to unify the three pretraining model families under a masked language modeling approach.
  • NLU as Generation: Generative models can solve some NLU tasks by directly predicting answers, but their unidirectional attention limits parameter efficiency.PET instead reformulates NLU tasks as cloze questions, extending generation-based task formulation.
  • Blank Language Modeling: Blank infilling has been studied previously, but this work differs by pretraining with blank infilling and evaluating both downstream NLU and generation tasks.

5 Conclusions

GLM is presented as a unified pretraining framework for NLU and generation. It uses autoregressive blank infilling with mixed attention masks and 2D position encodings, and reports stronger NLU performance with parameter sharing across tasks.

  • GLM unifies pretraining objectives for NLU and generation through autoregressive blank infilling, mixed attention masks, and 2D position encodings.
  • GLM formulates NLU tasks as conditional generation tasks, allowing autoregressive models to solve them.
  • GLM outperforms previous methods on NLU tasks and effectively shares parameters across different tasks.

A.1 Datasets

The GLMBase and GLMLarge models use the BookCorpus and Wikipedia datasets used by BERT. GLMRoBERTa follows RoBERTa’s broader pretraining dataset mixture.

  • GLMBase and GLMLarge are trained on BookCorpus and Wikipedia, matching the datasets used by BERT.

A.2 Hyperparameters

The appendix reports pretraining hyperparameters and resource-driven training choices for GLM variants. GLMRoBERTa receives fewer training steps than RoBERTa and BART, with additional scheduling and optimization differences.

  • GLMBase and GLMLarge use BERT-like hyperparameters, while GLMLarge uses batch size 1024 and 200,000 training steps for speed and comparison.GLMBase uses 120,000 training steps; GLMDoc and GLMSent use GLMLarge’s hyperparameters.
  • GLMRoBERTa is pretrained for 250,000 steps, half the training steps of RoBERTa and BART and close to T5 in trained tokens.
  • GLMRoBERTa uses cosine rather than linear learning-rate decay and applies gradient clipping with value 1.0.
  • The hyperparameters for all pretraining settings are summarized in Table 7.

A.3 Implementation

The implementation builds on Megatron-LM and DeepSpeed, with code provided in the supplementary material but pretrained models unavailable there because of space limits.

  • GLM’s pretraining implementation is based on Megatron-LM and DeepSpeed.
  • The authors include the implementation code in the supplementary material.
  • Pretrained models are not included because of the supplementary material’s size limit, with future public release planned.

B Downstream Tasks

GLM is evaluated across SuperGLUE, summarization, and question generation using blank-infilling or sequence-to-sequence formulations and standard benchmark metrics.

  • SuperGLUE: SuperGLUE’s 8 NLU tasks are formulated as blank-infilling tasks with task-specific cloze questions and verbalizers.Three tasks allow multi-token answers, while the other five use single-token answers.
  • SuperGLUE: For SuperGLUE, GLM scores answer candidates using verbalizer logits for single-token tasks and summed verbalizer log-probabilities for multi-token tasks.
  • SuperGLUE: Baseline classifiers concatenate task inputs and classify from the [CLS] representation, while cloze-style finetuning was also implemented for other pretrained models.The cloze-style performance was usually similar to standard classification in the reported ablation study.
  • Evaluation: Summarization uses Rouge-1, Rouge-2, and Rouge-L F1 scores, whereas question generation uses BLEU-1 through BLEU-4.
  • Baselines: T5Large results on XSum come from the Hugging Face Transformers summarization script, while other sequence-to-sequence baseline results come from the corresponding papers.

B.3 Text Infilling

The text-infilling evaluation masks contiguous token spans in Yahoo Answers documents and measures reconstruction against the original text, alongside language-modeling evaluations on BookWiki and LAMBDA.

  • Text Infilling: Yahoo Answers contains 100K/10K/10K training, validation, and test documents, averaging 78 words each.
  • Text Infilling: Text infilling masks 10%–50% of each document’s tokens, collapses contiguous masked tokens into one blank, and uses dynamic masking during finetuning.
  • Text Infilling: Infilled text is evaluated against the original document with BLEU and compared against BERT and BLM baselines.BLM fills blanks with arbitrary trajectories, while BERT generates masked tokens left to right from the blank representation.
  • Language Modeling: Language modeling is evaluated with perplexity on BookWiki and accuracy on the LAMBDA dataset.
  • Language Modeling: Perplexity is defined as the exponentiation of a corpus’s average cross entropy.
  • Language Modeling: Overlapping evaluation advances fixed-size transformer windows by an overlap and computes loss only for the final overlap tokens, using o = 256.
  • Language Modeling: LAMBDA tests long-range dependency modeling by requiring the missing final word of a 4–5 sentence passage.A prediction is correct only when all WordPiece tokens of the word are correct.

C Results on Other NLU Benchmarks

On GLUE and SQuAD, GLM continues to outperform BERT with the same parameter count, although the margin is smaller than on the paper’s other NLU results.

  • GLUE covers single-sentence and sentence-pair NLU tasks, while SQuAD evaluates extractive question answering.
  • GLM outperforms BERT on both GLUE and SQuAD with the same number of parameters, but by a smaller margin.

D Text Generation Samples

This section presents GLMDoc text-generation samples from randomly selected test contexts, alongside generation settings and references to benchmark result tables. The examples illustrate continuation-style outputs for factual prompts, though some texts are cut short.

  • Generation setup: GLMDoc generates continuations for randomly sampled test-set contexts, using top-k sampling with k = 40 and a maximum sequence length of 512.Some generated texts are cut short.
  • Generation samples: Example D.1 prompts a continuation about the Wyoming State Legislature, followed by generated factual text about its composition and elections.
  • Generation samples: Example D.2 prompts a continuation describing Jonathan Terry as an actor, followed by generated text listing voice-acting and television roles.
  • Evaluation references: The section also points readers to GLUE and SQuAD development-set result tables.
  • Generation samples: Example D.3 begins with a factual prompt about Corona station, followed by generated historical details about the station and related railway service.
Loading 2103.10360v2…