PaperlayerView exact revision on arXiv

Source-linked AI summary

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le

arXiv:1906.08237v2cs.CLcs.LG

TL;DR

Existing autoregressive pretraining struggles to model bidirectional context, while BERT’s masking neglects dependencies between masked positions and creates a pretrain-finetune discrepancy. XLNet uses permutation-based autoregressive pretraining with architectural improvements, and consistently outperforms BERT across diverse language-understanding tasks.

  • Problem

    Autoregressive pretraining lacks effective bidirectional context, while BERT neglects dependencies among masked tokens and introduces a pretrain-finetune discrepancy.

  • Method

    XLNet maximizes expected likelihood across factorization permutations and integrates Transformer-XL with two-stream attention for autoregressive pretraining.

  • Results

    XLNet consistently outperforms BERT across diverse tasks, including language understanding, reading comprehension, classification, and document ranking.

  • Takeaways & Limitations

    XLNet combines autoregressive and autoencoding advantages while substantially improving over previous pretraining objectives across varied language-understanding tasks.

  • Takeaways & Limitations

    The paper’s motivating limitations concern existing objectives, especially BERT’s pretrain-finetune discrepancy and independence assumption among predicted tokens.

Abstract

from arXiv · show

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

1 Introduction

XLNet is proposed to combine autoregressive language modeling with bidirectional context while avoiding BERT’s masking-related limitations. It uses permutation-based factorization and Transformer-XL components, and outperforms BERT across diverse language-understanding tasks under comparable settings.

  • AR language modeling captures only uni-directional context, whereas downstream language-understanding tasks often require bidirectional information.
  • BERT closes the bidirectional information gap through masked-token reconstruction but introduces a pretrain-finetune discrepancy and cannot model masked-token dependencies jointly with the product rule.
  • XLNet maximizes expected log likelihood over all factorization-order permutations, allowing each position to use contextual information from both left and right.
  • XLNet integrates Transformer-XL’s segment recurrence and relative encoding into pretraining, empirically improving performance especially on tasks involving longer text sequences.
  • Under comparable experiment settings, XLNet consistently outperforms BERT on GLUE, SQuAD, RACE, Yelp, IMDB, and ClueWeb09-B document ranking.

2 Proposed Method

XLNet proposes permutation language modeling, sharing parameters across autoregressive factorization orders to capture bidirectional contexts while avoiding BERT’s masked-token independence assumption and pretrain-finetune discrepancy. Its design preserves natural sequence order and adds target-aware two-stream attention, Transformer-XL memory reuse, and relative segment encodings.

  • Permutation Language Modeling: Because XLNet remains autoregressive, it avoids BERT’s independence assumption between masked tokens and its pretrain-finetune discrepancy.Unlike BERT’s separate reconstruction of masked tokens, the autoregressive objective factorizes likelihood using the universally valid product rule.
  • Permutation Language Modeling: Permutation language modeling samples among T! factorization orders while sharing parameters, enabling autoregressive learning from bidirectional contexts.The sequence order remains unchanged; only the factorization order is permuted through an appropriate Transformer attention mask.
  • Two-Stream Self-Attention: XLNet uses target-aware representations and two-stream self-attention so predictions depend on both the context and the target position.Standard Transformer representations do not depend on the predicted position, producing the same distribution regardless of target; the new representation explicitly receives that position.
  • Transformer-XL Integration: Transformer-XL memory can be cached and reused independently of the previous segment’s factorization order, allowing learning over all orders of the preceding segment.Positional encodings depend on original sequence positions, making the memory update independent of the prior permutation after representations are obtained.
  • Modeling Multiple Segments: For multiple input segments, XLNet concatenates two randomly sampled segments into one sequence for permutation language modeling and reuses memory only within the same context.This pretraining procedure follows BERT’s two-segment sampling while retaining the autoregressive framework.
  • Relative Segment Encodings: Relative segment encodings represent whether two positions belong to the same segment using learnable s+ and s− parameters for each attention head.This extends Transformer-XL’s relative encodings, whereas BERT adds absolute segment embeddings to word embeddings.

3 Experiments

Experiments show that XLNet outperforms BERT under matched training conditions and generally surpasses BERT and RoBERTa across diverse benchmarks. Ablations attribute its gains to permutation language modeling, the Transformer-XL backbone, memory caching, span-based prediction, and bidirectional input.

  • Fair comparison: Under the same data and hyperparameters, XLNet outperforms the best of three BERT variants by a sizable margin on all considered datasets.The comparison decouples improvements from the effects of using more pretraining data.
  • Comparison with state of the art: XLNet generally outperforms BERT and RoBERTa across reading comprehension, document ranking, question answering, text classification, and natural language understanding.The comparisons use similarly sized 24-layer architectures for BERT, RoBERTa, and XLNet.
  • Comparison with state of the art: XLNet’s performance gains are usually larger on explicit reasoning tasks involving longer contexts, while substantial gains remain on classification tasks with abundant supervised examples.The longer-context advantage may derive from XLNet’s Transformer-XL backbone; the cited classification datasets include MNLI, Yelp, and Amazon.
  • Ablation study: Ablations show that both Transformer-XL and permutation language modeling contribute to XLNet’s superior performance over BERT.The study compares XLNet variants, BERT-Base, and a Transformer-XL baseline using BERT’s denoising auto-encoding objective.
  • Ablation study: Removing memory caching clearly reduces performance, especially on RACE, while span-based prediction and the bidirectional input pipeline also play important roles.RACE involves the longest context among the four ablation tasks.

4 Conclusions

XLNet combines autoregressive and autoencoding pretraining advantages through permutation language modeling, supported by an architecture integrating Transformer-XL and two-stream attention, and improves performance across various tasks.

  • XLNet uses permutation language modeling to combine the advantages of autoregressive and autoencoding pretraining methods.
  • Its architecture integrates Transformer-XL and a carefully designed two-stream attention mechanism to work seamlessly with the autoregressive objective.
  • XLNet achieves substantial improvement over previous pretraining objectives across various tasks.

A Target-Aware Representation via Two-Stream Self-Attention · A.1 A Concrete Example of How Standard LM Parameterization Fails

The section presents a concrete example showing that standard language-model parameterization fails under the permutation objective. Specifically, different target positions can receive the same prediction even though their ground-truth distributions should differ.

  • A.1 A Concrete Example of How Standard LM Parameterization Fails: The section analyzes how standard language-model parameterization fails under the permutation objective.It frames the discussion as a concrete example related to the permutation objective.
  • A.1 A Concrete Example of How Standard LM Parameterization Fails: The example considers two different permutations, z(1) and z(2), satisfying a specified relationship.The supplied passage introduces the permutations but does not reproduce their relationship.
  • A.1 A Concrete Example of How Standard LM Parameterization Fails: The example links the failure to applying a naive parameterization across different permutation orders.The two permutations are substituted respectively into the naive parameterization, producing the shared-prediction issue.
  • A.1 A Concrete Example of How Standard LM Parameterization Fails: Substituting the two permutations into the naive parameterization yields the corresponding model predictions.The passage describes this substitution without including the resulting equations.
  • A.1 A Concrete Example of How Standard LM Parameterization Fails: Two different target positions, i and j, effectively share exactly the same model prediction.This is the concrete failure exposed by the example.
  • A.1 A Concrete Example of How Standard LM Parameterization Fails: The shared prediction conflicts with the fact that the ground-truth distributions of the two positions should be different.The example therefore identifies a mismatch between the model’s predictions and the target distributions.

A.2 Two-Stream Attention

The section details two-stream attention implemented with a Transformer-XL backbone, covering initialization, cached memory, layer updates, and target-aware prediction.

  • Implementation overview: Two-stream attention is implemented with a Transformer-XL backbone.The section introduces the implementation details before specifying its representations and prediction distribution.
  • Segment memory: The model caches the layer-m content representation from the previous segment as memory, written ˜h^(m).This cached representation supplies the memory carried across segments.
  • Layer updates: For Transformer-XL layers m = 1, · · · , M, relative positional attention and position-wise feed-forward operations consecutively update the representations.The layer updates use relative positional encoding followed by position-wise feed-forward processing.
  • Prediction: The implementation also specifies a target-aware prediction distribution.The passage identifies the prediction distribution as target-aware but gives no further formula here.

A.3 Datasets · A.3.1 RACE Dataset

RACE is a large, challenging reading-comprehension dataset built from English exams for Chinese middle- and high-school students. Its human-generated answers and unusually long passages make it distinct from datasets such as SQuAD.

  • A.3 Datasets: RACE contains near 100K questions from English exams for Chinese middle- and high-school students aged 12 to 18.
  • A.3.1 RACE Dataset: Human experts generated the answers for RACE questions.
  • A.3.1 RACE Dataset: RACE is described as one of the most difficult reading-comprehension datasets.Its questions involve challenging reasoning.
  • A.3.1 RACE Dataset: RACE passages have an average length exceeding 300.
  • A.3.1 RACE Dataset: RACE passages are significantly longer than those in other popular reading-comprehension datasets such as SQuAD.
  • A.3.1 RACE Dataset: The dataset’s questions are drawn from exams taken by Chinese students in middle and high school.

A.3.2 SQuAD

The SQuAD evaluation covers reading comprehension with answerable and unanswerable questions, using joint answerability classification and span extraction losses for SQuAD2.0 fine-tuning.

  • Dataset: SQuAD is a large-scale reading comprehension dataset with two tasks.The passage identifies SQuAD as a large-scale reading comprehension benchmark.
  • Dataset: SQuAD1.1 contains questions with corresponding answers in the provided passages.Every SQuAD1.1 question has an answer in its associated passage.
  • Dataset: SQuAD2.0 introduces unanswerable questions alongside answerable ones.Unlike SQuAD1.1, SQuAD2.0 includes questions without corresponding passage answers.
  • Fine-tuning: XLNet fine-tuning on SQuAD2.0 jointly uses logistic regression for answerability and standard span extraction for question answering.The answerability loss is similar to classification-task training, while span extraction uses a standard question-answering loss.

A.3.3 Text classification Datasets … A.7 Visualizing Memory and Permutation

The appendix specifies evaluation datasets and training hyperparameters, then analyzes XLNet’s advantages over BERT and standard autoregressive language modeling. It also examines XLNet’s distinctive attention patterns and visualizes permutation masking, memory reuse, and its two attention streams.

  • A.3.3 Text classification Datasets: XLNet is evaluated on IMDB, Yelp-2, Yelp-5, DBpedia, AG, Amazon-2, and Amazon-5 text-classification benchmarks.
  • A.3.4 GLUE Dataset: GLUE contains 9 natural language understanding tasks, with results reported for single-task, multi-task, single-model, and ensemble settings.
  • A.3.5 ClueWeb09-B Dataset: ClueWeb09-B evaluates document ranking by reranking the top 100 documents retrieved from 50M-document collections using TREC 2009–2012 Web Track queries.The dataset tests low-level representations and word-embedding quality because document ranking mainly concerns low-level representations rather than high-level semantics.
  • A.4 Hyperparameters: Pretraining and finetuning hyperparameters are specified in Tables 7 and 8, with finetuning using exponentially decayed layer-wise learning rates.If layer 24 uses learning rate l and decay rate α, layer m uses lα24−m.
  • A.5.1 Comparison with BERT: XLNet covers more target-context dependencies than BERT when context includes target tokens preceding a prediction in the factorization order.Both objectives maximize log p(T | N), but XLNet can provide training signals for dependencies that BERT cannot cover.
  • A.5.2 Comparison with Language Modeling: XLNet covers both directional dependencies in expectation over factorization orders, whereas standard autoregressive language modeling cannot cover dependencies that reverse the original sequence order.For example, standard autoregressive modeling covers (York, {New}) but not (New, {York}), while XLNet covers both in expectation.
  • A.5.3 Bridging the Gap Between Language Modeling and Pretraining: XLNet bridges language modeling and pretraining by addressing language modeling’s lack of bidirectional context modeling and its gap with downstream-task improvement.
  • A.6 Qualitative Analysis of Attention Patterns; A.7 Visualizing Memory and Permutation: XLNet shares four attention patterns with BERT but additionally exhibits self-exclusion, relative-stride, and one-side masked patterns; its permutation objective uses masked attention, memory reuse, and distinct query and content streams.The query stream cannot self-attend or access the token content at its own position, whereas the content stream follows the permutation-specific attention mask.
Loading 1906.08237v2…