Source-linked AI summary
Cloze-driven Pretraining of Self-attention Networks
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, Michael Auli
TL;DR
Language-model pretraining had used unidirectional or independently trained bidirectional objectives, motivating a stronger jointly bidirectional approach. The paper trains a two-tower transformer to predict every ablated center token from both contexts, achieving large GLUE gains and new state-of-the-art NER and parsing results.
Problem
Existing pretraining used unidirectional language models or independently trained bidirectional losses, leaving room to jointly exploit both directions.
Method
A bi-directional transformer separately computes forward and backward states, then jointly predicts each ablated center word from surrounding context with a cloze objective.
Results
Large GLUE gains include a 9.1 point RTE gain over Radford et al. (2018), while stacked architectures achieve new state-of-the-art NER and parsing performance.
Takeaways & Limitations
Performance improves with paragraph-structured pretraining data and up to 18B tokens, and the cloze regime outperforms separate left- and right-context prediction.
Takeaways & Limitations
The authors identify separate left- and right-context prediction tasks as too different from center-word prediction, with insufficiently complementary learning signals.
Abstract
from arXiv · showhide
We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each word is ablated and must be predicted given the rest of the text. Experiments demonstrate large performance gains on GLUE and new state of the art results on NER as well as constituency parsing benchmarks, consistent with the concurrently introduced BERT model. We also present a detailed analysis of a number of factors that contribute to effective pretraining, including data domain and size, model capacity, and variations on the cloze objective.
1 Introduction
The paper jointly pretrains both directions of a self-attention model with a cloze objective that predicts each center word from both contexts. This produces strong gains across GLUE and state-of-the-art results on NER and constituency parsing.
- Existing pretraining used either unidirectional language models or independently trained bidirectional objectives.
- The proposed model predicts every token by combining left-to-right and right-to-left context representations around an ablated center word.
- 9.1 points: the gain on RTE over Radford et al. (2018) within GLUE, alongside strong gains on every GLUE task.
- The pretrained representations support stacked task-specific architectures that achieve new state-of-the-art performance for NER and constituency parsing.
- Pretraining continues improving with up to 18B tokens, while cross-sentence data and the joint cloze regime are important according to the analysis.
2 Related work
The work builds on contextual representation learning and language-model pretraining, while differing from BERT in both architecture and training targets. Its central distinction is extracting learning signal from every token rather than a subset.
- Earlier approaches learned sentence representations from translation encoders or language-model pretraining without requiring bilingual data.
- The approach follows GPT while also using an ELMo module for NER and constituency parsing.
- Unlike BERT’s encoder and multi-task masked-token and next-sentence objectives, this model predicts every token from surrounding context with one loss function.
- Using all tokens as training targets extracts learning signal from every token in the sentence, not just a subset.
- The study additionally varies training-data amounts and sources, whereas prior comparisons considered fixed pretraining sources.
3 Two tower model
The two-tower model separately encodes left and right context with masked self-attention, then combines those representations to predict each target token without exposing it. Fine-tuning removes the final target-token mask to provide full context for downstream tasks.
- 3 Two tower model: The model represents p(t_i|t_1, . . . , t_{i−1}, t_{i+1}, . . . , t_n) using separate forward and backward towers.
- 3.1 Block structure: Forward and backward towers use stacked masked self-attention blocks, with an added zero state handling batches whose examples have uneven lengths.
- 3.1 Block structure: The blocks use multi-head attention, feed-forward layers, residual connections, pre-normalization, sinusoidal positions, character-CNN token encoding, and shared input embeddings.
- 3.2 Combination of representations: A top self-attention module combines forward and backward states, projects them into vocabulary classes, and predicts every center token while masking target-containing states.
- 3.2 Combination of representations: During fine-tuning, masking is disabled in the final combination layer so downstream models can access the target token and full context, especially for NER.
4 Fine-tuning
Fine-tuning adapts the pretrained representations to classification, regression, NER, and parsing through task-specific output structures. It uses boundary or separator representations for sentence tasks and separately tuned task-specific parameters.
- Classification and regression tasks: Single-sentence classifiers concatenate the representations of boundary tokens and project them to the downstream class set.
- Classification and regression tasks: Sentence-pair tasks concatenate the inputs, insert a separator token, and include its representation in the final projection.
- Structured prediction tasks: NER and parsing use task-specific architectures fine-tuned jointly with the pretrained language model at different learning rates.
- No Masking: Fine-tuning removes masking in the combination layer so token-level and sentence-level classifiers can use representations containing the current token.
- Optimization: New parameters receive larger learning rates than pretrained parameters, with task-dependent dropout and validation-based stopping procedures.
5 Experimental setup
The experiments use multiple datasets, model configurations, and large-scale distributed training to evaluate the two-tower architecture. The setups vary in corpus composition, vocabulary design, and model implementation while keeping core dimensionality and optimization choices specified.
- Datasets: The two-tower model is trained on Common Crawl, News Crawl, and BooksCorpus plus Wikipedia datasets.Common Crawl experiments use 9B tokens, News Crawl provides up to 4.5B words, and BooksCorpus plus Wikipedia contains about 3.3B words.
- Model configurations: The experiments compare two character-CNN models with base and large configurations against a Byte-Pair-Encoding model.The CNN models use unconstrained input vocabularies and capped output vocabularies, whereas the BPE model uses a vocabulary built from 30K merge operations.
- Reported measurements: Table 1 reports model hyperparameters, parameter counts excluding the adaptive softmax layer, and training times measured on different numbers of Volta GPUs.Training time is measured on 128 Volta GPUs for CNN models and 64 Volta GPUs for the BPE model.
- Optimization: Training uses Nesterov accelerated gradient with momentum 0.99, gradient renormalization above 0.1, linear warmup, and cosine learning-rate annealing.The learning rate warms from 10^-7 to 1 for 16K steps before annealing to 0.0001.
- Infrastructure: Experiments run on DGX-1 machines with 8 NVIDIA V100 GPUs, Infiniband, NCCL2, distributed PyTorch, and 16-bit floating-point precision.The BPE model trains much faster than the character-CNN models.
6 Results
The approach yields strong GLUE results and new state-of-the-art performance on NER and constituency parsing, while analyses identify effective pretraining conditions.
- GLUE: The GLUE evaluation covers sentence acceptability, sentiment, similarity, and natural language inference, using task-specific metrics including MCC, accuracy, F1, and Spearman correlation.The reported average excludes Winograd and several additional metrics, so it is not comparable to the official GLUE leaderboard average.
- GLUE: Our models outperform the unidirectional OpenAI GPT model across GLUE tasks despite being about 50% larger than GPT.The BPE model has more parameters than the CNN model but does not perform better in aggregate, although it trains faster.
- Structured prediction: Stacking task-specific architectures on pretrained representations produces new state-of-the-art results for named entity recognition and constituency parsing.NER uses task-specific biLSTM-CRF-style modeling, while parsing uses a task-specific constituency parser.
- Structured prediction: Both NER stacking methods outperform prior work, with fine-tuning producing the largest gain; parsing requires fine-tuning to surpass the previous state of the art.The NER comparison uses ELMoBASE and BERT results, while the parsing comparison uses ELMo embeddings as the prior state of the art.
- Objective functions: Cloze loss significantly outperforms bidirectional left/right token prediction, and combining the losses does not improve over cloze loss alone.The bidirectional loss has larger gradients because it predicts with less context; scaling that term by 0.15 improves the triplet-loss setup.
- Domain and amount of training data: Performance improves with more pretraining data up to 18B Common Crawl tokens, while paragraph-structured corpora outperform individual-sentence training examples.News Crawl generally performs worse than Common Crawl, including a 14 point RTE accuracy gap at 4.5B tokens, attributed to shorter examples.
7 Conclusion
The paper introduces a bi-directional transformer with a cloze objective and reports strong gains across language understanding tasks. Analysis identifies cross-sentence pretraining, more data, and cloze-driven training as important factors.
- The bi-directional transformer predicts every token using a cloze-style objective based on all left and right context.
- GLUE results show large gains over Radford et al., while task-specific stacking achieves new state-of-the-art performance for parsing and named entity recognition.
- Cross-sentence pretraining is crucial for many tasks, and pretraining continues to improve with up to 18B tokens.
- The cloze-driven training regime is more effective than predicting left and right tokens separately.
- Parameter sharing between the two towers initially enabled much deeper models without increasing parameter count, motivating future architectural investigation.