Source-linked AI summary
A Survey on Contextual Embeddings
Qi Liu, Matt J. Kusner, Phil Blunsom
TL;DR
The survey addresses the limits of global word representations and reviews how contextual embeddings represent tokens using their surrounding sequence. It synthesizes model architectures, multilingual pre-training, downstream applications, compression, and analyses, reporting transferable cross-language representations and broad task performance while identifying unresolved robustness and probing concerns.
Problem
Global word representations assign one representation per word and ignore context, motivating contextual embeddings that capture varied word uses and transferable knowledge.
Method
The survey reviews contextual embedding models, multilingual pre-training, downstream applications, model compression, and analyses of encoded linguistic knowledge.
Results
Contextual embeddings achieve state-of-the-art performance across many NLP tasks and learn useful, transferable representations across languages.
Takeaways & Limitations
Contextual embeddings provide context-dependent representations that support broad NLP applications, while compression methods target deployment in resource-constrained environments.
Takeaways & Limitations
Existing probing methods may not establish that pretrained representations encode linguistic knowledge because shallow probes can exploit high-dimensional data, and model robustness remains a concern.
Abstract
from arXiv · showhide
Contextual embeddings, such as ELMo and BERT, move beyond global word representations like Word2Vec and achieve ground-breaking performance on a wide range of natural language processing tasks. Contextual embeddings assign each word a representation based on its context, thereby capturing uses of words across varied contexts and encoding knowledge that transfers across languages. In this survey, we review existing contextual embedding models, cross-lingual polyglot pre-training, the application of contextual embeddings in downstream tasks, model compression, and model analyses.
1 Introduction
Contextual embeddings represent each token as a function of the entire input sequence, capturing context-dependent linguistic properties and transferable representations across languages. The survey reviews their models, multilingual pre-training, downstream applications, compression, analyses, and future challenges.
- Contextual embeddings assign token representations based on the entire input sequence, unlike global word representations that ignore context.
- Pretrained contextual embeddings achieve state-of-the-art performance across tasks including text classification, question answering, and summarization.
- Contextual embeddings can learn useful and transferable representations across languages.
- The survey covers contextual embedding methods, multilingual pre-training, downstream applications, model compression, linguistic analyses, and future research challenges.
2 Token Embeddings
Contextual token embeddings depend on the full input sequence rather than a single global word-type vector. This design better captures sequence-level semantics, including polysemy.
- Traditional embeddings map each vocabulary word type to one global vector, whereas contextual embeddings compute each token representation from the entire sequence.
- Contextual embedding models apply an aggregation function to token representations to produce context-dependent vectors.
- Contextual representations are better suited than non-contextual embeddings to capture sequence-level semantics such as polysemy.
3 Pre-training Methods for Contextual Embeddings
Contextual-embedding pre-training learns token representations from sequence context, primarily through unsupervised language modelling, and these representations are adapted to downstream tasks. The survey covers precursor and major models, their objectives, architectures, and reported results.
- Language modelling: Language modelling factorizes the probability of a token sequence into conditional probabilities and commonly learns these probabilities with neural networks on large unlabeled corpora.The survey describes maximum-likelihood estimation as the standard training objective for language modelling.
- ELMo: ELMo combines forward and backward LSTM representations at each layer and aggregates them with task-specific normalized weights for downstream use.These contextual representations can be concatenated with a task model’s global word representations.
- GPT: GPT uses unsupervised language-model pre-training followed by supervised fine-tuning, and its pretrained Transformer outperforms task-specific architectures in 9 out of 12 tasks.GPT is trained on BookCorpus and adapts inputs into a single contiguous token sequence using special tokens.
- GPT2: GPT2 trains on diverse WebText data and, without labeled training data, reaches an F1-score of 55 on CoQA, matching or outperforming 3 of 4 baseline systems.The survey notes that the contribution of the model versus its byte-level BPE input representation remains unclear.
- BERT: BERT uses masked language modelling and next-sentence prediction with a bidirectional Transformer encoder, obtaining state-of-the-art results on eleven NLP tasks and a GLUE score of 80.5%.Its [CLS] representation supports sentence-level tasks, while token hidden states support token-level tasks; the survey notes that comparisons are confounded by changes in objectives, data, and architecture.
4 Cross-lingual Polyglot Pre-training for Contextual Embeddings
Cross-lingual polyglot pre-training learns multilingual representations that support transfer from data-rich to data-scarce languages. Surveyed methods vary in their use of joint training and shared vocabularies, yet transfer can remain possible without a shared vocabulary.
- Cross-lingual polyglot pre-training aims to learn joint multilingual representations that transfer knowledge from data-rich languages to data-scarce languages.
- Joint training & shared vocabulary: A shared BiLSTM encoder-decoder with a shared BPE vocabulary was pretrained for 93 languages, enabling classifiers trained on English to transfer without modification.
- Polyglot learning showed benefits for low-resource languages, while multilingual BERT was pretrained on Wikipedia spanning more than 100 languages.
- XLM combines causal, masked, and translation language modeling, using parallel corpora to learn cross-lingual associations.
- Joint training & separate vocabularies: Cross-lingual transfer remains possible without a shared vocabulary, alongside universal latent symmetries between languages’ embedding spaces.
- Separate training & separate vocabularies: Separate-vocabulary transfer can replace a fine-tuned model’s vocabulary with another language’s embeddings for zero-shot transfer.
5 Downstream Learning
Downstream use of contextual embeddings includes feature extraction, fine-tuning, and adapters, while training methods address catastrophic forgetting. Multi-task learning extends transfer across tasks and can improve benchmark performance.
- Ways to Use Contextual Embeddings Downstream: Pre-trained contextual embeddings are used downstream through feature-based methods, fine-tuning methods, and adapter methods.
- Feature-based: Feature-based methods freeze the pre-trained model and combine internal representations as features for task-specific architectures.
- Fine-tuning: Fine-tuning adjusts pre-trained weights for a specific task, while structurally different tasks can initialize task architectures with as many pre-trained weights as possible.
- Adapters: Adapters insert trainable modules between fixed pre-trained layers, and shared adapters often require fewer parameters than separate fine-tuned models.
- Countering Catastrophic Forgetting: Downstream learning can overwrite information in pre-trained models, a problem known as catastrophic forgetting.
- Countering Catastrophic Forgetting: Freezing layers, adaptive learning rates, and regularization are surveyed as strategies for reducing catastrophic forgetting during fine-tuning.
- Multi-task Fine-tuning: MT-DNN fine-tunes BERT across GLUE tasks and improves the GLUE benchmark to 82.7%, with better domain adaptation than BERT.
- Multi-task Fine-tuning: T5 finds that multitask learning before task-specific fine-tuning performs best among its tested settings.
6 Model Compression
Model compression addresses the memory and latency costs that hinder deployment of pre-trained language models in resource-constrained environments. Surveyed approaches include low-rank approximation, knowledge distillation, and weight quantization.
- Pre-trained language models can have prohibitive memory footprints and latency, making deployment difficult in resource-constrained environments.
- Compression work uses low-rank approximation, knowledge distillation, and weight quantization to support embedded systems and edge devices.
- Low rank approximation: Low-rank approximation replaces full-rank weight matrices with smaller low-rank matrices, reducing effective parameters; ALBERT factorizes BERTBase’s embedding matrix, which accounts for 21% of parameters.
- Knowledge distillation: Knowledge distillation transfers teacher-network knowledge to a smaller student using soft target probabilities, aiming for a lightweight model near the teacher’s accuracy.
- Weight quantization: Weight quantization maps parameters to low-precision values; Q-BERT uses attention-head groups and Hessian-based mixed precision.
7 Analyzing Contextual Embeddings
Analyses of contextual embeddings use probes and visualization to identify the linguistic information they encode and explain their behavior. Findings show structured, layered, and sometimes redundant representations, while some fine-grained capabilities remain difficult to capture.
- Probe classifiers: Probe classifiers test whether contextual embeddings encode syntactic and semantic information.These constrained classifiers are designed to explore the information present in the representations.
- Probe classifiers: Contextual embeddings perform competitively on most probing tasks but fail on some fine-grained linguistic tasks, such as conjunct identification.
- Probe classifiers: BERT rediscovers the traditional NLP pipeline in an interpretable and localizable order, covering POS tagging, parsing, NER, semantic roles, and coreference.
- Probe classifiers: BERT captures phrase-level information in earlier layers and long-distance dependency information in deeper layers.
- Visualization: BERT reaches a good initial point during pre-training for downstream tasks, which can lead to better optima than random initialization.
- Visualization: Visualization studies find that BERT attention heads are highly redundant, and manually disabling certain heads can improve performance over full-head fine-tuned models.
- Visualization: BERT distinguishes word senses at a very fine-grained level, with those senses encoded in a relatively low-dimensional subspace.
8 Current Challenges
The survey identifies unresolved challenges involving pre-training objectives, interpretation of encoded knowledge, robustness, and controllable generation. These challenges concern efficiency, reliable analysis, deployment security, and adapting generated text to domain and world-knowledge requirements.
- Better pre-training objectives: It remains unclear whether pre-training objectives can be simultaneously more efficient and effective than BERT’s masked language modeling.Recent work explores new training methods, noise combination techniques, and multi-task learning approaches.
- Understanding the knowledge encoded in pre-trained models: Existing probe analyses do not yet establish whether contextual representations encode linguistic knowledge or whether probes exploit high-dimensional data.Control-task experiments found that most existing probes fail to perform well on linguistic tasks while performing badly on controls.
- Model robustness: Pre-trained models are vulnerable to universal adversarial triggers that can cause significant performance deterioration, while misuse concerns such as fake-news generation have also arisen.
- Controlled generation of sequences: Pretrained language models generate realistic-looking sequences but remain hard to adapt to domain-specific generation or agreement with common human knowledge.The survey advocates research on finer-grained control over sequence generation.