Source-linked AI summary
What do you learn from context? Probing for sentence structure in contextualized word representations
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, Ellie Pavlick
TL;DR
The paper asks what contextualized representations encode beyond lexical information and develops edge probing tasks to test sentence structure across syntactic and semantic phenomena. It probes four pretrained models with a common span-based framework and finds stronger gains for syntactic than semantic tasks, while ELMo captures information beyond local context.
Problem
The paper investigates what contextual representations encode beyond conventional word embeddings, including whether they capture syntactic or higher-level semantic information and local or long-range structure.
Method
The authors introduce edge probing over sub-sentence tasks derived from structured NLP, using a common classifier to probe CoVe, ELMo, OpenAI GPT, and BERT.
Results
Contextualized embeddings generally improve most on syntactic tasks rather than semantic tasks, while ELMo’s performance cannot be fully explained by access to local context.
Takeaways & Limitations
The results suggest that these representations encode syntax more strongly than higher-level semantics and include distant linguistic information useful for longer-range dependencies and higher-level syntactic structures.
Takeaways & Limitations
The study does not fine-tune encoder weights, so some information may remain inaccessible without fine-tuning.
Abstract
from arXiv · showhide
Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.
1 INTRODUCTION1
The paper asks what contextualized word representations encode beyond conventional word embeddings, introducing edge probing to examine sentence structure across syntactic, semantic, local, and long-range phenomena.
- The work investigates whether contextual representations improve over conventional word embeddings and what information they encode about a word’s role in a sentence.
- The authors introduce edge probing tasks covering syntactic, semantic, local, and long-range phenomena.
- The probing model accesses only contextual embeddings from a fixed pretrained encoder and predicts properties from one or two specified spans.
- The suite derives tasks from tagging, parsing, semantic roles, and coreference annotations, including OntoNotes resources.
- The study probes CoVe, ELMo, OpenAI GPT, and BERT, comparing contextual models with word-level baselines to isolate context and examining local versus long-range information.
2 EDGE PROBING
Edge probing represents structured NLP tasks as labeled span-based edges and evaluates their labeling versions with a common multi-label classification framework across eight core tasks.
- The edge probing framework represents any task expressible as a labeled graph anchored to sentence spans using uniform metrics and architectures.
- Each edge contains one or two spans and a set of task-specific labels, with unary edges omitting the second span.
- The framework uses known spans as inputs and predicts multi-label targets, enabling a uniform binary F1 metric across labeling tasks.
- The suite covers part-of-speech, constituent, dependency, named-entity, semantic-role, coreference, semantic proto-role, and relation classification tasks.
- The tasks include token-level syntactic labels, span-level constituent and entity labels, span-pair relations, and predicate-argument semantic labels.
- OntoNotes supplies five task annotations, while Universal Dependencies, SPR datasets, SemEval, and Winograd-based DPR provide the remaining resources and an additional coreference challenge set.
3 EXPERIMENTAL SET-UP
The experiments probe four pretrained contextual encoders with a span-restricted classifier, using multiple representation extraction strategies and aligned tokenizations.
- The probing architecture pools contextual vectors only within each input span, concatenates span representations, and classifies them with a two-layer MLP.
- The study evaluates CoVe, ELMo, OpenAI GPT, and BERT, each producing contextual vectors for the input tokens.
- CoVe uses translation-trained biLSTM activations concatenated with GloVe vectors, while ELMo uses a bidirectional language model trained on the Billion Word Benchmark.
- GPT is a left-to-right language-model Transformer, whereas BERT is trained with masked-language-model and next-sentence objectives on BooksCorpus and Wikipedia.
- For GPT and BERT, the study compares concatenated top-layer features with learned scalar mixtures of layer activations.
- The models use different vector dimensions and tokenization schemes, requiring heuristic alignment of source spans to model-specific tokens.
4 EXPERIMENTS
The experimental design compares contextual encoders with lexical, randomized, and local-context baselines to separate contextual, pretrained, architectural, and long-range effects.
- The experiments ablate aspects of contextual encoders to determine what information they capture beyond conventional word embeddings.
- Lexical baselines measure performance from context-independent word representations and lexical priors without access to surrounding words.
- Randomized ELMo replaces weights above the lexical layer with random orthonormal matrices to assess architecture-driven feature functions separately from pretrained knowledge.
- The experiments do not fine-tune encoder weights, leaving information accessible through fine-tuning for future work.
- A fixed-width CNN added to word representations serves as a local-context baseline, allowing comparison with contextual encoders’ long-range contributions.
5 RESULTS
Across representation models, contextual encoders improve recovery of sentence structure most strongly for syntactic phenomena, while semantic-task gains are less uniform. BERT-large performs best overall on most tasks, but genre matching contributes little and several improvements reflect local or lexical information.
- Comparison of representation models: BERT-large outperforms BERT-base by 1.1 F1 points and ELMo by 2.7 F1, corresponding to nearly 20% relative error reduction on most tasks.BERT-base also exceeds GPT by approximately 2 F1 points on average with scalar mixing.
- Comparison of representation models: BERT improvements are uneven: BERT-large gains 7.4 F1 points over ELMo on OntoNotes coreference and 7.8 points on Winograd-style coreference.The Winograd result averages runs with substantial variance, including scores in the mid-50s and high-60s F1.
- Encoding of syntactic vs. semantic information: Contextual encoders show their largest gains on syntactic tasks, while improvements are smaller on semantic tasks such as semantic proto-roles and Winograd.ELMo, CoVe, and GPT follow this trend across the task suite.
- Encoding of syntactic vs. semantic information: ELMo improves relation classification by 22 F1 points over its lexical baseline, while BERT-large adds another 4.6 points.A bag-of-words feature captures more than 70% of the full ELMo headroom on this task.
- Encoding non-local context: A width-3 CNN closes 72% of the lexical-to-ELMo gap and width 5 closes 79%, indicating that much syntactic information is local.For more semantic tasks, the larger remaining gap between CNN and ELMo suggests that ELMo’s improvements rely more on long-range information.
- Encoding non-local context: Full ELMo loses only 7 F1 points from dependency distance 0 to distance 8, whereas CNN and orthonormal models decline rapidly with distance.This supports the presence of useful long-distance dependency information in the pretrained encoder.
6 RELATED WORK
Prior probing work examined sentence encoders and token-level properties, but full-sentence pooling can obscure where information is encoded. This work extends sub-sentence probing across broader syntactic and semantic phenomena using existing annotations.
- Prior probing: Earlier studies probed sentence encoders for surface and syntactic properties, while other work used perplexity to examine linguistic knowledge in language models.
- Motivation: Full-sentence challenge sets confound whether relevant information is encoded in the target span or inferred from elsewhere in the sentence.Sub-sentence probing directly tests word- and phrase-level representations instead.
- Prior probing: Token-level probing has shown that neural translation representations encode part-of-speech, semantic tags, morphology, and pairwise dependency relations.
- This work: The probing suite uses structured NLP annotations spanning tagging, parsing, semantic roles, and coreference.
- This work: The proposed approach extends sub-sentence probing to syntactic and semantic tasks, including long-range and high-level relations such as predicate-argument structure.It uses existing annotated datasets rather than templated data and supports analysis by label and span distance.
7 CONCLUSION
The paper introduces edge probing to analyze sub-sentence structure in contextualized word embeddings from four models. The analysis finds stronger contextual improvements for syntax than semantics and evidence that ELMo captures distant linguistic information.
- Conclusion: Edge probing evaluates sub-sentential syntactic and semantic phenomena in CoVe, ELMo, GPT, and BERT representations.
- Conclusion: Contextualized embeddings generally improve more over non-contextual baselines on syntactic tasks than on semantic tasks.The reported examples contrast constituent labeling with coreference.
- Conclusion: ELMo’s performance cannot be fully explained by access to local context, indicating that its representations encode distant linguistic information.This information can help with longer-range dependency relations and higher-level syntactic structures.
- Conclusion: The released data-processing and model code is intended to support understanding and improvement of contextualized word embedding models.
A CHANGES FROM ORIGINAL VERSION
The updated version adds BERT probing results and several methodological or reporting changes. It also revises the comparison between ELMo and GPT after adding scalar mixing experiments.
- Changes: The revision adds probing results for BERT-base and BERT-large.
- Changes: The paper adds SemEval 2010 Task 8 relation classification to probe semantic information in pretrained encoders.
- Changes: Scalar mixing experiments on GPT improve performance slightly and change the conclusion that ELMo was overall superior to GPT.The two models are approximately equal on average, with slight task-level differences.
- Changes: Winograd coreference results now report averages over five runs to reduce noise.
B DATASET STATISTICS
Table 3 summarizes each probing corpus by labels, examples, tokens, and targets across train, development, and test splits. It also specifies task-specific counting and splitting conventions.
- Table 3: Table 3 reports the number of labels, examples, tokens, and targets for each probing task, split into train, development, and test sets.
- Counting conventions: Examples generally correspond to sentences, but semantic role labeling counts frames instead.
- Counting conventions: Targets count classification edges or spans as defined by the edge probing task formulation.
- Splits: For SemEval relation classification, 15% of training data forms a fixed development subset and the remaining 85% is used for training.
C MODEL DETAILS
The probing model projects contextual token vectors into span-specific spaces, pools each span into a fixed-length representation, and predicts labels with an MLP and independent sigmoid outputs.
- Span representation: Token vectors are first projected to 256 dimensions, with separate projections allowing different information to be extracted from each span.The two projections correspond to spans such as predicates and arguments.
- Span representation: A self-attentional pooling operator learns token weights and sums the projected vectors within each span.This converts variable-length spans into fixed-length representations.
- Classifier: The pooled span representations are concatenated and passed through a two-layer MLP followed by a sigmoid output layer for label prediction.The output estimates a probability independently for each label.
- Training objective: Binary cross entropy with per-label sigmoids avoids the exclusivity constraint of softmax loss and supports analysis of individual labels or label groups.This choice is used for stratified task analysis.
- Training objective: The probing classifier is trained while the sentence encoder remains fixed, except for ELMo scalar parameters.Training uses Adam with scheduled learning-rate reduction and early stopping.
D CONTEXTUAL REPRESENTATION MODELS
The study probes four contextual representation models built from translation, language modeling, or masked-language-modeling objectives, with representations combined with non-contextual embeddings in several cases.
- Models: CoVe is a two-layer biLSTM encoder trained for English-German translation and concatenated with pretrained GloVe vectors to form 900-dimensional token representations.The translation model was trained on approximately 7 million English sentences.
- Models: ELMo is a two-layer bidirectional language model built over a context-independent character CNN, producing 1024-dimensional token representations from a weighted sum of layers.The model is trained on the Billion Word Benchmark.
- Models: OpenAI GPT is a 12-layer Transformer trained as a left-to-right language model with masked attention.For probing, its pretrained Transformer weights are held fixed and its contextual activations are combined with subword embeddings.
- Models: BERT uses stacked Transformer layers trained jointly with masked-language-modeling and next-sentence-prediction objectives.The study evaluates pretrained uncased base and large models.
E RETOKENIZATION
Because probing annotations and pretrained encoders use different tokenizations, the study retokenizes inputs and projects annotated spans across tokenizations using byte-level Levenshtein alignments.
- Tokenization mismatch: Encoder-specific retokenization is required because native probing tokenization may differ from Moses or subword tokenization.The mismatch can change one annotated word span into differently indexed spans under each encoder tokenizer.
- Tokenization mismatch: For example, the native span for “pineapples” becomes [5, 6) under Moses tokenization and [5, 7) under the illustrated subword model.The subword tokenizer splits “pineapples” into two pieces.
- Alignment procedure: The alignment procedure treats the target tokenizer as a black box, aligns joined target tokens to the source string with byte-level Levenshtein distance, and derives token-to-byte alignments.The resulting token-to-token alignment is composed from these alignments.
- Alignment procedure: Source spans are represented as boolean vectors, projected through the token alignment, and recovered on the target side from the minimum and maximum nonzero indices.This transfers annotated span boundaries to the encoder’s tokenization.