PaperlayerView exact revision on arXiv

Source-linked AI summary

Linguistic Knowledge and Transferability of Contextual Representations

Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, Noah A. Smith

arXiv:1903.08855v5cs.CL

TL;DR

The paper asks what linguistic knowledge contextual word representations encode and how transferable that knowledge is across layers and pretraining choices. It probes representations from pretrained contextualizers across seventeen tasks and finds broad transferability alongside fine-grained weaknesses and architecture-dependent layer behavior.

  • Problem

    Contextual word representations are highly successful, but their linguistic knowledge and transferability are not yet well understood.

  • Method

    The paper evaluates pretrained contextualizers with seventeen probing tasks and analyzes transferability across layers, architectures, and pretraining objectives.

  • Results

    Frozen CWRs support strong performance across many tasks, but fine-grained tasks expose failures; LSTM higher layers are more task-specific, unlike transformer layers.

  • Takeaways & Limitations

    Transferability depends on both representation layer and architecture, while task-specific contextual features can help encode information missing from pretrained representations.

  • Takeaways & Limitations

    The probing models underperform on several tasks, including named entity recognition, grammatical error detection, and conjunct identification.

Abstract

from arXiv · show

Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language. To shed light on the linguistic knowledge they capture, we study the representations produced by several recent pretrained contextualizers (variants of ELMo, the OpenAI transformer language model, and BERT) with a suite of seventeen diverse probing tasks. We find that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge (e.g., conjunct identification). To investigate the transferability of contextual word representations, we quantify differences in the transferability of individual layers within contextualizers, especially between recurrent neural networks (RNNs) and transformers. For instance, higher layers of RNNs are more task-specific, while transformer layers do not exhibit the same monotonic trend. In addition, to better understand what makes contextual word representations transferable, we compare language model pretraining with eleven supervised pretraining tasks. For any given task, pretraining on a closely related task yields better performance than language model pretraining (which is better on average) when the pretraining dataset is fixed. However, language model pretraining on more data gives the best results.

1 Introduction

The paper investigates what linguistic knowledge contextual word representations encode, how transferability varies across layers, and how pretraining tasks affect learned representations. It uses probing models over diverse tasks to show that contextual representations are broadly useful but have task-specific weaknesses.

  • Contextual word representations assign each word a vector based on the entire input sequence, enabling words to be modeled in context.
  • The paper studies CWR linguistic knowledge with seventeen diverse probing tasks spanning phenomena such as coreference, semantic relations, and entity information.
  • Probing generates word features from pretrained contextualizers and trains a model to predict linguistic information from those features alone.
  • Linear models on frozen CWRs are competitive with task-specific state-of-the-art models in many cases but fail on tasks requiring fine-grained linguistic knowledge.
  • The first LSTM layer is consistently most transferable, whereas transformer middle layers are most transferable.
  • Higher LSTM layers are more task-specific, while transformer layers lack the same monotonic increase in task-specificity.
  • Language model pretraining produces more generally transferable representations than eleven other candidate pretraining tasks, while related tasks yield the strongest individual-task results.

2 Probing Tasks

The paper builds seventeen English probing tasks for individual or paired contextual word representations, covering token labels, segmentation, and relationships between words. These tasks target syntactic, semantic, entity, factuality, and dependency information.

  • The probing suite contains seventeen diverse English tasks focused on contextual representations of individual words or word pairs.The authors release the analysis toolkit for future probing research.
  • Token Labeling: Token-labeling tasks probe part-of-speech, fine-grained syntactic roles, hierarchical syntax, semantic tags, preposition supersenses, and event factuality.
  • Token Labeling: Preposition supersense disambiguation classifies a preposition’s lexical-semantic function and the semantic role or relation it mediates.
  • Segmentation: Segmentation tasks use single-word contextual representations to identify chunks, named entities, grammatical errors, and conjunct tokens.
  • Pairwise Relations: Pairwise relation tasks represent two words and their combination to predict or classify syntactic, semantic, and coreference relationships.

3 Models

The study evaluates six publicly available English contextualizers, including recurrent and transformer architectures trained on language-modeling or related objectives. The models vary in depth, directionality, and pretraining data.

  • The probing model is linear, limiting capacity so the analysis focuses on information easily extractable from contextual representations.
  • The study examines six publicly available English contextualized-word-representation models.
  • ELMo: ELMo variants use bidirectional language modeling with either 2-layer or 4-layer LSTMs, or a 6-layer transformer, trained on 800M tokens.
  • Transformers: The OpenAI transformer is a left-to-right 12-layer language model trained on 800M tokens from more than 7,000 unpublished books.
  • Transformers: BERT uses a bidirectional transformer trained jointly on masked language modeling and next sentence prediction over approximately 3300M tokens.

4 Pretrained Contextualizer Comparison

Linear probes over frozen contextual representations perform strongly on many tasks, but performance varies across contextualizers and tasks requiring specific linguistic information. Task-trained contextualization and higher-capacity probes improve these difficult cases, while entity and coreference information remains limited.

  • Results and Discussion: Linear probing models using contextual word representations outperform noncontextual GloVe and often rival or exceed task-specific state-of-the-art systems.They surpass published state of the art on grammatical error detection and preposition supersense identification.
  • Results and Discussion: BERT significantly improves over ELMo and the OpenAI transformer, while recurrent ELMo models outperform the ELMo transformer.The OpenAI transformer also underperforms the ELMo models and BERT; its unidirectional training and lowercased text are identified as relevant differences.
  • Results and Discussion: Current contextual word representations capture limited transferable information about entities and coreference phenomena.The authors suggest augmenting pretrained contextualizers with explicit entity representations.
  • Results and Discussion: Probing failures occur on tasks including NER, grammatical error detection, and conjunct identification because information may be absent from the representation or difficult for the probe to extract.The experiments compare task-trained LSTM contextualization with an MLP to distinguish these possibilities.
  • Results and Discussion: Task-trained contextual features provide large gains on tasks requiring highly specific syntactic information that pretrained contextualizers do not encode adequately.On conjunct identification and great-grandparent prediction, task-trained contextualization helps more than simply increasing probe capacity.

5 Analyzing Layerwise Transferability

Layerwise transferability differs systematically between recurrent and transformer contextualizers. LSTM lower layers are more general, whereas transformer layers show no monotonic increase in task specificity.

  • Layerwise transferability: The first contextualization layer in recurrent models is consistently the most transferable across probing tasks.It can outperform a scalar mix of layers on most tasks.
  • Language-model probing: Higher recurrent-model layers achieve lower bidirectional language-model perplexity, while transformer layers do not follow the same monotonic pattern.Perplexity is measured by relearning softmax classifiers from each layer’s representations.
  • Language-model probing: Representations that perform better on language modeling perform worse on probing tasks, revealing a trade-off between task-specific and general features.The analysis compares language-model probing with the seventeen linguistic probing tasks.
  • Layerwise transferability: Higher LSTM layers produce representations that are more task-specific and less general.This pattern motivates greater fine-tuning of higher layers during transfer.
  • Architecture differences: Transformer layers do not exhibit the LSTMs’ monotonic increase in task specificity.The paper leaves systematic explanation of the architecture difference to future work.

6 Transferring Between Tasks

The paper compares language-model pretraining with eleven supervised objectives under controlled architecture and data conditions. Language modeling transfers best on average, while related supervised tasks can be strongest for individual targets and more data improves results.

  • Controlled pretraining comparison: Bidirectional language-model pretraining is the most effective on average across target tasks.The comparison evaluates twelve pretraining tasks using the same contextualizer architecture and PTB data.
  • Related-task transfer: Related-task pretraining often produces the highest performance on individual target tasks.For syntactic dependency arc classification on EWT, pretraining on the same task with PTB data gives the largest gains.
  • Related-task transfer: Pretraining on syntactic dependency arcs, CCG supertagging, chunking, ancestor prediction, or semantic dependency arcs can outperform bidirectional language-model pretraining.These comparisons concern particular target tasks rather than average performance.
  • Data scale: More pretraining data yields stronger results than related-task transfer alone.The ELMo original BiLM trained on the Billion Word Benchmark outperforms the PTB-pretrained settings, emphasizing large-corpus self-supervision.
  • Layerwise results: Layer 0 of the BiLM is the highest-performing single layer among PTB-pretrained contextualizers.Because layer 0 comes from a character-level CNN without token-level context, the result suggests lexical information contributes to initial generalizability.

7 Related Work

Related work studies linguistic knowledge in neural representations through probing, architecture comparisons, pretraining-task analyses, and other interpretability methods. This paper focuses specifically on contextual word representations and a broader range of linguistic phenomena.

  • Probing representations: Earlier probing studies often examine a single linguistic phenomenon, whereas this work studies a broader range of contextual word-representation properties.The paper’s methodological comparisons include probing internal representations to predict linguistic properties.
  • Probing representations: Prior work evaluates sentence embeddings, while this paper analyzes contextual representations of individual words or word pairs.The distinction concerns the representational unit being probed.
  • Contextualizer analyses: Contemporaneous evaluations suggest pretrained contextualizers encode stronger syntax than higher-level semantics and benefit from scalar mixes of output layers.These findings are reported for CoVe, ELMo, the OpenAI Transformer, and BERT.
  • Alternative analyses: Other studies investigate how architecture design, pretraining objectives, multitask learning, neurons, ablations, and attention affect learned representations.These approaches complement probing-based analyses.
  • Methodological distinction: The paper’s approach differs from intrinsic word-similarity evaluations because its probing models require training additional parameters.Intrinsic methods directly assess vectors without extra parameters.

8 Conclusion

The paper uses seventeen probing tasks to analyze contextualized word representations and finds distinct transferability patterns across architectures. It concludes that language-model pretraining is broadly transferable, while related tasks can excel for specific targets.

  • Contributions: Seventeen diverse probing tasks provide a broad assessment of linguistic knowledge in contextualized word representations.The paper focuses on representations for individual words or word pairs and releases an analysis toolkit.
  • Layerwise findings: The lowest LSTM layer is most transferable, transformer middle layers are most transferable, and higher LSTM layers are more task-specific.Transformer layers do not show the same monotonic increase in task specificity.
  • Pretraining findings: Bidirectional language-model pretraining is more transferable in general, whereas related-task pretraining produces the strongest results for individual end tasks.The comparison concerns transferability across target tasks.

A Probing Task Setup Details

The probing setup restricts several tasks to linguistically relevant structures or tokens and explicitly includes a “None” label for shallow syntactic positions.

  • Syntactic constituency ancestor tagging removes the top-level ROOT node and assigns “None” when a word lacks a parent, grandparent, or great-grandparent.
  • Preposition supersense disambiguation trains and evaluates probing models only on single-word adpositions.
  • Conjunct identification trains and evaluates probing models only on sentences containing a coordination construction.

B Probing Model Training Details

The study trains probing models with a fixed optimization procedure and converts subword contextualizer outputs into token representations, while its tables organize performance by task and contextualizer family.

  • Training procedure: Probing models use Adam with a 0.001 learning rate for up to 50 epochs and early stopping with patience 3.
  • Representation construction: For subword contextualizers, each token representation is taken from its final subword representation.
  • Task-wise reporting: Tables 5 and 6 report token-labeling performance for linear probes over ELMo, OpenAI, and BERT contextualizers.
  • Task-wise reporting: Tables 7 and 8 report segmentation performance for linear probes over ELMo, OpenAI, and BERT contextualizers.
  • Task-wise reporting: Tables 9 and 10 report pairwise-relation performance for linear probes over ELMo and OpenAI, and BERT contextualizers, respectively.
  • Pretraining comparisons: Tables 11–13 report target labeling, segmentation, and pairwise prediction performance for ELMo contextualizers pretrained on varied tasks.
Loading 1903.08855v5…