PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissecting Contextual Word Embeddings: Architecture and Representation

Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, Wen-tau Yih

arXiv:1808.08949v2cs.CL

TL;DR

Pre-trained biLMs improve many NLP tasks, but it remains unclear how their architectures produce effective contextual representations. The paper empirically compares LSTM, CNN, and self-attention biLMs and finds that all learn strong, depth-organized linguistic representations, while trading some speed against accuracy.

  • Problem

    Pre-trained biLMs improve many NLP tasks, but the paper notes that why and how they work remain insufficiently understood.

  • Method

    The paper empirically compares LSTM, CNN, and self-attention biLM architectures using downstream task evaluation and quantitative probes of learned representations.

  • Results

    All three architectures outperform traditional word vectors across four benchmark NLP tasks, with a modest speed–accuracy tradeoff and depth-dependent representations spanning morphology, syntax, and semantics.

  • Takeaways & Limitations

    Across architectures, biLM layers form a hierarchy from morphology through local syntax to longer-range relationships such as coreference, making biLMs general-purpose language feature extractors.

  • Takeaways & Limitations

    BiLMs learn deeper linguistic phenomena from surface forms and token order alone, leaving room for explicit syntactic biases or supervised signals.

Abstract

from arXiv · show

Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks. However, many questions remain as to how and why these models are so effective. In this paper, we present a detailed empirical study of how the choice of neural architecture (e.g. LSTM, CNN, or self attention) influences both end task accuracy and qualitative properties of the representations that are learned. We show there is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging NLP tasks. Additionally, all architectures learn representations that vary with network depth, from exclusively morphological based at the word embedding layer through local syntax based in the lower contextual layers to longer range semantics such coreference at the upper layers. Together, these results suggest that unsupervised biLMs, independent of architecture, are learning much more about the structure of language than previously appreciated.

1 Introduction

The paper investigates why pre-trained biLMs work and how architecture affects both task accuracy and the linguistic information encoded in contextual representations. It finds that LSTM, CNN, and self-attention models all learn strong representations, with depth corresponding to increasingly broad linguistic structure.

  • 1 Introduction: Pre-training improves many NLP tasks, but the mechanisms behind these gains remain insufficiently understood.Reported typical relative error reductions range from 10–25%.
  • 1 Introduction: The study empirically compares LSTM, CNN, and self-attention architectures for end-task accuracy and syntax- and semantics-related representation properties.
  • 1 Introduction: All three architectures substantially outperform traditional word vectors across four benchmark NLP tasks, despite a modest speed–accuracy tradeoff.
  • 1 Introduction: Representation content changes with depth, progressing from word morphology to local syntax and then longer-range semantics such as pronominal coreference.
  • 1 Introduction: The architectures consistently learn rich contextual structure, suggesting that unsupervised biLMs capture more language information than previously appreciated.

2 Contextual word representations from biLMs

The paper describes biLMs as bidirectional language models whose internal contextual states can be transferred to downstream tasks. Character-aware input encoding and layer-weighted representations provide the basis for comparing contextual information across layers.

  • 2 Contextual word representations from biLMs: A biLM jointly trains forward and backward language models to maximize token log likelihood in both directions.
  • 2 Contextual word representations from biLMs: Each direction begins with a token representation, computes layered context-dependent states, and uses its top state to predict the next token.
  • 2 Contextual word representations from biLMs: Forward and backward states are concatenated at each token position to form bidirectional contextual representations.
  • 2 Contextual word representations from biLMs: The character-to-word encoder combines character embeddings, 2048 character n-gram CNN filters, max pooling, highway layers, and a linear projection.
  • 2 Contextual word representations from biLMs: ELMo representations combine all biLM layers through a learned weighted average transferred into downstream task models.

3 Architectures for deep biLMs

The study compares LSTM, Transformer, and gated CNN biLM architectures, evaluating their language-modeling quality, computational efficiency, and downstream representation performance. The architectures differ in structure and speed while providing comparable pre-trained models for analysis.

  • 3 Architectures for deep biLMs: The central research question is whether contextual-layer architecture determines the quality of learned biLM representations.
  • 3.2 Transformer: The Transformer uses multi-headed attention followed by position-wise feed-forward computation in each encoder layer.
  • 3.3 Gated CNN: The gated CNN biLM uses causal convolutions in both directions and 16 residual layers to obtain a wide top-layer receptive field.
  • 3.4 Pre-trained biLMs: Across the compared models, averaged forward and backward perplexities range from 37.5 for the 4-layer LSTM to 44.5 for the Gated CNN.
  • 3.4 Pre-trained biLMs: Transformer and CNN contextual layers run 3–5X faster than the 2-layer LSTM for the reported hyperparameter choices.

4 Evaluation as word representations

The evaluation tests whether frozen contextual representations from LSTMs, a Transformer, and a gated CNN transfer across four benchmark NLP tasks. All architectures substantially outperform GloVe, with LSTMs generally strongest and task-specific differences remaining modest.

  • 13%–25% relative improvements over GloVe occur for most tasks and architectures, while MultiNLI gains range from 6% to 13%.
  • Across the four tasks, LSTM contextual representations perform best overall, although the alternatives remain competitive.The trials replace GloVe with contextual vectors from each biLM without fine-tuning the language-model weights.
  • 4.1 MultiNLI: The 4-layer LSTM reaches a new MultiNLI state of the art, exceeding the prior best by 1.3% matched and 1.9% mismatched.The Transformer is 0.2% / 0.6% lower than the 2-layer LSTM on matched/mismatched accuracy.
  • 4.2 Semantic Role Labeling: The 4-layer LSTM improves semantic role labeling by 0.6% absolute over the Transformer and CNN representations.
  • 4.3 Constituency parsing: LSTM models lead constituency parsing by 0.2% over the Transformer and 1.0% over the CNN.
  • 4.4 Named entity recognition: For named entity recognition, the 2-layer LSTM is 0.4%–0.8% higher in averaged F1 than the other biLMs.The comparison averages results across five random seeds.

5 Properties of contextual vectors

Deep biLMs organize contextual information by depth: morphology appears in the word embedding layer, local syntax in lower contextual layers, and semantic relationships such as coreference in upper layers. These representations support probing of token-level, phrasal, and semantic structure across architectures.

  • Contextual representations progress from morphology at the embedding layer to local syntax and then longer-range semantics at upper layers.
  • 5.1 Contextual similarity: Nearest-neighbor analysis shows lower layers capture local information and syntactic-constituent similarity, while upper layers represent longer-range relationships.
  • 5.3 Probing contextual information: Span representations cluster by chunk type and support constituent prediction, with parsing-favorable layers positioned above POS-optimal layers but below coreference-optimal layers.
  • 5.1 Contextual similarity: A simple coreference method subtracts constituent context from the pronoun vector, then selects the most similar preceding noun matching in number.
  • 5.1 Contextual similarity: 52%–57% coreference accuracy peaks near the top layers, while removing number agreement reduces accuracy only 2–3% rather than the 12–14% baseline drop.
  • 5.2 Context independent word representation: The biLM embedding layer encodes morphology with little semantics, unlike traditional word vectors that perform well on both analogy types.
  • 5.3 Probing contextual information: 97.2–97.4% POS accuracy is highest in lower layers, whereas constituency parsing reaches near 80% F1 compared with 18.1% for GloVe.
  • 5.3 Probing contextual information: Task-learned layer weights show parsing preferences near peak linear parsing layers, while NER and MultiNLI emphasize the embedding and first contextual layers.

6 Related work

Related work has studied contextual representations, efficient sequence architectures, and prediction attribution from several angles. This paper distinguishes itself by probing internal biLM information across broader architectures, spans, and semantic as well as syntactic tasks.

  • CoVe provides contextualized vectors, but prior work found biLM-based representations outperformed it across all considered tasks.
  • Densely connected RNNs and layer pruning were proposed to accelerate context-vector prediction, and the method could also combine with other architectures.
  • Earlier studies examined neuron-level phenomena and RNN linguistic abilities, whereas this paper also probes span representations, semantic transfer, and multiple architectures.
  • Prediction-attribution methods explain outputs by manipulating representations, gradients, or decompositions, while this paper examines the contextual information encoded in internal biLM states.

7 Conclusions and future work

Deep biLMs learn hierarchical contextual information across three architectures, with lower layers specializing in syntax and higher layers modeling longer-range relationships. The authors identify scaling and linguistically informed training as directions for future work.

  • All three architectures learn rich contextual information at both word and span levels.
  • Lower biLM layers specialize in local syntax, while higher layers model longer-range relationships such as coreference.
  • Future work: The authors propose exploring whether larger models or datasets improve biLM representation quality.
  • Future work: Explicit syntactic structure, linguistically motivated inductive biases, multitask learning, or semi-supervised training may address limitations of sequential biLMs.

A biLM hyperparameters

The biLMs use a shared 1024-dimensional contextual representation at each layer and residual connections, while their architectures differ in depth, hidden dimensions, attention, and optimization.

  • All biLMs use 512 dimensions in each direction, yielding a 1024-dimensional contextual representation per layer.
  • All systems use residual connections between contextual layers and share a character-based word embedding layer, except the 4-layer LSTM.
  • LSTM: The 4-layer LSTM uses 4096-dimensional hidden states and 512-dimensional projections in each direction and layer.
  • Transformer: The Transformer uses six layers, eight attention heads per layer, 2048-dimensional feed-forward hidden layers, and 10% dropout at several locations.
  • Gated CNN: The gated CNN uses 16 layers of [4, 512] residual blocks with 5% dropout between blocks.

B Task model hyperparameters

The downstream task models combine biLM or word representations with task-specific recurrent, feed-forward, convolutional, or conditional-random-field components and regularization settings.

  • MultiNLI: The MultiNLI implementation uses 300 dimensions for all LSTMs and feed-forward layers, with 50% dropout and Adam optimization at learning rate 0.0004.
  • Semantic Role Labeling: The semantic role labeling model concatenates word representations with a 100-dimensional binary predicate representation before an 8-layer bidirectional LSTM.
  • Constituency Parsing: The constituency parser feeds biLM word representations through a two-layer bidirectional LSTM and then a feed-forward span classifier.
  • NER: The NER model adds 128 character CNN filters to pretrained word representations, uses two LSTM layers, and predicts with a CRF enforcing valid tag sequences.
  • Figures 5, 6, and 7 visualize contextual similarities across layers for the LSTM, Transformer, and gated CNN biLMs.

D Layer diagnostics

The diagnostics report full results for selected layers and visualize contextual similarities across biLM depths. The associated tables cover pronominal accuracy, POS tagging, and parsing metrics.

  • Tables 4, 5, and 6 provide full results corresponding to the top three rows in Figure 3.
  • Figures: Figure 5 visualizes contextual similarities for the 4-layer LSTM biLM, with layers ordered from top-left toward bottom-right.
  • Figures: Figure 6 visualizes contextual similarities for the Transformer biLM, with layers ordered from top-left toward bottom-right.
  • Figures: Figure 7 visualizes contextual similarities for the gated CNN biLM, with layers ordered from top-left toward bottom-right.
  • Tables: Table 4 reports unsupervised pronominal accuracies on the CoNLL 2012 development set.
  • Tables: Table 5 reports POS tagging accuracies for linear models on the PTB development set, while Table 6 reports labeled bracketing F1, precision, and recall for linear parsing models.
Loading 1808.08949v2…