PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep contextualized word representations

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer

arXiv:1802.05365v2cs.CL

TL;DR

Existing word representations struggle to capture both context-sensitive syntax and semantics and variation across contexts. ELMo addresses this with deep contextualized representations from all layers of a pretrained bidirectional language model, significantly improving state-of-the-art performance across six challenging NLP tasks, including up to 20% relative error reductions.

  • Problem

    Existing word representations have limited ability to capture both complex word-use characteristics and context-dependent variation such as polysemy.

  • Method

    ELMo represents each token using a learned combination of all internal layers of a bidirectional language model pretrained on a large text corpus.

  • Results

    ELMo significantly improves state-of-the-art performance across six challenging language-understanding tasks, with up to 20% relative error reductions.

  • Takeaways & Limitations

    Different biLM layers encode distinct syntactic and semantic information, and exposing them together improves task performance.

Abstract

from arXiv · show

We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

1 Introduction

ELMo is a deep contextualized word representation whose token vectors depend on the entire input sentence and the internal states of a biLM. It captures contextual meaning and syntax, and improves performance when added to models for six challenging language-understanding problems.

  • 1 Introduction: ELMo assigns each token a representation that is a function of the entire input sentence, unlike traditional word type embeddings.The representations are derived from a bidirectional LSTM trained with a coupled language-model objective on a large text corpus.
  • 1 Introduction: ELMo is deep because each task learns a linear combination of vectors from all internal biLM layers.The vectors are stacked above each input word, enabling task-specific mixing of the network’s internal states.
  • 1 Introduction: Higher-level LSTM states capture context-dependent word meaning, while lower-level states model syntax.The higher-level states support supervised word sense disambiguation without modification, whereas lower-level states can support part-of-speech tagging.
  • 1 Introduction: ELMo representations can be easily added to existing models for six diverse and challenging language-understanding problems.The introduction identifies textual entailment, question answering, and sentiment analysis among these problems and states that adding ELMo alone significantly improves the state of the art in every case.
  • 1 Introduction: Deep representations outperform those derived from just the top layer of an LSTM.This conclusion is reported from an analysis comparing ELMo and CoVe.

2 Related work

Prior word-vector methods capture syntactic and semantic information but are context-independent, while newer approaches add subword, multi-sense, or contextual information. This work combines character convolutions and multi-sense information with fixed, pretrained biLM representations and added task-specific capacity.

  • Traditional word vectors: Pretrained word vectors are standard in state-of-the-art NLP architectures because they capture syntactic and semantic information from large-scale unlabeled text, but remain context-independent.The cited applications include question answering, textual entailment, and semantic role labeling.
  • Subword and multi-sense methods: Previous methods address traditional word-vector limitations by adding subword information or learning separate vectors for each word sense.The approach described here uses character convolutions for subword units and incorporates multi-sense information without explicitly training to predict predefined senses.
  • Context-dependent representations: Context-dependent representation methods include context2vec and embeddings computed from supervised or unsupervised language-model and machine-translation encoders.The passage identifies context2vec as using a bidirectional LSTM to encode context around a pivot word.
  • Layer-specific information: Different layers of deep biRNNs encode different information, and lower-level syntactic supervision can improve higher-level tasks such as dependency parsing and CCG supertagging.The cited examples use part-of-speech tags as multi-task syntactic supervision.
  • Pretraining and downstream adaptation: Unlike approaches that fine-tune pretrained encoder-decoder pairs, this work fixes the pretrained biLM weights and adds task-specific model capacity.This design leverages large, rich, universal biLM representations when downstream data supports only a smaller supervised model.

3 ELMo: Embeddings from Language Models

ELMo represents each token using context-sensitive states from a pretrained two-layer bidirectional language model with character convolutions. Task models learn to combine the biLM’s layers and incorporate the resulting vector into existing architectures.

  • 3.1 ELMo representations: ELMo representations are functions of the entire input sentence, computed from two-layer biLMs with character convolutions as linear functions of internal network states.The pretrained biLM can be incorporated into a wide range of existing neural NLP architectures.
  • 3.1 Bidirectional language models: A biLM jointly models forward and backward language directions, tying token-representation and Softmax parameters while maintaining separate directional LSTM parameters.The backward language model predicts each token from future context, complementing the forward model’s history-based prediction.
  • 3.2 ELMo layer combinations: ELMo is a task-specific combination of the 2L + 1 representations produced by a biLM, using softmax-normalized layer weights and a task-specific scaling parameter.The simplest version selects the top layer, while the general formulation weights all biLM layers; layer normalization can also be applied before weighting.
  • 3.3 Incorporating ELMo: To use ELMo, the supervised model freezes the biLM, concatenates its task-specific vector with token representations, and may also add ELMo at the task-RNN output.These additions preserve the remainder of the supervised architecture and can be used within more complex models.
  • 3.4 Pretraining: The pretrained model uses L = 2 biLSTM layers with 4096 units, 512-dimensional projections, residual connections, and purely character-based input representations.After 10 epochs on the 1B Word Benchmark, its average forward and backward perplexity is 39.7.

4 Evaluation

Across six benchmark NLP tasks, adding ELMo produced new state-of-the-art results with relative error reductions of 6–20% over strong base models. Reported gains included improvements in semantic role labeling, coreference resolution, and sentiment classification.

  • Overall evaluation: 6–20% relative error reductions over strong base models were achieved across all six evaluated NLP tasks, establishing new state-of-the-art results.Table 1 reports accuracy for SNLI and SST-5, F1 for SQuAD, SRL, and NER, and average F1 for Coref.
  • Semantic role labeling: 3.2% F1 improvement raised semantic role labeling performance from 81.4% to 84.6%.The gain came from adding ELMo to a re-implementation of the 8-layer deep biLSTM SRL system.
  • Coreference resolution: 3.2% average F1 improvement raised coreference resolution from 67.2 to 70.4, exceeding the previous best ensemble by 1.6% F1.The baseline was an end-to-end span-based neural model evaluated on OntoNotes CoNLL 2012 annotations.
  • Sentiment analysis: 1.0% absolute accuracy improvement over the state of the art resulted from replacing CoVe with ELMo in the biattentive classification network.The result concerns fine-grained five-label sentiment classification on the Stanford Sentiment Treebank.

5 Analysis

The analysis shows that combining all biLM layers and adding ELMo at task-appropriate locations improves downstream performance, while different layers encode distinct transferable syntactic and semantic information. ELMo also improves training and data efficiency, and its learned weights vary by task and insertion point.

  • Layer ablations: 3.9%: Using only the last biLM layer improves SQuAD development F1 over the baseline, while averaging all layers adds another 0.3%.These comparisons show that contextual representations from the last layer help, and representations from all layers improve performance further.
  • Layer ablations: 88.7%: Averaging all CoVe layers improves SNLI development accuracy from 88.2%, while SRL F1 rises 0.1% to 82.2% versus the last layer.The overall CoVe trend is similar to the biLM trend but produces smaller gains over the baseline.
  • ELMo placement: Adding ELMo at both input and output improves SNLI and SQuAD over input-only ELMo, whereas SRL performs best with ELMo only at the input.Coreference resolution also performs best when ELMo is included only at the input layer.
  • Contextual information: Different biLM layers represent different types of information: intrinsic evaluations test semantic WSD and syntactic POS tagging, explaining why combining all layers benefits downstream tasks.The representations are used directly with a nearest-neighbor WSD method and a linear POS classifier, limiting additional model capacity.
  • Contextual information: 69.0 F1: The biLM top layer outperforms its first layer on fine-grained WSD and is competitive with supervised, task-specific systems.CoVe also performs better at its second layer than its first, but the biLM representations are more transferable to WSD and POS tagging.
  • Learned layer weights: The task model favors the first biLSTM layer at the input, while output-layer weights are relatively balanced with a slight preference for lower layers.The input-layer preference is strongest for coreference and SQuAD and less peaked for other tasks.

6 Conclusion

The paper introduces a general approach for learning deep, context-dependent representations from biLMs and reports large improvements across diverse NLP tasks. Controlled experiments show that biLM layers encode different syntactic and semantic information, while using all layers improves overall task performance.

  • 6 Conclusion: The approach learns high-quality deep context-dependent representations from bidirectional language models.These representations are presented as a general approach for using biLMs.
  • 6 Conclusion: Applying ELMo produces large improvements across a broad range of NLP tasks.The conclusion characterizes the gains as large and spanning diverse NLP tasks.
  • 6 Conclusion: Controlled experiments show that biLM layers encode different syntactic and semantic information about words in context, and using all layers improves overall task performance.The findings come from ablations and other controlled experiments.

A Supplemental Material to accompany Deep contextualized word representations

The supplement documents the architectures, training routines, and hyperparameter choices of the state-of-the-art models in Section 4. These models share common lower layers with context-independent token representations beneath stacked recurrent layers, using LSTMs except for SQuAD’s GRUs.

  • The supplement details model architectures, training routines, and hyperparameter choices for the state-of-the-art models in Section 4.
  • All models use context-independent token representations below several stacked RNN layers, with LSTMs in every case except SQuAD, which uses GRUs.

A.1 Fine tuning biLM

Fine-tuning the biLM for one epoch on task-specific training data substantially reduced perplexity for nearly all tasks, while its effect on supervised performance varied by task. The fine-tuned biLM was then fixed during task training.

  • Fine-tuning procedure: Fine-tuning ignored supervised labels, adapted the biLM for one training epoch, evaluated it on development data, and fixed its weights during task training.This procedure was applied separately for each task.
  • Perplexity: Large perplexity improvements occurred for every considered task except CoNLL 2012, including SNLI’s decrease from 72.1 to 16.8.The reported values are development-set perplexities.
  • Supervised performance: 0.6%: SNLI development accuracy increased from 88.9% to 89.5% with fine-tuning, whereas sentiment accuracy was approximately unchanged.The SNLI comparison concerns the single best model.

A.2 Importance of γ in Eqn. (1)

The γ parameter was practically important for optimization because biLM internal and task-specific representations have different distributions. Its importance was especially pronounced in the last-only setting, where removing γ caused poor SNLI performance and complete SRL training failure.

  • A.2 Importance of γ in Eqn. (1): γ aided optimization by addressing distribution differences between biLM internal representations and task-specific representations.The passage identifies this distribution mismatch as the practical reason γ was important.
  • A.2 Importance of γ in Eqn. (1): Without γ, the last-only setting performed well below the baseline on SNLI and training failed completely for SRL.The passage states that γ was especially important in the last-only case in Sec. 5.1.

A.3 Textual Entailment

On SNLI textual entailment, ELMo was added to an ESIM sequence model, improving accuracy by 0.7% to a new single-model state of the art of 88.7%; a five-member ensemble reached 89.3%.

  • Model setup: The baseline SNLI system used the ESIM sequence model with 300-dimensional LSTM and feed-forward layers and fixed 300-dimensional pretrained GloVe embeddings.This configuration followed the original implementation.
  • Optimization: Training used 50% variational dropout in LSTM inputs, 50% dropout before the final two fully connected layers, ReLU activations, and Adam optimization.Gradient norms were clipped at 5.0, with an initial learning rate of 0.0004 and batch size 32.
  • Results: 0.7% accuracy improvement from adding ELMo to ESIM established a new single-model state of the art of 88.7%.The comparison was reported on the test set against previously published systems.
  • Results: 89.3% overall accuracy was achieved by a five-member ensemble.The ensemble further improved the overall accuracy beyond the single model.

A.4 Question Answering

The QA system combines GloVe and character-based token representations with BiDAF-style attention and residual self-attention, then predicts answer spans. Adding ELMo produced the highest performance, with the submission improving prior and baseline results and reaching 87.4% F1 with an ensemble.

  • Model architecture: The model concatenates case-sensitive 300-dimensional GloVe vectors with character-CNN embeddings, processes them using a shared bidirectional GRU, and applies BiDAF attention.The resulting representations are further processed by a residual self-attention layer using a GRU and context-to-context attention.
  • Model architecture: The system uses linear ReLU layers and predicts the start and end tokens of the answer span.At test time, output spans are limited to at most 17 tokens.
  • ELMo configuration: Performance was highest when ELMo was added without layer normalization to both inputs and outputs of the contextual GRU, with unregularized ELMo weights (λ = 0).The model also uses variational dropout at rate 0.2 and does not update word vectors during training.
  • Results: 1.4% F1 improvement over SAN and 4.2% over the baseline made the submission the highest-scoring single model and ensemble at submission time.These comparisons are reported for SQuAD leaderboard results as of November 17, 2017.
  • Results: 87.4% F1 was achieved by an 11-member ensemble, a 1.0% increase over the previous ensemble best.The ensemble result is reported on the SQuAD leaderboard.

A.5 Semantic Role Labeling … A.8 Sentiment classification

Across semantic role labeling, coreference resolution, and named entity recognition, ELMo-enhanced models achieved strong improvements using task-specific integration and weighting choices. The sentiment-classification section describes a biattention architecture adapted with simpler feedforward layers and dropout.

  • A.5 Semantic Role Labeling: The SRL baseline uses 200-dimensional token representations with an 8-layer interleaved biLSTM, highway connections, variational recurrent dropout, and a final softmax tagger.The token representation concatenates 100-dimensional GloVe vectors with a 100-dimensional predicate feature.
  • A.5 Semantic Role Labeling: 84.6 F1 from the single ELMo-augmented SRL model establishes a new state of the art, exceeding the prior single model by 2.9 F1 and the 5-model ensemble by 1.2 F1.
  • A.6 Coreference resolution: The coreference baseline is the end-to-end neural model of Lee et al. (2017), with hyperparameters following the original implementation.The best configuration adds ELMo to the lowest-layer biLSTM input, uses unregularized layer weighting (λ = 0), and applies 50% dropout to ELMo representations.
  • A.6 Coreference resolution: 3.2% average F1 improves the single-model coreference state of the art, while the single model also surpasses the previous ensemble best by 1.6% F1.Adding ELMo to the biLSTM output as well as its input reduced F1 by approximately 0.7%.
  • A.7 Named Entity Recognition: The NER baseline combines 50-dimensional Senna vectors with CNN character features and two biLSTM layers before dense and softmax output layers.Training uses variational dropout, gradient rescaling above an ℓ2 norm of 5.0, Adam at learning rate 0.001, fine-tuning, early stopping, and five random-seed runs.
  • A.7 Named Entity Recognition: 92.22% F1 from the ELMo-enhanced biLSTM-CRF tagger establishes a new state of the art on CoNLL 2003 NER.ELMo is added to the lowest task-biLSTM layer, with λ = 0.1 selected for the relatively small dataset; using all biLM layers provides a modest improvement.
  • A.8 Sentiment classification: The sentiment classifier largely follows McCann et al. (2017)'s biattention architecture but replaces its final maxout network with two dropout-equipped ReLU feedforward layers.A batch-normalized maxout variant achieved significantly lower validation accuracies in the authors' experiments, though implementation discrepancies may exist.
Loading 1802.05365v2…