Source-linked AI summary
Learned in Translation: Contextualized Word Vectors
Bryan McCann, James Bradbury, Caiming Xiong, Richard Socher
TL;DR
NLP commonly transfers only pretrained word vectors, motivating richer contextual representations. The paper transfers an attentional machine-translation LSTM encoder as CoVe and concatenates its outputs with word vectors. CoVe improves performance across diverse NLP tasks and brings SST and SNLI baselines to state-of-the-art performance.
Problem
NLP typically initializes only the lowest layer with pretrained word vectors, leaving an opportunity to transfer representations of words in sentence context.
Method
The paper transfers token-level outputs from an attentional machine-translation LSTM encoder and concatenates them with GloVe vectors for downstream models.
Results
CoVe improves downstream performance over word-vector and character n-gram baselines across NLP tasks, reaching state of the art on SST and SNLI.
Takeaways & Limitations
Machine translation is a promising source task for reusable NLP representations, with larger MT datasets associated with better downstream performance.
Abstract
from arXiv · showhide
Computer vision has benefited from initializing multiple deep layers with weights pretrained on large supervised training sets like ImageNet. Natural language processing (NLP) typically sees initialization of only the lowest layer of deep models with pretrained word vectors. In this paper, we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation (MT) to contextualize word vectors. We show that adding these context vectors (CoVe) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks: sentiment analysis (SST, IMDb), question classification (TREC), entailment (SNLI), and question answering (SQuAD). For fine-grained sentiment analysis and entailment, CoVe improves performance of our baseline models to the state of the art.
1 Introduction
The paper transfers contextual representations learned by an attentional machine-translation encoder to downstream NLP models. Adding these context vectors improves baseline performance across tasks, reaching state of the art on SST and SNLI.
- NLP commonly initializes only word vectors, although representing words in sentence context could further improve transfer learning.
- The paper trains an encoder for machine translation and transfers it to other NLP tasks, mirroring ImageNet-CNN transfer in computer vision.
- The approach appends MT-LSTM outputs, called context vectors (CoVe), to the word vectors used by downstream models.
- CoVe improves downstream models over baselines using pretrained word vectors alone and reaches state-of-the-art performance on SST and SNLI.
- More MT training data is positively correlated with downstream performance, supporting MT as a promising transfer-learning resource.
2 Related Work
Prior work establishes transfer learning across synergistic domains and develops neural machine-translation encoders as transferable representations. This paper positions token-level NMT representations as compatible with sequence-based NLP models.
- Transfer Learning: Transfer learning adapts representations across independently collected datasets when the source and target domains have synergistic relationships.
- Neural Machine Translation: Neural machine translation uses encoder-decoder sequence-to-sequence models, with attention giving the decoder access to encoder representations during generation.
- Transfer Learning and Machine Translation: Machine translation offers abundant data and requires preserving source-sentence information while producing a target-language sentence.
- Transfer Learning and Machine Translation: Unlike fixed-length NMT representations, the paper transfers token-level representations that fit subsequent LSTMs, attention mechanisms, and other sequence-processing layers.
3 Machine Translation Model
The machine-translation source model is an attentional sequence-to-sequence system with a two-layer bidirectional LSTM encoder and an attentional decoder. It processes source and target word sequences to predict target-language words.
- The paper trains an attentional sequence-to-sequence model for English-to-German translation to obtain an encoder transferable to other tasks.
- The model represents source and target sentences as word sequences, using GloVe source vectors and randomly initialized target word vectors.
- A two-layer bidirectional LSTM, called the MT-LSTM, computes hidden states from the source-word representations.
- At each decoding step, a unidirectional LSTM uses the previous target embedding and context-adjusted hidden state to produce a decoder state.
- The decoder computes attention weights over encoder time steps, combines the weighted encodings with its state, and transforms the result into output-word probabilities.
4 Context Vectors (CoVe)
CoVe transfers the MT-LSTM’s token-level outputs as contextual representations. Downstream models concatenate each CoVe vector with its corresponding GloVe vector before task-specific processing.
- The MT-LSTM outputs are transferred to downstream tasks as context vectors, or CoVe.
- For an input sequence, each GloVe word vector is concatenated with its corresponding CoVe vector.
- The resulting contextualized sequences are fed to the task-specific model shown in Figure 1b.
5 Classification with CoVe
The BCN is a general classification architecture that combines task-specific sequence representations with biattention, conditioning, pooling, and a maxout classifier. It supports both single- and two-sentence tasks, using CoVe-augmented input vectors.
- The BCN handles single- and two-sentence classification tasks, duplicating single inputs so the model processes two sequences.
- A feedforward ReLU network and bidirectional LSTM convert each input-vector sequence into task-specific representations.
- Biattention computes an affinity matrix between the two sequences and uses context summaries to condition each representation on the other.
- The conditioning LSTMs combine original representations with differences and element-wise products to preserve, compare, and modulate information.
- Max, mean, min, and self-attentive pooling aggregate conditioned sequences before a three-layer batch-normalized maxout network predicts class probabilities.
6 Question Answering with CoVe
For question answering, CoVe-compatible sequence representations are produced for the document and question, then passed through coattention and a dynamic decoder. The question-answering encoder uses tanh rather than the classification model’s ReLU activation.
- Question answering uses the document and question as the two input sequences before applying coattention and the original Dynamic Coattention Network decoder.
- The question-answering representation function replaces ReLU with tanh when producing the input sequences.
7 Datasets
The paper evaluates CoVe across three English–German MT pretraining corpora and classification, sentiment, entailment, and question-answering benchmarks. The MT datasets range from 30,000 to roughly 7 million sentence pairs and differ in domain and linguistic complexity.
- Machine Translation: Three English–German datasets train separate MT-LSTMs: Multi30k, IWSLT 2016, and WMT 2017.
- Machine Translation: The corpora differ in content: Multi30k uses shorter, simpler image captions, IWSLT uses conversational TED presentations, and WMT uses web, news, parliamentary, and press-release text.
- Machine Translation: The MT training sets contain 30,000, 209,772, and roughly 7 million sentence pairs, respectively.
- Machine Translation: The resulting context-vector variants are named CoVe-S, CoVe-M, and CoVe-L according to the MT encoder used.
- Downstream Tasks: Downstream evaluation covers SST and IMDb sentiment analysis, six- and fifty-class TREC question classification, SNLI entailment, and SQuAD question answering.
8 Experiments
The experiments evaluate CoVe across machine translation, classification, and question answering, finding gains from contextual vectors and larger MT training datasets.
- 8.1 Machine Translation: MT-LSTMs trained on MT-Small, MT-Medium, and MT-Large achieved BLEU scores of 38.5, 25.54, and 28.96, respectively, on their stated test sets.The smallest dataset had the highest BLEU score but was also simpler and domain-restricted.
- 8.2 Classification and Question Answering: Models combining CoVe with GloVe achieved higher validation performance than models using GloVe alone.The experiments compared GloVe, character n-grams, CoVe, and their combinations.
- 8.2 Classification and Question Answering: CoVe produced larger improvements than character n-gram embeddings, while combining both representations improved performance further on some tasks.The results suggest CoVe complements both word-level GloVe and character-level embeddings.
- 8.2 Classification and Question Answering: Larger MT datasets, containing more complex and varied language, were positively correlated with larger downstream improvements from CoVe.The trend supports MT data as a resource for transfer learning in NLP.
- 8.2 Classification and Question Answering: Final test performances on SST-5 and SNLI reached a new state of the art.The best classification models used GloVe, CoVe, and character n-gram embeddings.
- 8.2 Classification and Question Answering: Adding CoVe pushed the original DCN's validation performance above the published R-NET validation performance on SQuAD.The comparison concerned exact match and F1 validation scores; the model was not submitted for testing.
9 Conclusion
The paper introduces transfer from machine-translation encoders to downstream NLP tasks. Its best CoVe models outperformed several initialization baselines, supporting reusable shared representations for NLP.
- 9 Conclusion: The paper transfers knowledge from an encoder pretrained on machine translation to a variety of downstream NLP tasks.The authors provide code and the pretrained MT-LSTM to encourage research on shared NLP representations.
- 9 Conclusion: CoVe models from the best pretrained MT-LSTM outperformed baselines using random vectors, GloVe, or GloVe with character n-gram embeddings.The conclusion frames this as progress toward unified NLP models with increasingly general reusable weights.