Source-linked AI summary
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, Quoc V. Le
TL;DR
The paper tackles sequence-to-sequence learning when DNN inputs and outputs have variable lengths. It uses encoder and decoder LSTMs with reversed source sentences, achieving higher English–French translation scores than a phrase-based SMT baseline and strong performance on long sentences.
Problem
DNNs cannot directly handle sequence-to-sequence problems when inputs and targets require variable, non-fixed dimensionality.
Method
One deep LSTM encodes the input sequence into a fixed-dimensional vector, while another deep LSTM decodes the target sequence from that representation, using reversed source sentences.
Results
34.81 BLEU for direct WMT’14 English–French translation exceeded the SMT baseline’s 33.30, while rescoring its 1000-best lists reached 36.5 BLEU.
Takeaways & Limitations
The simple LSTM approach outperformed a standard SMT system and suggests promise for other sequence-learning problems with sufficient training data.
Abstract
from arXiv · showhide
Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT'14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous best result on this task. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.
1 Introduction
The paper addresses the fixed-dimensionality limitation of DNNs for variable-length sequence-to-sequence problems with an end-to-end LSTM architecture. On English–French translation, it outperforms a phrase-based SMT baseline, handles long sentences, and benefits substantially from reversing source-word order.
- Motivation: DNNs are limited when sequence inputs and targets cannot be encoded as fixed-dimensional vectors, although many important tasks have unknown sequence lengths.The paper motivates a domain-independent method for mapping sequences to sequences.
- Translation results: 34.81 BLEU was achieved by an ensemble of deep LSTMs directly translating WMT’14 English–French sentences, versus 33.30 for the SMT baseline.The score was penalized for reference words outside the 80k-word vocabulary.
- Translation results: 36.5 BLEU was obtained by using the LSTM to rescore the SMT system’s 1000-best hypotheses, compared with the 33.3 baseline.This was close to the previous best published result of 37.0 BLEU.
- Optimization: Reversing source sentences but not target sentences introduced short-term dependencies that made optimization easier and enabled strong performance on very long sentences.The authors identify source-word reversal as a key technical contribution.
- Representations: The model learns variable-length sentence representations that capture meaning, with qualitative evidence of sensitivity to word order and relative invariance to active versus passive voice.Translations encourage sentences with similar meanings to have nearby representations.
2 The model
The model uses one LSTM to encode an input sequence into a fixed-dimensional vector and another LSTM to decode an output sequence of potentially different length. Separate deep LSTMs and reversed source-word order improve the model’s trainability and performance.
- Motivation: The RNN sequence-to-sequence strategy is difficult when input and output lengths differ and their alignment is complicated or non-monotonic.LSTMs are used because they can learn long-range temporal dependencies.
- Sequence-to-sequence architecture: The encoder LSTM maps an input sequence to a fixed-dimensional representation, which initializes a decoder LSTM that models the target sequence probability.The input and output lengths may differ.
- Decoding: The decoder assigns probabilities to successive target tokens and uses an <EOS> symbol to represent output sequences of arbitrary length.Each conditional distribution is represented with a vocabulary-wide softmax.
- Architecture choices: The implemented model uses separate four-layer LSTMs for input and output sequences, because separate networks increase parameters cheaply and support multiple language pairs.The authors also found deep LSTMs substantially better than shallow ones.
- Architecture choices: Reversing the input words places source tokens closer to corresponding target tokens, making it easier for SGD to establish communication between sequences.This data transformation greatly improved LSTM performance.
3 Experiments
Experiments evaluated the deep LSTM on WMT’14 English–French translation through direct decoding, SMT rescoring, and representation analysis. Reversing source sentences improved optimization and long-sentence performance, while the learned representations captured word order and voice variation.
- Experimental setup: The experiments evaluated direct translation, SMT n-best rescoring, and learned sentence representations on WMT’14 English–French data.The models were trained on 12M selected sentence pairs with fixed source and target vocabularies.
- Source reversal: Reversing source sentences reduced test perplexity from 5.8 to 4.7 and increased decoded-translation BLEU from 25.9 to 30.6.The authors attribute the improvement to shorter dependencies between source and target words, which simplified optimization.
- Long-sentence behavior: The LSTM showed no degradation below 35-word sentences and only minor degradation on the longest sentences.The paper reports that reversed-source models performed better on long sentences than models trained on raw source order.
- Representation analysis: PCA projections showed representations sensitive to word order and relatively insensitive to replacing active voice with passive voice.The projections formed meaning-based clusters with similar internal structure.
4 Related work
Prior neural machine translation work mainly improved existing systems through rescoring or source-information integration, while related sequence-to-sequence models differed in representation, integration, or generation strategy.
- Neural language models: RNNLMs and NNLMs were primarily used to rescore n-best lists from strong machine-translation baselines.This strategy reliably improved translation quality.
- Source information: Source-language information was incorporated through topic models, decoder alignment information, or other inputs to neural language models.These approaches focused on improving rescoring or decoder integration rather than direct sequence generation.
- Sequence-to-sequence models: Kalchbrenner and Blunsom mapped sentences to vectors with convolutional networks, which lose word ordering, while Cho et al. used an LSTM-like architecture mainly within SMT.Both approaches were closely related to mapping sentences into vectors and back.
- Direct generation: Hermann et al. mapped inputs and outputs to similar vector-space points but required database lookup or rescoring to generate translations.Their approach could not generate translations directly.
5 Conclusion
The authors conclude that a simple deep LSTM can outperform SMT on large-scale translation, with source reversal supporting optimization and long-sentence translation. They suggest broader sequence-learning potential when sufficient training data is available.
- Main conclusion: A large deep LSTM with limited vocabulary and minimal structural assumptions outperformed a standard SMT system on a large-scale machine-translation task.The authors present this as the central contribution of the work.
- Optimization: Reversing source words introduced short-term dependencies that simplified learning, while the authors did not experimentally verify the corresponding standard-RNN claim.The authors were surprised by the improvement from reversal.
- Long sentences: LSTMs trained on reversed data translated very long sentences with little difficulty despite concerns about limited memory and earlier reports of poor long-sentence performance.This result contrasts with the authors’ initial expectation that the LSTM would fail on long sentences.
- Broader implications: The relatively unoptimized approach suggests that further work could improve translation accuracy and that the method may transfer to other challenging sequence-to-sequence problems.The broader suggestion is explicitly conditioned by the paper’s results and approach.