Source-linked AI summary
On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, Yoshua Bengio
TL;DR
Neural machine translation was new enough that its sentence-level properties, vocabulary effects, and failure cases remained insufficiently analyzed. This paper evaluates RNN Encoder–Decoder and gated recursive convolutional models, finding rapid degradation with sentence length alongside automatic grammatical-structure learning by grConv.
Problem
The study addresses limited analysis of neural machine translation’s sentence properties, vocabulary effects, and failure cases.
Method
The paper analyzes RNN Encoder–Decoder and gated recursive convolutional models on French-to-English translation.
Results
Performance degrades quickly with sentence length, while grConv learns a kind of syntactic structure without supervision.
Takeaways & Limitations
The findings identify sentence length as a weakness of neural translation and suggest grConv may suit other natural-language-processing applications.
Takeaways & Limitations
The fixed-length representation may lack sufficient capacity for long sentences with complicated structure and meaning.
Abstract
from arXiv · showhide
Neural machine translation is a relatively new approach to statistical machine translation based purely on neural networks. The neural machine translation models often consist of an encoder and a decoder. The encoder extracts a fixed-length representation from a variable-length input sentence, and the decoder generates a correct translation from this representation. In this paper, we focus on analyzing the properties of the neural machine translation using two models; RNN Encoder--Decoder and a newly proposed gated recursive convolutional neural network. We show that the neural machine translation performs relatively well on short sentences without unknown words, but its performance degrades rapidly as the length of the sentence and the number of unknown words increase. Furthermore, we find that the proposed gated recursive convolutional network learns a grammatical structure of a sentence automatically.
1 Introduction
The paper analyzes neural machine translation as an encoder–decoder approach, evaluating an RNN Encoder–Decoder and gated recursive convolutional neural network on French-to-English translation. It finds that performance degrades with longer source sentences and vocabulary changes, while the gated recursive model learns a kind of syntactic structure without supervision.
- Approach: Neural machine translation uses an encoder to extract a fixed-length representation and a decoder to generate a variable-length target translation.
- Motivation: Neural machine translation requires 500MB of memory in the trained models, compared with tens of gigabytes for existing statistical machine translation systems.
- Research questions: The analysis examines how sentence properties, vocabulary choices, and failure cases affect neural machine translation behavior.
- Models and evaluation: The paper analyzes the RNN Encoder–Decoder and a gated recursive convolutional neural network on French-to-English translation.The gated recursive convolutional network replaces the RNN Encoder–Decoder’s encoder.
- Findings: The models’ performance degrades quickly as source-sentence length increases, and vocabulary size has a high impact on translation performance.Despite these effects, both models generate correct translations most of the time, and gated recursive convolutional networks learn a kind of syntactic structure without supervision.
2 Neural Networks for Variable-Length Sequences
This section presents recurrent and gated recursive convolutional neural networks for processing variable-length sequences. RNNs maintain hidden states over time, while the proposed grConv recursively transforms sequences into fixed-length vectors using adaptive gating that can form tree-like structures.
- Recurrent neural network: RNNs process variable-length sequences by maintaining a hidden state over time.The hidden state is updated at each timestep as the network reads the sequence.
- Recurrent neural network: RNNs can learn sequence distributions by modeling the conditional distribution of each next input given preceding inputs.For 1-of-K vectors, the model represents probabilities over all possible symbols and thereby defines a joint sequence distribution.
- Gated recursive convolutional network: The proposed grConv recursively applies shared convolutional parameters until an input sequence becomes a single fixed-length vector.Its architecture augments recursive convolution with gating units and uses four weight matrices: W_l, W_r, G_l, and G_r.
- Gated recursive convolutional network: Its gating adaptively chooses a new activation from both children or copies the left or right child activation.The gates therefore allow the recursive convolution’s overall structure to change according to each input sample.
- Gated recursive convolutional network: With hard 1-of-K gating, grConv adapts to each input and forms a tree-like structure, suggesting a form of unsupervised parsing.The paper leaves further investigation of the learned structure for future research.
3 Purely Neural Machine Translation
Purely neural machine translation learns a conditional distribution over target sentences given source sentences, using an encoder–decoder architecture. This paper analyzes direct translation with two gated model configurations.
- 3 Purely Neural Machine Translation: Neural translation models learn p(f | e) from bilingual parallel data and generate target sentences by sampling or approximate maximum-search decoding.Here, e denotes the source sentence and f the target sentence.
- 3 Purely Neural Machine Translation: The encoder processes a variable-length source sentence into a fixed-length representation z, which conditions the decoder’s variable-length target sequence.This encoder–decoder architecture is the common core of recent neural translation work.
- 3 Purely Neural Machine Translation: Earlier encoder–decoder systems mainly supported statistical machine translation by reranking n-best lists or adding scores to phrase tables.The paper instead focuses on direct translation performance, following Sutskever et al. (2014).
- 3 Purely Neural Machine Translation: Both analyzed configurations use an RNN with a gated hidden unit, while their encoders differ between a gated RNN and a proposed gated recursive convolutional network.The gated hidden unit avoids requiring a non-trivial method for determining target length.
4 Experiment Settings
The experiments evaluate two neural encoder–decoder models on English-to-French translation using a selected parallel corpus, length and vocabulary restrictions, and beam-search decoding. Training uses minibatch stochastic gradient descent with AdaDelta, while decoding normalizes log-probability by translation length and excludes unknown-word hypotheses.
- Data and evaluation: 348M selected parallel-corpus words support English-to-French evaluation on news-test2012, news-test2013, and news-test2014.The corpus combines Europarl, news commentary, UN, and two crawled corpora; no separate monolingual data are used.
- Data and evaluation: Training retains sentence pairs with at most 30 words per English and French sentence and limits each language to its 30,000 most frequent words.All other rare words are mapped to the special [UNK] token.
- Models and training: The RNNenc and grConv models are trained with minibatch stochastic gradient descent using AdaDelta.Both models use an RNN decoder with gated hidden units; their recurrent nonlinearities are tanh for RNNenc and rectifier max(0, x) for grConv.
- Decoding: Beam search uses beam-width s = 10, excludes hypotheses containing unknown words, and retains candidates with the highest decoder log-probabilities.For finding the k best translations, the model uses translation-length-normalized log-probability to prevent preference for shorter outputs.
5 Results and Analysis
Neural machine translation performs well on short, known-word sentences but degrades as sentence length or unknown-word count increases. Its performance remains below phrase-based SMT overall, while grConv learns a plausible grammatical structure automatically.
- Translation quality by sentence difficulty: Both RNNenc and grConv perform relatively well on short sentences but suffer significantly as sentence length increases.A similar degradation occurs as the number of unknown words increases, including for grConv.
- Hybrid translation systems: Combining neural machine translation with the existing phrase-based system improves overall translation performance.The paper also reports similar behavior when models are trained using sentences of up to 50 words.
- Limitations of fixed-length encoding: Phrase-based SMT achieves higher BLEU scores on longer sentences, consistent with fixed-length representations lacking capacity for complicated sentence structure and meaning.The neural approach may sacrifice important input topics when encoding variable-length sequences.
- Comparison with phrase-based SMT: 27.81 is the RNNenc BLEU score versus 33.08 for Moses on 10–20-word sentences with no unknown words.Under these conditions, the performance difference between neural translation and phrase-based SMT diminishes substantially.
- Structure learned by grConv: The grConv encoder learns a hierarchical structure that merges “of the United States” with “is the President of,” then combines this with “Obama is” and “.”.This learned structure is well correlated with linguistic intuition despite grConv’s lower performance than the RNN Encoder–Decoder.
6 Conclusion and Discussion
The study evaluates RNN and gated recursive convolutional encoders for neural machine translation, finding strong qualitative translations but substantial degradation with sentence length. The grConv also learns grammatical structure without syntactic supervision, while future work targets scalability, long-sentence performance, and decoder architectures.
- Conclusion and Discussion: The comparison is limited because grConv received one-third as many gradient updates as RNNenc, and neither model was trained to convergence.The models were compared after equal training time, so longer training could change the result.
- Conclusion and Discussion: The study compares an RNN with gated hidden units and a gated recursive convolutional neural network as encoder–decoder translation models.Both models were trained on English–French sentence pairs and evaluated using BLEU scores.
- Conclusion and Discussion: Neural machine translation performance suffers significantly as sentence length increases, although both models can qualitatively generate correct translations.The analysis examined performance by sentence length and the presence of unknown or rare words.
- Conclusion and Discussion: Future research should scale computation and memory for larger vocabularies, address long-sentence underperformance, and explore alternative neural architectures, especially decoders.The authors suggest decoder representational power may contribute to the shared sentence-length limitation.
- Conclusion and Discussion: The grConv mimics an input sentence’s grammatical structure without supervision, suggesting suitability for natural language processing applications beyond machine translation.This property was observed specifically for the proposed gated recursive convolutional neural network.