PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation

Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, Wei Xu

arXiv:1606.04199v3cs.CLcs.LG

TL;DR

Single NMT models were shallower and less competitive than the best conventional machine-translation system, motivating deeper architectures. The paper introduces fast-forward connections and an interleaved bi-directional LSTM stack, achieving state-of-the-art single-model performance and 40.4 BLEU after ensembling and unknown-word processing.

  • Problem

    Single NMT models were generally shallow and had not matched the best conventional system on WMT’14 English-to-French.

  • Method

    The paper uses fast-forward connections and an interleaved bi-directional architecture to build deep LSTM-based NMT models.

  • Results

    37.7 BLEU was achieved by a single deep attention model, while ensembling and unknown-word processing produced 40.4 BLEU on WMT’14 English-to-French.

  • Takeaways & Limitations

    A single deep NMT model achieved state-of-the-art performance and outperformed the best conventional SMT system on WMT’14 English-to-French.

  • Takeaways & Limitations

    Further improvements were not obtained with the current depth-16 topology and training techniques, and deeper models remained future work.

Abstract

from arXiv · show

Neural machine translation (NMT) aims at solving machine translation (MT) problems using neural networks and has exhibited promising results in recent years. However, most of the existing NMT models are shallow and there is still a performance gap between a single NMT model and the best conventional MT system. In this work, we introduce a new type of linear connections, named fast-forward connections, based on deep Long Short-Term Memory (LSTM) networks, and an interleaved bi-directional architecture for stacking the LSTM layers. Fast-forward connections play an essential role in propagating the gradients and building a deep topology of depth 16. On the WMT'14 English-to-French task, we achieve BLEU=37.7 with a single attention model, which outperforms the corresponding single shallow model by 6.2 BLEU points. This is the first time that a single NMT model achieves state-of-the-art performance and outperforms the best conventional model by 0.7 BLEU points. We can still achieve BLEU=36.3 even without using an attention mechanism. After special handling of unknown words and model ensembling, we obtain the best score reported to date on this task with BLEU=40.4. Our models are also validated on the more difficult WMT'14 English-to-German task.

1 Introduction

NMT models were promising but remained shallow and behind the best conventional system, motivating deeper architectures with improved gradient propagation. The proposed fast-forward connections enabled depth-16 NMT and substantially improved WMT’14 English-to-French performance.

  • 37.0 BLEU was achieved by the best conventional system, versus 31.5 for the best six-layer single NMT model on WMT’14 English-to-French.
  • Deep topology was pursued because NMT had reached only six successfully used layers, unlike computer-vision systems with tens or hundreds of layers.The authors attribute the NMT depth barrier partly to gradient attenuation through nonlinear and recurrent LSTM computations.
  • Fast-forward connections and an interleaved bi-directional encoder supported a depth-16 topology applicable to both encoder-decoder and attention NMT models.
  • 37.7 BLEU was obtained with a single deep attention model, improving 6.2 BLEU points over the six-layer shallow model.
  • 0.7 BLEU points separated the single deep NMT model from the best conventional SMT system, while ensembling and unknown-word processing raised the score to 40.4 BLEU.

2 Neural Machine Translation

Neural machine translation generates target sequences from source sequences with neural models, using encoding, an interface representation, and decoding. Existing NMT systems are generally shallow, motivating new connections to facilitate gradient propagation in deeper LSTM-based architectures.

  • NMT formulation: NMT maximizes the likelihood p(y | x, θ) of a target sequence given a source sequence.The source and target are represented as word sequences, with θ denoting the learnable parameters.
  • NMT architecture: The NMT process consists of encoding the source into vectors, forming a representation at the interface, and decoding the target sequence.The encoder produces vectors e for source time steps, interface operations derive c, and the decoder generates the target from c.
  • NMT architecture: Encoder-decoder models use a single vector from the encoder, whereas attention models dynamically obtain c from target–source relationships.The two model types differ specifically in how the interface representation is constructed.
  • Motivation for depth: Most existing NMT topologies are shallow, with one LSTM layer in each attention component and at most six encoder-decoder LSTM layers reported.The passage frames machine translation as difficult and motivates more complex encoding and decoding networks.
  • Motivation for depth: Deep vision models benefit from shortcut connections, while training LSTM-based language models is more challenging because gradient propagation must be facilitated.The paper therefore proposes new connections to support gradient propagation in deeper NMT networks.

3 Deep Topology

The model builds deep recurrent networks by combining fast-forward connections between adjacent feed-forward blocks with interleaved bi-directional LSTM stacking. The resulting topology supports separate encoder, interface, and decoder components for Deep-ED and Deep-Att.

  • Network topology: The full network contains an encoder, an interface, and a decoder, with Deep-ED and Deep-Att differing in how the interface extracts source representations.Deep-ED uses static representations, whereas Deep-Att concatenates time-step outputs and applies soft attention.
  • Recurrent layer: The recurrent computation is separated into a feed-forward block f and a subsequent recurrent block r, whose composition is equivalent to a basic RNN.The same decomposition underlies the fast-forward design between stacked layers.
  • Recurrent layer: Fast-forward connections link feed-forward blocks across adjacent recurrent layers, giving each block both feed-forward and recurrent outputs from the previous layer.These connections provide a fast information path and a channel for gradient propagation through the deep topology.
  • Encoder and decoder: The encoder alternates processing directions across adjacent LSTM layers, while the decoder uses a fixed forward direction.The direction term (−1)^k marks the alternating encoder directions.
  • LSTM layer: The LSTM maps its input and previous state-output pair to the current state-output pair using gated nonlinear computations.The notation explicitly defines element-wise multiplication, gate activations, and LSTM parameters.
  • Implementation: The final model applies Half(f) and dropout Dr(h) in the fast-forward computation; Half(f) reduces parameter size without affecting performance in the reported experiments.Using only the first third of f caused noticeable performance degradation.

4 Experiments

The experiments evaluate the proposed models primarily on WMT’14 English-to-French and additionally on the more difficult English-to-German translation task.

  • Evaluation: The method is evaluated mainly on WMT’14 English-to-French and also on WMT’14 English-to-German to validate it on a more difficult language pair.The models are implemented in the PADDLE parallel distributed deep learning platform.

4.1 Data sets

The experiments use the full WMT’14 parallel corpora for English-to-French and English-to-German translation, with consistent development and test partitioning.

  • Corpus composition: The English-to-French corpus contains 36 million sentence pairs, while the English-to-German corpus contains 4.5 million.The English-to-French data combine Europarl v7, Common Crawl, UN, News Commentary, and Gigaword; English-to-German uses Europarl v7, Common Crawl, and News Commentary.
  • Data partition: The news-test-2012 and news-test-2013 sets form the development set, and news-test-2014 is the test set.This partition matches previous NMT work for fair comparison.
  • Vocabulary: The source vocabulary contains the 200K most frequent words; target vocabularies contain 80K French words and 160K German words.Out-of-vocabulary words are replaced with the unknown symbol ⟨unk⟩.

4.2 Model settings

Deep-ED and Deep-Att share the same configuration and layer sizes, differing only in their interface components. The models use relatively narrow embeddings and LSTM layers compared with prior work.

  • Model configuration: Deep-ED and Deep-Att have identical configurations and layer sizes except for the interface P-I.Their context-vector dimensions are 5120 for Deep-ED and 1280 for Deep-Att.
  • Model dimensions: Both models use 256-dimensional word embeddings and LSTM layers with 512 memory cells.The encoder contains 2×n_e LSTM layers and the decoder contains n_d layers.
  • Model dimensions: The output layer matches the target vocabulary size, and gate, input, and output activations are sigmoid, tanh, and tanh respectively.The context-vector dimensions differ between Deep-ED and Deep-Att because of their distinct interfaces.
  • Scale study: Larger-scale models did not provide further improvements, despite the proposed network being narrower than earlier models.Previous work used 1000-dimensional word embeddings and LSTM layers, compared with the 256-dimensional embeddings and 512-cell layers here.

4.3 Optimization

The model uses separate optimization settings for recurrent and non-recurrent computations, with regularization and large-scale training supporting the deep network. Performance is reported as robust to small setting variations.

  • Learning rates: Recurrent computation uses lr = 5 × 10−4, while feed-forward computation, embeddings, and softmax use lf = 4 × 10−5.The distinction reflects the different computation types within each LSTM layer.
  • Regularization: L2 regularization constrains parameter matrix v, with r = 2 for all layers except the two embedding layers.The regularization strength is applied to the other model layers.
  • Initialization: Recurrent parameters are initialized to zero, while non-recurrent parameters use random initialization with zero mean and standard deviation 0.07.These initialization rules distinguish the two parameter groups.
  • Training setup: Training uses dropout ratio pd = 0.1 and batches of 500 ∼800 sequences, depending on sequence lengths and model size.Larger batches improve convergence slightly but are constrained by GPU memory.
  • Robustness: Small variations in the optimization settings do not affect the deep model’s final performance.The authors characterize the model as insensitive to these settings.

4.4 Results

The experiments show that fast-forward connections and increased depth substantially improve NMT performance, while interleaved bidirectional encoding and post-processing provide additional gains. The topology also performs competitively on English-to-German translation and supports efficient generation.

  • English-to-French single models: BLEU=37.7 is achieved by the deepest Deep-Att model, improving over the previous six-layer single NMT score of BLEU=31.5.The best Deep-Att configuration uses ne = 9 and nd = 7.
  • Generation efficiency: Beam size = 3 gives the best generation results, compared with about 12 in previous NMT systems.The smaller beam size significantly improves generation efficiency.
  • Fast-forward connections: Fast-forward connections improve shallow Deep-Att from BLEU=31.2 to BLEU=32.3 and enlarge the gain from 1.1 to 1.4 BLEU points at greater depth.Without fast-forward connections, larger-depth training frequently encounters parameter exploding.
  • Fast-forward connections: A two-times wider LSTM without fast-forward connections reaches BLEU=33.8, remaining behind the corresponding Deep-Att model with fast-forward connections.This comparison controls for the reduced model size caused by removing the connections.
  • English-to-German: BLEU=20.6 on English-to-German is similar to the conventional SMT result of 20.7 and outperforms the shallow attention models.The authors report consistency with the English-to-French findings.
  • Post processing: BLEU=40.4 is obtained after ensembling Deep-Att models and recovering unknown words with PosUnk, while the single post-processed model reaches BLEU=39.2.The ensemble alone reaches 38.9, and unknown-word recovery contributes an additional 1.5 BLEU points.

4.5 Analysis

The analysis examines how depth, unknown-word handling, and sequence length affect translation performance and generalization. Deep attention models remain strong on long inputs, while unknown-word processing substantially improves BLEU.

  • Source sequence length: Deep-Att outperforms the previous Enc-Dec and SMT models on nearly all source lengths.Deep-ED also outperforms shallow Enc-Dec across nearly all lengths.
  • Source sequence length: For sequences longer than 70 words, Deep-Att does not degrade relative to Enc-Dec, whereas Deep-ED degrades and falls behind Deep-Att.
  • Unknown words: 40.3 BLEU is achieved by Deep-Att on the 1705-sentence subset without unknown target words, versus 37.7 on the full test set.The subset comprises 56.8% of the original test set.
  • Unknown words: 41.4 BLEU is achieved by the ensemble on the subset without unknown words, compared with 37.5 for SMT on that subset.
  • Over-fitting: Deep models show better test-set performance at the shallow models’ training-set token error rate.Token error rate is the ratio of incorrectly predicted words in the target sequence with correct historical input.

5 Conclusion

Fast-forward connections create a low-decay gradient path that enables depth-16 NMT topologies. These models improve translation quality, remain efficient in generation, and show advantages on long sequences, although deeper models did not improve further with the current design.

  • Conclusion: Fast-forward connections provide a non-linear- and recurrence-free gradient path, enabling depth-16 NMT topologies.Gradients decay much more slowly on this path than in standard deep networks.
  • Conclusion: 37.7 BLEU marks a 6.2-point improvement over the previous best single model, while ensemble processing and unknown-word handling reach 40.4 BLEU.The single model outperforms the best conventional SMT system; 40.4 improves the previous best result by 2.9 points.
  • Conclusion: Beam size 3 produces the best single-model and ensemble results, compared with about 12 in previous NMT systems.
  • Conclusion: The analysis finds deep models advantageous for learning long sequences and resistant to over-fitting.
  • Conclusion: Deeper models produced no further improvements with the current topology and training techniques.The authors identify deeper designs and training techniques as future work.
Loading 1606.04199v3…