PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Architectures for Neural Machine Translation

Antonio Valerio Miceli Barone, Jindřich Helcl, Rico Sennrich, Barry Haddow, Alexandra Birch

arXiv:1707.07631v1cs.CL

TL;DR

The paper addresses the lack of thorough empirical comparison among architectures that add depth to neural machine translation. It evaluates stacked and deep-transition recurrent architectures, including their combination, and finds that BiDeep gives better accuracy than single-type-depth systems at the same parameter count.

  • Problem

    Different architectures add depth to NMT, but their empirical performance had not been thoroughly compared.

    Source: §1 · p. 1

  • Method

    The study evaluates stacked and deep-transition recurrent architectures alone and in combination, varying their connection schemes and depth.

    Source: §1 · p. 1

  • Results

    BiDeep, combining stacked and transition depth in the encoder and decoder, yields better accuracy at the same parameter count than systems using only one kind of depth.

    Source: §4 · p. 8

  • Takeaways & Limitations

    Deep-transition architectures perform comparably to stacked ones in accuracy and better in speed and parameter count.

    Source: §4 · p. 8

    The paper reports results for only one translation direction.

    Source: §4 · p. 8

Abstract

from arXiv · show

It has been shown that increasing model depth improves the quality of neural machine translation. However, different architectural variants to increase model depth have been proposed, and so far, there has been no thorough comparative study. In this work, we describe and evaluate several existing approaches to introduce depth in neural machine translation. Additionally, we explore novel architectural variants, including deep transition RNNs, and we vary how attention is used in the deep decoder. We introduce a novel "BiDeep" RNN architecture that combines deep transition RNNs and stacked RNNs. Our evaluation is carried out on the English to German WMT news translation dataset, using a single-GPU machine for both training and inference. We find that several of our proposed architectures improve upon existing approaches in terms of speed and translation quality. We obtain best improvements with a BiDeep RNN of combined depth 8, obtaining an average improvement of 1.5 BLEU over a strong shallow baseline. We release our code for ease of adoption.

1 Introduction

Neural machine translation commonly uses attention-based sequence-to-sequence models with single-layer recurrent networks. Because depth can be defined through different connection schemes and sub-networks, the paper compares stacked and deep-transition architectures for NMT.

1 Introduction

  • NMT commonly uses attention-based sequence-to-sequence models with single-layer recurrent networks in both encoder and decoder.The passage describes NMT as yielding the best results on most language pairs.Source: §1 · p. 1
  • Depth in recurrent and sequence-to-sequence models has multiple definitions and configurations, complicating comparison across architectures.Different connection schemes create distinct notions of depth, and several sub-networks in a sequence-to-sequence model can each be deep.Source: §1 · p. 1
  • The study compares stacked and deep-transition recurrent architectures, individually and in combination, varying their connections and depth on WMT news translation.Deep-transition architectures had been used for language modeling but not previously for NMT; empirical comparisons of deep architectures were lacking.Source: §1 · p. 1
  • Related architecture studies examined embedding size, cell type, depth, attention, and beam size, while convolutional and pervasive-attention models also varied depth.Britz et al. (2017) explored NMT architectures; Gehring et al. (2017) proposed convolutional models, and Vaswani et al. (2017) proposed pervasive attention.Source: §1 · pp. 1–2

2 NMT Architectures

The evaluated NMT architectures use GRU sequence-to-sequence transducers with attention, and the section describes the baseline and its variants.

2 NMT Architectures

  • All evaluated architectures are GRU sequence-to-sequence transducers with attention.The section describes the baseline system and the evaluated variants.Source: §2 · p. 2

2.1 Baseline Architecture

The baseline is Nematus augmented with layer normalization, and it already contains recurrent transition depth in the decoder and feed-forward depth in attention and output networks.

2.1 Baseline Architecture

  • The Nematus baseline is augmented with layer normalization, which the authors report improves translation quality and speeds training.The passage describes Nematus as the baseline architecture.Source: §2.1 · p. 2
  • The baseline decoder has two GRU transitions per output word, with attention between them.The first transition receives the previous target word, and the second receives an attention context vector.Source: §2.1 · p. 2
  • The baseline attention and output networks each have one tanh hidden layer.Both are multilayer perceptrons.Source: §2.1 · p. 2

2.2 Deep Transition Architectures

Deep-transition RNNs add sequential GRU transitions within each time step. The paper applies this design to the encoder and decoder, with word and attention inputs entering selected transitions.

2.2 Deep Transition Architectures

  • A deep-transition RNN applies multiple independently parameterized GRU transitions sequentially within each time step, with recurrence across the whole cell.The final transition's state is carried to the first transition at the next time step; applying this architecture to NMT is a novel contribution.Source: §2.2 · p. 2

2.2.1 Deep Transition Encoder

  • The deep-transition encoder remains bidirectional, with the source-word embedding entering the first transition and the previous word's final state supplying recurrence.The reverse states are computed similarly and concatenated with the forward states.Source: §2.2.1 · p. 2

2.2.2 Deep Transition Decoder

  • The deep-transition decoder extends the baseline to arbitrary transition depth, using the previous target-word embedding and attention context as inputs to its first two transitions.Later transitions have no external inputs, and the final state feeds the output network; Figure 1 depicts the decoder.Source: §2.2.2 · pp. 2–3, Figure 1 · p. 2
  • The decoder's feed-forward output network can also be deepened by adding hidden layers.Source: §2.2.2 · p. 3

2.3 Stacked architectures

The section defines stacked RNN architectures and compares encoder and decoder variants, including their combinations with deep-transition architectures. Stacked networks run multiple RNNs over the same time steps, with higher layers receiving states from lower layers and residual connections supporting information flow.

2.3 Stacked architectures

  • Stacked RNNs run multiple recurrent networks over the same time steps, passing lower-layer states upward and using residual connections between levels.Unlike deep-transition GRUs, each stacked GRU transition block has its own recurrent state carried across time.Source: §2.3 · p. 3

2.3.1 Stacked Encoder

  • The study empirically compares stacked encoder architectures alone and in combination with deep-transition architectures, while excluding stacked unidirectional encoders.The authors cite prior evidence that bidirectional encoders outperform unidirectional ones as the reason for that exclusion.Source: §2.3.1 · p. 3
  • Alternating encoders switch processing direction across layers and concatenate top-layer forward and backward states; biunidirectional encoders stack forward-only GRUs above shallow bidirectional states.The biunidirectional design has shorter maximum information-propagation paths, which may make it less expressive, but enables higher model parallelism; its higher GRUs have twice the base state size.Source: §2.3.1 · pp. 3–4

2.3.2 Stacked Decoder

  • Stacked decoders vary how higher levels use states and attention: GRU and rGRU variants reuse base-level context, while cGRU levels have independent attention.In stacked crGRU, higher conditional GRUs reuse the context vector computed at the base level; the base RNN is always a conditional GRU with at least two transitions per step.Source: §2.3.2 · p. 4

2.4 BiDeep architectures

BiDeep combines stacked depth with deep-transition depth. Its encoder and decoder replace recurrent cells with multi-layer deep-transition cells, and transition depth can vary across stacking levels.

2.4 BiDeep architectures

  • BiDeep is a novel architecture that combines deep transitions with stacking.The paper also identifies applying this architecture to NMT as a novel contribution.Source: §2.4 · p. 5
  • A BiDeep encoder replaces each recurrent GRU cell in a stacked encoder with a multi-layer deep-transition cell composed of GRU transition blocks.The alternating BiDeep encoder applies these cells within the alternating stack and retains residual addition across levels.Source: §2.4 · p. 5
  • BiDeep decoders similarly replace recurrent cells with deep-transition multi-layer cells, and transition depths can differ across stacking levels.Source: §2.4 · p. 5

3 Experiments

On English–German WMT-2017, the experiments compare deep architectures for translation quality, speed, and size. The depth-8 BiDeep model gives the largest reported BLEU gains over the best deep encoder and the shallow baseline.

3 Experiments

  • Experiments use WMT-2017 English–German training data, newstest2013 validation, and BLEU on newstest2014–2016; training speed is measured on one Titan X (Pascal) GPU.Nematus is used with prior preprocessing and hyperparameters; deep models are usually four times the baseline depth, with diminishing returns observed in preliminary deeper-model experiments.Source: §3 · p. 5

3.1 Layer Normalization

  • 0.8–1 BLEU average improvement: layer normalization benefits both the shallow baseline and the depth-4 alternating encoder and substantially reduces training time.The authors use layer normalization in subsequent experiments.Source: §3.1 · p. 5, Table 1 · p. 6

3.2 Deep Encoders

  • +0.5–+1.2 BLEU over baseline: all deep encoders improve quality, with no consistent quality differences; the deep transition encoder is best on parameters and training speed.On a single GPU, the biunidirectional encoder may be comparatively faster with multiple GPUs because of its higher model parallelism.Source: §3.2 · p. 5, Table 2 · p. 6

3.3 Deep Decoders

  • +0.6 BLEU on average: stacked cGRU with independent multi-step attention has the strongest deep-decoder gains, but is slowest and largest; deep transition is fastest and smallest among improving deep decoders.Reused-attention crGRU gains +0.3 BLEU on average, with intermediate speed and size; stacked GRU is near baseline and deep output4 decreases BLEU.Source: §3.3 · pp. 5–6, Table 3 · p. 7

3.4 Deep Encoders and Decoders

  • +1.5 BLEU over the shallow baseline and +0.6 BLEU over the best deep encoder: depth-8 BiDeep also improves cross-entropy, while the stacked-only model performs similarly to smaller models.The comparison uses models with combined depth eight times that of the baseline.Source: §3.4 · p. 6, Table 4 · p. 7
  • Deepening both encoder and decoder generally improves cross-entropy but not BLEU over the deep-transition encoder alone; equal-total-depth BiDeep adds +0.2 BLEU on average.Alternating encoder plus stacked GRU performs better than the other compared encoder–decoder architecture on BLEU, model size, and training speed; other variants do not beat the best deep encoder with a shallow decoder.Source: §3.4 · p. 6, Table 4 · p. 7

3.5 Error Analysis

  • On Lingeval97 contrastive subject–verb agreement pairs, BiDeep achieved the highest accuracy at 94.3%; the deep recurrent transition network scored over 92.5% (N = 560), above the shallow decoder (91.6%) and similar to the stacked GRU (92.7%).Figure 5 plots accuracy by subject–verb distance; the text reports that information is successfully passed over distances of 16 or more words (at least 128 GRU transitions).Source: §3.5 · p. 7, Figure 5 · p. 7

4 Conclusions

The study compares deeper NMT architectures, finding that depth improves BLEU especially in the encoder, while decoder depth improves cross-entropy more than BLEU. BiDeep yields better accuracy at the same parameter count, while deep transition architectures are recommended when speed or model size matters.

4 Conclusions

  • Alternating stacked encoders outperform biunidirectional stacked encoders in accuracy and single-GPU speed; deep transition architectures match stacked accuracy and improve speed and parameter count.Source: §4 · pp. 7–8
  • Depth improves BLEU especially in the encoder, while decoder depth improves cross-entropy without strongly improving BLEU.Source: §4 · p. 8
  • BiDeep combines stacked and transition depth in the alternating encoder and decoder, yielding better accuracy for the same number of parameters than systems using only one kind of depth.Source: §4 · p. 8
  • Combined architectures are recommended for maximum accuracy, while deep transition architectures are recommended when speed or model size matters.Source: §4 · p. 8
  • The paper reports one translation direction; follow-up work confirmed effectiveness across data conditions and language pairs.Source: §4 · p. 8

Related papers

Loading 1707.07631v1…