PaperlayerView exact revision on arXiv

Source-linked AI summary

A Convolutional Encoder Model for Neural Machine Translation

Jonas Gehring, Michael Auli, David Grangier, Yann N. Dauphin

arXiv:1611.02344v3cs.CL

TL;DR

Neural machine translation commonly uses bi-directional LSTMs to encode source sentences, but recurrent computation is constrained by temporal dependencies. The paper proposes convolutional encoders that process source features simultaneously and reports competitive accuracy across translation benchmarks, while more than doubling CPU decoding speed at the same or higher accuracy as a strong bi-directional LSTM baseline.

  • Problem

    Bi-directional recurrent encoders constrain source-sentence computation through temporal dependencies, motivating a faster and simpler alternative.

  • Method

    The paper replaces recurrent source encoding with stacked convolutional layers, using position embeddings and residual connections for deep encoders.

  • Results

    The convolutional encoder is competitive with bi-directional LSTM baselines and state-of-the-art systems across WMT translation tasks, including 27.8 BLEU on WMT’15 English-German.

  • Takeaways & Limitations

    Convolutional encoders provide a more parallelizable source-encoding approach while retaining competitive or better translation performance than bi-directional LSTM baselines.

  • Takeaways & Limitations

    The paper identifies effectiveness on other sequence-to-sequence tasks, such as summarization, constituency parsing, and dialog modeling, as future work.

Abstract

from arXiv · show

The prevalent approach to neural machine translation relies on bi-directional LSTMs to encode the source sentence. In this paper we present a faster and simpler architecture based on a succession of convolutional layers. This allows to encode the entire source sentence simultaneously compared to recurrent networks for which computation is constrained by temporal dependencies. On WMT'16 English-Romanian translation we achieve competitive accuracy to the state-of-the-art and we outperform several recently published results on the WMT'15 English-German task. Our models obtain almost the same accuracy as a very deep LSTM setup on WMT'14 English-French translation. Our convolutional encoder speeds up CPU decoding by more than two times at the same or higher accuracy as a strong bi-directional LSTM baseline.

1 Introduction

The paper proposes convolutional encoders as a faster, simpler alternative to recurrent source encoders, exploiting parallel computation and shorter paths for sequence relationships. Experiments target competitiveness with recurrent and state-of-the-art systems.

  • Convolutional networks compute all source-sequence features simultaneously, unlike RNNs whose hidden-state dependencies prevent parallel computation within a sequence.
  • A succession of convolutional layers provides shorter paths for capturing sequence relationships and applies the same number of transformations to every word.
  • The architecture investigates average pooling and parameterized convolutions while using residual connections to enable very deep convolutional encoders.
  • The experiments compare convolutional encoders with uni-directional and bi-directional LSTM variants across several standard translation datasets.
  • The method achieves competitive accuracy on WMT’16 English-Romanian, WMT’15 English-German, and WMT’14 English-French translation tasks.

2 Recurrent Neural Machine Translation

The recurrent translation system uses an encoder-decoder architecture with soft attention: a bi-directional LSTM encodes the source, and an LSTM decoder generates target words sequentially. Dot-product attention connects decoder states to encoder outputs.

  • The encoder-decoder architecture maps a source sentence to a sequence of states that supplies representations to the decoder through soft attention.
  • The decoder LSTM updates its hidden state from the previous decoder state, the previous target-word embedding, and an encoder-derived conditional input.
  • The translation model converts each LSTM output into a distribution over possible next target words using a linear layer and softmax.
  • Dot-product attention transforms the decoder state, scores each encoder state, and forms the conditional input as a weighted sum of encoder outputs.
  • Preliminary experiments found no significant BLEU or perplexity advantage for MLP attention, while dot-product attention was more favorable for training and evaluation speed.
  • The BiLSTM encoder processes the source in forward and reversed directions, then concatenates the top-level states and projects them into encoder outputs.

3 Non-recurrent Encoders

The paper develops non-recurrent encoders using pooling and stacked convolutional networks with position embeddings, residual connections, and separate networks for attention and decoder conditioning. These designs address context size, positional information, and deep-network optimization.

  • 3.1 Pooling Encoder: The pooling encoder averages embeddings over consecutive word windows and adds position embeddings to represent each source word’s absolute position.
  • 3.1 Pooling Encoder: The pooling encoder pads inputs so its output length matches the input length and uses a kernel width of 5 in all experiments.
  • 3.2 Convolutional Encoder: Stacked convolutions expand the receptive field, while nonlinearities can adjust whether representations use the full available context or fewer words.
  • 3.2 Convolutional Encoder: Five convolutions with kernel width k = 3 produce an input field of 11 words for each output.
  • 3.2 Convolutional Encoder: Residual connections ease learning in deep encoders, while the convolutional network retains the full source-sequence length without down-sampling.
  • 3.2 Convolutional Encoder: Separate CNN-a and CNN-c networks compute attention-related encoder outputs and decoder conditional inputs, respectively.
  • 3.2 Convolutional Encoder: Using two different CNNs improved perplexity and BLEU compared with using a single CNN.

4 Experimental Setup

The experiments evaluate convolutional encoders on IWSLT’14 and three WMT translation tasks, using standardized preprocessing, model settings, optimization, and decoding procedures.

  • Datasets: The study evaluates IWSLT’14 German-English and WMT’16 English-Romanian, WMT’15 English-German, and WMT’14 English-French translation tasks.IWSLT’14 contains 167K training sentence pairs after filtering; the WMT datasets contain 2.8M, 3.9M, and 10.7M training pairs, respectively.
  • Datasets: For IWSLT’14, training sentences are limited to 175 words, while WMT’14 English-French removes sentences longer than 150 words.The IWSLT’14 test set is not length-filtered, and WMT results use the specified newstest or ntst evaluation sets.
  • Model configuration: The models use 512 hidden units in recurrent encoders and CNN-a, 256 units in CNN-c, and 256-dimensional embeddings.CNN-c uses linear transformations to match the smaller embedding size.
  • Optimization: Convolutional models are trained with SGD and learning-rate annealing, whereas recurrent models use Adam with early stopping based on validation perplexity.The convolutional learning rate starts at 0.1 and is reduced by an order of magnitude after validation perplexity stops improving.
  • Training procedure: Training uses mini-batches, truncated back-propagation through time for target sequences, gradient normalization, dropout, and single-GPU Torch implementations.Mini-batches contain 32 sentences for IWSLT’14 and 64 for WMT; target sequences are limited to 25 words per mini-batch.
  • Decoding and evaluation: Translations use beam search with sentence-length-normalized log-likelihoods, tuned beam widths and word penalties for WMT, and attention-based unknown-word replacement.BLEU is case-sensitive and tokenized except for WMT’16 English-Romanian, where detokenized BLEU is used for comparability.

5 Results

Across IWSLT’14 and three WMT tasks, convolutional encoders are competitive with recurrent baselines and prior systems, while depth, residual connections, and separate encoder stacks matter for accuracy and speed.

  • 5.1 Recurrent vs. Non-recurrent Encoders: A six-layer CNN-a with a three-layer CNN-c outperforms the BiLSTM by 0.7 BLEU on IWSLT’14 German-English.The six-three configuration was selected after independently varying CNN-a and CNN-c depth from 1 to 10 layers.
  • 5.1 Recurrent vs. Non-recurrent Encoders: Position information is crucial for shallow convolutional encoders, while deeper models are less affected by its removal.On IWSLT’14, the BiLSTM is 2.3 BLEU better than the uni-directional LSTM, while pooling is 1.3 BLEU below the uni-directional LSTM and 3.6 BLEU below the BiLSTM.
  • 5.1 Recurrent vs. Non-recurrent Encoders: Perplexity improvements translate into smaller BLEU gains for convolutional encoders than for recurrent counterparts.This relationship is also observed on larger datasets.
  • 5.2 Evaluation on WMT Corpora: On WMT’16 English-Romanian, an 8/4 convolutional encoder reaches 27.8 BLEU, outperforming the 512-unit BiLSTM baseline and approaching the state of the art.A single-layer convolutional encoder reaches 27.1 BLEU, while the baseline is 0.6 BLEU below the state-of-the-art result.
  • 5.2 Evaluation on WMT Corpora: On WMT’15 English-German, a 15/5 convolutional model outperforms BiLSTM encoders with either one or two decoder layers.The compared recurrent and convolutional encoders use 512 hidden units.
  • 5.2 Evaluation on WMT Corpora: On WMT’14 English-French, deep convolutional encoders outperform BiLSTM baselines by 0.3 BLEU with one decoder layer and 0.4 BLEU with two.The very deep convolutional encoder has 40% fewer parameters than the BiLSTM and is 0.2 BLEU below the best cited deep-LSTM system.
  • 5.3 Convolutional Encoder Architecture Details: Increasing CNN-c depth helps up to three layers and CNN-a depth up to six on IWSLT’14, while residual connections are needed to benefit from deeper encoders.A two-to-three-times larger CNN-a than CNN-c is reported as a useful depth rule of thumb.
  • 5.4 Training and Generation Speed: The convolutional encoder speeds overall IWSLT’14 generation by 1.35x while improving accuracy by 0.7 BLEU, and speeds WMT’15 English-German generation by 2.1 times.With equal embedding sizes, the convolutional encoder remains 1.34x faster despite requiring roughly 1.6x as many FLOPs.

6 Conclusion

The paper introduces convolutional encoders for neural machine translation as a more parallelizable alternative to recurrent encoders, with competitive translation performance. It also identifies position embeddings and separate CNNs for attention and conditional aggregation as essential design choices.

  • Convolutional encoders are more parallelizable than recurrent networks and provide a shorter path for capturing long-range source dependencies.
  • Position embeddings and different CNNs for attention scoring and conditional input aggregation are essential design choices.
  • Convolutional encoders perform on par or better than bi-directional LSTM encoder baselines.
  • The paper identifies summarization, constituency parsing, and dialog modeling as other sequence-to-sequence tasks for evaluating the architecture.

A Alignment Visualization

The alignment visualizations compare BiLSTM and deep convolutional encoders across English-German and English-French examples. BiLSTM attention can be sharp without correct alignment, while convolutional attention is broader and may exhibit systematic positional biases.

  • Attention plots place the translation on the x-axis and the source sentence on the y-axis.
  • BiLSTM attention is sharp but does not necessarily represent a correct alignment.
  • Convolutional attention is less focused but can still indicate an approximate source location during reordered translation.
  • Convolutional encoders tend to focus on the last token, or on both the first and last tokens, in the illustrated examples.
  • Providing the decoder and attention module with an explicit representation of source length changed neither attention patterns nor translation accuracy.

B Performance by Sentence Length

Performance by sentence length is evaluated by sorting WMT’15 English-German test sentences into length buckets and comparing model BLEU scores. The reported results provide no clear evidence that limited convolutional context causes sub-par translation on longer sentences.

  • The WMT’15 English-German test set is partitioned into 15 equally-sized buckets according to source sentence length for BLEU comparison.
  • The evaluation compares per-bucket BLEU scores for models with different encoder architectures and contexts.
  • There is no clear evidence that sentences longer than an encoder output’s observable context receive sub-par translations.
  • A 6-layer CNN-c with a 3-layer CNN-a performs worse than a 2-layer BiLSTM, at 23.3 BLEU versus 23.6.
  • With six convolutional layers of kernel width 3, each encoder output contains information from 13 adjacent source words.
Loading 1611.02344v3…