Convolutional Sequence to Sequence Learning

1. Introduction

• Convolutional networks parallelize computation across sequence elements, unlike recurrent networks whose previous-time-step dependence prevents within-sequence parallelization.Stacking layers also allows control over the maximum dependency length.Source: §1 · p. 1
• Fixed-depth convolutional processing applies a constant number of nonlinearities, easing optimization compared with recurrent processing whose operations grow with input length.Hierarchical representations provide shorter paths for modeling long-range dependencies.Source: §1 · p. 1
• Earlier convolutional sequence models had not demonstrated improved state-of-the-art results on large benchmark datasets, while some retained recurrent decoders or relied on restricted evaluations.This paper targets large-scale translation and summarization benchmarks with a fully convolutional design.Source: §1 · pp. 1–2
• The proposed sequence-to-sequence architecture is entirely convolutional, with gated linear units, residual connections, and attention in every decoder layer.Each attention layer adds only negligible overhead, enabling the model to address large-scale problems.Source: §1 · p. 2
• The approach achieves a new WMT’16 English-Romanian state of the art by 1.8 BLEU and beats Wu et al. (2016) by 0.5 BLEU on English-German and 1.5 BLEU on English-French.It also translates unseen sentences at an order of magnitude faster speed on GPU and CPU hardware.Source: §1 · p. 2

2. Recurrent Sequence to Sequence Learning

• An encoder RNN maps the input sequence to state representations, and a decoder RNN generates the variable-length output from left to right.Each new decoder state depends on the previous state, the previous target-word embedding, and a conditional representation.Source: §2 · p. 2
• Attention computes a weighted sum of encoder states at each decoder step, allowing the model to focus on different input positions during generation.The weights are attention scores normalized over the input elements.Source: §2 · p. 2
• LSTMs and GRUs extend recurrent networks with gating mechanisms that help retain information across time and model long-term dependencies.Bidirectional encoders additionally represent both past and future contexts.Source: §2 · p. 2

• The architecture is entirely convolutional, replacing recurrent encoder and decoder state computation with CNNs.Source: §3 · p. 2

3.1. Position Embeddings

• Position embeddings are added to token embeddings for both source and previously generated target elements, supplying sequence-order information.These representations are fed into the encoder and decoder networks.Source: §3.1 · p. 2

3.2. Convolutional Block Structure

• Stacking convolutional blocks expands each state’s receptive field; six blocks with kernel width 5 make each output depend on 25 inputs.Non-linearities let the network use the full receptive field or focus on fewer elements.Source: §3.2 · pp. 2–3
• Each convolution maps k d-dimensional inputs to a 2d-dimensional output, then GLUs gate the resulting context representation.The gates control which current-context inputs remain relevant, and the GLU output has half the convolution output dimensionality.Source: §3.2 · p. 3

3.3. Multi-step Attention

• Each decoder layer has its own attention mechanism, whose weighted encoder-context result is added to the corresponding decoder-layer output.Attention operates through multiple hops: earlier layers provide source context that later layers use when computing their own attention.Source: §3.3 · pp. 3–4

3.4. Normalization Strategy

• The convolutional design batches attention across all sequence elements, while residual connections and activation scaling support deep networks and stable variance.Residual blocks and attention are scaled to limit changes in activation variance, under assumptions described as imperfect but effective in practice.Source: §3.2 · p. 3, §3.3 · p. 4, §3.4 · p. 4

• The evaluation covers WMT’16 English-Romanian, WMT’14 English-German, WMT’14 English-French, and abstractive summarization.Summarization uses Gigaword training data and DUC-2004 and Gigaword test sets with ROUGE-1, ROUGE-2, and ROUGE-L.Source: §4.1 · pp. 4–5

WMT’16 English-Romanian

• The English-Romanian setup contains 2.8M training sentence pairs after removing sentences longer than 175 words and evaluates on newstest2016.Source: §4.1 · p. 4

WMT’14 Translation

• The English-German setup uses 4.5M sentence pairs and 40K BPE sub-word tokens, while English-French uses 35.5M filtered pairs and 40K BPE types.Source: §4.1 · p. 5

• Models generally use 512 hidden units and embeddings, with validation subsets controlling early stopping and learning-rate annealing.Source: §4.2 · p. 5, §4.1 · p. 5
• Training uses Nesterov accelerated gradient, dropout, weight normalization, and GPU computation; reported results average three runs with beam-search decoding.English-French training uses up to eight GPUs synchronously, and beam width is 5.Source: §4.2 · p. 5, §4.3 · p. 5

5.1. Recurrent vs. Convolutional Models

• 1.8 BLEU with BPE and 1.3 BLEU with a word-factored vocabulary over the WMT’16 English-Romanian winner establish ConvS2S’s strongest reported translation gains.The model used 20 encoder and 20 decoder layers with width-3 kernels and trained for 6–7.5 days on one GPU.Source: §5.1 · p. 6
• 1.5 BLEU over likelihood-trained GNMT and 0.5 BLEU over its reinforcement-learning models show gains on the much larger WMT’14 English-French task.The comparison uses a simple token-level likelihood objective for ConvS2S.Source: §5.1 · p. 6

5.3. Generation Speed

• 9.3× faster generation on K40, 13.7× on M40, and 21.3× on GTX-1080ti show substantial GPU inference-speed advantages over GNMT.On CPU, ConvS2S is up to 9.3× faster overall and 17× faster per core, although the compared systems used different core counts.Source: §5.3 · p. 7

5.5. Multi-step Attention

• Attention in every decoder layer gives the best validation perplexity, while adding four attention modules causes only a 4% training slowdown.The all-layer setup processes 3624 target words per second versus 3772 words per second with attention in one layer.Source: §5.5 · p. 8

5.6. Kernel size and Depth

• Deeper encoders help more than deeper decoders, with encoder accuracy improving through nine layers before plateauing; narrow kernels also outperform wider ones.Decoder configurations with two layers already perform well, while many-layer narrow-kernel encoders can reduce computation.Source: §5.6 · pp. 8–9

6. Conclusion and Future Work

• The fully convolutional model outperforms strong recurrent models on very large benchmark datasets at an order of magnitude faster speed.The approach uses gating and multiple attention steps.Source: §6 · p. 9
• 1.8 BLEU, 1.5 BLEU, and 0.5 BLEU improvements are reported on WMT’16 English-Romanian, WMT’14 English-French, and WMT’14 English-German, respectively.The paper describes these results as new state-of-the-art performance on several public translation benchmarks.Source: §6 · p. 9

A. Weight Initialization

• The initialization scheme is tailored to GLU activations by tracking activation variance in both forward and backward passes.It also modifies initialization to account for dropout.Source: A. Weight Initialization · p. 12

A.1. Forward Pass

• The forward analysis assumes i.i.d. inputs and weights to characterize the variance of a convolutional layer’s output.The convolutional output is written as y_l = W_l x_l + b_l.Source: §A.1 · p. 12
• GLU output variance is bounded using symmetry and sigmoid relations, with a lower bound based on one quarter of the linear-part variance.The analysis also derives an upper bound on the sigmoid-square expectation.Source: §A.1 · p. 12
• Small embedding variance permits dropping the quadratic term when approximating GLU output variance.The embeddings are initialized with variances around 0.01.Source: §A.1 · p. 12
• The analysis generalizes the variance relation across L GLU layers and targets stable activations rather than exponential magnification or reduction.The corresponding initialization uses the number of layer inputs to set weight variance.Source: §A.1 · p. 12

A.2. Backward Pass

• Backpropagation expresses convolutional-layer gradients through rearranged weights and derives the variance of gradients entering successive layers.The same weights appear in forward and backward variance calculations.Source: §A.2 · p. 12
• The backward analysis reuses the forward approximation for GLU-input gradients and extends it across multiple successive layers.The resulting condition is expressed in terms of the next layer’s input count and weight variance.Source: §A.2 · p. 13
• Gradients are typically around 0.001 at the start of training, motivating the omission of quadratic terms in the approximation.This assumption is specific to the networks considered.Source: §A.2 · p. 13
• The approximations become invalid for arbitrarily large input or activation variances, requiring balanced initialization or explicit variance control.Batch normalization and layer normalization are offered as alternatives.Source: §A.2 · p. 13

• Dropout retains each activation with probability p, zeros it otherwise, and scales retained activations by 1/p during training.The resulting multiplier r has E[r] = 1.Source: §A.3 · p. 13
• For convolutional layers preceded by dropout, forward and backward activation variations are approximated using the retain probability p.Source: §A.3 · p. 13
• Weights are initialized from a modified zero-mean normal distribution to calibrate for preceding dropout.Layers without a succeeding GLU use a corresponding initialization adjustment.Source: §A.3 · p. 13

• The sigmoid can be rewritten through tanh(x) = 2 σ(2x) −1, while tanh′(x) = 1 − tanh^2(x).Source: B. Upper Bound on Squared Sigmoid · p. 13
• The resulting relation is also expressible directly in terms of σ(x).Source: B. Upper Bound on Squared Sigmoid · p. 13
• Rotational symmetry about zero applies to both terms of the inequality.Source: B. Upper Bound on Squared Sigmoid · p. 13

• Different decoder layers capture different portions of the source sentence, with layers 1, 3, and 6 exhibiting linear alignment.The first layer has the clearest, though slightly offset, alignment and often attends to the source word corresponding to the previous target word.Source: C. Attention Visualization · pp. 13–14, Figure 3 · p. 15
• Layers 2 and 8 lack clear alignment structure and presumably collect information about the whole source sentence.Source: C. Attention Visualization · p. 14
• Layer 4 assigns high attention to nouns such as “festival”, “way” and “work” for target nouns and their preceding words.Source: C. Attention Visualization · p. 14