PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Sequence to Sequence Learning

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin

arXiv:1705.03122v3cs.CL

TL;DR

Sequence-to-sequence learning commonly relies on recurrent networks, which limit parallel computation within sequences. This paper introduces a fully convolutional alternative with gated units and decoder-layer attention, achieving state-of-the-art translation results and order-of-magnitude faster inference than strong recurrent baselines.

  • Problem

    Recurrent sequence-to-sequence models prevent parallel computation within a sequence, motivating architectures that can parallelize sequence processing.

  • Method

    The paper introduces a fully convolutional sequence-to-sequence architecture using gated linear units, residual connections, and attention in every decoder layer.

  • Results

    1.9 BLEU improvement over the previous best on WMT’16 English-Romanian accompanied state-of-the-art translation results and order-of-magnitude faster translation than Wu et al. (2016).

  • Takeaways & Limitations

    Fully convolutional sequence-to-sequence models can outperform strong recurrent models on large translation benchmarks while translating unseen sentences substantially faster.

  • Takeaways & Limitations

    Comparisons did not use identical vocabulary sizes because word-piece and BPE methods estimate vocabulary differently.

Abstract

from arXiv · show

The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU.

1. Introduction

The paper introduces an entirely convolutional sequence-to-sequence architecture designed to improve sequence modeling through hierarchical representations, gated linear units, residual connections, and attention in every decoder layer. It reports state-of-the-art or improved BLEU results on several machine-translation benchmarks.

  • Motivation: Convolutional networks offer controllable dependency lengths and shorter paths for modeling long-range relationships than recurrent networks.Stacking layers expands effective context size, while hierarchical representations connect nearby elements at lower layers and distant elements at higher layers.
  • Motivation: A fixed number of convolutional operations and nonlinearities simplifies learning compared with recurrent processing that can grow with sequence length.The passage contrasts constant processing in convolutional networks with up to n operations and nonlinearities applied to the first recurrent input.
  • Contributions: The proposed sequence-to-sequence model is entirely convolutional and combines gated linear units, residual connections, and attention in every decoder layer.Each attention layer adds only negligible overhead, enabling the model to address large-scale problems.
  • Results: 1.9 BLEU is the improvement over the previous best result on WMT’16 English-Romanian translation, achieving a new state of the art.The evaluation covers several large datasets for machine translation and summarization.
  • Results: 0.5 BLEU and 1.6 BLEU are the gains over Wu et al. (2016) on WMT’14 English-German and English-French translation, respectively.The English-German comparison uses Wu et al.’s strong LSTM setup, while the English-French comparison uses its likelihood trained system.

2. Recurrent Sequence to Sequence Learning

Recurrent sequence-to-sequence models use encoder-decoder RNNs to transform input sequences into variable-length outputs one element at a time. Attention augments this process by dynamically combining encoder states, while gated recurrent units and bidirectional encoders address long-term dependencies and contextual representation.

  • Recurrent encoder-decoder architecture: Encoder-decoder RNNs process input elements into state representations and generate outputs left to right, one element at a time.The encoder maps x = (x1, . . . , xm) to z = (z1, . . . , zm), while the decoder generates y = (y1, . . . , yn).
  • Attention mechanisms: Without attention, decoders rely on the final encoder state zm, either as every context vector or as the initial decoder state.In the latter formulation, the context vector ci is not used.
  • Attention mechanisms: Attention computes each context vector ci as a weighted sum of encoder states, with scores normalized over input elements.The scores compare encoder states with the previous decoder state and last prediction.
  • Recurrent network variants: LSTMs and GRUs extend Elman RNNs with gating that memorizes information from previous time steps to model long-term dependencies.Recent approaches also use bidirectional encoders to represent both past and future contexts.

3. A Convolutional Architecture

The model replaces recurrent encoder and decoder states with a fully convolutional architecture, using position-aware embeddings, gated convolutional blocks, residual connections, and separate attention mechanisms for each decoder layer. Its design supports parallel attention computation across sequence elements while preserving causal decoding.

  • Convolutional architecture: The architecture uses convolutional neural networks instead of RNNs to compute intermediate encoder states z and decoder states h.
  • Input representations: Input representations combine distributional embeddings w_j with absolute position embeddings p_j, producing e_j = w_j + p_j; decoder inputs are formed similarly.The representations provide both element content and order information.
  • Convolutional blocks: Each encoder and decoder block applies a one-dimensional convolution over a fixed number of input elements followed by a non-linearity.Stacking 6 blocks with kernel width k = 5 gives each state an input field of 25 elements.
  • Convolutional blocks: Gated linear units transform each convolution output Y = [A B] ∈ R2d into a d-dimensional state, with σ(B) gates controlling which A inputs are relevant.Residual connections from each convolution’s input to its block output enable deeper convolutional networks.
  • Causal decoding: The decoder uses padding and removes k output elements so that no future information is available during decoding.Encoder layers are padded to preserve input length.
  • Layer-wise attention: Each decoder layer has a separate attention mechanism that computes source interactions from decoder states and encoder outputs, then adds the resulting conditional input to that decoder layer.The attention uses dot products and incorporates input element embeddings e_j alongside encoder outputs.
  • Layer-wise attention: Attention can be batched across all sequence elements, unlike recurrent approaches that compute attention sequentially.The model batches computations separately for each decoder layer.

4. Experimental Setup

The experiments cover three WMT translation tasks and abstractive summarization, using standardized datasets, vocabulary configurations, and evaluation protocols. Models generally use 512-dimensional representations and are assessed with translation decoding and ROUGE metrics.

  • Datasets and tasks: The study evaluates three WMT translation tasks and an abstractive summarization task.The translation tasks are English-Romanian, English-German, and English-French.
  • Datasets and tasks: 2.8M sentence pairs train WMT’16 English-Romanian models, evaluated on newstest2016 with 200K source and 80K target word vocabularies or 40K joint BPE types.Sentences longer than 175 words are removed.
  • Datasets and tasks: 4.5M sentence pairs train WMT’14 English-German models tested on newstest2014, while 35.5M filtered pairs train English-French models tested on newstest2014; both use 40K BPE types.The English-French data remove sentences longer than 175 words and pairs with source/target length ratios above 1.5.
  • Datasets and tasks: 3.8M Gigaword examples train summarization models, with 190K validation examples and evaluation on DUC-2004’s 500 pairs using ROUGE-1, ROUGE-2, and ROUGE-L.A second evaluation uses a Gigaword test set of 2000 pairs.
  • Model and training configuration: 512 hidden units and 512-dimensional embeddings are used for encoders, decoders, decoder outputs, and intervening linear layers unless otherwise stated.Training uses Nesterov accelerated gradient with momentum 0.99, learning rate 0.25, and gradient renormalization above norm 0.1.
  • Evaluation protocol: Results average three random-seed runs, with beam-search translation using width 5 and sentence-length-normalized log-likelihoods.For English-German, the length-normalization constant is tuned on newstest2015.

5. Results

ConvS2S outperforms strong recurrent and ensemble baselines across WMT translation tasks while delivering substantial GPU and CPU speedups. Additional experiments show effective translation behavior, favorable depth and kernel-width choices, and competitive abstractive summarization performance.

  • Translation accuracy: 1.9 BLEU on WMT’16 English-Romanian with BPE and 1.3 BLEU with a word-factored vocabulary overtake the winning entry.The model used 20 encoder and 20 decoder layers with width-3 kernels and 512 hidden units, training for 6–7.5 days on one GPU.
  • Translation accuracy: 0.5 BLEU over GNMT on WMT’14 English-German demonstrates stronger accuracy from the convolutional model.This model used 15 encoder and 15 decoder layers and was trained for 18.5 days on one GPU with batch size 48.
  • Translation accuracy: 1.6 BLEU over GNMT on average for WMT’14 English-French, plus 0.5 BLEU over GNMT’s reinforcement-learning models, is achieved with token-level likelihood training.The reported comparison is against GNMT in the same setting; reinforcement learning is also applicable to ConvS2S.
  • Ensembles: Eight-model ensembles outperform the best current ensembles on both WMT’14 English-German and English-French.For English-German, the evaluation also includes an ensemble of 10 models.
  • Inference speed: 21.3 times faster on a GTX-1080ti, up to 13.7 times faster on an M40, and 9.3 times faster on a K40, the model generates translations substantially faster than GNMT.On the K40, the model also achieves 2.25 higher BLEU; larger beam size 5 reduces speed while improving quality.
  • Inference speed: 17 times faster per CPU core and 2.7 times faster than GNMT on a custom TPU, the model also reaches higher CPU speed on commodity hardware.Overall CPU speed is up to 9.3 times faster, despite GNMT using 88 cores versus just over half that number for ConvS2S.

6. Conclusion and Future Work

The paper introduces a fully convolutional sequence-to-sequence model that outperforms strong recurrent systems while running an order of magnitude faster. It achieves state-of-the-art translation results and proposes extending convolutional architectures to other sequence-to-sequence problems.

  • Conclusion: The fully convolutional model outperforms strong recurrent models on very large benchmark datasets at an order of magnitude faster speed.The approach uses gating and performs multiple attention steps.
  • Conclusion: Hierarchically built representations make compositional structure easier to discover than in recurrent networks.The model builds sequence representations hierarchically through convolutional processing.
  • Results: 1.9 BLEU improvement over the previous best result is achieved on WMT’16 English-Romanian translation.This result contributes to a new state of the art across several public translation benchmarks.
  • Results: 1.6 BLEU improvement over Wu et al. (2016) is achieved on WMT’14 English-French translation in a comparable setting.The comparison is against the LSTM model of Wu et al. (2016).
  • Results: 0.5 BLEU improvement over Wu et al. (2016) is achieved on WMT’14 English-German translation.The result is reported against the same LSTM model.
  • Future Work: Future work will apply convolutional architectures to other sequence-to-sequence learning problems.The paper identifies this as a direction for extending the approach beyond the reported translation tasks.

A. Weight Initialization

The paper derives a weight initialization scheme tailored to GLU activations by controlling activation variance during forward and backward passes, including modifications for dropout.

  • A. Weight Initialization: The scheme targets GLU activations by focusing on activation variance in both forward and backward passes.It is developed in relation to Glorot & Bengio (2010) and He et al. (2015b).
  • A. Weight Initialization: The initialization procedure is modified to account for dropout.

A.1. Forward Pass

The forward-pass analysis tracks output variance through convolutional layers with GLU activations. It derives bounds for GLU variance and motivates initialization choices that prevent activation magnification or decay across layers.

  • Variance propagation: The layer output is modeled as y_l = W_lx_l + b_l under i.i.d. inputs and weights, with n_l inputs determining the output variance.For one-dimensional convolutions, n_l equals the kernel width times the input dimension, kc.
  • GLU variance bounds: The analysis bounds the variance of GLU outputs by assuming symmetric zero-mean preactivations and using a lower bound together with an upper bound on E[σ(x)^2].The lower bound includes (1/4)Var[y_a^(l−1)].
  • Initialization: Embedding matrices are initialized with small variances around 0.01, allowing the quadratic term to be dismissed when approximating GLU output variance.This approximation relies on the small initial variance of the embeddings.
  • Initialization: For L equal-sized layers with GLU activations, the final output variance is analyzed to keep activations from being exponentially magnified or reduced.The initialization follows a condition involving n_l and Var, with W_l initialized from a zero-mean normal distribution.

A.2. Backward Pass

The backward-pass analysis derives convolutional and GLU gradients using re-arranged weights, then extends variance approximations across successive layers. It relies on typically small gradients and notes limitations for arbitrarily large input or activation variances.

  • Gradient computation: Backpropagation computes a convolutional layer’s gradient as ∆x_l = ˆW_l y_l, with ˆW_l containing the same values as W_l in rearranged order.The rearrangement enables back-propagation while preserving the weights’ values.
  • Gradient computation: The GLU backward pass separately considers gradients for its two inputs and uses the forward-pass approximation to estimate the variance of ∆y_a.The resulting variance expressions use the convolutional layer’s input count and weight variance.
  • Variance approximation: 0.001 is the typical gradient magnitude at training start, motivating an approximation that discards quadratic terms in the backward-pass variance analysis.The same approximation is generalized through many successive convolutional layers.
  • Limitations: For arbitrarily large input or activation variances, the approximations become invalid and the initial values for W_a would need balancing to retain the input distribution.Batch normalization or layer normalization are offered as alternative methods for explicitly controlling network variance.

A.3. Dropout

Dropout randomly removes activations while scaling retained activations during training to preserve their expected value. In convolutional layers, this changes activation-variation estimates and motivates modified weight initialization, including calibration after dropout when no GLU follows.

  • Dropout: Dropout retains each activation with probability p and otherwise sets it to zero, scaling retained activations by 1/p during training.This preserves E[r] = 1 and avoids modifying network weights when p is set to 1 at test time.
  • Dropout: With input dropout before a convolutional layer, the forward and backward activation variations are approximated using the retain probability p.The approximation is introduced from the preceding forward- and backward-propagation analyses.
  • Dropout: Dropout requires modified initialization of W_l, while layers without a succeeding GLU are initialized to calibrate immediately preceding dropout.The passages specify normal-distribution initialization for W_l and separate calibration for layers lacking a subsequent GLU.

B. Upper Bound on Squared Sigmoid

This section derives an upper bound on the squared sigmoid by rewriting sigmoid through tanh, using tanh’s derivative, and exploiting symmetry around zero.

  • The sigmoid is expressed through the identity tanh(x) = 2 σ(2x) − 1.This provides the transformation used to derive the bound.
  • The derivation uses tanh′(x) = 1 − tanh2(x) for tanh(x) ∈ [0, 1] and x ≥ 0.These derivative and domain properties establish the relevant inequality.
  • The resulting relation is rewritten in terms of σ(x).The sigmoid-form expression is presented as an equivalent formulation of the preceding relation.
  • Rotational symmetry of both inequality terms with respect to 0 extends the relation across the symmetric domain.The symmetry argument supports the final form of the bound.

C. Attention Visualization

Attention visualization shows that different decoder layers learn distinct source-target alignment patterns. Some layers align linearly, while others appear to gather information from the whole source sentence.

  • Attention Visualization: Figure 3 visualizes attention across 8 decoder layers for an English-to-German sentence using an 80K BPE vocabulary.The visualization reports attention scores with the generated sentence on the y-axis and the source sentence on the x-axis.
  • Attention Visualization: Layers 1, 3, and 6 exhibit a linear alignment, with layer 1 providing the clearest but slightly offset alignment.Layer 1 frequently attends to the source word corresponding to the previously generated target word.
  • Attention Visualization: Layers 2 and 8 lack clear structure and presumably collect information about the whole source sentence.
  • Attention Visualization: Layer 4 shows high alignment scores for nouns such as “festival”, “way” and “work” in generated target nouns and their preceding words.
Loading 1705.03122v3…