PaperlayerView exact revision on arXiv

Source-linked AI summary

Pay Less Attention with Lightweight and Dynamic Convolutions

Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, Michael Auli

arXiv:1901.10430v2cs.CL

TL;DR

Long-range self-attention has quadratic input-length complexity, motivating more efficient alternatives. The paper introduces lightweight convolutions and dynamic convolutions with time-step-specific kernels. Across translation, language modeling, and summarization, dynamic convolutions match or improve strong self-attention results, including 29.7 BLEU on WMT’14 English-German and 20% faster runtime.

  • Problem

    Self-attention's quadratic complexity makes unlimited context computationally challenging for long sequences.

  • Method

    The paper introduces lightweight convolutions and dynamic convolutions that predict separate kernels from the current time-step rather than the entire context.

  • Results

    Dynamic convolutions improve over strong self-attention models across translation, language modeling, and summarization, achieving 29.7 BLEU on WMT’14 English-German.

  • Takeaways & Limitations

    Lightweight convolutions can achieve good accuracy on the considered language tasks without self-attention, while dynamic convolutions provide a more efficient alternative.

  • Takeaways & Limitations

    The experiments cover the considered language tasks, while application to question answering and computer vision remains future work.

Abstract

from arXiv · show

Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

1 INTRODUCTION

Self-attention models unrestricted context by comparing elements, but their quadratic input-length complexity makes long sequences computationally challenging. The paper introduces lightweight and dynamic convolutions as simpler alternatives, with dynamic convolutions predicting time-step-specific kernels.

  • Self-attention directly summarizes all context by assigning weights based on comparisons between the current time-step and context elements.
  • Unlimited self-attention context has quadratic complexity in input length, creating computational challenges for long sequences.
  • Lightweight convolutions are depth-wise separable, softmax-normalized, and channel-shared, yielding several orders of magnitude fewer weights than standard non-separable convolutions.
  • Dynamic convolutions predict a different convolution kernel at every time-step using only the current time-step, unlike self-attention's dependence on the entire context.
  • 29.7 BLEU: dynamic convolutions achieve a new state of the art on WMT English-German translation while improving over strong self-attention results across evaluated tasks.They also achieve 20% faster runtime than a highly optimized self-attention baseline.

2 BACKGROUND

The paper reviews sequence-to-sequence architectures and explains self-attention and depthwise convolution as foundations for its convolutional models. Self-attention forms weighted context summaries from projected representations, while depthwise convolution processes channels independently with far fewer parameters.

  • Sequence-to-sequence learning maps a source sequence to a target sequence using separate encoder and autoregressive decoder networks.
  • Self-attention projects input X into keys, queries, and values, computes scaled key-query dot products, and softmax-normalizes the resulting attention weights.
  • Self-attention produces context representations by computing a weighted sum from the value projection.
  • Depthwise convolution applies convolution independently over each channel, reducing parameters from d^2k to dk for kernel width k.

3 LIGHTWEIGHT CONVOLUTIONS

LightConv is a depthwise convolution with softmax-normalized temporal weights and channel sharing, designed to retain useful context modeling with far fewer parameters than standard convolutions.

  • Lightweight convolutions: LightConv combines depthwise convolution, softmax normalization across the temporal dimension, and shared output-channel weights.Its weights are normalized over the kernel width and reused across specified channel groups.
  • Lightweight convolutions: Unlike self-attention, LightConv uses a fixed context window and time-invariant weights to determine context importance.This design reuses the same weights for context elements regardless of the current time-step.
  • Weight sharing: 112 weights replace 7,340,032 weights for d = 1024 and k = 7 when H = 16.The corresponding depthwise separable convolution uses 7,168 weights.
  • Module: The module projects inputs from d to 2d, applies a gated linear unit, performs LightConv, and then uses an output projection.The GLU creates gates with sigmoid units before the pointwise product.
  • Implementation: LightConv implementation expands normalized weights into a band matrix and uses batch matrix multiplication for short sequences.The authors expect a dedicated CUDA kernel to be more efficient.

4 DYNAMIC CONVOLUTIONS

Dynamic convolutions extend lightweight convolutions by generating a time-step-specific kernel from the current input rather than the entire context. They retain changing context weights while reducing attention-weight computation from quadratic to linear sequence-length scaling.

  • Efficiency: DynamicConv builds on LightConv because its drastically reduced parameter count makes dynamically generated kernels practical on current GPUs.A dynamic version of standard convolutions would otherwise require impractical memory.
  • Dynamic kernel generation: DynamicConv predicts a separate convolution kernel at every time-step as a function of that individual time-step only.The kernel-generating function maps the current representation to H × k weights.
  • Dynamic kernel generation: DynamicConv changes context-element weights over time without making them depend on the entire context.This distinguishes it from context-based self-attention.
  • Efficiency: Linear sequence-length scaling replaces self-attention’s quadratic computation for generating attention weights.The linear scaling applies to dynamic-kernel computation.
  • Results: DynamicConv models match or exceed state-of-the-art models using context-based self-attention.The experiments report this comparison across the evaluated settings.

5 EXPERIMENTAL SETUP

The experiments evaluate convolutional and self-attention sequence models across machine translation, language modeling, and summarization using encoder-decoder or decoder-only setups. Translation experiments cover four benchmarks with BLEU evaluation, while language modeling and summarization use perplexity and F1-Rouge.

  • Model architecture: The models use encoder-decoder architectures with N blocks, residual connections, layer normalization, and feed-forward sub-blocks.Encoder sub-blocks use self-attention, LightConv, or DynamicConv; decoder blocks additionally include source-target attention.
  • Model architecture: LightConv and DynamicConv replace Transformer Big self-attention modules, use fewer parameters per block, and increase encoder depth to N = 7 to roughly match parameter counts.Both convolutional models generally use H = 16 and encoder and decoder kernel sizes of 3, 7, 15, and 31x4 across blocks.
  • Datasets and evaluation: Translation experiments evaluate WMT English-German, English-French, English-Chinese, and IWSLT German-English benchmarks with dataset-specific training sizes, validation sets, test sets, and BPE vocabularies.WMT En-De uses 4.5M sentence pairs, WMT En-Fr 36M, WMT Zh-En 20M, and IWSLT De-En 160K.
  • Datasets and evaluation: Translation quality is measured with BLEU, using case-sensitive tokenized scoring for WMT En-De and En-Fr, detokenized scoring for WMT Zh-En, and dataset-specific beam widths.Beam widths are 5 for WMT En-De and En-Fr, 4 for IWSLT, and 8 for WMT Zh-En.
  • Datasets and evaluation: Language modeling uses the 768M-token Billion Word dataset and reports validation and test perplexity, while summarization uses over 280K CNN-DailyMail article-summary pairs and reports Rouge-1, Rouge-2, and Rouge-L F1.Summarization articles are truncated to 400 tokens and use a 30K-type BPE vocabulary; language modeling uses an adaptive softmax for its nearly 800K-type vocabulary.
  • Training and hyperparameters: Training uses task-specific optimization and regularization settings, including Adam with cosine schedules for translation and summarization, Nesterov acceleration for language modeling, dropout, label smoothing, and mixed-precision gradient accumulation.Translation training uses floating point 16 precision and generally accumulates gradients for 16 batches, except on IWSLT.

6 RESULTS

Across translation, language modeling, and summarization, lightweight and dynamic convolutions match or exceed strong self-attention baselines while reducing computational cost.

  • Machine Translation: LightConv trails the WMT English-French state of the art by only 0.1 BLEU despite using few fixed weights.Self-attention computes dot-products with all context elements at every time-step.
  • Machine Translation: DynamicConv surpasses the best known WMT English-German result by 0.4 BLEU and matches the WMT English-French state of the art.The WMT English-German result establishes a new state of the art.
  • Machine Translation: 0.8 BLEU is DynamicConv’s improvement over the self-attention baseline on IWSLT German-English, while LightConv improves by 0.4 BLEU.On WMT Chinese-English, LightConv improves by 0.5 BLEU and DynamicConv by 0.6 BLEU.
  • Model Ablation: 20% higher inference speed is achieved by DynamicConv than the self-attention baseline, with slightly fewer parameters and equal validation accuracy in ablations.Softmax normalization is important for DynamicConv because removing it caused training to diverge.
  • Language Modeling: DynamicConv achieves slightly better perplexity than a highly competitive self-attention baseline on the Billion Word language-modeling benchmark.The DynamicConv model uses 17 blocks versus 16 for the baseline to assimilate parameter count.
  • Abstractive Summarization: DynamicConv performs better than LightConv, the self-attention baseline, and comparable previous work on CNN-DailyMail summarization.The benchmark evaluates models on documents of up to 400 words and multi-sentence summaries.

7 CONCLUSION

The paper concludes that lightweight convolutions can achieve competitive language-task accuracy without self-attention, while dynamic convolutions adapt kernels by time-step and improve efficiency and results.

  • Conclusion: Lightweight convolutions achieve competitive results despite a very small parameter footprint and kernels that remain fixed across time-steps.The conclusion states that self-attention is not critical for good accuracy on the considered language tasks.
  • Conclusion: Dynamic convolutions predict a different kernel at every time-step, using only the current time-step rather than the entire context.This design resembles the changing attention weights of self-attention.
  • Conclusion: 20% faster runtime is achieved by both lightweight and dynamic convolutions than by self-attention, while DynamicConv reaches a new WMT’14 English-German state of the art.Dynamic convolutions also improve further over lightweight convolutions on several reported tasks.
  • Conclusion: The paper’s future scope includes applying dynamic convolutions to question answering and computer vision with larger inputs.These applications are presented as planned future work rather than evaluated results.
  • Conclusion: An earlier CNN-DailyMail comparison was erroneous because it used full-text data instead of the paper’s entity-anonymized version.The note specifically concerns comparison with Gehrmann et al. (2018).

A COMPARISON OF SOFTMAX-NORMALIZATION TO ALTERNATIVES

The paper compares softmax-normalization with alternative weight transformations in DynamicConv on WMT English-German newstest2013. Softmax-normalization performs best among the tested alternatives.

  • Three seeds provide the mean and standard deviation of BLEU for each setting, with normalization computed over the kernel dimension.
  • Softmax-normalization performs best, while absolute-value and squared weights do not make training more stable.
  • Table 6 compares softmax-normalization with alternative weight transformations in DynamicConv on WMT English-German newstest2013.The experiment uses ϵ = 10^-6.

B ON THE CURRENT STATE OF NON-AUTOREGRESSIVE GENERATION

The paper evaluates DynamicConv against existing non-autoregressive models using comparable single-example GPU speed measurements. A single-layer DynamicConv decoder outperforms prior reported results in both speed and accuracy, with only two efforts achieving higher speed at a small BLEU cost.

  • Generation speed is measured on a P100 GPU with batch size one to compare DynamicConv against reported non-autoregressive results.Published results use NVIDIA GTX-1080 or P100 GPUs; GPU-type effects are considered likely negligible under batch-size-one underutilization.
  • A single-decoder-layer DynamicConv outperforms all previously reported non-autoregressive results in both speed and accuracy.
  • Only Guo et al. (2019) and Li et al. (2019) achieve higher speed, each with a small BLEU drop after distilling autoregressive models.
Loading 1901.10430v2…