PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Transformer Modifications Transfer Across Implementations and Applications?

Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, Colin Raffel

arXiv:2102.11972v2cs.LGcs.CL

TL;DR

Although many Transformer modifications have been proposed, relatively few have been widely adopted, and their transfer across applications and implementations is uncertain. The paper evaluates many variants in a shared setting and finds that most do not meaningfully improve performance; beneficial variants tend to be minor changes or developed in the same codebase.

  • Problem

    Many proposed Transformer modifications have seen little widespread adoption, and it is uncertain whether their benefits generalize across applications and implementations.

    Source: §1 · pp. 1–2

  • Method

    The authors reimplement and evaluate many Transformer variants in a shared experimental setting covering common Transformer tasks.

    Source: §1 · p. 2

  • Results

    Most modifications do not meaningfully improve performance, while beneficial variants tend to be minor changes or developed in the evaluation codebase.

    Source: §1 · p. 2, §3.1 · pp. 5–6

  • Takeaways & Limitations

    The authors conjecture that gains from Transformer modifications may depend on implementation details and recommend improving the generality of experimental results.

    Source: §1 · p. 2

    The study is limited to the encoder-decoder architecture, and the authors suggest that modifications may not generalize across implementations and tasks.

    Source: §2 · p. 2, §4 · p. 8

Abstract

from arXiv · show

The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.

1 Introduction

The paper asks why few proposed Transformer modifications have been widely adopted, including whether their gains generalize across applications and implementations. It evaluates many variants in a shared setting and finds that most do not improve performance, while beneficial variants tend to be minor or developed in the evaluation codebase.

1 Introduction

  • Researchers cannot test every applicable downstream task, while adopted proposals tend to reliably improve performance across varied real-world tasks.Source: §1 · p. 1
  • The paper frames cross-task and cross-implementation generalization as possible explanations for why proposed modifications have not seen widespread adoption.The authors contrast this possibility with the idea that the original Transformer was already near-perfect.Source: §1 · pp. 1–2
  • Most evaluated Transformer modifications did not improve performance; beneficial variants tended to be small changes or developed in the evaluation codebase.The authors suggest that modifications may not generalize across implementations and tasks.Source: §1 · p. 2
  • The study tests reimplemented Transformer variants on a suite of tasks commonly using Transformers to investigate why few modifications have been widely adopted.Source: §1 · p. 2

2 Modifications

The paper surveys Transformer modifications while restricting its study to encoder-decoder models; space constraints mean many modifications are not thoroughly defined. The modifications span activations, normalization, model depth, embeddings, parameter sharing, softmax computation, and architecture variants.

2 Modifications

  • The study enumerates architectural modifications but is limited to encoder-decoder models and seldom defines each modification thoroughly.Readers are directed to original sources for more detail.Source: §2 · p. 2

2.1 Activations

  • Activation experiments replace feedforward-block ReLU with alternatives, including GeLU, Swish, ELU, SeLU, Softplus, and gated linear units.Gated variants combine two learned affine transformations elementwise and vary the activation function.Source: §2.1 · p. 2

2.2 Normalization

  • Normalization experiments compare RMS norm with layer normalization and test Rezero and Fixup initialization schemes, including combinations with normalization.Fixup rescales initializations to address vanishing or exploding gradients.Source: §2.2 · p. 2

2.3 Depth

  • Depth experiments vary feedforward width and layer count while scaling feedforward width and attention heads to hold parameter count constant.Source: §2.3 · p. 2

2.4 Embeddings

  • Embedding experiments vary which encoder and decoder input/output matrices are tied, factorize embedding matrices, and test adaptive input embeddings.Adaptive input embeddings cluster vocabulary items by frequency, assigning larger dimensions to more frequent words before projection and concatenation.Source: §2.4 · p. 2

2.5 Parameter sharing

  • Parameter-sharing experiments share each Transformer subblock’s weights across layers, with factorized embeddings and variants that tie decoder softmax and vocabulary embeddings.The study also applies sharing separately to encoder and decoder.Source: §2.5 · p. 3

2.6 Softmax

  • Softmax variants include adaptive softmax, with and without projection, and Mixture of Softmaxes, which combines softmax distributions using learned coefficients.Adaptive softmax uses word-frequency clusters to reduce computation time.Source: §2.6 · p. 3

2.7 Architectures

  • Architecture variants include transparent attention’s weighted residual connections, evolved architectures, and Synthesizer patterns that can be combined with dot-product attention.Transparent attention is intended to facilitate gradient flow; Synthesizer adds synthetic attention patterns.Source: §2.7 · p. 3

3 Experiments

Most architecture variants performed poorly, while significant gains clustered in minor, resource-costly, or Mesh TensorFlow-originated changes; most beneficial modifications improved across training settings, with some exceptions.

3 Experiments

  • Variants were compared with task, hyperparameters, optimizer, and either parameter count or FLOP budget held constant across transfer learning and supervised WMT’14 EnDe.Transfer learning included T5-style pre-training and fine-tuning on SuperGLUE, XSum, and WebQuestions.Source: §3 · p. 4

3.1 Results

  • SwiGLU, GeGLU, RMS normalization, and untied embeddings improved performance; deeper models traded more compute for gains, parameter sharing hurt, and mixture-of-softmaxes was almost 40% slower.Source: §3.1 · p. 5
  • Two Synthesizer variants were beneficial; Switch Transformer, mixture of experts, and product key memories improved performance with more parameters but roughly baseline-equivalent operations.These architectures use only a subset of their parameters at each step.Source: §3.1 · p. 5
  • Most other architecture variants performed poorly; significant gains clustered in minor changes, resource-costly variants, or modifications developed in the Mesh TensorFlow codebase.Examples of the latter include mixture of experts, Switch Transformer, and Synthesizer.Source: §3.1 · p. 6
  • Most beneficial modifications improved pre-training, fine-tuning, and supervised training, but transparent attention, Synthesizer-random, and Fixup harmed transfer learning without harming WMT’14 EnDe.Source: §3.1 · p. 6

3.2 Impact of hyperparameter tuning

  • Only 2 of 25 Universal Transformer sweeps beat the initial result, improving loss from 2.40 to 2.265 (6% relative) but not matching vanilla’s 2.182.Successful settings reduced recurrent steps from 16 to 2 and slightly increased model size.Source: §3.2 · p. 6

3.3 Correlation of perplexity and task performance

  • Spearman’s ρ was 0.87 for SuperGLUE, 0.80 for XSum, and 0.69 for WebQuestions; the weaker association may reflect less alignment for knowledge-intensive question answering.Source: §3.3 · p. 6

4 Conjectures and Recommendations

Most modifications that improved performance were simple, developed in the evaluation codebase, or required more parameters or FLOPs; other modifications generally did not improve performance. The authors consider several possible explanations, including task mismatch and limited hyperparameter tuning, while judging codebase mismatch unlikely.

4 Conjectures and Recommendations

  • Modifications that improved performance were generally simple, developed in the evaluation codebase, or more costly in parameters or FLOPs; other modifications generally did not improve performance.Examples of the latter groups include activation-function changes, Synthesizer variants, Switch Transformer, and Universal Transformer.Source: §4 · pp. 6, 8
  • The authors consider codebase mismatch an unlikely explanation because Mesh TensorFlow was created by an original Transformer coauthor and had been used for state-of-the-art results.Source: §4 · p. 8
  • The evaluated tasks cover most Transformer NLP use cases, including transfer learning and supervised training, though task mismatch remains a proposed explanation.Source: §4 · p. 8
  • Not tuning each modification may handicap methods, but the authors argue useful improvements should be reasonably hyperparameter-agnostic.They note that per-modification tuning might improve results and that few methods gained performance under the shared settings.Source: §4 · p. 8

4. We implemented many of the modifications

The authors conclude that Transformer modifications often fail to transfer across implementations and applications. They recommend testing changes across codebases and downstream tasks, controlling hyperparameters where possible, and reporting results across multiple trials.

4. We implemented many of the modifications

  • To check implementation correctness, the authors contacted many modification creators, who confirmed correctness in all cases.Source: §4 · p. 8
  • Transformer modifications often do not transfer across implementations and applications.Source: §4 · p. 8
  • The authors view limited adoption as support for transferability concerns, reasoning that easily transferable modifications would likely have been more widely adopted.Source: §4 · p. 8
  • They recommend testing modifications in multiple disparate codebases and across applications such as transfer learning, supervised learning, and language modeling.They also suggest possibly including domains beyond NLP, such as computer vision.Source: §4 · p. 8
  • They recommend holding hyperparameters fixed as much as possible across implementations and tasks, or measuring modification robustness to hyperparameter changes.Source: §4 · p. 8
  • They recommend reporting means and standard deviations across multiple trials, or at least avoiding selection of only the best run.Source: §4 · pp. 8–9

A Experiments with positional embeddings

With learned positional embeddings instead of relative attention, relative-attention architectures significantly outperformed positional-embedding architectures. Some variants that surpassed the relative-attention baseline performed worse than it in the positional-embedding setup.

A Experiments with positional embeddings

B Implementation details for modifications

The implementation details specify embedding cluster sizes, model-specific settings for several architecture variants, and selected configurations for softmax and attention experiments.

B Implementation details for modifications

  • Factorized embeddings used an inner dimension of 128; adaptive input embeddings used clusters of 2,500, 6,000, and 23,628, with adaptive softmax splitting the last cluster into 23,500 and 128.Source: B Implementation details for… · p. 13
  • The reported Mixture of Softmaxes model used 15 softmaxes; convolution kernels had width 9, Universal Transformer used 24 recurrent steps and a 0.5 halting threshold, and Mixture of Experts used 32 experts.Source: B Implementation details for… · p. 13
  • Product key memory used k_nn = 32, 128 keys, and 512 memory slots before the decoder’s last layer; Funnel Transformer used mean pooling across three encoder blocks every four layers.Source: B Implementation details for… · p. 13
  • For the weighted Transformer, branch-attention weights were frozen during the final 20,000 pretraining steps.Source: B Implementation details for… · p. 13

C Reproducing the original Transformer experiments

The authors judged their codebase to closely replicate the original Transformer and found their baseline comparable to Transformer-Big. Their reported BLEU scores provide the two comparisons.

C Reproducing the original Transformer experiments

  • The authors judged their codebase to closely replicate the original Transformer and found their baseline comparable to Transformer-Big.The comparisons use different reported evaluations: the original Transformer dev set without checkpoint averaging and the WMT EnDe validation set.Source: C Reproducing the original Transformer… · pp. 13–14
  • 25.52 BLEU was the replication score for the Transformer-Base architecture, versus 25.8 BLEU reported for the original model on the dev set without checkpoint averaging.The authors use this comparison to support their assessment that their codebase closely replicates the original.Source: C Reproducing the original Transformer… · p. 13
  • 26.62 BLEU was the baseline score on the WMT EnDe validation set, versus 26.4 BLEU for Transformer-Big on that same set.The paper describes its baseline as marginally better than the originally reported result.Source: C Reproducing the original Transformer… · p. 14

D Transformer Background

This section outlines the encoder-decoder Transformer used in the experiments: token and position embeddings feed encoder and decoder blocks, with attention, feedforward layers, residual connections, and layer normalization. The experiments use encoder-decoder Transformers, and Table 2 reports variant results across pre-training and fine-tuning.

D Transformer Background

  • The experiments use encoder-decoder Transformers, whose encoder embeds input tokens with position information before processing them through layer blocks.Encoder blocks combine multi-headed self-attention and position-wise feedforward networks, with residual connections and layer normalization.Source: D Transformer Background · p. 14
  • Each encoder block applies multi-headed self-attention followed by a position-wise feedforward network, with residual connections and layer normalization.Layer normalization uses learned scale and offset parameters.Source: D Transformer Background · p. 14
  • Decoder blocks use causal self-attention and insert encoder-decoder attention between self-attention and the feedforward network.The decoder’s final-layer activations are projected through an output logit matrix.Source: D Transformer Background · p. 15
  • Table 2 reports early and final pre-training perplexity alongside validation results for SuperGLUE, XSum, and WebQuestions.The reported fine-tuning metrics are average, ROUGE-2, and accuracy, respectively; boldface marks scores above the vanilla Transformer.Source: Table 2 · p. 16

Related papers

Loading 2102.11972v2…