Synthesizer: Rethinking Self-Attention in Transformer Models

Yi Tay; Dara Bahri; Donald Metzler; Da-Cheng Juan; Zhe Zhao; Che Zheng

Source-linked AI summary

Synthesizer: Rethinking Self-Attention in Transformer Models

Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, Che Zheng

arXiv:2005.00743v3May 2, 2020cs.CLcs.IRcs.LG

TL;DR

The paper examines whether Transformer models truly require dot-product self-attention and pairwise token interactions. It proposes Synthesizers, which learn attention weights through synthetic functions, and finds competitive performance without token-token interactions, improved results when combined with dot-product attention, and strong efficiency comparisons.

Problem
The paper investigates whether dot-product self-attention and pairwise query-key interactions are necessary for strong Transformer performance.
Method
SYNTHESIZER learns self-alignment matrices through synthetic functions instead of manually computing pairwise dot products, using inputs that may exclude token-token interactions.
Results
Synthetic attention performs competitively across evaluated tasks, complements dot-product attention, and simple variants remain competitive with Dynamic Convolutions and Linformers.
Takeaways & Limitations
Attention weights can be synthesized without token-token interactions for competitive self-attention, while combining synthetic and dot-product attention is broadly beneficial.
Takeaways & Limitations
Random and dense Synthesizers use sequence-length-dependent parameters, requiring a defined maximum length and dynamic truncation to batch lengths.

Abstract

from arXiv · show

The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose \textsc{Synthesizer}, a model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product attention, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only $60\%$ faster but also improves perplexity by a relative $3.5\%$. Finally, we show that simple factorized Synthesizers can outperform Linformers on encoding only tasks.

1. introduction

This paper questions whether Transformer performance requires dot-product self-attention and investigates synthetic alternatives that avoid token-token interactions. Experiments show competitive results from synthetic attention, while combining it with dot-product attention improves Transformer performance.

1. introduction: The paper asks whether dot-product self-attention and pairwise query-key interactions are necessary for strong Transformer performance.The question targets self-alignment and content-based retrieval at the core of conventional attention.
1. introduction: 27.27 BLEU is achieved by a random-matrix SYNTHESIZER model on WMT 2014 English-German.The result uses fully global attention weights without token-token or instance-level information.
1. introduction: +3.5% relative improvement in perplexity and 60% faster performance are reported for Synthesizers compared with Dynamic Convolutions.Factorized Synthesizers also outperform Linformers on encoding tasks.
1. introduction: Dot-product attention remains helpful: composing it with synthesizing functions consistently improves Transformer performance.The findings suggest that token-token interactions are useful, although synthetic attention can perform competitively without them.
1. introduction: Synthetic Attention learns alignment matrices without explicitly using dot-product or content-based attention, instead exploring parameterized synthesizing functions.The synthesized matrix can be generated independently of token-token dependencies and may use diverse source information.
1. introduction: SYNTHESIZER achieves competitive performance with state-of-the-art Transformers across language tasks including machine translation and language modeling.The model replaces conventional self-attention with Synthetic Attention modules.

2. Related Work

Related work has improved or replaced self-attention while generally retaining content-based or query-key mechanisms. Synthesizer instead learns attention matrices through functions that may not depend on input tokens, and Random Synthesizers connect to MLP-Mixers.

2. Related Work: Attention research originated in soft word alignment and memory-based retrieval before self-attention was popularized by Transformer models.Related paradigms include intra-attention, self-matching networks, and LSTMN.
2. Related Work: Prior self-attention research improves efficiency through sparsification and reduced alignment-matrix complexity while retaining the query-key-value paradigm.These approaches remain coupled to memory-based content retrieval.
2. Related Work: Synthesizer replaces pairwise dot products with synthesizing functions that learn attention matrices with optional dependence on input tokens.This extends earlier work using fixed, manually defined attention patterns in Transformer encoders.
2. Related Work: Random Synthesizers are a form of MLP-Mixer token mixing that applies a learned L × L weight matrix across the sequence-length dimension.They differ through softmax kernel normalization and multi-headed structure.

3. The Proposed Method

Synthetic Attention replaces query-key-value dot-product attention with functions that directly synthesize alignment matrices, using token-wise, random, factorized, or mixed designs. These variants can preserve competitive performance while reducing reliance on pairwise token interactions and, in factorized forms, parameter costs.

Synthetic Attention: Synthetic Attention directly synthesizes the alignment matrix instead of using query-key-value dot products.The module is described per attention head and layer, replacing self-attention modules in a Transformer.
Dense Synthesizer: Dense Synthesizer projects each input token from d dimensions to N dimensions, so each token predicts attention weights over the sequence.Its parameterized function is applied positionwise to the input sequence.
Random Synthesizer: Random Synthesizer learns or fixes an N×N alignment matrix without conditioning attention weights on input tokens or pairwise token interactions.The trainable random matrix represents a task-specific global alignment shared across samples; each head adds N^2 parameters.
Factorized Models: Factorized Synthesizers replace dense N×N matrices with low-rank or tiled constructions to reduce parameter costs and help prevent overfitting.The factorized random variant uses matrices in R^(N×k), while the factorized dense variant constructs C in R^(N×N) from tiled projections.
Mixture of Synthesizers: Synthesizing functions can be combined additively with learnable weights, including mixtures of random, factorized, dense, and standard attention components.The paper investigates several mixture variants experimentally.
Sequence-Length Dependence: Random and dense Synthesizers use parameters dependent on sequence length, with maximum-length definitions and dynamic truncation to batch lengths.The authors note a possible projection-and-tiling solution but leave that exploration for future work.

4. Experiments

The experiments evaluate Synthesizer variants against Transformers, Dynamic Convolutions, and Linformers across translation, language modeling, generation, and language-understanding tasks. Synthesizers are often competitive, while mixtures with dot-product attention frequently improve results.

Experimental setup: Experiments cover machine translation, autoregressive language modeling, text generation, and GLUE/SuperGLUE multitask language understanding.The study compares multiple Synthesizer variants with vanilla Transformers across five task areas.
Machine translation and language modeling: 27.27 BLEU on EnDe and 41.12 BLEU on EnFr were achieved by the Random Synthesizer baseline.Fixed random attention still yielded approximately 24 BLEU, although it performed substantially worse than the trainable random variant.
Machine translation and language modeling: Random Synthesizers performed within 1-2 PPL points of vanilla Transformers on LM1B, while Synthesizer (D+V) achieved the best performance in that setting.On EnDe, mixing standard dot-product attention increased performance by +0.8 BLEU points.
Text generation: Synthesizers (R) and (D) outperformed vanilla Transformers on dialogue generation by approximately 1-3 points across most or all metrics.For summarization, the (D) model was approximately 2 Rouge-L points below Transformers, whereas (R+V) and (R+D) outperformed them; (+V) variants performed poorly on dialogue.
Masked language modeling: +3.5% relative perplexity improvement and 60% faster training were reported for Synthesizers (R) versus Dynamic Convolutions on masked language modeling.Against Lightweight Dynamic Convolutions, Synthesizers matched performance while being +5% faster; Random Synthesizers also used fewer FLOPS.
GLUE and SuperGLUE: +1.9 points on SuperGLUE and +0.6 points on GLUE were reported for Syn (R+V) over T5.The comparison also used T5 (Base+) with an equal number of parameters, and Synthesizers (+V) still outperformed it.
Encoding-only tasks: Factorized Synthesizers were competitive with Linformers and Transformers on encoding tasks, while Syn (FR+V) outperformed both.The Linformer comparison used AGnews and movie reviews without contextualized embeddings, so results are not comparable with other work.

5. Qualitative Analysis

The qualitative analysis examines learned synthetic attention weights and their relationship to vanilla Transformer attention. Random Synthesizers can learn local, smoother attention patterns, while overall results show both standalone competitiveness and complementarity with dot-product attention.

Language-model attention patterns: Synthesizers on LM1B learned a local attention window that closely emulated vanilla Transformer attention despite starting from completely random weights.Their weights appeared smoother and less coarse, consistent with lacking token-specific information.
Overall findings: Synthetic attention functions achieved performance reasonably close to dot-product self-attention across all evaluated tasks.Dialogue generation was the one task where dot-product self-attention degraded performance, while machine translation was least affected by its removal.
Overall findings: On multitask language understanding, synthesized attention performed considerably worse because concatenated sentence pairs make encoder self-attention function as cross-attention.This benchmark therefore depends more strongly on pairwise interactions than the other evaluated tasks.
Complementarity: Synthetic attention and dot-product attention were highly complementary, with their composition improving Transformer performance on almost all investigated tasks.Standalone synthetic attention was generally competitive and fast, while combining multiple synthetic and dot-product attention mechanisms produced gains.
Efficient alternatives: Random Synthesizers remained competitive with Dynamic Convolutions and Linformers on selected fast or efficient Transformer comparisons.Factorized random Synthesizers were evaluated on two encoding tasks, while Random Synthesizers were also competitive on large-scale masked language modeling.

6. Conclusion

SYNTHESIZER uses Synthetic Attention to replace pairwise dot-product computation with learned self-alignment. Across multiple tasks, synthetic attention performs competitively with vanilla self-attention, while pairwise interactions can hurt dialogue generation.

SYNTHESIZER learns a self-alignment matrix through synthetic attention rather than computing pairwise dot products.
Synthetic attention demonstrates competitive performance against vanilla self-attention across machine translation, language modeling, dialogue generation, masked language modeling, and document classification.
Pairwise interactions hurt performance on the dialogue generation task.
Synthesizers can outperform or match Dynamic Convolutions, while Factorized Synthesizers can outperform other low-rank Linformer models.