Source-linked AI summary
Fastformer: Additive Attention Can Be All You Need
Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, Xing Xie
TL;DR
Long-sequence Transformer modeling is limited by the quadratic cost of pairwise self-attention and by shortcomings in existing acceleration methods. Fastformer uses additive attention to summarize global contexts and then transforms token representations through their interactions with those contexts. Across five benchmark datasets, it is reported as more efficient than many Transformer models while achieving competitive or better long-text performance.
Problem
Transformer self-attention has quadratic complexity in sequence length, while existing acceleration methods can remain inefficient on long sequences or model context less effectively.
Method
Fastformer uses additive attention to summarize queries and keys into global contexts, then combines those contexts with token representations through element-wise products and linear transformation.
Results
Fastformer is much more efficient than many Transformer models and achieves competitive or even better performance in long-text modeling across five benchmark datasets.
Takeaways & Limitations
Effective context modeling can be achieved with linear complexity by replacing pairwise token interactions with global-context interactions.
Takeaways & Limitations
Latency improvements are relatively smaller for shorter sequences because the model includes multiple additional MLPs.
Abstract
from arXiv · showhide
Transformer is a powerful model for text understanding. However, it is inefficient due to its quadratic complexity to input sequence length. Although there are many methods on Transformer acceleration, they are still either inefficient on long sequences or not effective enough. In this paper, we propose Fastformer, which is an efficient Transformer model based on additive attention. In Fastformer, instead of modeling the pair-wise interactions between tokens, we first use additive attention mechanism to model global contexts, and then further transform each token representation based on its interaction with global context representations. In this way, Fastformer can achieve effective context modeling with linear complexity. Extensive experiments on five datasets show that Fastformer is much more efficient than many existing Transformer models and can meanwhile achieve comparable or even better long text modeling performance.
1 Introduction
Transformers model sequence context effectively but become inefficient on long inputs because self-attention has quadratic complexity. Fastformer replaces pairwise attention with additive global-context modeling and reports efficient, competitive long-text performance.
- Self-attention computes pairwise interactions between input positions, giving Transformer quadratic complexity in sequence length.
- Existing acceleration methods include sparse, approximate, and other attention mechanisms, but sparse attention may not fully model global context.
- Fastformer summarizes queries into a global query, forms context-aware keys through element-wise products, and transforms values using global context.
- Fastformer uses additive attention and element-wise products to model context with linear complexity instead of pairwise token interactions.
- Extensive experiments on five datasets show Fastformer is more efficient than many Transformer models while achieving competitive performance.
2 Related Work
Transformer self-attention captures interactions across sequence positions but incurs quadratic cost. Related efficient variants use sparse, hashing-based, or approximate attention, while Fastformer models global contexts with element-wise interactions to reduce computation.
- Multi-head self-attention captures interactions between every pair of sequence positions, making its computational complexity quadratic in sequence length.
- Sparse methods such as Longformer and BigBird combine local, global, and random attention patterns to reduce self-attention cost.
- Reformer uses multiround hashing to group similar representations and theoretically reduce complexity to O(N log(N)), but its large constant can limit efficiency.
- Linformer and Linear Transformer approximate self-attention, but their context-agnostic approximations may be suboptimal for text modeling and costly on very long sequences.
- Fastformer uses element-wise products between token representations and global contexts to reduce computational cost while capturing contextual information.
3 Fastformer
Fastformer replaces pairwise self-attention with additive attention and element-wise interactions involving global query and key vectors. This design models contextual information with O(N · d) complexity and fewer parameters than standard Transformer layers.
- Architecture: Fastformer transforms input embeddings into query, key, and value sequences for each attention head.The matrices Q, K, and V each contain N token representations of hidden dimension d.
- Architecture: Additive attention summarizes the query matrix into a global query vector containing global contextual information.The global query is formed by weighting query vectors across the sequence.
- Architecture: Fastformer uses element-wise products between the global query and key vectors to form context-aware keys, then summarizes them into a global key vector.This interaction allows different keys to receive different influences from the global query while retaining linear efficiency.
- Architecture: The global key interacts with each attention value through element-wise products before linear transformation and addition with the query matrix.The resulting hidden representations are assembled into the final output, with outputs from attention heads concatenated.
- Efficiency: Stacking Fastformer layers enables contextual information modeling, while shared transformation parameters reduce memory and parameter costs.The paper also shares parameters across Fastformer layers to reduce parameter size and mitigate overfitting risk.
- Efficiency: O(N · d) total complexity makes Fastformer more efficient than standard Transformer self-attention at O(N^2 · d).With weight sharing, Fastformer uses 3hd^2 + 2hd parameters per layer versus at least 4hd^2 for Transformer.
4 Experiments
Experiments across classification, recommendation, summarization, and efficiency settings evaluate Fastformer's effectiveness, computational cost, and practical deployment behavior.
- Datasets and Experimental Settings: Experiments cover sentiment and topic classification, news recommendation, and text summarization across five benchmark datasets.The datasets include Amazon, IMDB, MIND, CNN/DailyMail, and PubMed.
- Effectiveness Comparison: Fastformer achieves competitive or better performance than other efficient Transformer variants on both long and short text classification.The authors attribute this to modeling global contexts and their relationships to different tokens.
- Effectiveness Comparison: Fastformer achieves the best performance among compared Transformer architectures in news recommendation and improves PLM-NR when substituted into its user encoder.The ensemble model achieves the best results on the MIND leaderboard.
- Effectiveness Comparison: Fastformer achieves the best performance in most summarization metrics on CNN/DM, while several other efficient variants underperform vanilla Transformer.The comparison includes sparse-attention and approximated-self-attention methods.
- Efficiency Comparison: Fastformer's theoretical complexity has the least dependence among compared methods, using only sequence length and hidden dimension rather than additional method-specific factors.Vanilla Transformer has complexity O(N2 ·d), whereas the other compared methods have linear sequence-length complexity with varying additional dependencies.
- Efficiency Comparison: Fastformer improves model performance while reducing parameter size through query-value and layer-wise parameter sharing.Query-value sharing performs similarly or slightly better, whereas head-wise sharing causes notable performance drops.
5 Conclusion and Future Work
Fastformer is presented as an additive-attention Transformer variant for efficient long-sequence processing, with experiments showing strong efficiency and competitive long-text performance. Future work targets pretraining for long-document NLP and applications to user-behavior modeling.
- Conclusion: Fastformer uses additive attention and linear complexity to handle long sequences efficiently.It summarizes queries and keys into global context vectors and combines token representations through element-wise products.
- Conclusion: Experiments on five benchmark datasets show Fastformer is much more efficient than many Transformer models while achieving competitive or better long-text performance.
- Future Work: Future work includes pretraining Fastformer-based language models for long-document modeling.
- Future Work: The authors also plan to apply Fastformer to e-commerce recommendation and Ads CTR prediction using long user-behavior sequences.
A.1 Experimental Environment
Experiments used a cloud Linux environment with Ubuntu 16.04, Python 3.6, Keras 2.2.4, TensorFlow 1.12, and a Tesla V100 GPU.
- Environment: Experiments ran on a cloud Linux server using Ubuntu 16.04.
- Environment: The implementation used Python 3.6 with Keras 2.2.4 and a TensorFlow 1.12 backend.
- Environment: Each experiment used a single thread on an Nvidia Tesla V100 GPU with 32GB of memory.
A.2 Preprocessing
Text preprocessing used NLTK tokenization to convert inputs into token sequences, with random-vector initialization for out-of-vocabulary words.
- Preprocessing: The preprocessing pipeline used NLTK to tokenize the input texts.
- Preprocessing: The word tokenize function converted texts into token sequences.
- Preprocessing: Out-of-vocabulary word embeddings were filled with random vectors matching the mean and covariance of other words.
A.3 Hyperparameter Settings
Dataset-specific hyperparameter settings are documented in Table 9.
- Hyperparameter Settings: The paper provides detailed hyperparameter settings for each dataset.
- Hyperparameter Settings: These settings are collected in Table 9.
- Hyperparameter Settings: Table 9 is labeled as the source for detailed hyperparameter settings on each dataset.