Source-linked AI summary
Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, Ruslan Salakhutdinov
TL;DR
The paper addresses how to better understand and design Transformer attention, particularly its component structure and positional-embedding integration. It recasts attention as kernel smoothing, uses that view to organize existing and new variants, and reports competitive performance with less computation for a proposed variant.
Problem
The paper seeks a clearer understanding of Transformer attention and a broader framework for designing its kernel and positional-embedding components.
Method
The paper reformulates attention as kernel smoothing and constructs attention through kernels, including a valid symmetric positive semi-definite kernel on a joint feature-position space.
Results
The proposed attention variant requires fewer parameters and empirically reaches competitive performance with previous state-of-the-art models.
Takeaways & Limitations
Kernel construction provides a unified lens for understanding attention variants and expanding the space of attention mechanisms.
Takeaways & Limitations
The paper discusses kernels beyond the rigorous definition requiring symmetry and positive semi-definiteness, while Section 3 examines semi-positive and symmetric kernels.
Abstract
from arXiv · showhide
Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction. At the core of the Transformer is the attention mechanism, which concurrently processes all inputs in the streams. In this paper, we present a new formulation of attention via the lens of the kernel. To be more precise, we realize that the attention can be seen as applying kernel smoother over the inputs with the kernel scores being the similarities between inputs. This new formulation gives us a better way to understand individual components of the Transformer's attention, such as the better way to integrate the positional embedding. Another important advantage of our kernel-based formulation is that it paves the way to a larger space of composing Transformer's attention. As an example, we propose a new variant of Transformer's attention which models the input as a product of symmetric kernels. This approach achieves competitive performance to the current state of the art model with less computation. In our experiments, we empirically study different kernel construction strategies on two widely used tasks: neural machine translation and sequence prediction.
1 Introduction
Transformers process entire sequences concurrently through attention, whose similarity-based weighting is order-agnostic unless positional information is added. The paper reframes attention as kernel smoothing to clarify its components and explore kernel-based variants.
- Transformers outperform recurrent and temporal-convolutional models across sequence modeling and several other tasks.
- Attention concurrently processes the entire input sequence and integrates dependencies among its elements.
- Attention forms a weighted combination of inputs, with weights determined by similarities between input elements.
- Because attention is order-agnostic, positional embeddings encode sequence order for sequence modeling.
- The paper formulates attention as a kernel smoother over inputs, where the kernel measures pairwise similarity and enables a broader family of attention mechanisms.
- The formulation highlights attention components, identifies kernel construction as especially important, and evaluates kernel forms and positional-embedding integration on NMT and sequence prediction.
2 Attention
The paper reformulates Transformer attention as kernel smoothing, exposing its components and enabling systematic alternatives. This perspective motivates valid symmetric kernels, joint feature–position constructions, and design choices for values and filtering.
- Original attention: Transformer attention concurrently processes inputs through similarity-weighted combinations, while decoder self-attention additionally filters out future tokens.Encoder self-attention and encoder-decoder attention use no additional mask in the original formulation.
- Kernel reformulation: The kernel formulation represents attention as a linear smoother over a query, filtered keys, and a value function.Its inputs are a non-negative kernel, a set-filtering function, and a value function.
- Kernel construction: The original Transformer uses an asymmetric exponential kernel with query and key mappings, whereas the paper examines symmetric and other kernel constructions.The examined alternatives include linear, polynomial, and RBF kernels.
- Kernel reformulation: The formulation decomposes attention into kernel feature space, kernel construction, value function, and set filtering, providing a framework for categorizing prior variants.These components correspond to choices about representations, similarities, outputs, and visibility.
- Kernel construction: A product of feature and temporal kernels captures similarities in both non-temporal and positional components.This joint-space construction contrasts with applying the exponential kernel directly to summed feature and positional embeddings.
- Value function: Constructing the value function without positional embedding constantly outperforms including positional embedding in the reported experiments.The comparison is between Eq. (8), v((f_k,t_k)) := f_kW_v, and the positional-value construction in Eq. (7).
3 Experiments
Experiments evaluate positional-embedding strategies, kernel forms, symmetry, order sensitivity, and positional information in values across neural machine translation and sequence prediction. Results favor product-kernel positional integration and infinite-feature kernels, while showing decoder self-attention is not permutation equivariant and value-function positional embedding adds little.
- Experimental setup: Experiments on neural machine translation and sequence prediction compare positional embedding, kernel, symmetry, order-invariance, and value-function choices.NMT includes three attention types, whereas SP uses decoder self-attention only.
- Incorporating Positional Embedding: Relative-position look-up embeddings outperform direct-sum embeddings, especially for sequence prediction.The look-up table is indexed by relative position tq − tk rather than absolute position.
- Incorporating Positional Embedding: The proposed product kernel achieves the best NMT result and is competitive with the best SP result, while Dai et al.’s product-kernel positional embedding has a lower NMT BLEU score.The experiments compare four positional-embedding integrations, including direct sum, look-up, and asymmetric or symmetric product kernels.
- Kernel Types: Linear kernels do not converge for either NMT or SP, while RBF performs best for NMT and exponential performs best for SP.The authors attribute linear-kernel failure to potentially negative scores violating the positive-score assumption of kernel smoothing.
- Kernel Types: Symmetric and asymmetric kernels show little performance difference, while the symmetric design saves parameters when the projection size is fixed.Symmetry is tested by forcing Wq = Wk.
- Order-Invariance in Attention: Decoder self-attention is not permutation equivariant because its masked decoding process already encodes order information.Removing positional embedding only from decoder self-attention reduces NMT performance from 34.71 to 34.49, whereas removing it throughout the model reduces performance to 14.47.
- Positional Embedding in Value Function: Adding positional embedding to the value function produces little performance difference compared with omitting it.This conclusion is reported from the Eq. (7) versus Eq. (8) comparison.
4 Related Work
Related work connects Transformer attention to graph-structured learning, including non-local image operations and message passing in graphical models. The paper distinguishes its kernel-learning perspective from these graph-based connections.
- Graph-structured connections: Prior work relates attention to graph-structured learning, including non-local operations in image processing and message passing in graphical models.Examples include Non-Local Neural Networks and attention variants for positional relations or multimodal feature types.
5 Conclusions
The paper formulates Transformer attention through kernels, unifying existing variants while expanding the space of attention designs. It also identifies kernel construction as especially important and studies kernel forms and positional-embedding integration across two tasks.
- The kernel formulation unifies previous Transformer attention variants by clarifying their individual components.
- The formulation enables a larger space for designing attention mechanisms.
- A proposed attention variant achieves competitive performance against previous state-of-the-art models.
- Kernel-function construction is identified as the most important component of Transformer attention.
- The study examines different kernel forms and positional-embedding integration strategies on neural machine translation and sequence prediction.