Source-linked AI summary
An Empirical Study of Spatial Attention Mechanisms in Deep Networks
Xizhou Zhu, Dazhi Cheng, Zheng Zhang, Stephen Lin, Jifeng Dai
TL;DR
Prior work had not closely examined how spatial-attention factors and weight-computation methods affect performance across modules and applications. The paper unifies Transformer attention, deformable convolution, and dynamic convolution, then ablates their elements across language and vision tasks. It finds that query-key content comparison is minor in self-attention but vital in encoder-decoder attention, while combining deformable convolution with key-content-only saliency offers the best self-attention accuracy-efficiency tradeoff.
Problem
Prior work had limited examination of how spatial-attention factors and their weight-computation methods affect performance, including across Transformer and convolutional modules.
Method
The paper views Transformer attention, deformable convolution, and dynamic convolution as spatial-attention variants and ablates their elements within a generalized formulation across three applications.
Results
Query-key content comparison is negligible for self-attention but vital for encoder-decoder attention, while deformable convolution combined with key-content-only saliency achieves the best self-attention accuracy-efficiency tradeoff.
Takeaways & Limitations
The findings challenge conventional understanding of query-sensitive attention and indicate substantial room for improving spatial-attention design.
Abstract
from arXiv · showhide
Attention mechanisms have become a popular component in deep neural networks, yet there has been little examination of how different influencing factors and methods for computing attention from these factors affect performance. Toward a better general understanding of attention mechanisms, we present an empirical study that ablates various spatial attention elements within a generalized attention formulation, encompassing the dominant Transformer attention as well as the prevalent deformable convolution and dynamic convolution modules. Conducted on a variety of applications, the study yields significant findings about spatial attention in deep networks, some of which run counter to conventional understanding. For example, we find that the query and key content comparison in Transformer attention is negligible for self-attention, but vital for encoder-decoder attention. A proper combination of deformable convolution with key content only saliency achieves the best accuracy-efficiency tradeoff in self-attention. Our results suggest that there exists much room for improvement in the design of attention mechanisms.
1. Introduction
The paper unifies major spatial attention modules around a small set of input factors and empirically disentangles their effects across language and vision tasks. Its findings challenge the view that query-sensitive content comparison is universally central, while identifying substantial room to improve spatial attention design.
- Attention factors: Spatial attention weights commonly use query content, key content, and relative position, yielding four attention factors in a generalized formulation.The factors are query and key content; query content and relative position; key content only; and relative position only.
- Unified perspective: The study treats Transformer attention, deformable convolution, and dynamic convolution as spatial-attention instantiations with different factor subsets and weight functions.It evaluates these mechanisms across neural machine translation, semantic segmentation, and object detection.
- Key findings: Query-sensitive terms, especially query-key content comparison, play a minor role in self-attention but are vital in encoder-decoder attention.In self-attention, query-key content can be removed; in encoder-decoder attention, removing it causes a noticeable accuracy drop.
- Key findings: Self-attention modules using only query-sensitive terms perform on par with modules using only query-irrelevant terms, suggesting the degeneration reflects module design rather than self-attention itself.Deformable convolution effectively and efficiently exploits query content and relative position in image recognition.
- Implication: The analysis concludes that spatial attention mechanisms have substantial design headroom and aims to motivate further investigation of their operational mechanisms.The study presents initial progress toward improving their design.
2. Related Work
Prior work developed attention across NLP and vision, while related convolutional modules can also be interpreted as spatial attention. The paper addresses limited component-level analysis with a cross-domain empirical study that directly measures disentangled factors and weight functions.
- Attention applications: Attention mechanisms expanded from neural machine translation to object reasoning, image captioning, generation, image recognition, and video recognition.Vision systems often use formulations similar to Transformer attention, with keys and queries representing visual elements.
- Study scope: Because attention-factor effects may vary by application, the study evaluates neural machine translation, object detection, and semantic segmentation.These tasks span NLP and computer vision settings strongly influenced by attention modeling.
- Related mechanisms: Deformable and dynamic convolution are included because they focus on selected input parts and can be viewed as attention mechanisms using subsets of factors with different weight functions.This places them alongside Transformer attention in a unified spatial-attention analysis.
- Research gap: Earlier spatial-attention analysis largely visualized whole-module weights, especially in NLP, rather than directly measuring individual elements across NLP and vision.The paper describes its study as the first comprehensive empirical analysis of these elements across both domains.
- Research gap: The paper disentangles attention factors and weight-assignment functions and measures their effects using final task performance.This design targets the limited prior evidence on the relative significance of different spatial-attention components.
3. Study of Spatial Attention Mechanisms
The study represents Transformer attention, regular and deformable convolution, and dynamic convolution within a generalized spatial-attention formulation. It compares their attention factors, computational costs, and design choices across self-attention and encoder-decoder settings.
- Generalized formulation: The generalized formulation expresses multi-head attention as weighted aggregation of key contents over a query-specific supporting region.Each head uses attention weights and learnable projections before the head outputs are linearly aggregated.
- Transformer attention: Transformer attention combines four factors: query-key content, query content with relative position, key content only, and relative position only.The E1 and E2 terms are query-sensitive, whereas E3 and E4 are query-irrelevant.
- Transformer attention: Query-key content compatibility is widely treated as central to Transformer attention, motivating variants that retain only E1.The study tests this assumption by ablating attention terms within the generalized formulation.
- Convolutional attention: Regular convolution uses fixed local sampling and relative position only, whereas deformable convolution predicts query-dependent offsets to adjust sampling locations.Deformable convolution is therefore modeled as a self-attention instantiation with dynamically selected spatial support.
- Efficiency: Transformer terms E1, E2, and E4 require query-key-pair traversal, while E3 has computation linear in the number of key elements.Deformable convolution uses sparse sampling and is much faster than E2 for image recognition; dynamic convolution can also reduce overhead with smaller kernels.
- Convolutional attention: Dynamic convolution assigns query-dependent kernel weights within a local range, with computational overhead proportional to kernel size times the number of query elements.Its formulation treats input channels as separate attention heads and uses shared dynamic kernel weights.
4. Experiments and Analysis
Experiments disentangle spatial-attention factors across object detection, semantic segmentation, and neural machine translation, comparing Transformer configurations with deformable and dynamic convolution. Results show that attention factors have task-dependent importance, with query-key content vital for encoder-decoder attention but largely removable in self-attention, where deformable convolution plus key-content saliency offers the best tradeoff.
- Disentanglement in Transformer attention: The study evaluates all 16 configurations of Transformer attention factors, controlling activation and deactivation through βTrans values.The four factors are examined across self-attention and encoder-decoder attention settings.
- Disentanglement in Transformer attention: In self-attention, query and key content has a negligible accuracy effect, while removing it substantially reduces computational overhead in image recognition tasks.Removing the term from configuration “1111” to “0111” causes only a tiny accuracy drop.
- Disentanglement in Transformer attention: In encoder-decoder attention, query and key content is vital, and using it alone achieves accuracy almost equal to the full Transformer configuration.This term supports source-target word alignment in neural machine translation.
- Disentanglement in Transformer attention: In self-attention, configurations using query content with relative position and key content only approach full-model accuracy while reducing computational overhead.Configuration “0110” is especially effective in image recognition, and key-content-only saliency improves performance with little additional overhead.
- Disentanglement in Transformer attention: Deformable convolution surpasses E2 in image tasks but is only on par with E2 for neural machine translation, reflecting differences in local feature smoothness.Dynamic convolution matches E2 at lower cost for NMT but performs considerably worse for object detection and semantic segmentation.