Source-linked AI summary
A General Survey on Attention Mechanisms in Deep Learning
Gianni Brauwers, Flavius Frasincar
TL;DR
Attention research spans many deep-learning domains, but its mechanisms and terminology require a unified account. This survey develops a general framework and taxonomy, reviews evaluation and structural analysis, and concludes that attention has broad performance benefits while interpretability claims require caution and coverage beyond supervised learning remains limited.
Problem
Existing attention surveys generally focus on individual domains, while taxonomies lack the depth and structure needed to distinguish the diverse mechanisms comprehensively.
Method
The survey explains attention through a general model and uniform queries-keys-values notation, organizes mechanisms into a taxonomy, and reviews evaluation and structural-analysis methods.
Results
The survey synthesizes attention mechanisms across deep-learning domains and reports that they have produced significant performance improvements and state-of-the-art results across varied tasks.
Takeaways & Limitations
Attention remains an active field with opportunities to apply existing mechanisms to new tasks, including semi-supervised and unsupervised learning.
Takeaways & Limitations
The survey mainly covers supervised attention models, while semi-supervised and unsupervised attention research has received limited and only recently increasing attention.
Abstract
from arXiv · showhide
Attention is an important mechanism that can be employed for a variety of deep learning models across many different domains and tasks. This survey provides an overview of the most important attention mechanisms proposed in the literature. The various attention mechanisms are explained by means of a framework consisting of a general attention model, uniform notation, and a comprehensive taxonomy of attention mechanisms. Furthermore, the various measures for evaluating attention models are reviewed, and methods to characterize the structure of attention models based on the proposed framework are discussed. Last, future work in the field of attention models is considered.
1 INTRODUCTION
The survey introduces attention as a widely useful deep-learning mechanism and positions itself as a cross-domain framework for explaining, classifying, and evaluating attention techniques.
- Attention mechanisms improve performance across many deep-learning tasks and can be trained jointly with base models using regular backpropagation.They also introduce a degree of interpretation into otherwise complex neural-network models.
- The Transformer’s initial success drove extensions targeting predictive performance, computational efficiency, or both, alongside adoption in image, video, and recommender-system tasks.Transformer-XL addresses longer dependencies through recurrence, while remaining more efficient during evaluation.
- This survey provides a cross-domain overview of attention techniques rather than restricting its analysis to a single application domain.It aims to make attention techniques understandable and applicable across varied domains.
- The paper proposes a structured taxonomy to distinguish attention mechanisms more deeply than previous domain-specific surveys.The taxonomy organizes techniques according to the task-model structure and addresses techniques that earlier surveys had not properly covered.
- The paper reviews performance measures and evaluation methods for attention models, then uses the taxonomy to assess their structural organization.Its final section presents conclusions and suggestions for further research.
2 GENERAL ATTENTION MODEL
The survey formulates attention as a general module that transforms task inputs into feature vectors, uses queries to score keys, aligns those scores, and produces output predictions from weighted value averages.
- 2 GENERAL ATTENTION MODEL: The task model comprises feature, query, attention, and output submodels that transform inputs into predictions.Feature and query models prepare inputs for attention; attention and output models produce the result.
- 2.1 Attention Input: Feature models transform original inputs into feature vectors that the attention model attends to.Inputs may include words, image pixels, acoustic sequences, or other collections.
- 2.1 Attention Input: Queries specify which feature vectors to attend to, with their design generally determined by the desired output.In image captioning, a decoder RNN hidden state can serve as the query at each prediction step.
- 2.2 Attention Output: The attention module derives keys and values from feature vectors, scores keys against the query, and aligns the scores into attention weights.Keys support scoring, while values are combined to form the module output.
- 2.2 Attention Output: Attention weights form a weighted average of value vectors to create a context vector, which the output model converts into a prediction.A simple softmax layer is one possible output model.
- 2.3 Attention Applications: Because attention does not depend on feature-vector organization, the general model applies across structured and irregular domains whenever feature vectors can be defined or extracted.Examples include language, images, audio, video, protein structures, traffic flows, and communication networks.
3 ATTENTION TAXONOMY
The survey organizes attention mechanisms into a taxonomy that supports classification and architectural analysis across feature-related, query-related, and general dimensions.
- 3 ATTENTION TAXONOMY: The taxonomy classifies mechanisms as feature-related, query-related, or general according to their relationship to feature models and query models.Each mechanism is either a modification of the general module's inner mechanisms or an extension of it.
- 3 ATTENTION TAXONOMY: Attention models can combine techniques from any or all taxonomy categories, while some mechanisms are generally required and others are optional.Scoring and alignment functions are generally required; multi-head and co-attention are not necessary in every situation.
- 3 ATTENTION TAXONOMY: The taxonomy includes query types such as basic, specialized, and self-attentive queries, alongside mechanisms including singular-query, multi-head, multi-hop, and capsule-based attention.These labels appear as taxonomy entries and subcategories.
- 3 ATTENTION TAXONOMY: The survey provides uniform notation for feature, key, and value matrices and uses the taxonomy to characterize attention-model architecture.Table 1 gives an overview of the notation and corresponding descriptions.
3.1 Feature-Related Attention Mechanisms
Feature-related attention mechanisms adapt attention to the multiplicity, levels, and representations of features. The survey describes co-attention variants for multiple inputs, attention-via-attention for hierarchical features, and multi-representational attention for combining embeddings.
- Feature-related mechanisms address feature multiplicity, feature levels, or feature representations.These mechanisms accommodate structures that may require attention beyond a single feature matrix.
- Multiplicity of Features: Co-attention uses two feature matrices, with coarse-grained variants using a compact representation as a query and fine-grained variants using all feature vectors as queries.The survey distinguishes these groups by how attention scores are calculated across the two inputs.
- Multiplicity of Features: Alternating co-attention uses each module’s context vector as the query for the other module, calculating the two context vectors sequentially.In the VQA example, image and question features are produced by separate feature models before attention is applied.
- Multiplicity of Features: Interactive co-attention calculates attention on both feature matrices in parallel, while parallel co-attention uses all feature vectors of each input as queries.Parallel co-attention uses an affinity matrix to translate one keys matrix into the space of the other.
3.2 General Attention Mechanisms
General attention mechanisms are characterized by score functions and alignment methods that convert queries and keys into weights over values. The survey compares score-function choices and alignment strategies, including soft, hard, local, and reinforced alignment.
- General attention mechanisms can be described through the attention score function, attention alignment, and attention dimensionality.The score and alignment functions are generally required, whereas mechanisms such as multi-head or co-attention are optional.
- Attention Score Functions: Score functions compute score(q, k_l) from a query vector and a key vector, with concatenate and multiplicative forms among the most popular choices.The survey also covers general, biased general, activated general, and similarity-based score functions.
- Attention Score Functions: The multiplicative score function is computationally inexpensive, but large key dimensions can produce suboptimal results; scaled multiplicative scoring addresses large dot-product magnitudes.The survey notes that multiplicative or scaled multiplicative scoring is typically preferred when efficiency is vital.
- Attention Alignment: Alignment transforms attention scores into weights used to compute a weighted-average context vector.A softmax function is a common alignment choice, and the resulting weights indicate the relative importance of feature vectors.
- Attention Alignment: Soft or global alignment attends to all feature vectors, whereas hard alignment selects exactly one feature vector.Hard alignment is typically more efficient at inference but requires sampling, simulation, or reinforcement learning during training instead of regular backpropagation.
- Attention Alignment: Local alignment applies softmax within a window centered at a position p, which can be set monotonically or predicted by the model.The window spans [p − D, p + D], and predictive alignment learns the location of interest in the sequence.
- Attention Alignment: Reinforced alignment uses a reinforcement-learning agent to select a subset of feature vectors before calculating soft alignment.This approach compromises between soft and hard alignment by replacing a fixed window with an agent-selected subset.
3.3 Query-Related Attention Mechanisms
Query-related attention mechanisms organize attention around how queries are formed and how many are used. The survey covers self-attention, multi-head attention, multi-hop attention, and their roles in constructing improved feature representations.
- Types of queries: Queries determine which information attention extracts from feature vectors and may be based on the task model, auxiliary variables, or specialized inputs.Some mechanisms use trainable vectors or feature-derived queries instead.
- Self-attention: Self-attention derives query-related information from the feature vectors themselves and captures relations among those vectors.Its context vectors summarize information relevant to each feature-vector representation.
- Self-attention: Self-attention can update feature vectors directly or add context vectors through normalization, producing improved word or sentence embeddings that enhance model accuracy.The latter approach combines previous features with the acquired context vectors before normalization.
- Multiplicity of queries: Multi-head attention applies multiple attention modules in parallel to differently transformed versions of one query, allowing heads to focus on different input information.For machine translation, heads may learn distinct noun–verb and noun–pronoun relations.
- Multiplicity of queries: In multi-head attention, head-specific context vectors are concatenated and linearly transformed to produce one context vector with a desired dimension.The output dimension can be set equal to the per-head value dimension so the context-vector dimension remains unchanged.
- Multiplicity of queries: Multi-hop attention applies attention modules sequentially, iteratively adjusting context vectors, whereas multi-head attention processes modules in parallel.The Transformer combines forms of multi-hop and multi-head attention, but its computational complexity scales quadratically with the number of input feature vectors.
4 EVALUATION OF ATTENTION MODELS
The survey evaluates attention models structurally, extrinsically, intrinsically, and through manual inspection. It also emphasizes that attention weights can support diagnosis or plausible interpretation, but do not reliably establish feature importance or human-like attention.
- Structural evaluation: The proposed taxonomy evaluates model structure by treating attention-mechanism categories as orthogonal dimensions and identifying which mechanism each model uses.Table 3 summarizes attention models using this taxonomy.
- Extrinsic evaluation: Extrinsic evaluation measures task performance using domain-specific metrics, comparisons with state-of-the-art models, or ablations that replace attention with alternative mechanisms.One example replaces a weighted context-vector average with an unweighted average.
- Intrinsic evaluation: Intrinsic evaluation measures attention itself using annotated attention vectors, human attention behavior, or metrics such as Alignment Error Rate and Attention Correctness.Attention Correctness sums weights assigned to ground-truth regions and ranges from 0 to 1.
- Evaluation via human attention: Human-attention comparisons report that standard attention maps generally do not correspond to human attention, often covering larger or multiple non-adjacent regions.Supervising attention with human gaze data is presented as a way to improve alignment with human patterns.
- Manual evaluation: Manual inspection can reveal whether models focus on relevant inputs, but using attention weights as explanations remains contested.Experiments found weak correspondence with important features and showed that different attention distributions can preserve the same model output.
- Manual evaluation: Attention patterns should support only cautious, plausible explanations because their interpretive validity can depend on model architecture.Incorrect focus can help diagnose problems when relevant information is known, but attention weights do not establish that attended parts are significant.
5 CONCLUSION
The survey organizes attention mechanisms through a unified framework and taxonomy, reviews their evaluation, and identifies practical opportunities and research gaps. It emphasizes broad performance benefits while noting limited coverage of unsupervised settings and insufficient theoretical support.
- The survey presents a comprehensive taxonomy organized around feature, attention, query, and output models, using queries, keys, and values as a common framework.The taxonomy categorizes diverse attention mechanisms and supports analysis of their structure.
- Extrinsic and intrinsic measures are presented for evaluating attention models, while the taxonomy is used to analyze their structural organization.
- Attention mechanisms are typically simple to understand and implement, can significantly improve performance, and remain applicable to many tasks and models.The survey highlights multi-dimensional attention as broadly applicable but underused, and multi-head attention as efficiently parallelizable yet concentrated in Transformer architectures.
- The survey mainly covers supervised models, while semi-supervised and unsupervised attention research has received limited and relatively recent attention.Future work may incorporate hand-annotated data and human attention behavior as larger datasets become available.
- Despite attention's intuitive clarity and practical performance improvements, the field still lacks substantial theoretical support.The survey anticipates further theoretical studies alongside continued practical development.