Source-linked AI summary
Attention, please! A survey of Neural Attention Models in Deep Learning
Alana de Santana Correia, Esther Luna Colombini
TL;DR
Neural attention models address the need to select and prioritize relevant information within limited computational processing across diverse deep-learning tasks. The paper systematically surveys their architectures, mechanisms, applications, and interpretability, analyzing 650 papers and identifying trends and research opportunities. It concludes that attention has influenced major advances across domains, while its role as a complete explanation of model decisions remains unresolved.
Problem
Existing surveys provide limited coverage by focusing on particular domains, while neural attention models have developed across architectures and applications.
Method
The paper conducts a systematic review of neural attention models and critically analyzes 650 papers across convolutional, recurrent, generative, and multimodal applications.
Results
The survey identifies common attention mechanisms, architectural uses, application patterns, interpretability findings, and research opportunities across deep learning.
Takeaways & Limitations
Attention is presented as a broad and influential component of deep learning, with notable advances especially in computer vision and natural language processing.
Takeaways & Limitations
Attention can help identify representations associated with outputs but does not fully explain the model’s decisions.
Abstract
from arXiv · showhide
In humans, Attention is a core property of all perceptual and cognitive operations. Given our limited ability to process competing sources, attention mechanisms select, modulate, and focus on the information most relevant to behavior. For decades, concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. For the last six years, this property has been widely explored in deep neural networks. Currently, the state-of-the-art in Deep Learning is represented by neural attention models in several application domains. This survey provides a comprehensive overview and analysis of developments in neural attention models. We systematically reviewed hundreds of architectures in the area, identifying and discussing those in which attention has shown a significant impact. We also developed and made public an automated methodology to facilitate the development of reviews in the area. By critically analyzing 650 works, we describe the primary uses of attention in convolutional, recurrent networks and generative models, identifying common subgroups of uses and applications. Furthermore, we describe the impact of attention in different application domains and their impact on neural networks' interpretability. Finally, we list possible trends and opportunities for further research, hoping that this review will provide a succinct overview of the main attentional models in the area and guide researchers in developing future approaches that will drive further improvements.
1 Introduction
Attention is a selective process that prioritizes relevant information across perception and cognition, inspiring computational models and later neural architectures. This survey reviews the field’s development, mechanisms, applications, and research opportunities.
- Attention in perception and cognition: Attention selectively focuses processing on relevant information while ignoring competing perceptible inputs.In biological systems, it allocates limited perceptual and cognitive resources across sensory processing and decision making.
- From computational models to deep learning: Computational attention systems built on psychophysical and neurobiological models preceded deep-learning attention mechanisms.Earlier systems supported object recognition, image compression, segmentation, tracking, robotics, navigation, and SLAM.
- From computational models to deep learning: Since 2014, neural attention models have expanded across deep-learning applications by dynamically managing information, features, and computational resources.The survey links attention with filtering task-irrelevant stimuli and handling long-time dependencies.
- Survey scope and contributions: Existing surveys often focus on particular domains, leaving a need for a broader review of neural attention models.Prior reviews separately emphasize recurrent networks, computer vision, NLP, graph neural networks, or shorter general coverage.
- Survey scope and contributions: The authors provide a replicable review methodology and critically analyze 650 papers using metrics extracted from 6,567 items.The survey also describes attention mechanisms, neural architectures, application domains, trends, and research opportunities.
2 Overview
Neural attention networks emerged in NLP around mid-2014 and subsequently expanded across computer vision, multimodal learning, memory-augmented systems, and end-to-end architectures. The overview traces how attention selects relevant information, addresses bottlenecks, and supports increasingly integrated neural designs.
- Origins and expansion: Around mid-2014, neural attention networks emerged in NLP and enabled advances in conversational comprehension, sentiment analysis, translation, question-answering, and transfer learning.Attention later appeared in other application fields.
- Origins and expansion: The encoder-decoder bottleneck motivated attention because compressing a source sentence into one fixed-length vector can degrade performance.Attention research addressed the need to retain more source information during decoding.
- Attentional interfaces and end-to-end models: Attentional interfaces plug modules into conventional networks, whereas end-to-end models use attentional layers across the network’s abstraction levels.Interfaces became the more explored direction because of their simplicity, generality, and reported generalization.
- Computer vision: In vision, RAM adaptively selects high-resolution image or video glimpses sequentially, while DRAW constructs scenes iteratively through repeated refinement.Both approaches connect attention with sequential visual processing.
- Multimodality: Multimodal attentional interfaces connect different sensory sources and help reason over misaligned data, extending early attention applications beyond isolated NLP and vision tasks.Image-captioning systems combine CNN image features with LSTM word generation, using attention to select relevant image information for each word.
- Attention-augmented memory: Memory-augmented models add addressable or explicit memory to represent history, helping address long-term dependencies that conventional recurrent states may forget.Memory Networks assign dedicated vector representations to sequence-history elements, making memory size a tunable hyper-parameter.
- End-to-end attention models: Transformers and Graph Attention Networks made attention the full architecture: self-attention replaces recurrence and convolution for sequences, while graph attention filters noisy graph parts.The Transformer captures relationships regardless of sequence distance and is described as simple, parallelizable, and low-cost to train.
3 Attention Mechanisms
Deep attention mechanisms are categorized as soft, hard, and self-attention, differing in how they weight inputs and model dependencies. Soft and hard attention focus on task-relevant elements, while self-attention models interactions among the inputs themselves.
- Soft Attention: Soft attention assigns continuous weights from 0 to 1, producing deterministic and differentiable attention through softmax functions.It can operate over spatial or temporal contexts to emphasize relevant features.
- Hard Attention: Hard attention assigns binary weights of 0 or 1, selecting input parts through a non-differentiable sequence of discrete decisions.In temporal settings, each selection helps determine where to attend next.
- Soft Attention: In visual question answering, soft attention weights visual features before answer generation, using feature magnitude as a relevance premise.High-magnitude feature vectors are associated with image regions containing relevant semantic content.
- Hard Attention: In visual question answering, hard attention selects only important visual features for subsequent processing.Unlike soft attention, it does not retain all visual features with varying weights.
- Self-Attention: Self-attention quantifies interdependence among input elements by allowing them to interact and determine what deserves greater attention.Its matrix-based computations support parallel processing of long inputs.
- Self-Attention: Self-attention examples show relationships among representative query locations in sentences and images through color-coded attention links.The links summarize the regions receiving the greatest attention for each query location.
4 Attention-based Classic Deep Learning Architectures
Attention-based interfaces extend classic convolutional, recurrent, and generative architectures by filtering inputs, reallocating processing, modeling dependencies, and sharing relevant representations. Across these architectures, attention is used to improve robustness, efficiency, contextual modeling, and information integration.
- CNN Architectures: The survey organizes CNN attention into six groups: attention pool, input, layer, prediction, residual attention, and attention out.These groups describe attention replacing pooling, filtering inputs, operating between layers, assisting prediction, creating residual connections, or sharing feature maps.
- CNN Architectures: CNN input attention filters irrelevant image regions, improving robustness to distractors and changes in lighting, scale, and viewpoint.The survey also associates it with scalability, complex tasks, simpler encoding, and prioritized learning.
- CNN Architectures: CNNs have limited receptive fields and treat channels equally, restricting long-range pixel dependencies and flexible feature use.Larger receptive fields provide more context but may increase training instability or hinder convergence.
- CNN Architectures: Squeeze-and-Excitation Networks pioneered attention between convolutional layers by modeling channel interdependencies through squeeze and excitation blocks.The squeeze operation encodes spatial information into channel descriptors for recalibrating filter responses.
- CNN Architectures: Attention-controlled sparsity reduced forward-pass runtime by more than 200% in one layer during training.A k-selection filter controls data sparsity and therefore computational resources during training and inference.
- CNN Architectures: Attention out-groups share relevant feature maps across architectures or instances to support feature fusion, multimodality, and external knowledge.Some approaches turn CNNs into recurrent convolutional networks for challenging image problems.
- RNN Architectures: Recurrent attention input uses item-wise or location-wise hard and soft selection to choose discrete inputs or assign continuous weights.These mechanisms operate over sequence items or input features according to task relevance.
- RNN Architectures: Recurrent hidden attention dynamically builds context vectors for inter-alignment, contextual embedding, multiple-input processing, memory management, and long-term dependencies.The encoder-decoder attention interface helps reduce the bottleneck problem and has been applied beyond machine translation.
5 Applications
Attention models are applied across language, vision, multimodal tasks, recommendation, and robotics to select relevant information, align inputs, model dependencies, and improve representations. The survey describes attention’s roles in sequence alignment, fine-grained visual recognition, multimodal reasoning, user-interest modeling, navigation, and interpretability.
- Natural Language Processing (NLP): Attention aligns input and output sequences in machine translation, question answering, and speech recognition, capturing long-range dependencies and selecting relevant information.In speech recognition, it aligns acoustic frames with anchor words while ignoring background noise and interfering speech; in question answering, it links queries with contextual content.
- Natural Language Processing (NLP): Attention improves text summarization and classification by building embeddings that incorporate contextual, semantic, and hierarchical information.For summarization, the survey identifies benefits for keyword modeling, abstract sentence generation, hierarchical structure, phrase repetition, and concise output.
- Computer Vision (CV): In computer vision, attention amplifies fine-grained recognition, handles occlusion and viewpoint changes, and recalibrates features toward discriminative image regions.Reported approaches include recurrent convolutional networks for bird subspecies and reinforcement-learning architectures for vehicle subtypes.
- Computer Vision (CV): Attention supports image generation by modeling dependencies between distant regions, enabling fine details, high-resolution images, and intricate geometric patterns.Self-attention complements convolution, addressing limitations of stacked convolutions in representing distant-region dependencies.
- Multimodal Tasks (CV/NLP): Multimodal attention aligns textual and visual features and can construct shared semantic spaces or hierarchical representations for visual question answering.Hypergraph Attention Networks use modality-specific symbolic graphs and co-attention maps to extract an inter-modal representation.
- Recommender Systems (RS) and Robotics: Attention models user behavior through long- and short-term interests, supports memory-based representations, and helps agents retain relevant history for navigation and long-horizon tasks.In robotics, self-attention models human-human and human-robot interactions, while scene memory stores past observations for action selection.
6 Trends and Opportunities
Attention has become central to state-of-the-art deep learning, while the field continues to contain important gaps and opportunities for further research.
- The survey identifies substantial room for growth and highlights gaps and opportunities for future contributions in neural attention research.
- State-of-the-art deep-learning models use attention, including end-to-end architectures such as Transformers and Graph Attention Networks across multiple domains.
6.2 Learning Multimodality
Attention supports learning from multimodal data by modeling relationships among sensory signals, but scalable architectures for more than paired modalities remain underexplored.
- Attention helps models learn synergistic relationships between different sensory data, including unsynchronized signals, for emotion recognition, feelings, and language-based image generation.
- Most multimodal research relates paired visual and textual data, while architectures that scale beyond one sensor pair are not yet widely explored.
- Multimodal applications increasingly incorporate voice, RGBD images, monocular-camera images, navigation sensors, and biomedical sensors, but these combinations remain scarce.
6.3 Cognitive Elements
Attention has enabled neural architectures to incorporate cognitive elements such as memory, while recurrent attention models offer flexible image computation through selective glimpses.
- Attention enables neural networks to incorporate cognitive elements that were traditionally neglected, including memory and logic flow control.
- Memory Networks and Neural Turing Machines use attention to update and recover information from external memory.
- Recurrent Attention Models introduced image computation with glimpses and hard attention through a simple, scalable, and flexible architecture.
- RAM is described as potentially useful for tasks involving large, high-resolution images where convolutional neural networks have difficulties.
6.5 Capsule Neural Network
Attention has been incorporated into capsule networks through routing and self-attention mechanisms, while neural-symbolic reasoning remains limited in its treatment of symbolic relevance.
- Capsule Neural Network: Attention-based capsule models include dynamic routing, dual attention, self-attention layers, and attentional routing between daughter and parent capsules.
- Capsule Neural Network: The survey suggests attention may improve the relational and hierarchical properties that capsule networks are designed to represent.
- Neural-Symbolic Learning and Reasoning: Neural-symbolic systems use attention to select operations, aggregate graph-neighbor information, and complete missing knowledge-graph relations.
- Neural-Symbolic Learning and Reasoning: Recent neural reasoning approaches primarily use attention for efficient memory management and do not adequately address symbolic reasoning.
6.7 Incremental Learning
Attention has been little explored for incremental learning, despite the need to add classes from limited data without retraining on the complete dataset.
- Incremental learning aims to add classes from limited data without retraining on the complete training set.
- Attention could contribute significantly to incremental learning, but this application remains little explored.
6.8 Credit Assignment Problem (CAP)
The Credit Assignment Problem concerns identifying which actions contribute to future rewards, so actions leading to higher cumulative reward receive more credit.
- Credit assignment seeks to determine individual actions’ contributions to future rewards.
- Actions leading to higher final cumulative reward should receive more credit than actions leading to lower final reward.
- The temporal CAP variation identifies which actions were useful over time.
6.9 Attention and Interpretability
Research on attention as an interpretability tool is mixed: some studies find attention informative, while others show that attention weights may diverge from other explanations and predictions.
- Attention’s usefulness as an interpretability tool remains contested across studies.
- Jain and Wallace found that attention is inconsistent with other explainability metrics and that similar attention distributions can yield different results.
- Other studies report that attention captures syntax and co-reference, while Vashishth et al. found interpretability and feature-importance correlation in some NLP cases.
6.10 Unsupervised Learning
Attention has been applied in unsupervised generative and language models, but important gaps remain in simplifying and stabilizing generative systems and developing purely bottom-up attentional learning.
- In GANs, attention improves global perception by learning which image parts should receive more attention.
- DRAW uses sequential attention to refine image patches as more information is captured, but it was tested only on simple datasets.
- Attention remains underexplored for making generative models simpler, scalable, and more stable.
- Unsupervised NLP attention is used mainly for sentiment-aspect extraction, while purely bottom-up attentional systems remain unexplored.
6.11 New Tasks and Robotics
The survey identifies time-series prediction, medical applications, and robotics as relatively little-explored areas where attention may improve results. It highlights multimodal, uncertain, and planning-intensive robotics as a particularly broad opportunity.
- 6.11 New Tasks and Robotics: Time-series prediction, medical applications, and robotics remain little-explored areas that may benefit from attention.The survey specifically notes increasing difficulty in time-series prediction as series length grows.
- 6.11 New Tasks and Robotics: Robotics requires deep-learning techniques to handle multimodality, active and incremental learning, uncertainty, semantics, reasoning, awareness, and planning.
- 6.11 New Tasks and Robotics: RAM, DRAW, and Transformer architectures are identified as potentially useful for advancing deep-learning applications in robotics.
7 Conclusions
The survey systematically reviews attention in deep learning across approaches, architectures, applications, and interpretability. It analyzes more than 600 papers, reporting attention’s links to performance, efficiency, and understanding while identifying theoretical taxonomy as an ongoing need.
- 7 Conclusions: More than 600 relevant papers published from 2014 onward were critically analyzed in the survey.The survey presents this as its broadest literature coverage compared with existing domain-specific reviews.
- 7 Conclusions: The review connects attention mechanisms across CNNs, RNNs, and generative models, while covering applications and interpretability.
- 7 Conclusions: Attention is discussed in relation to performance gains, improved computational efficiency, and better understanding of networks’ knowledge.
- 7 Conclusions: The authors identify a need for taxonomies grounded in theoretical concepts of attention and extensible across architectures and application domains.Existing taxonomies are described as lacking theoretical concepts and being difficult to extend broadly.
Appendix
The appendix describes a reproducible systematic-review process that searched broad machine-learning and related repositories, then applied successive automatic and manual filters. The process narrowed a large collection through attention- and deep-learning-specific criteria and additional abstract-based screening.
- Appendix: The survey uses a systematic-review approach to collect, critically evaluate, and synthesize primary studies on attention in deep learning.The approach is intended to be objective, transparent, meticulous, and replicable.
- Appendix: The search covered major machine-learning repositories and related sources during two collection stages spanning 2014 to 2021.
- Appendix: The filtering sequence combined general attention terms, deep-learning terms, and specific attention-in-deep-learning terms before further screening.
- Appendix: For papers with abstracts, later screening required more than five Filter II terms and additional cascade conditions.