Attention, please! A survey of Neural Attention Models in Deep Learning

1 Introduction

• Attention selects and modulates relevant information across sensory processing, cognition, decision-making, and awareness because available inputs, memories, and actions exceed processing capacity.Its effects have been studied through introspection, electrophysiology, and brain imaging, with roots in experiments by Helmholtz and James.Source: §1 · pp. 1–2

1.1 Pre-Deep Learning Models of Attention

• Before deep learning, computational attention models established theoretical foundations and supported visual selection, recognition, compression, robotics, tracking, navigation, and SLAM.Visual systems commonly computed intensity, orientation, and color, combined them into salience maps, and routed prominent regions for further processing.Source: §1.1 · p. 2

1.2 Deep Learning Models of Attention: the beginning

• Neural attention became fundamental to deep learning around 2014 by dynamically filtering information, features, and resources to improve learning and handle long-time dependencies across applications [36] [37] [38].The field’s rapid growth motivates questions about which problems benefit from attention, how performance improves, and why those benefits arise.Source: §1.2 · p. 3

1.3 Contributions

• The survey contributes a replicable methodology, an in-depth analysis of 650 papers and 6,567 metrics, and comprehensive coverage of mechanisms, architectures, applications, trends, and research opportunities.It also explains how attention extends classic deep-learning architectures and augments Neural Network Zoo diagrams.Source: §1.3 · pp. 3–4

2 Overview

• Attention research expanded from NLP into computer vision, reinforcement learning, and robotics, with RNNSearch [44], Transformer [37], Memory Networks [38], “show, attend and tell”, and RAM [46] identified as key developments.Figure 2 ranks architectures by citation impact and innovation level, with larger circles indicating greater impact.Source: §2 · p. 4, Figure 2 · p. 4
• RNNSearch [44] addressed the encoder–decoder bottleneck by adaptively selecting source representations during translation, outperforming classic encoder–decoders on long sentences.It replaced a single fixed-length context vector with a sequence of vectors and propagated relevant information through attention.Source: §2 · p. 5

2.1 Attentional interfaces

• Attentional interfaces became the dominant research direction because they are modular, broadly applicable, and generalize well, while end-to-end attention applies recursive or cascaded attention across network abstraction levels.Interfaces include soft attention, hierarchical attention, pointer mechanisms, visual glimpses, and spatial transformation; RAM [46] and STN selectively process or transform relevant image regions.Source: §2 · p. 5, §2.1 · p. 6

2.2 Multimodality

• Multimodal attentional interfaces integrate misaligned sensory sources by selecting complementary information, beginning with CNN–LSTM image captioning and extending to video-text, image-text, audio-video, and other sensor combinations.Attention can condition source selection on decoder state or jointly model cross-modal, temporal, and hierarchical interactions.Source: §2.2 · p. 7

2.3 Attention-augmented memory

• Attention-augmented memory introduced addressable storage through Neural Turing Machines [86] and Memory Networks [38], enabling selective read/write operations and explicit representations of long-term history.Memory Networks treat memory capacity as a tunable hyperparameter and support question answering in which external memory acts as a dynamic knowledge base.Source: §2.3 · pp. 8–9

2.4 End-to-end attention models

• The Transformer [37] and Graph Attention Networks [97] established purely attentional, end-to-end architectures: Transformers remove recurrence and convolution for parallel sequence processing, while GATs focus on relevant graph structure and improve interpretability.Subsequent variants target convergence speed, memory cost, quadratic sequence scaling, and decoder parallelization, while extending Transformers to multimodal learning, transfer learning, and image generation.Source: §2.4 · pp. 10–11

2.5 Attention today

• Recent attention research combines Transformers, GATs, memory, temporal modeling, and hyperbolic geometry, while also exploring bidirectional bottom-up/top-down routing and adaptive computation time.Adaptive computation dynamically assigns more processing to complex inputs and less to straightforward ones; the timeline identifies ACT [126] and BRIMs [125] among these developments.Source: §2.5 · p. 11, Figure 6 · p. 12
• Soft attention forms the output as a weighted sum of intermediate outputs, while a halting unit determines whether computation should stop or continue.A time penalty in the cost function discourages processing data longer than necessary.Source: §2.5 · p. 12
• Spatially Adaptive Computation Time (SACT) [127] adjusts computation independently at each spatial position in convolutional layers, focusing on regions of interest and stopping when feature maps are “good enough”.Source: §2.5 · p. 12
• Differentiable Adaptive Computation Time (DACT) introduced the first differentiable end-to-end approach to computation time on recurring networks.Source: §2.5 · p. 12

Mechanism Categories

• The survey categorizes deep attention mechanisms into soft, hard, and self-attention, corresponding to global, local, and intra-attention.The mechanisms differ in whether they weight all inputs, select particular inputs, or model interactions within the input.Source: §3 · p. 12

Soft Attention

• Soft attention assigns continuous weights from 0 to 1 using softmax, operates spatially or temporally, and remains deterministic and differentiable despite high computational cost for large inputs.Spatial attention emphasizes relevant features, while temporal attention weights samples within sliding time windows according to their contributions.Source: §3 · p. 12

Hard Attention

• Hard attention makes binary 0-or-1 selections over input parts, requiring stochastic policies and reinforcement learning because the objective is non-differentiable.By avoiding storage or processing of the entire input, hard attention reduces inference time and computational costs compared with soft attention.Source: §3 · p. 12

Self-Attention

• Self-attention models interactions among input elements and uses parallelizable matrix calculations, providing an advantage for processing long inputs.It evaluates attention among all elements of the same input and can be applied to sentences and images.Source: §3 · p. 13, Figure 9 · p. 13

4.1 Attention-based Convolutional Neural Networks (CNNs)

• CNN attention is grouped into six uses: pooling, input filtering, inter-layer recalibration, prediction, residual attention, and feature-map output.These mechanisms select relevant stimuli, fuse information, support recurrent processing, and directly influence predictions.Source: §4.1 · pp. 13–14
• CNN input attention improves robustness by suppressing distractors and reducing sensitivity to background, lighting, scale, and viewpoint changes.It also supports scalable models, complex tasks, simpler encoding, and prioritized learning.Source: §4.1 · p. 15
• Attention between convolutional layers recalibrates features to capture long-range dependencies and unequal relevance across channels and frequencies.Squeeze-and-Excitation Networks [132] model channel interdependencies through squeeze and excitation operations, while residual attention can preserve relevant stimuli.Source: §4.1 · p. 15
• Attention-controlled sparsity reduced one-layer forward-pass runtime by more than 200%, with k selecting the retained convolutional responses.The filter preserves sparse feature maps and lowers memory consumption during training and inference.Source: §4.1 · pp. 15–16

4.2 Attention-based Recurrent Neural Networks (RNNs)

• RNN input attention uses hard or soft item- and location-wise selection, with soft attention providing differentiable, low-cost noise filtering and feature correlation learning.Hard selection is more common in computer vision, whereas soft mechanisms are often used in natural language processing and graph-to-sequence tasks.Source: §4.2 · p. 17
• RNN attention is organized into input, memory, and hidden attention, with hidden attention most prominent for alignment, contextual embedding, memory management, and long-term dependencies.RNNSearch [44] dynamically builds context from encoder and decoder states to reduce the encoder-decoder bottleneck.Source: §4.2 · pp. 17, 19

4.3 Attention-based Generative Models

• Generative-model attention is grouped into input masks, memory attention, and attentive encoder-decoder designs that augment memory and improve generalization.MONet [159] applies masked attention at the VAE input, while self-attentive autoencoders [164] can reuse encoder and decoder components without previous-step information.Source: §4.3 · pp. 20–21

5 Applications

• Attention networks span NLP, computer vision, multimodal tasks, recommender systems, reinforcement learning, robotics, and graph-related applications.Their use is attributed to versatility, interpretability, and significant results across diverse domains.Source: §5 · p. 22, Figure 16 · p. 22

5.1 Natural Language Processing (NLP)

• In NLP, attention aligns input-output sequences for MT and ASR, filters interfering speech, captures long-range dependencies, and links queries with relevant QA context.It also improves document embeddings and summarization by modeling contextual, semantic, and hierarchical information while addressing keywords, abstraction, repetition, and concise meaning preservation.Source: §5.1 · pp. 22–23

5.2 Computer Vision (CV)

• In computer vision, attention improves fine-grained recognition under occlusion and viewpoint changes, captures spatiotemporal action features, and enables detailed, high-resolution image generation.Self-attention complements convolution by modeling long-range dependencies, while recurrent and temporal mechanisms refine focus on informative joints, regions, or frames.Source: §5.2 · pp. 25–27

5.3 Multimodal Tasks (CV/NLP)

• In multimodal tasks, attention aligns visual and textual features for captioning, video captioning, and visual question answering while building richer cross-modal representations.Approaches include adaptive visual-language gating, bilinear higher-order interactions, co-attention over modality-specific graphs, and self-attention for global actions, objects, and linguistic descriptions.Source: §5.3 · pp. 28–29

5.4 Recommender Systems (RS)

• In recommender systems, attention weights interacted items to represent users’ changing short- and long-term interests more effectively than classical approaches.STAMP separates general and current interests through long-term and short-term memories, while Collaborative Filtering and SASRec [338] use self-attention for long-term semantics.Source: §5.4 · p. 30

5.5 Reinforcement Learning (RL)

• In reinforcement learning, attention addresses partial observability and long-term dependencies by retaining past observations, rewards, and actions in policy or state representations.These mechanisms aim to improve agent generalizability and reduce long-term memory problems in domains including visual navigation, graph reasoning, NLP, and virtual navigation.Source: §5.5 · pp. 30–31
• In reinforcement learning, policy-guided attention selects informative environmental glimpses, while recurrent memory and self-attention represent state history for image classification and model-free policies.RAM uses glimpse, core, action, and location networks trained with REINFORCE; related approaches modulate CNN features through attentional bottom-up and top-down connections.Source: §5.5 · p. 31

5.6 Robotics

• Robotic attention addresses difficult navigation and trajectory-prediction settings by modeling physical constraints, social interactions, relative importance, and long-horizon scene memory.Sophie predicts physically and socially feasible crowd trajectories; self-attention models human-robot and human-human interactions, while scene memory transformers retain observations as environments grow.Source: §5.6 · pp. 31–32
• In visual odometry, adaptive memory preserves global sequence information for refining poses, while self-attention reweights features, selects semantic regions, and replaces inaccurate recurrent temporal integration.These mechanisms target accumulated tracking errors, dynamic objects, non-textured surfaces, and the training cost of recurrent networks.Source: §5.6 · pp. 32–33

5.7 Interpretability

• Attention can identify representations influencing outputs, but attentional weights generally do not fully explain model decisions, and their explanatory validity varies by architecture and task.Higher weights often have greater decision impact, yet they may not identify the most relevant representations; recurrent encoders showed low correlation with gradient- or erasure-based importance measures.Source: §5.7 · pp. 33–34
• Interpretability depends on the explanation standard: attention may provide plausible insights, but faithful input-output explanations vary with dataset, architecture, and task.Weights can act as gates in single-sequence tasks, explain reasoning in pair-sequence tasks, and resemble human judgments, while alternative attention distributions can sometimes yield similar predictions.Source: §5.7 · p. 34

6 Trends and Opportunities

• Attention has produced profound advances mainly in computer vision and natural language processing, but many contributions and research opportunities remain.The survey frames this area as still early, with substantial space for new attentional mechanisms.Source: §6 · p. 34, §6.1 · p. 34

6.1 End-To-End Attention models

• End-to-end attention networks, including Transformers [37] and Graph Attention Networks [97], are expanding significantly across multiple application domains.Their successful use across tasks is identified as a major trend in recent deep learning research.Source: §6.1 · p. 34
• Transformer introduced a neural architecture whose core computation is fully attentional.However, its decoder remains autoregressive and it does not incorporate other cognitive mechanisms such as memory.Source: §6.1 · p. 34
• BERT [286], GPT2, and GPT3 [117] are described as the most advanced Transformer-based language models in NLP.These models exemplify the growing influence of end-to-end attention in natural language processing.Source: §6.1 · p. 34
• Image GPT has recently revolutionized unsupervised learning results in imaging.This extends the reported impact of end-to-end attention beyond language tasks into computer vision.Source: §6.1 · p. 34
• Sparse-attention Transformer models are being proposed to reduce computational complexity from quadratic to linear.Attention mechanisms are also being used to address multimodality in Graph Attention Networks.Source: §6.1 · p. 34
• End-to-end attention architectures may be highly influential in the future development of deep learning models.This outlook follows the continuing expansion of attention-based models and the field’s early-stage status.Source: §6.1 · p. 34

6.2 Learning Multimodality

• Attention helps models learn synergistic relationships among different sensory data, even when the signals are not synchronized.This has supported increasingly complex multimodal applications.Source: §6.2 · p. 35
• Multimodal attention has contributed to emotion recognition [75], feelings, and language-based image generation.Source: §6.2 · p. 35
• Most research relates pairs of sensory data, especially visual and textual inputs, while architectures that scale beyond one sensor pair remain underexplored.Multimodal learning using voice, RGBD images, monocular-camera images, diverse sensors, and biomedical sensors is still scarce.Source: §6.2 · p. 35

6.3 Cognitive Elements

• Attention enables neural networks to incorporate cognitive elements such as external memory and logic flow control, through approaches including Memory Networks [38] and Neural Turing Machines [86].These models use attention to update and recover information from external memory.Source: §6.3 · p. 35
• Neural Turing Machines [86] remain limited to simple algorithmic datasets because their convergence is slow and unstable.The survey calls for more stable and efficient models tested across broader application domains.Source: §6.3 · p. 35
• Attention-based memory research rarely manages complex hierarchical or multiple memory types simultaneously, including working, declarative, semantic, and long- and short-term memory.Dynamic Memory Networks [93] provide the most significant cited advances through episodic memory, while connecting external knowledge to memory and accelerating training remain open challenges.Source: §6.3 · p. 35
• The survey identifies imagination, reasoning, creativity, and consciousness as further human cognitive elements that should work in harmony with attentional structures.These capabilities are presented as major unresolved challenges for future neural-network architectures.Source: §6.3 · p. 35

6.4 Computer Vision

• Recurrent Attention Models (RAM) [46] introduced image computing with glimpses and hard attention, while Spatial Transformers (STN) learn plug-in image transformations.RAM is described as simple, scalable, and flexible, whereas STN modules can be integrated into different architectures.Source: §6.4 · p. 35
• RAM [46] could address tasks challenging for convolutional networks, especially large high-resolution images, but has so far been tested mainly on simple datasets.The survey recommends validation on complex classification and regression tasks and extensions involving memory, multimodal glimpses, and scaling.Source: §6.4 · p. 35
• Combining RAM [46] with STN could improve robustness to spatial transformations and support visual odometry, multiple-instance learning, and multiple viewpoints.The survey also proposes using STN to predict transformations between image sets.Source: §6.4 · p. 35

6.5 Capsule Neural Network

• Capsule networks have achieved strong results in image recognition and natural language processing, but few studies have implemented attention within them [553].Source: §6.5 · p. 35
• AR CapsNet, DA-CapsNet, SACN [556], and Tsai et al. [557] introduce attention through dynamic routing, dual attention, self-attention, or attentional capsule routing.AR CapsNet preserves spatial information during a fast forward pass; DA-CapsNet places attention after convolution and primary capsules, while SACN integrates self-attention as a layer.Source: §6.5 · p. 35
• Attention is viewed as important for improving the relational and hierarchical properties of capsule networks, especially through future dynamic and attentional routing mechanisms.The proposed directions include self-attention, soft attention, and hard attention in capsules.Source: §6.5 · p. 35

6.6 Neural-Symbolic Learning and Reasoning

• Neural-symbolic learning seeks to combine neural-network robustness with symbolic representations for complex reasoning, addressing a longstanding challenge identified by LeCun.Symbolic systems are recursive and declarative, whereas neural networks encode knowledge implicitly through adjusted weights.Source: §6.6 · p. 35
• Hybrid approaches including statistical relational learning and neural-symbolic learning [560] have re-emerged, with attention recently added to some models; Memory Networks [38] and Neural Turing Machines [86] were early reasoning systems.Research combining these paradigms grew again mainly during the past decade, but attention-integrated neural-symbolic work remains at an early stage.Source: §6.6 · p. 36
• Neural Logic Programming, Logic Attention Networks, and pGAT use attention to select operations or memory, aggregate graph information, and complete missing knowledge-graph relations.Despite advances, current systems mainly use attention for efficient memory management rather than identifying knowledge relevant to forming hypotheses and correct answers.Source: §6.6 · p. 36

6.7 Incremental Learning

• Incremental learning aims to add classes from limited data without retraining on the complete training set.Conventional classifiers are trained for a fixed class set.Source: §6.7 · p. 36
• Attention remains relatively unexplored in incremental learning despite its potential to advance the area.Source: §6.7 · p. 36
• Ren et al. introduced Attention Attractor Networks, which regularize new-class learning while preserving recognition of previously learned classes without revising the original training set.The method trains new weights episodically until they converge on the new classes.Source: §6.7 · p. 36

6.8 Credit Assignment Problem (CAP)

• CAP requires assigning more credit to actions producing higher final cumulative rewards, but measuring each action’s contribution to future rewards is difficult.This challenge has been studied in reinforcement learning for years.Source: §6.8 · p. 36
• Temporal CAP identifies useful or useless actions, structural CAP finds sensory situations yielding the same result for an action sequence, and transfer CAP generalizes action sequences across tasks.These are the three explored CAP variations described in the passage.Source: §6.8 · p. 36
• Ferret et al. [564] initiated attention-based CAP research by learning credit assignment through a separate supervised problem and transferring it to new environments.The passage characterizes this as a seminal first work in the area and argues that attention will be fundamental for advancing credit assignment research.Source: §6.8 · p. 36

6.9 Attention and Interpretability

• Attention weights are interpretable and correlate with feature-importance metrics when they are essential for prediction rather than reducible to a gating unit [551].Vashishth et al. investigated when attention is interpretable across multiple NLP tasks and experiments.Source: §6.9 · p. 36
• Jain and Wallace [550] found that attention can disagree with other explainability metrics and vary without significantly changing model predictions.They showed that distributions resembling those of a trained model can produce different results, challenging attention-based analysis.Source: §6.9 · p. 36
• Attention in neural models has also been reported to capture aspects of syntax and co-reference [547] [549] [548].These findings contribute to the broader evidence that attention may encode linguistically meaningful structure.Source: §6.9 · p. 36
• Systematic methodologies are still needed because existing conclusions rely on few architectures and a specific set of NLP applications.The survey identifies further research opportunities for evaluating attention as an interpretability tool.Source: §6.9 · p. 36

Generative models

• Attention improved unsupervised image generation through global perception, high-resolution synthesis, multimodal feature generation, and progressively detailed image features in GANs and related models.SAGAN [119] incorporated self-attention into convolutional GANs, Image Transformer surpassed ImageNet state-of-the-art in 2018, and AttGan used word-context and image regions to generate detailed features.Source: §6.10 · p. 37
• DRAW [36] explored sequential attention for refining image patches, while Bornschein et al. [163] used attentional memory addressing in VAEs, but attention remains little explored for simpler, scalable, stable generative models.DRAW [36] was tested only on simple datasets, and attention has received limited exploration in autoencoders.Source: §6.10 · p. 37

Natural Language Processing

• Unsupervised attention in NLP has extracted sentiment aspects and generated semantic phrase representations, yet most attention studies remain supervised and purely bottom-up systems remain unexplored.The survey identifies bottom-up attentional systems, particularly with inhibition and return mechanisms, as an open research direction for unsupervised learning.Source: §6.10 · p. 37

New Tasks and Robotics

• Attention is identified as a promising approach for larger time-series prediction, biomedical signals, medical diagnosis, segmentation, classification, and chronic-disease treatment support.The survey specifically proposes extending RAM [46] with multiple glimpses over different series segments or frequency ranges.Source: §6.11 · p. 37
• Robotics offers broad opportunities for attention-based deep learning in visual odometry, SLAM, and mapping, using architectures such as RAM [46], DRAW [36], and Transformer [37].Robotic systems must address multimodality, active and incremental learning, unknowns, uncertainty, semantics, reasoning, awareness, and planning while pursuing safer, more human-like behavior.Source: §6.11 · p. 37

• The survey connects attention mechanisms across CNNs, RNNs, and generative models, documenting gains in performance and computational efficiency, improved understanding of network knowledge, applications, interpretability, and future research directions.It critically analyzes more than 600 relevant papers and presents what the authors consider the broadest survey in the literature.Source: §7 · p. 37
• The review process collected more than 20,000 papers, selected approximately 650 for critical analysis and more than 6,000 for quantitative analyses, and provides its methodology and search-code links for future revisions.Automatic and manual filtering removed noise and used originality and citation counts among the selection criteria.Source: §7 · p. 38
• A complementary theoretical survey is being developed to create taxonomies grounded in psychology and neuroscience, clarifying attention’s roles across architectures and applications and guiding new attentional structures.The study is in its final development stages and aims to address limitations of existing taxonomies that lack theoretical concepts and broad extensibility.Source: §7 · p. 38

Systematic review methodology

• The survey follows a systematic-review process that defines questions, searches databases, applies selection criteria, critically evaluates studies, and synthesizes the relevant literature.The review is designed to be objective, transparent, meticulous, reproducible, and replicable; its workflow is summarized in Figure 23.Source: Appendix · p. 38, Figure 23 · p. 38

Scope and search strategy

• The review covers attention uses, mechanisms, applications, interpretability, trends, and challenges, using databases and sources searched across 01/01/2014–06/30/2019 and 07/01/2019–02/15/2021.Searches used the keywords attention, attentional, and attentive, with metadata stored in a JSON file after automated collection.Source: Appendix · pp. 38–39

Filtering strategy

• The selection pipeline removed duplicates and non-deep-learning papers, then applied three sequential filters based on general attention, deep-learning, and specific attention terms.The filtering strategy is depicted in Figure 24 and includes exclusion of papers focused on visual attention, saliency, and eye-tracking.Source: Appendix · pp. 39–40, Figure 24 · p. 39

Study selection and analysis

• 6,338 works remained for manual selection, 3,567 were stored in Zotero, 650 were judged relevant for critical analysis, and 6,567 supported quantitative analyses.Manual and citation-based screening excluded works lacking relevant titles, abstracts, introductions, or citation counts above the yearly group average.Source: Appendix · p. 40