Source-linked AI summary
Attention Mechanism in Neural Networks: Where it Comes and Where it Goes
Derya Soydaner
TL;DR
Attention mechanisms were inspired by visual attention and developed across neural-network tasks, but earlier encoder–decoder models struggled to retain all information from long source sentences. This survey reviews the field’s milestones through modern self-attention architectures, including the Transformer, and discusses advances and complexity challenges.
Problem
Encoder–decoder neural machine translation models compress source sentences into fixed-length vectors, making it difficult to capture all semantic details of very long sentences.
Method
The survey traces attention mechanisms from early visual-attention-inspired neural networks through recent self-attention mechanisms and Transformer variants across tasks.
Results
Attention mechanisms have produced state-of-the-art results across many tasks, while the Transformer achieved strong results in machine translation and constituency parsing without recurrence or convolution.
Takeaways & Limitations
The field has reached a stage where self-attention and GPT-n models motivate further exploration of advanced architectures and mechanisms beyond current attention designs.
Abstract
from arXiv · showhide
A long time ago in the machine learning literature, the idea of incorporating a mechanism inspired by the human visual system into neural networks was introduced. This idea is named the attention mechanism, and it has gone through a long development period. Today, many works have been devoted to this idea in a variety of tasks. Remarkable performance has recently been demonstrated. The goal of this paper is to provide an overview from the early work on searching for ways to implement attention idea with neural networks until the recent trends. This review emphasizes the important milestones during this progress regarding different tasks. By this way, this study aims to provide a road map for researchers to explore the current development and get inspired for novel approaches beyond the attention.
1 Introduction
Attention mechanisms draw on visual attention, where processing dynamically restricts the visual field and sequentially selects informative regions. This survey traces how that idea became useful across neural-network tasks and reviews its development toward modern architectures.
- 1 Introduction: Human visual recognition proceeds through successive fixations and saccades until the foveal information is sufficient for recognition.This sequential process is rapid enough to appear simultaneous.
- 1 Introduction: Visual attention restricts processing to a subset of the visual field while addressing what and where to look.The human eye combines high-resolution foveal processing with low-resolution peripheral information.
- 1 Introduction: Researchers adapted visual-attention ideas to neural networks because attention proved useful across tasks from image recognition to machine translation.The resulting mechanisms have been developed over many years and draw inspiration from psychology and neuroscience.
- 1 Introduction: The survey follows attention mechanisms from early neural-network integrations through recent trends, including Transformers that replace convolution and recurrence with attention layers.It also considers how the field might progress beyond current attention mechanisms.
2 From the Late 1980s to Early 2010s: The Attention Awakens
From the late 1980s through the early 2010s, researchers developed attention-based neural architectures for selective perception, recognition, detection, and tracking. These efforts explored biologically inspired models, selective processing, recurrent computation, and reinforcement learning across varied visual tasks.
- From the Late 1980s to Early 2010s: The Attention Awakens: The earliest neural-network attention studies, dating to the late 1980s, included selective-attention Neocognitron variants, VISIT, and SCAN.These models addressed cursive-character recognition and segmentation, brain-area relationships, and attentional scanning.
- From the Late 1980s to Early 2010s: The Attention Awakens: Early attention models targeted visual tasks including target detection, region-of-interest extraction, selective image recognition, and selective tuning.Some systems combined bottom-up and top-down image information or integrated results over time.
- From the Late 1980s to Early 2010s: The Attention Awakens: By the 2000s, attention architectures combined orienting and recognition pathways, modeled eye movements, and applied selective attention to pattern recognition.Reported applications included object detection, handwritten-digit recognition, and face recognition.
- From the Late 1980s to Early 2010s: The Attention Awakens: Around 2010, attention systems incorporated fixation controllers, gaze-driven tracking, and reinforcement-learning-based recurrent networks for image classification.These studies extended attention to fovea-inspired classification and simultaneous object tracking and recognition.
3 2015: The Rise of Attention
In 2015, attention mechanisms addressed the fixed-length bottleneck in encoder-decoder translation and extended neural networks across language and visual tasks. These models adaptively weight input representations, with hard and soft variants differing in how attention locations are selected and trained.
- Motivation: 2015 attention mechanisms addressed the difficulty of compressing long source sentences into a single fixed-length vector.The encoder-decoder bottleneck could prevent capturing all semantic details of very long sentences.
- Neural machine translation: The NMT extension encodes a source sentence as bidirectional annotations and adaptively selects relevant vectors during decoding.Each annotation summarizes preceding and following context, while the decoder uses selected source information to predict the target word.
- Neural machine translation: Attention weights are computed from alignment energies, and the context vector is formed as a weighted sum of encoder annotations.The alignment model is jointly trainable, and the weights express each annotation’s importance for the decoder state.
- Visual attention: In image captioning, convolutional image features provide location-specific annotations, while a learned mechanism computes the relevant context at each generation step.The context vector represents the relevant image region, with energies conditioned on the previous hidden state.
- Attention variants: Hard attention samples discrete locations and uses REINFORCE, whereas soft attention computes an expected context vector and supports standard backpropagation.Doubly stochastic attention adds a training constraint encouraging equal attention across image regions.
- Attention variants: Global NMT attention considers all source words, while local attention uses a differentiable window centered on an aligned source position.The two mechanisms derive context from all encoder states versus a selectively focused subset.
- Broader applications: After these 2015 mechanisms, attention was applied to image captioning and visual question answering, focusing on relevant parts of input images.The visual examples include object-focused captioning and word-, phrase-, and question-level co-attention maps.
4 2015-2016: Attack of the Attention
From 2015 to 2016, attention became embedded in new memory-oriented architectures and was applied across language, vision, speech, video, and reinforcement-learning tasks. Self-attention and related models expanded the range of neural architectures using selective information processing.
- Expansion across tasks: From 2015 onward, attention mechanisms supported new architectures across multiple tasks and application areas.Reported applications include image captioning, summarization, speech recognition, video captioning, translation, textual entailment, and visual question answering.
- Memory architectures: Neural Turing machines enabled end-to-end training without the supervision signal required by memory networks through content-based soft attention.End-to-end memory networks subsequently used recurrent attention.
- Self-attention: Self-attention was implemented in LSTMN by replacing the standard LSTM memory cell with a memory network.Memory networks use key and value vectors, unlike LSTMs’ hidden and memory vectors.
- New architectures: New attention-based architectures included SAN for image question answering and DARQN, which combines soft and hard attention within DQN.Other examples include WS-RAM for faster training and alignDRAW for generating images from captions.
5 The Transformer: Return of the Attention
The Transformer made self-attention the basis of an architecture without recurrence or convolution, extending attention across neural-network tasks while motivating research on variants and efficiency.
- The Transformer: The Transformer replaces recurrent encoder-decoder layers with self-attention and achieves strong results in machine translation and English constituency parsing.Its architecture contains neither recurrence nor convolution.
- The Transformer: Each Transformer stack contains six identical encoder layers and six identical decoder layers, using attention mechanisms, feedforward networks, and positional encodings.Positional encodings provide relative or absolute position information because the architecture lacks recurrence and convolution.
- Self-attention: Self-attention maps each word to query, key, and value vectors, then computes outputs as weighted sums of values based on query-key relationships.The vectors are produced using learned matrices Wq, Wk, and Wv.
- Attention mechanisms: Multi-head attention performs attention in parallel across multiple learned projections of queries, keys, and values, while decoder masking restricts prediction to previous words.The Transformer uses multi-head attention based on scaled dot-product calculations and applies masked attention first in the decoder.
- Applications: Self-attention has been applied successfully to sentence embedding, abstractive summarization, constituency parsing, speech recognition, and acoustic modeling.The surveyed literature also applies self-attention across multiple domains and tasks.
- Recent directions: Since 2017, research has focused on new self-attention mechanisms, adapting Transformers to varied tasks, improving efficiency, and making attention more understandable.Examples include relation-aware, directional, reinforced, lightweight, graph, and structured-text attention models.
- Complexity: Transformer attention has quadratic computation and memory requirements with sequence length, prompting linear or sparse alternatives such as Linear Transformer, Reformer, Routing Transformer, and BIGBIRD.Linear Transformer changes softmax attention to feature-map-based dot-product attention, while other approaches use hashing, clustering, or sparse attention.
6 Concluding Remarks: A New Hope
Attention mechanisms have developed from visual-system inspiration into a broad research area with state-of-the-art results across many tasks. Self-attention and GPT-family models mark a new stage, while important questions about future architectures remain open.
- A New Hope: Attention mechanisms have evolved over a long period from human visual-system inspiration to state-of-the-art results across a large variety of tasks.The paper reviews this development from its roots through current trends.
- A New Hope: Self-attention mechanisms and GPT-n family models have become a new hope for developing more advanced neural-network models.The paper presents this progress as an important stage in the field’s development.
- Open questions: Whether attention can support further development, replace popular neural-network layers, or improve on existing attention mechanisms remains an active research question.The conclusion states that much remains to be learned.