PaperlayerView exact revision on arXiv

Source-linked AI summary

An Attentive Survey of Attention Models

Sneha Chaudhari, Varun Mithal, Gungor Polatkan, Rohan Ramanath

arXiv:1904.02874v3cs.LGstat.ML

TL;DR

Attention models address information compression and alignment limitations in traditional encoder-decoder networks. This survey organizes attention techniques through a taxonomy, reviews architectures and applications, and concludes that attention improves performance, interpretability, and computational efficiency while retaining open explainability challenges.

  • Problem

    Traditional encoder-decoder networks may lose information by compressing long inputs into one fixed-length vector and cannot model input-output alignment.

  • Method

    The survey synthesizes attention formulations through a taxonomy and reviews attention-based neural architectures, application domains, and interpretability findings.

  • Results

    The survey reports that incorporating attention has produced significant gains in performance, greater insight into neural-network inner workings, and improved computational efficiency by eliminating sequential input processing.

  • Takeaways & Limitations

    Attention research spans diverse domains, and techniques developed in one area may be applied to other domains.

  • Takeaways & Limitations

    Attention weights remain contested as explanations because experiments found they are often not correlated with typical feature-importance analyses.

Abstract

from arXiv · show

Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy which groups existing techniques into coherent categories. We review salient neural architectures in which attention has been incorporated, and discuss applications in which modeling attention has shown a significant impact. We also describe how attention has been used to improve the interpretability of neural networks. Finally, we discuss some future research directions in attention. We hope this survey will provide a succinct introduction to attention models and guide practitioners while developing approaches for their applications.

1 INTRODUCTION

Attention models dynamically focus on relevant parts of inputs and have become widely used across neural-network applications. This survey organizes the field through a taxonomy, reviews architectures and applications, and examines interpretability.

  • 1 INTRODUCTION: Attention models selectively emphasize input elements that help perform a task, such as relevant sentences and sentiment-bearing words in reviews.In the Yelp example, the first and third sentences receive more attention, with “delicious” and “amazing” carrying greater meaning for sentiment classification.
  • 1 INTRODUCTION: Attention is used across natural language processing, speech, computer vision, cross-modal tasks, and recommender systems.Reported applications include translation, summarization, sentiment analysis, image classification, object detection, multimedia description, visual question answering, and recommendation.
  • 1 INTRODUCTION: Attention models can improve interpretability and address recurrent-network problems involving long inputs and sequential computation.The survey identifies interpretability and reduced sequential-processing inefficiency as benefits beyond task performance.
  • 1 INTRODUCTION: The survey covers attention intuitions and functions, a taxonomy, neural architectures, applications, interpretability, and future research directions.Its organization proceeds from basic intuition through attention models, taxonomy, architectures, applications, and interpretability.

2 ATTENTION BASICS

Attention basics can be introduced through a regression estimator that predicts a query’s target from training instances. Unlike a naive average, the estimator weights training targets by their relevance to the query.

  • 2 ATTENTION BASICS: The regression setup contains n training instances with features and targets, and predicts the target value for a new query instance.The training data are {(x_1,y_1), (x_2,y_2), ..., (x_n,y_n)}, while the query is x.
  • 2 ATTENTION BASICS: A naive estimator predicts the simple average of the training targets.This provides the unweighted baseline for understanding attention-style estimation.
  • 2 ATTENTION BASICS: The attention-style estimator instead uses a weighted average whose weights represent the relevance of each training instance to the query.The weighting function α(x,x_i) encodes the relevance of instance x_i for predicting x.
  • 2 ATTENTION BASICS: A normalized Gaussian kernel is a common weighting function, although other normalized similarity measures can also be used.The regression estimator is described as consistent with sufficient data and simple because it has no free parameters.

3 ATTENTION MODEL

Attention models address fixed-vector compression and missing input-output alignment in traditional encoder-decoder systems by dynamically weighting relevant encoded states. The section explains their context-vector mechanism and the alignment and distribution functions used to compute attention weights.

  • Traditional encoder-decoder models compress the entire input into one fixed-length vector, risking information loss for long sequences and lacking selective input-output alignment.
  • Attention lets the decoder access the entire encoded input sequence while prioritizing positions relevant to generating each output token.
  • At each decoding step, the context vector combines encoder hidden states using learned attention weights rather than using only the final encoder state.
  • An alignment function scores the relevance of each encoder state to the decoder state, and a distribution function converts scores into attention weights.
  • Alignment functions: Alignment functions compare or jointly represent keys and queries, with specialized variants including location-based alignment for position-dependent scoring.
  • Distribution functions: Sigmoid and softmax distributions constrain attention weights to [0,1] and sum to 1, while sparse alternatives assign nonzero probability to fewer plausible outputs.

4 TAXONOMY OF ATTENTION

The survey organizes attention models into four non-exclusive categories, covering input relationships, abstraction levels, positions, and representations. These dimensions can be combined to characterize attention techniques across applications.

  • Four broad, non-mutually-exclusive categories organize attention models as dimensions for analyzing application-specific designs.The survey also summarizes technical papers by the multiple attention types they use.
  • Input relationships: Distinctive attention connects keys and queries from separate input and output sequences, whereas self attention relates tokens within the same sequence.Distinctive attention is common in translation, image captioning, and speech recognition; self attention supports tasks such as classification and recommendation.
  • Input relationships and abstraction levels: Co-attention jointly learns attention over multiple input sequences, while Lu et al.’s visual question answering model combines visual-text co-attention with word-, phrase-, and question-level attention.This combines attention over the image and question at three textual abstraction levels.
  • Number of abstraction levels: Multi-level attention applies attention sequentially across abstraction levels, using lower-level context vectors as queries for higher levels.The hierarchical attention model builds sentence representations from words, then aggregates sentences into a document representation for classification.
  • Number of abstraction levels: Stacked Attention Networks iteratively refine attention using query information and previous attention results to locate increasingly fine-grained image regions.In image question answering, two layers outperformed one, while three or more layers did not further improve performance.
  • Number of positions: Soft attention averages all hidden states, hard attention samples states stochastically, and local attention attends within a window around a selected position.Local attention provides a trade-off between soft and hard attention, combining efficiency and differentiability within the window.
  • Number of representations: Multi-representational attention weights alternative input representations, while multi-dimensional attention weights individual embedding dimensions according to contextual relevance.These approaches can select preferred embeddings or features for downstream tasks and help address polysemy in word embeddings.

5 NETWORK ARCHITECTURES WITH ATTENTION

The survey reviews four attention-enabled architectures: encoder–decoder models, Transformers, memory networks, and graph attention networks. These architectures use attention to align or select information, remove sequential constraints, access external memory, or model graph structure.

  • The survey covers encoder–decoder models, Transformers, memory networks, and graph attention networks as salient attention-enabled architectures.It presents these architectures as popular choices across application domains while noting that the set continues to grow.
  • Encoder-Decoder Framework: Encoder–decoder attention compresses variable input representations into a fixed-length context vector, decoupling input representation from output decoding.This formulation was introduced for long input sentences in recurrent encoder–decoder systems.
  • Encoder-Decoder Framework: Pointer networks produce discrete outputs that point to input positions, with target classes varying according to input length.They address sequential input-output problems that the standard encoder–decoder formulation cannot solve, including sorting and travelling-salesman problems.
  • Transformer: Transformers eliminate recurrent connections and sequential processing, using self-attention to capture global dependencies and multi-head attention to process projections in parallel.The architecture stacks six encoder and decoder layers containing attention and feed-forward sublayers.
  • Transformer: Transformers support parallel processing, long-range dependencies, scalability, minimal inductive bias, and domain-agnostic processing across text, images, and speech.The survey describes Transformer as a significant milestone whose variants are widely used across NLP, vision, and cross-modal tasks.
  • Transformer: Transformer limitations include context fragmentation, high computational cost, large data requirements, and difficulty interpreting self-attention contributions.These limitations have motivated multiple lines of work, including longer or adaptive attention spans and analyses of attention-head roles.
  • Memory Networks: Memory networks use attention to access only relevant facts in external memory, while Dynamic Memory Networks iteratively revisit inputs conditioned on the question and prior memory.Memory networks have reported superior performance to RNNs and LSTMs on question answering and language modeling, while DMNs have reported state-of-the-art sentiment analysis and part-of-speech tagging results.
  • Graph Attention Networks (GAT): Graph attention networks apply attention over graph neighborhoods, and hierarchical attention combines node-level neighbor weighting with semantic-level meta-path weighting in heterogeneous graphs.The shared edge-wise attention mechanism does not require upfront access to the global graph structure.

6 APPLICATIONS

Attention models are applied across NLP, computer vision, multimodal tasks, recommender systems, and graphical systems. They focus computation on relevant inputs, align modalities or sequences, capture long-range dependencies, and support richer representations.

  • The survey organizes attention applications into NLP, computer vision, multimodal tasks, recommender systems, and graphical systems.It presents selected seminal applications as a starting point for further investigation.
  • Natural Language Processing (NLP): In NLP, attention focuses on relevant sequence parts, aligns input and output sequences, and captures long-range dependencies.Machine translation benefits from attention through improved alignment, especially for longer sentences.
  • Natural Language Processing (NLP): Attention has produced performance improvements in machine translation and supports question answering, abstractive summarization, sentiment analysis, text classification, and text representation learning.Different NLP tasks use attention to identify relevant words, store facts, generate summaries, or construct sentence and document representations.
  • Natural Language Processing (NLP): Pre-trained Transformer language models learn broad language representations, require less labeled data for downstream fine-tuning, and simplify NLP model development.The survey discusses models including Transformer-XL, BART, BERT, ALBERT, XLNet, GPT, GPT-2, and GPT-3.
  • Computer Vision (CV): Visual attention selects relevant image regions and captures structural long-range dependencies for image classification and object detection.Sequential glimpses can process one recognized object at a time in multiple-object detection.
  • Computer Vision (CV): Vision Transformers apply Transformer processing to image patches, while DETR generates object classes and bounding boxes and removes hand-designed components such as anchors and non-maximum suppression.Deformable attention reduced DETR training time by 10X in the cited work.
  • Multi-Modal Tasks: Attention supports multimodal applications by identifying relevant image regions or video frames for generating natural-language descriptions.Its role is to model relationships between different modalities.

7 ATTENTION FOR INTERPRETABILITY

The survey presents attention as a way to inspect which inputs influence neural predictions, while emphasizing that attention weights are not yet reliable explanations.

  • Interpretability: Attention weights can be visualized to inspect the input regions associated with particular outputs.The survey describes this as a direct way to examine internal model behavior.
  • Interpretability: Examples show attention identifying relevant words, user interests, image regions, and cross-language alignments.Reported cases include summarization, recommendation, image captioning, and French–English translation.
  • Interpretability: Attention identifies non-monotonic alignments such as “environnement marin” with “marine environment.”This demonstrates alignment despite differing subject–verb–noun positions across languages.
  • Interpretability: Attention can expose context dependence and potential bias, including gendered words receiving higher attention in occupation classification.The sentiment example shows that the importance of “good” and “bad” varies with context.
  • Limitations: Attention weights should not automatically be treated as explanations because they may be weakly related to feature importance and prediction sensitivity.Experiments found that permutations or adversarial changes to attention weights sometimes leave predictions unchanged.

8 CONCLUSION

The survey organizes attention research through a taxonomy, architectures, applications, interpretability, and future directions. Its conclusion highlights performance, interpretability, computational efficiency, and online attention as continuing priorities.

  • Conclusion: The survey synthesizes attention formulations through a taxonomy, neural architectures, application domains, and emerging research directions.It aims to provide an overview of techniques and how developments may transfer across domains.
  • Conclusion: Attention is associated with performance gains, greater insight into neural-network operation, and improved computational efficiency through eliminating sequential input processing.These are the survey’s stated overall conclusions about incorporating attention.
  • Future Directions: Online attention supports translation before the entire source sentence has been read, addressing real-time applications such as live captions.Monotonic chunkwise and monotonic multi-head attention are described as enabling online decoding.

8.2 Stand-alone Attention

The survey examines whether attention can serve as a stand-alone vision primitive rather than merely augmenting convolutional networks.

  • Stand-alone Attention: Pure self-attention vision models replace spatial convolutions with local-region self-attention and can compete with state-of-the-art models on benchmark datasets.The cited work investigates stand-alone self-attention as an alternative architectural primitive.
  • Stand-alone Attention: Factorizing 2D self-attention into two 1D attentions enables attention over larger or global regions.This design expands the spatial region over which attention can be performed.

8.3 Model Distillation

The survey describes model distillation as a response to the latency costs of large attention-based models and discusses improved attention interpretability as a related research direction.

  • Model Distillation: Recommender and search systems face strict online-serving latency constraints, limiting the applicability of large models such as BERT.BERT is described as having hundreds of millions of parameters.
  • Model Distillation: Model distillation compresses a large complex model into a simpler model while retaining its accuracy.The passage introduces distillation as the relevant approach for online serving.
  • Attention for Interpretability: Diversity-driven LSTM cells produce attention weights that better rank important hidden states and correlate better with gradient-based attribution.The approach uses orthogonalization to make hidden states farther apart in their spatial dimensions.

8.5 Auto-learning Attention

Auto-learning attention applies neural architecture search to discover high-order attention modules beyond the backbone architecture. The approach introduces Higher Order Group Attention and uses differentiable search to identify an effective module efficiently.

  • Neural architecture search extends beyond backbone design to search for optimal plug-and-play attention modules.This addresses the open question of automatically designing high-order attention modules.
  • Higher Order Group Attention represents high-order attentions within a searchable attention-module design.
  • A differentiable search method efficiently searches for the optimal attention module.

8.6 Multi-instance Attention

Multi-instance attention extends attention from individual memory items to structurally adjacent groups with dynamic shapes and sizes. Attention also supports modeling interactions and group structure in multi-agent systems.

  • Multi-instance Attention: Multi-instance attention attends to structurally adjacent groups rather than individual words or image pixels.Examples include 2D image areas and natural-language subsequences.
  • Multi-instance Attention: Area attention can represent a group using the mean of its item keys or richer derived features such as within-area standard deviation.
  • Multi-instance Attention: Group-based attention formulations support attention over groups with dynamic shapes and sizes.
  • Multi-agent Systems: Attention in deep generative models captures behavior generation, agent groups, and interactions within multi-agent systems.Applications include autonomous vehicles and multiplayer games.

8.8 Scalability

Transformer self-attention becomes prohibitively costly for long sequences because its time and space complexity grows quadratically with sequence length. A central scalability goal is reducing this complexity to linear.

  • O(n^2) time and space make standard Transformer self-attention costly for long sequences.The passage identifies bioinformatics as an example setting involving long sequences.
  • Reducing Transformer complexity from quadratic to linear is an important research theme.
Loading 1904.02874v3…