Survey on the attention based RNN model and its applications in computer vision

1.1 What does ”attention” mean?

• Psychological attention selects which input part to focus on and allocates limited processing resources; the survey relates this idea to selective data access in computing.Its CPU-cache analogy describes a hierarchy of caches, main memory, and storage for allocating computing resources.Source: §1.1 · p. 3

1.2 What are the applications of the attention based model in computer vision?

• Computer-vision attention includes local image features, saliency detection, and sliding windows, whose task specificity or independent detections limit broader use.The survey notes that many saliency methods use low-level cues and do not capture task-specific semantics or sequentially linked attention.Source: §1.2 · pp. 4–5

1.3 What is the attention based RNN model?

• Here, an attention-based RNN means an end-to-end trainable sequence-to-sequence RNN with an attention mechanism learned alongside the RNN.The survey treats attention as a methodology rather than a strict mathematical definition.Source: §1.3 · p. 5

1.3.1 Recurrent neural network (RNN)

• RNNs process sequential data of variable sizes by holding a hidden state, unlike CNNs described here as accepting fixed-length inputs and outputs.The survey positions RNNs as useful for sequences that CNNs cannot handle in this setup.Source: §1.3.1 · p. 5

1.3.2.1. Sequence to sequence problem

• The survey defines sequence-to-sequence learning as mapping an input sequence to an output sequence, while allowing varied input forms and focusing on explicit output items.Input may be an item sequence or a feature map; an output sequence may contain only one item.Source: §1.3.2.1 · pp. 5–6

1.3.2.2. The structure of the RNN model for sequence to sequence problem

• A common RNN uses an encoder to compress input information into an intermediate code, which a decoder uses to generate the output sequence.Input and output lengths can vary independently, and a feature-map input may use a non-recurrent encoder.Source: §1.3.2.2 · pp. 6–7

1.3.3 Attention based RNN model

• Because a fixed-length intermediate code cannot explicitly weight input items differently for each output, attention lets the model focus on needed inputs during decoding.The survey presents this addressing ability as a practical response to fixed-code compression limits when relevant input items can be located.Source: §1.3.3 · pp. 8–9

• The survey introduces item-wise soft, item-wise hard, location-wise hard, and location-wise soft attention.These four mechanisms are introduced at a high level before their mathematical descriptions.Source: §1.4 · p. 9
• Item-wise attention weights codes for explicit input items, combining them softly or stochastically selecting codes for the decoder.Hard selection usually picks one code according to the weights; soft attention forms a linear combination.Source: §1.4 · p. 9
• Location-wise attention operates on a feature map, either selecting a sub-region discretely or transforming the whole map to highlight relevant parts.The location is calculated by the attention module in the hard variant.Source: §1.4 · p. 9
• Soft attention is differentiable with respect to inputs, whereas hard attention makes discrete selections and uses reinforcement-learning techniques for learning.Both attention modules are learned with the raw RNN in an end-to-end model.Source: §1.4 · p. 9

• Attention-based RNNs assign different weights to input parts, potentially exposing input-output relations and providing a visualization tool.The survey also describes this weighting as a way to boost performance on some tasks.Source: §1.5 · p. 10
• Hard attention can process only selected input items, making it useful for partially observable environments such as game playing.Source: §1.5 · p. 10
• The survey notes applications beyond vision, including machine translation [6, 37], machine comprehension, sentence summarization, word representation, speech recognition [8], and game play.Source: §1.5 · p. 10

• The survey is organized into attention-model mathematics, computer-vision applications, and a closing discussion of insights and future research directions.Source: §1.6 · p. 11

• The section first introduces neural networks and then gives an abstract mathematical description of the RNN.Source: §2 · p. 11
• It analyzes four attention mechanisms on RNN models in detail.Source: §2 · p. 11

• A neuron reweights and sums its inputs, then passes the sum through an activation function; its computation includes a bias parameter.Source: §2.1 · p. 11
• Activation functions are usually nonlinear, with sigmoid, tanh, ReLU, and PReLU given as examples.Source: §2.1 · p. 11
• Neural networks connect neurons, commonly layer by layer, with each layer's outputs connected to the next layer.Source: §2.1 · p. 11, Figure 6 · p. 12
• CNNs use convolutional layers that accept regions of 2D feature maps, making them suitable for images; pretrained deep CNNs can transfer to other tasks.Source: §2.1 · pp. 12–13

• At each step, an RNN uses the current input and previous hidden state to produce an output and new hidden state, which serves as memory of prior inputs.Source: §2.2 · p. 13

2.3 The model for sequence to sequence problem

• Input and output sequences need not have the same length, and the encoder-decoder system models their mapping.Source: §2.3 · p. 13

2.3.1 Encoder

• A common RNN encoder compresses all input items into one code without modeling their correspondences to output items; the encoder may be CNN-based for image input or recurrent for sequences.Source: §2.3.1 · pp. 13–14

2.3.2 Decoder

• The decoder is recurrent in the surveyed setup, using the code, its previous hidden state, and its previous prediction to generate each output and updated state.Source: §2.3.2 · p. 14

2.3.3 Learning

• Training maximizes the sum of output-item log-likelihoods by gradient ascent on the model parameters.Source: §2.3.3 · pp. 14–15

2.4.1 Item-wise soft attention

• Item-wise soft attention encodes each input item separately, then weights the resulting codes using the decoder’s previous hidden state to form the intermediate code.The normalized weights are interpreted as relevance probabilities, and the code is their weighted expectation.Source: §2.4.1 · pp. 15–16

2.4 Attention based RNN model

• The four mechanisms cover soft or hard attention over explicit input items and soft or hard selection of regions from feature maps.Hard location-wise attention can use multiple glimpses, while STN-based soft attention transforms the feature map before encoding.Source: §2.4.1 · p. 15, §2.4.2 · p. 16, §2.4.3 · p. 16

2.4.2 Item-wise hard attention

• Item-wise hard attention samples one input code according to its relevance probabilities instead of combining all codes.When there are two codes, the categorical selection distribution becomes Bernoulli.Source: §2.4.2 · p. 16

2.4.3 Location-wise hard attention

• Location-wise hard attention samples a feature-map sub-region at each decoding step and encodes that region into the intermediate code.A single region may not match the output being predicted; multiple sequential glimpses can address this, but their number may be fixed or model-selected. One glimpse is faster, while multiple glimpses may perform better but take longer; choosing the number remains open.Source: §2.4.3 · pp. 16–18

2.4.4 Location-wise soft attention

• Location-wise soft attention uses a spatial transformer network (STN) to transform the input feature map into an output map used to calculate the decoder’s intermediate code.At each step, the localization network uses the input map and previous decoder state to generate transformation parameters; the transformed sampling positions are interpolated to produce output values. Affine transformations can provide translation, rotation, scale, skew, and cropping.Source: §2.4.4 · pp. 19, 20, 22, 23

2.5.1 Soft attention

• Soft attention trains with gradient ascent on the log-likelihood when its attention components are differentiable with respect to their parameters.For STNs, differentiability depends on the location network, transformation, and sampling kernel; the survey gives affine transformation and bilinear interpolation as an example.Source: §2.5 · p. 23, §2.5.1 · pp. 23–24

2.5.2 Hard attention

• Hard attention’s discrete selections have zero gradients, so the survey uses reinforcement-learning techniques to train the attention-based RNN.The item-wise and location-wise hard-attention learning methods are essentially the same; item-wise hard attention implicitly uses one glimpse.Source: §2.5 · p. 23, §2.5.2 · p. 24
• The hard-attention objective is a variational lower bound on the sequence log-likelihood, and its gradient is estimated by Monte Carlo sampling.The reward may equal the output log-likelihood, be proportional to it, or be a zero/one value.Source: §2.5.2 · pp. 24–25
• The unbiased hard-attention gradient estimate may have high variance, so the survey describes adding a variance-reduction baseline and a weight to balance gradient components.A common baseline is the expected reward; the text says variance reduction makes training faster and more robust.Source: §2.5.2 · p. 26

2.6 Summary

• The survey covers neural networks, CNNs, RNNs, sequence-to-sequence models, four attention mechanisms, and their optimization.Source: §2.6 · p. 26

3.1 Overview

• Location-wise attention accepts feature maps directly, while item-wise attention requires explicit input items and codes for every item.Soft attention is differentiable and trained with gradient ascent; hard attention is non-differentiable and typically uses reinforcement learning, while STN combines location-wise input with direct log-likelihood optimization.Source: §3.1 · p. 27

3.2 Image classification and object detection

• For classification, extracts an image patch at an attention-generated location, encodes it, and uses the decoder to predict a label.The extension in adds a separate attention network and a whole-image context vector to generate the first glimpse; the original model was shallow, manually set that glimpse, and was tested only on toy datasets.Source: §3.2 · p. 28, Figure 12 · p. 29
• On MNIST pairs, outperformed, and the context vector improved first-glimpse accuracy.The dataset combines two MNIST digits in a noisy 100×100 image.Source: §3.2 · p. 29, Table 1 · p. 29
• On SVHN sequence recognition, used fewer parameters and less training time than state-of-the-art CNN methods at the same error rates.Source: §3.2 · p. 29
• On Stanford Dogs, the attention-based RNN retained competitive performance on real-world fine-grained images with clutter, occlusion, and pose variation.The model used a pre-trained GoogLeNet encoder.Source: §3.2 · p. 30, Table 2 · p. 31

3.3 Image caption

• NIC encodes the whole image into one code for its LSTM decoder, whereas applies item-wise attention over CNN feature-map patches at each decoding step.The survey also identifies location-wise hard attention and STN as options when extracting image items is not straightforward.Source: §3.3 · pp. 31, 32, 34
• In the MS COCO Challenge, the attention model performed worse than NIC on both human and automatic evaluations.The survey suggests the evaluation gap may reflect imperfect automatic metrics and captions that humans find more natural.Source: §3.3 · p. 33, Table 3 · p. 33
• Figure 16 illustrates word-region alignment, such as “people” with people and “water” with the lake; object-proposal inputs lack a direct performance comparison with.Source: §3.3 · p. 33, Figure 16 · p. 34

Model trade-offs

• Item-wise attention requires explicit input items and slows training without improving test efficiency over no-attention RNNs; location-wise attention improves efficiency over item-wise attention.Source: §4 · p. 35
• Soft attention is differentiable and uses standard gradient optimization, whereas non-differentiable hard attention is optimized with reinforcement-learning techniques.Source: §4 · p. 35

Applications

• Image classification and object detection use location-wise hard attention, while image captioning uses item-wise soft and hard attention.Source: §4 · p. 35

Conclusion

• Attention-based RNNs are better than, or at least comparable to, models without attention in performance.Source: §4 · p. 35
• Attention makes visualization more intuitive; captioning examples show “people” aligned with people and “water” with a lake.Source: §4 · p. 35

Future research

• Future work includes scaling to larger datasets, comparing mechanisms across tasks, and converting explicit-item inputs into feature maps for location-wise attention.Source: §4 · p. 35
• Hard attention's approximate log-likelihood optimization sacrifices performance, and improving its learning method remains an open question.Source: §4 · p. 36
• Proposed extensions include parallel attention modules and cache-like RNN architectures; the survey excludes STN applications because few STN-based RNN applications exist.Source: §4 · pp. 35–36