Source-linked AI summary
Recurrent Models of Visual Attention
Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu
TL;DR
Large-image vision is computationally costly when processing scales with every pixel. This paper introduces a recurrent attention model that selects high-resolution regions sequentially, outperforming convolutional networks on cluttered images and learning dynamic visual tracking without explicit tracking supervision.
Problem
Convolutional vision models achieve strong recognition but incur high computational costs because processing scales with the number of image pixels.
Method
A recurrent neural network sequentially selects image or video locations, processes glimpse representations, and learns task-specific policies end-to-end with reinforcement learning.
Results
RAM learned effective task-specific visual strategies across image classification and dynamic control, reaching 7.1% error on cluttered translated MNIST and outperforming a convolutional baseline by 0.7%.
Takeaways & Limitations
The experiments support attention-based visual processing for handling clutter while controlling computation independently of input-image size.
Takeaways & Limitations
Learning policies over more than 25 discrete attention locations was difficult in experiments using discrete location representations.
Abstract
from arXiv · showhide
Applying convolutional neural networks to large images is computationally expensive because the amount of computation scales linearly with the number of image pixels. We present a novel recurrent neural network model that is capable of extracting information from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution. Like convolutional neural networks, the proposed model has a degree of translation invariance built-in, but the amount of computation it performs can be controlled independently of the input image size. While the model is non-differentiable, it can be trained using reinforcement learning methods to learn task-specific policies. We evaluate our model on several image classification tasks, where it significantly outperforms a convolutional neural network baseline on cluttered images, and on a dynamic visual control problem, where it learns to track a simple object without an explicit training signal for doing so.
1 Introduction
The paper introduces a recurrent visual-attention model that selectively processes image or video regions over time instead of entire scenes, reducing computation while building an internal representation. It frames attention as a task-specific control problem and demonstrates effective strategies for image classification and dynamic visual control.
- Motivation: Large convolutional neural networks achieve strong recognition accuracy but incur high computational costs during training and testing.Typical models can take days to train on multiple GPUs, even with downsampled inputs.
- Motivation: Human perception selectively focuses on visual regions and combines information across fixations, concentrating computational resources where they are needed.Selective processing saves visual “bandwidth” by reducing the number of pixels processed at high resolution.
- Approach: The model treats attention-based processing as a control problem applicable to static images, videos, and agents interacting with dynamic visual environments.The framework is intended for settings such as robots and computer game-playing agents.
- Approach: The recurrent neural network sequentially attends to image or video locations, using past information and task demands to select each next fixation.It incrementally combines information from different fixations into a dynamic internal representation of the scene or environment.
- Results: The model learns effective task-specific viewing strategies on several image-classification tasks and a dynamic visual-control problem.The results suggest attention-based processing may outperform convolutional networks in cluttered scenes and scale better to large input images.
2 Previous Work
Previous work reduces visual-computation costs through selective evaluation, prioritizes salient regions, or formulates vision as sequential decision-making. This paper is positioned as a more general deep-learning attentional framework using recurrent integration and end-to-end optimization.
- Computational limitations: Object-detection research reduces sliding-window costs by evaluating fewer windows, removing regions through branch and bound, or proposing candidate regions.These approaches target the computational burden of applying the full classifier across many image windows.
- Saliency-based attention: Saliency detectors prioritize potentially interesting regions using local low-level feature contrast and capture some properties of human eye movements.They typically do not integrate information across fixations, according to the passage.
- Sequential decision-making: Sequential-vision approaches gather image information over time and choose subsequent attention locations based on previous fixations.Related work includes object detection using a learned Bayesian observer model and policy-gradient learning.
- Deep-learning attentional frameworks: Compared with related deep-learning attention models, this work uses an RNN to integrate visual information over time and optimize sequential decisions end-to-end.Its formulation is described as more general and avoids relying on greedy action selection.
3 The Recurrent Attention Model (RAM)
RAM formulates visual attention as a partially observed sequential decision process in which a recurrent agent selectively senses local image regions, integrates information over time, and chooses both sensor placements and environment actions. Its stochastic policy is trained with reinforcement learning to maximize cumulative reward, using REINFORCE with a learned value-function baseline to reduce variance.
- Sensor and glimpse network: The glimpse sensor extracts a retina-like representation centered on the previous location, preserving high resolution near that location while reducing resolution farther away.This glimpse has much lower dimensionality than the original image.
- Recurrent architecture: The glimpse network converts each partial observation into a feature, and the recurrent core updates hidden state ht from that feature and the previous state ht−1.The hidden state summarizes past observations and supports decisions about future sensing and actions.
- Actions: At each step, RAM stochastically selects the next sensor location and environment action from distributions parameterized by networks operating on the recurrent state.Sensor deployment controls where information is acquired, while the environment action may affect the environment state.
- Sequential decision process: RAM treats attention as a POMDP: the agent receives bandwidth-limited observations, maintains an internal history-dependent state, and acts without observing the environment’s true state.The policy maps past interactions to a distribution over current sensor and environment actions.
- Policy learning: REINFORCE increases the log-probability of sampled actions followed by high cumulative reward and decreases it for low-reward actions.A learned value-function baseline compares obtained cumulative reward with expected reward to reduce gradient-estimate variance without changing its expectation.
4 Experiments
The experiments evaluate recurrent visual attention on image classification and dynamic control, using multi-scale glimpses, recurrent processing, and learned location policies. The model improves classification on cluttered and translated digits while controlling computation independently of image size, and learns ball tracking without explicit tracking supervision.
- Model design: The retina extracts k multi-scale patches centered at a location, resizes them to gw × gw, and concatenates them with normalized coordinate encodings.Each successive patch doubles the previous patch’s width; locations use real-valued (x, y) coordinates centered at (0, 0).
- Model design: The glimpse network combines rectified visual and location pathways into a 256-dimensional representation, while the location policy is a fixed-variance two-component Gaussian.The visual and location intermediate representations each have dimensionality 128, and the location-policy mean is produced from the recurrent state.
- Model design: Classification uses a rectifier recurrent core and makes its decision only at the final timestep, whereas the dynamic environment uses LSTM units.The attention model’s recurrent state has dimensionality 256, and training uses stochastic gradient descent with momentum 0.9.
- Classification experiments: 2.3% error was achieved by fully connected networks on centered digits, while RAM with 6 and 8 glimpses reached roughly 1.9% error and outperformed the convolutional network.RAM with 4 glimpses performed roughly as well as the convolutional network, while additional glimpses improved performance by focusing on the digit.
- Classification experiments: On cluttered translated MNIST, attention models were less affected by clutter and showed similar improvements over convolutional networks as image size increased from 60 × 60 to 100 × 100.The attention model’s capacity and computation stayed unchanged across these image sizes, unlike the convolutional network’s connected hidden layer.
- Dynamic visual control: 85% of catches was achieved by the best agent after training for 20 million frames, despite receiving no explicit instruction to track the ball.The recurrent attention model learned to play by tracking the ball near the bottom of the screen and was rewarded only for catching it.
5 Discussion
The paper introduces a unified recurrent visual attention model that selects where to focus next and generates control signals from glimpse inputs and internal state. Despite being non-differentiable, it is trained end-to-end from pixels to actions using policy gradients.
- The model is a single recurrent neural network that receives a glimpse window as input.
- Its internal state selects the next location to focus on and generates control signals in a dynamic environment.
- The unified architecture is trained end-to-end from pixel inputs to actions with a policy gradient method despite being non-differentiable.
Supplementary Material
Figures 4–6 illustrate the learned policy on the 60 × 60 cluttered-translated MNIST task through glimpse paths and six selected glimpses. The visualizations distinguish correct from false classifications and show full-resolution central regions surrounded by upscaled low-resolution areas.
- Policy visualizations: Figures 4–6 show learned glimpse paths overlaid on MNIST test images, with green indicating correct classifications and red indicating false classifications.The examples come from the 60 × 60 cluttered-translated MNIST task.
- Policy visualizations: Each example presents the six glimpses selected by the network.The glimpse sequence occupies columns 2–7 of each figure.
- Glimpse representations: The center of each glimpse contains a full-resolution view, while the outer areas are upscaled from low-resolution glimpses.This layout displays high-resolution information at the selected location together with broader low-resolution context.