Source-linked AI summary
Attention for Fine-Grained Categorization
Pierre Sermanet, Andrea Frome, Esteban Real
TL;DR
The paper addresses whether recurrent attention can support fine-grained categorization in cluttered, variable visual scenes. It applies an attention RNN with a pre-trained GoogLeNet visual core to Stanford Dogs, learning glimpses without bounding-box supervision. The model outperforms the full GoogLeNet baseline while using fewer input pixels, though additional glimpses provide only modest gains.
Problem
Attention models had largely been evaluated in toy or constrained visual environments, leaving fine-grained categorization in cluttered scenes as a less constrained test.
Method
The paper applies a recurrent attention model to Stanford Dogs, using a pre-trained GoogLeNet visual core, multi-resolution glimpses, and no bounding boxes.
Results
76.3% mA with one glimpse and 76.8% with three glimpses exceeded the full GoogLeNet model’s 75.5% mA.
Takeaways & Limitations
The model learned to focus high-resolution processing on discriminative features such as faces and fur without spatial supervision.
Takeaways & Limitations
With three resolutions, performance increased only slightly from 76.3% for one glimpse to 76.8% for three, limiting the benefit of sequential input.
Abstract
from arXiv · showhide
This paper presents experiments extending the work of Ba et al. (2014) on recurrent neural models for attention into less constrained visual environments, specifically fine-grained categorization on the Stanford Dogs data set. In this work we use an RNN of the same structure but substitute a more powerful visual network and perform large-scale pre-training of the visual network outside of the attention RNN. Most work in attention models to date focuses on tasks with toy or more constrained visual environments, whereas we present results for fine-grained categorization better than the state-of-the-art GoogLeNet classification model. We show that our model learns to direct high resolution attention to the most discriminative regions without any spatial supervision such as bounding boxes, and it is able to discriminate fine-grained dog breeds moderately well even when given only an initial low-resolution context image and narrow, inexpensive glimpses at faces and fur patterns. This and similar attention models have the major advantage of being trained end-to-end, as opposed to other current detection and recognition pipelines with hand-engineered components where information is lost. While our model is state-of-the-art, further work is needed to fully leverage the sequential input.
1 INTRODUCTION
The paper extends recurrent attention models to fine-grained dog-breed classification in cluttered, variable scenes, contrasting end-to-end attention with proposal-based pipelines. It aims to learn discriminative localization without bounding-box supervision.
- Unlike proposal-based detection and recognition pipelines, the model is trained end-to-end to incorporate information across observations.Proposal pipelines may lose information between candidate generation and classification or aggregate regions ad hoc.
- Fine-grained categorization on Stanford Dogs tests attention in scenes with clutter, occlusion, pose and lighting variation, and difficult class boundaries.
- The model avoids Stanford Dogs bounding boxes while simultaneously learning to localize and classify objects within scenes.
- Attention is useful because backgrounds can distract from sub-category distinctions such as German Shepherd versus Poodle.
2 MODEL DESCRIPTION
The model uses a recurrent attention architecture that starts from a low-resolution context image, selects multi-resolution glimpses, and produces classification scores. Its visual core is a fixed, pre-trained GoogLeNet-derived network, while recurrent components learn glimpse locations and classification.
- Model structure: The recurrent model processes N glimpses, using coordinates from prior steps to extract image patches and output either the next location or final classification scores.The full system accepts images of any size and uses a softmax classifier.
- Multi-resolution glimpses: Each glimpse concatenates resized low-, medium-, and high-resolution patches to emulate sharp central detail with lower-resolution peripheral context.The patches are resized to 96×96 before concatenation; experiments vary which resolutions are used.
- Context and attention: A randomly placed training context image, centered during inference, initializes the first glimpse location and influences later selections through recurrent connections.The context image is a low-resolution square patch resized to 96×96 and does not directly reach the classifier except through predicted coordinates.
- Context and attention: The attention network predicts glimpse coordinates using recurrent processing trained with backpropagation and policy-gradient updates.The coordinate-producing layers receive both recurrent backpropagation and policy-gradient learning signals.
- Model design differences: The system adapts patch sizes to input-image dimensions, using high-, medium-, and low-resolution side lengths of 1/4, 1/2, and the full shorter image dimension.This design addresses variation in image size while preserving relative object scale.
- Model design differences: Compared with the earlier model, it uses a vanilla RNN, linearly combines visual and location features, and replaces the glimpse visual network with a more powerful GoogLeNet-based core.The visual core is pre-trained and fine-tuned outside the attention RNN using multi-scale ImageNet patches and shared-parameter towers.
- Training and evaluation: Changing the first convolution stride increased three-resolution, three-glimpse attention accuracy from 68% to 76.8%.The improvement followed modifications made to accommodate small 96×96 glimpse inputs.
- Training and evaluation: During attention-RNN training, the pre-trained visual core remains fixed, and de-duplicated ImageNet data excludes Stanford Dogs test and validation images.Fine-tuning the visual core on Stanford Dogs training data did not improve performance.
3 EXPERIMENTAL RESULTS
On Stanford Dogs, the attention model classifies fine-grained breeds without bounding boxes and selects informative regions despite clutter, pose, lighting, and occlusion. Its three-resolution attention variant slightly surpasses full GoogLeNet, while additional glimpses provide limited gains unless each glimpse is information-restricted.
- Dataset and evaluation: 8,580 test images cover 120 dog breeds, with 100 training images per class and about 71 test images per class on average.
- Comparison with prior work: The model outperforms published results with medium- and low-resolution patches and matches the best published result using two small high-resolution patches, without bounding boxes.The cited prior results use ground-truth bounding boxes for training and testing.
- Learned attention: The high-resolution single-glimpse model often selects the dog’s face or fur, learning discriminative regions without spatial supervision despite its lowest performance among the tested variants.It can still choose the face under clutter, occlusion, and uncommon poses, although two dogs can cause an intermediate-location error.
- Comparison with GoogLeNet: 76.3% mA from the three-resolution, one-glimpse attention model exceeds full GoogLeNet’s 75.5% while using 73% of its pixels.Three glimpses increase performance only slightly to 76.8% but almost triple the number of input pixels.
- Ablation and interpretation: 70.3% versus 58.8% accuracy for low-resolution attention and GoogLeNet indicates that selecting an informative image region, rather than convolution stride, drives the performance difference.Both models share the same visual network and similar resolution inputs.
- Sequential input: With high-resolution-only glimpses, accuracy rises from 43.5% for one glimpse to 49.6% for three, but gains flatten quickly because the model has limited capacity to use more than two or three glimpses.The authors hypothesize that the RNN does not pass enough information from early glimpses to the classification layer.
A RANDOM VALIDATION SAMPLES
Figure 6 compares four attention-model variants on ten randomly selected validation samples, showing both classification outcomes and the locations of successive glimpses.
- Four model variants are compared using ten randomly selected validation samples, with glimpse locations and composite inputs shown for each example.
- Green borders mark correct classifications, while red borders mark errors.
- The examples show that attention can select informative dog regions even when classification fails.