PaperlayerView exact revision on arXiv

Source-linked AI summary

Diversified Visual Attention Networks for Fine-Grained Object Classification

Bo Zhao, Xiao Wu, Jiashi Feng, Qiang Peng, Shuicheng Yan

arXiv:1606.08572v2cs.CV

TL;DR

Fine-grained classification must distinguish subtle differences amid large appearance variation, while existing attention models may repeatedly focus on similar regions. DVAN uses diversified recurrent attention over multiple canvases and reports competitive performance across three benchmark datasets without bounding-box or part-location information.

  • Problem

    Fine-grained classification is challenged by subtle inter-class differences, large intra-class variation, and attention models that may fail to gather multiple discriminative regions.

    Source: §I · p. 2, C. Diversity Promoting Attention Models · pp. 4–5

  • Method

    DVAN generates canvases at different locations and scales, uses LSTM for recurrent attentive feature integration, and explicitly diversifies attention maps.

    Source: §I · p. 2, 1) Attention Feature Integration · p. 4, 4) Classification · p. 4

  • Results

    DVAN achieves competitive performance with state-of-the-art approaches on CUB-2011, Stanford Dogs, and Stanford Cars without bounding-box or part-location information.

    Source: §V · p. 10

  • Takeaways & Limitations

    A coarse global view and diversified fine-grained local parts are incrementally combined to capture discriminative information for classification.

    Source: §I · p. 2

Abstract

from arXiv · show

Fine-grained object classification is a challenging task due to the subtle inter-class difference and large intra-class variation. Recently, visual attention models have been applied to automatically localize the discriminative regions of an image for better capturing critical difference and demonstrated promising performance. However, without consideration of the diversity in attention process, most of existing attention models perform poorly in classifying fine-grained objects. In this paper, we propose a diversified visual attention network (DVAN) to address the problems of fine-grained object classification, which substan- tially relieves the dependency on strongly-supervised information for learning to localize discriminative regions compared with attentionless models. More importantly, DVAN explicitly pursues the diversity of attention and is able to gather discriminative information to the maximal extent. Multiple attention canvases are generated to extract convolutional features for attention. An LSTM recurrent unit is employed to learn the attentiveness and discrimination of attention canvases. The proposed DVAN has the ability to attend the object from coarse to fine granularity, and a dynamic internal representation for classification is built up by incrementally combining the information from different locations and scales of the image. Extensive experiments con- ducted on CUB-2011, Stanford Dogs and Stanford Cars datasets have demonstrated that the proposed diversified visual attention networks achieve competitive performance compared to the state- of-the-art approaches, without using any prior knowledge, user interaction or external resource in training or testing.

I. INTRODUCTION

Fine-grained classification is difficult because visually similar categories differ subtly while instances within a category vary substantially. DVAN addresses this challenge by diversifying recurrent visual attention across multiple image locations and scales to build an incremental representation.

Challenge

  • Fine-grained classification distinguishes visually and semantically similar subordinate categories, including bird species, dog breeds, and car classes.Its applications include multimedia information retrieval and content analysis.Source: §I · p. 1
  • Subtle inter-class differences and large intra-class variation arise from scale, viewpoint, background, and occlusion changes.These differences can be difficult even for humans to recognize.Source: §I · p. 1

Motivation

  • Existing approaches exploit discriminative local parts, but visual attention models struggle to find multiple discriminative regions simultaneously.The paper motivates diversity because fine-grained differences often occur in small object regions.Source: §I · pp. 1–2

Approach

  • DVAN uses LSTM to find multiple attentive regions over time and generates attention canvases at different locations and scales.Some canvases cover the whole object, while others contain local parts.Source: §I · p. 2
  • DVAN combines a coarse-grained global view with fine-grained diversified local parts to dynamically build an incremental object representation.This representation is designed to capture both the general picture and local details.Source: §I · p. 2

Results

  • Experiments on three benchmark datasets show competitive performance with state-of-the-art methods without prior knowledge or user interaction.The paper identifies CUB-2011, Stanford Dogs, and Stanford Cars as evaluation datasets.Source: §I · p. 2

II. RELATED WORK

Related work approaches fine-grained classification through deep representations, part alignment or localization, and diversified visual-attention frameworks. These methods differ in how they identify and combine discriminative object regions.

Overview

  • Fine-grained classification methods are organized into four groups according to their use of additional information or human interaction.The supplied related-work passages include representation, alignment, part-localization, and human-in-the-loop approaches.Source: A. Fine-Grained Image Classification · p. 2

Discriminative Representation

  • Deep representation methods use convolutional features, while bilinear architectures model local pairwise feature interactions with two independent subnetworks.These approaches build discriminative representations rather than relying on handcrafted features.Source: 1) Discriminative Feature Representation · p. 2

Alignment

  • Alignment approaches discover or segment common object-part patterns, align images, and extract features from corresponding regions.The cited examples include unsupervised template learning and alignment categorization.Source: 2) Alignment Approaches · p. 2

Part Localization

  • Part-based methods use predefined or automatically detected parts, combining local appearance features with global features or learning separate object and part detectors.Examples include dog-part features, POOF, and part-based R-CNNs.Source: 3) Part Localization-based Approaches · pp. 2–3

Visual Attention

  • DVAN is presented as a framework that prepares multiple attention canvases, learns CNN features, predicts diversified attention maps, and classifies from pooled features.Its framework is described as combining attention canvas generation, CNN feature learning, diversified visual attention, and classification.Source: Figure 2 · p. 3

4) Human-in-the-Loop Approaches:

Human-in-the-loop methods require interactive localization of discriminative regions and therefore face scalability concerns. DVAN instead uses automatically generated canvases and diversified attention to localize object parts without such interaction.

Human-in-the-Loop Approaches

  • Human-in-the-loop approaches require people to identify the most discriminative regions interactively.The cited limitation is reduced scalability for large-scale image classification.Source: 4) Human-in-the-Loop Approaches · p. 3

Visual Attention

  • Visual-attention models provide an alternative by automatically localizing discriminative regions, with soft attention trained differentiably and hard attention using stochastic points.Soft attention is generally more efficient because it can be trained end-to-end without sampling.Source: B. Visual Attention · p. 3

DVAN Architecture

  • The diversity-promoting attention model and canvas-generation approach are designed to guarantee diversity and information gain during visual attention.This design is identified as a core contribution of the model.Source: §III · p. 3
  • DVAN generates several image regions at different scales as attention canvases, then applies CNN feature learning and diversified attention to highlight important locations.The architecture uses these components before attentive feature integration and classification.Source: A. Overall Architecture · p. 3

B. Visual Attention Model

DVAN’s visual attention model integrates attentive features with an attention-map prediction process. LSTM recurrent processing combines sequential attentive features while CNN feature maps and prior hidden states determine successive attention maps.

B. Visual Attention Model

Attention Feature Integration

Attention Map Prediction

  • At each time step, DVAN can focus on K^2 locations, with each location represented by a feature vector whose dimension equals the number of feature maps.The feature maps are partitioned into spatial slices X_t,i.Source: 2) Attention Map Prediction · p. 4
  • CNN feature maps and the previous LSTM hidden state jointly determine the new attention map.Weights connect the previous hidden state and feature maps to each spatial attention-map location.Source: 2) Attention Map Prediction · p. 4

3) Attention Model Initialization:

DVAN initializes its LSTM states from averaged convolutional feature maps and uses the resulting hidden state for classification and subsequent attention prediction.

3) Attention Model Initialization:

  • The LSTM cell and hidden states are initialized by two MLPs using the average over all feature maps.These initial states calculate the first attention map and initial attentive feature.Source: 3) Attention Model Initialization · p. 4
  • The LSTM hidden state, after tanh activation, serves as the classification feature and guides the next attention-map prediction.Classification uses a fully convolutional layer followed by softmax to produce category probabilities.Source: 4) Classification · p. 4

4) Classification:

Vanilla attention can repeatedly select the same discriminative region, limiting information gain when fine-grained differences occur elsewhere. DVAN promotes temporally diverse attention through a diversity loss, spatial-overlap constraint, and multiple attention canvases.

Attention diversity

  • Vanilla attention repeatedly focuses on the same bird head and neck across time steps, missing potentially informative wings or tails.These regions may distinguish visually similar species when head and neck features are insufficient.Source: C. Diversity Promoting Attention Models · pp. 4–5
  • Fine-grained classification requires collecting information from multiple small regions because subtle differences may be spatially localized.The diversity of the attention process is introduced to capture sufficient discriminative information.Source: C. Diversity Promoting Attention Models · p. 5

Diversity objective

  • DVAN penalizes correlation between temporally adjacent attention maps, although minimizing correlation alone may not ensure sufficient diversity.The diversity metric assigns larger values to more similar neighboring maps.Source: C. Diversity Promoting Attention Models · p. 5
  • A hard spatial constraint limits overlap between neighboring attention regions, shifting attention toward different image locations over time.The threshold is expressed relative to the number of pixels in the original image.Source: C. Diversity Promoting Attention Models · p. 5
  • The final loss combines classification loss, a diversity penalty, and a hard attention-canvas constraint.The penalty coefficient λ controls the extent of the diversity penalty when neighboring attention locations change little.Source: C. Diversity Promoting Attention Models · p. 5

D. Multi-scale Attention Canvas Generation

DVAN generates attention canvases from different image locations and scales, then sequences them from coarse views to enlarged local regions. This supplies diverse candidates whose attention maps can cover bird parts from head to tail.

Canvas construction

  • DVAN crops multiple attention canvases at different locations and scales to provide diverse candidates for visual attention.Some canvases contain the main object, while others contain enlarged local regions.Source: D. Multi-scale Attention Canvas… · p. 5
  • Smaller window sizes enlarge local regions after resizing, while smaller strides generate more attention canvases.All canvases are normalized to a uniform size, such as 224 × 224 for VGG Net.Source: D. Multi-scale Attention Canvas… · pp. 5–6

Canvas sequence

  • The canvases cover most input-image regions and are ordered so coarse views precede fine local views.This sequence lets the model attend to the object’s main body before detecting local parts.Source: D. Multi-scale Attention Canvas… · p. 6
  • Diversified attention maps attend to different bird locations and scales, including the head, body, legs, and tail.The passage characterizes this pattern as reasonable for fine-grained classification.Source: D. Multi-scale Attention Canvas… · p. 6

IV. EXPERIMENTS

The experiments evaluate DVAN on three fine-grained classification benchmarks with complex image contents and available annotations. The model is assessed through ablations, comparisons with state-of-the-art methods, and attention-map visualization without using bounding-box or part annotations.

Datasets

  • The experiments evaluate DVAN on CUB-200-2011, Stanford Dogs, and Stanford Cars datasets.These benchmarks cover birds, dog breeds, and car classes.Source: A. Datasets · p. 6
  • The datasets contain complex image contents that make fine-grained classification challenging.Representative examples include cluttered backgrounds and substantial variation across the benchmark images.Source: A. Datasets · p. 6
  • CUB-200-2011 contains 11,778 images from 200 bird categories, with 5,994 training and 5,794 testing images.It provides image-level labels, bounding boxes, attribute annotations, and part landmarks.Source: A. Datasets · p. 6
  • Stanford Dogs contains 20,580 images across 120 breeds, while Stanford Cars contains 16,185 images across 196 car classes.Stanford Dogs provides tight bounding boxes, and Stanford Cars commonly contains cluttered backgrounds.Source: A. Datasets · p. 6

Experimental setting

  • DVAN does not use the datasets’ bounding-box or part-level annotations during the experiments.The authors identify this annotation independence as an advantage of the proposed method.Source: A. Datasets · p. 6

B. Implementation Details

DVAN is evaluated with multiple attention canvases, alternative pooling strategies, and varying diversity parameter λ. Dynamic attention pooling performs best among the compared pooling designs, reaching 79.0%.

Implementation

  • Three window sizes—224 × 224, 168 × 168, and 112 × 112—generate attention canvases from resized images.Their strides are 32, 44, and 48, respectively.Source: B. Implementation Details · p. 6

Pooling comparisons

  • DVAN is compared with single-image and multiple-canvas VGG-16 baselines, replacing diversified attention with fully connected layers in the latter.The multiple-canvas baseline averages softmax predictions across canvases, as does DVAN.Source: 1) Effect of Attention Mechanism · p. 7
  • DVAN-Avg and DVAN-Max outperform VGG-16-multi-canvas, supporting LSTM-based sequential modeling of attention canvases.DVAN-Avg reaches 76.8%, while average pooling retains more information than max pooling in this comparison.Source: 2) Effect of Different Pooling Methods · p. 7
  • Dynamic attention pooling raises performance to 79.0%, outperforming both DVAN-Avg and DVAN-Max.The model learns discriminative regions and pools feature maps according to attention probabilities.Source: 2) Effect of Different Pooling Methods · p. 7

Parameter setting

  • λ controls diversity among neighboring attention maps: λ = 0 imposes no successive-map diversification, whereas larger values force attention toward different regions.Performance improves initially but declines when excessive diversity scatters fixations and loses important information; the best setting is λ = 1.Source: 3) Effect of Parameter · p. 7, Figure 7 · p. 7, Figure 8 · p. 7

3) Effect of Parameter

DVAN benefits from diversified attention across scales, with performance generally improving as more canvas scales are added. However, the smallest scale is not uniformly helpful, as shown by Stanford Cars.

Effect of Parameter

  • With λ = 0, attention covers major image areas but includes substantial background, whereas λ = 1 focuses more accurately on the bird torso or local parts.Larger λ values scatter attention unevenly and can shift it away from discriminative regions, degrading performance.Source: 3) Effect of Parameter · pp. 7–8

Performance of Different Scales

  • Multi-scale canvases at different locations contribute to attention diversity and let DVAN observe objects from coarse to fine granularity.Large windows tend to capture whole objects, while small windows attend detailed regions.Source: 4) Performance of Different Scales · p. 8
  • 79.0% accuracy on CUB-200-2011 and 81.5% on Stanford Dogs are achieved with three-scale attention.The reported trend is that adding canvas scales significantly improves performance.Source: 4) Performance of Different Scales · p. 8
  • Two-scale attention outperforms three-scale attention on Stanford Cars, indicating that a small window does not always add useful information.Very small canvases may contain only localized regions such as tires or windshields.Source: 4) Performance of Different Scales · p. 8

5) Effect of CNN Feature Learning:

DVAN learns convolutional features that support stronger SVM classification than VGG-16 and VGG-16-multi-canvas features. Comparisons with prior methods are also reported across fine-grained datasets, with more complex part-based models sometimes performing better.

Effect of CNN Feature Learning

  • DVAN is end-to-end trainable because gradients from diversified visual attention back-propagate into CNN feature learning.The reported feature comparison associates this training design with superior learned convolutional features relative to the two VGG baselines.Source: 5) Effect of CNN Feature Learning · p. 8
  • 68.2% accuracy is obtained by an SVM using DVAN pool5 features, exceeding SVMs trained on VGG-16 or VGG-16-multi-canvas pool5 features.The comparison is intended to assess the quality of learned convolutional features.Source: 5) Effect of CNN Feature Learning · p. 8

Comparison with state-of-the-art methods

  • On CUB-200-2011, POOF achieves 73.3% using bounding-box and part annotations for part feature extraction and comparison.The paper compares DVAN with state-of-the-art methods using additional localization information.Source: 1) Performance on CUB-200-2011 · p. 8, Table V · p. 8
  • More complicated NAC and PD models outperform DVAN in the cited comparison because they use detected parts and richer part-based representations.The passage describes NAC and PD as relying on detected parts, with PD using Spatial Weighted Fisher Vector pooling.Source: 1) Performance on CUB-200-2011 · p. 9
  • Alignment localization on Stanford Dogs reaches only 50.1% despite using pregiven bounding boxes during training and testing.The paper characterizes this unsupervised alignment approach as poorly adapted in that comparison.Source: 2) Performance on Stanford Dogs · p. 9, Table VI · p. 9

3) Performance on Stanford Cars:

On Stanford Cars, DVAN achieved competitive performance while diversifying attention across locations and scales, though extremely small local parts could reduce classification accuracy.

Performance comparison

  • 82.7% accuracy was achieved by Fisher vector, while ELLF and symbiotic segmentation obtained 73.9% and 78.0%, respectively.The passage identifies these methods and values in the Stanford Cars comparison.Source: 3) Performance on Stanford Cars · p. 9

Diversified attention visualization

  • DVAN visualizations show attention progressing from broad object regions to higher-resolution local parts across time steps and canvas scales.Small canvases cover more of the object, whereas large canvases focus on smaller regions at higher resolution.Source: E. Diversified Attention Visualization · p. 9
  • For cars, attention moves from the frontal part to hoods, bumpers, side doors, front bumps, and wheels.These regions are attended sequentially after enlarging local parts.Source: E. Diversified Attention Visualization · p. 9

Limitation

Overall conclusion

  • Across three datasets, DVAN achieves competitive performance without bounding-box or part-location information.The conclusion states that the method uses no such strongly supervised localization information.Source: §V · p. 10

Related papers

Loading 1606.08572v2…