PaperlayerView exact revision on arXiv

Source-linked AI summary

Diversified Visual Attention Networks for Fine-Grained Object Classification

Bo Zhao, Xiao Wu, Jiashi Feng, Qiang Peng, Shuicheng Yan

arXiv:1606.08572v2cs.CV

TL;DR

Fine-grained classification must distinguish subtle differences amid large appearance variation, while existing attention models may repeatedly focus on similar regions. DVAN uses diversified recurrent attention over multiple canvases and reports competitive performance across three benchmark datasets without bounding-box or part-location information.

  • Problem

    Fine-grained classification is challenged by subtle inter-class differences, large intra-class variation, and attention models that may fail to gather multiple discriminative regions.

  • Method

    DVAN generates canvases at different locations and scales, uses LSTM for recurrent attentive feature integration, and explicitly diversifies attention maps.

  • Results

    DVAN achieves competitive performance with state-of-the-art approaches on CUB-2011, Stanford Dogs, and Stanford Cars without bounding-box or part-location information.

  • Takeaways & Limitations

    A coarse global view and diversified fine-grained local parts are incrementally combined to capture discriminative information for classification.

Abstract

from arXiv · show

Fine-grained object classification is a challenging task due to the subtle inter-class difference and large intra-class variation. Recently, visual attention models have been applied to automatically localize the discriminative regions of an image for better capturing critical difference and demonstrated promising performance. However, without consideration of the diversity in attention process, most of existing attention models perform poorly in classifying fine-grained objects. In this paper, we propose a diversified visual attention network (DVAN) to address the problems of fine-grained object classification, which substan- tially relieves the dependency on strongly-supervised information for learning to localize discriminative regions compared with attentionless models. More importantly, DVAN explicitly pursues the diversity of attention and is able to gather discriminative information to the maximal extent. Multiple attention canvases are generated to extract convolutional features for attention. An LSTM recurrent unit is employed to learn the attentiveness and discrimination of attention canvases. The proposed DVAN has the ability to attend the object from coarse to fine granularity, and a dynamic internal representation for classification is built up by incrementally combining the information from different locations and scales of the image. Extensive experiments con- ducted on CUB-2011, Stanford Dogs and Stanford Cars datasets have demonstrated that the proposed diversified visual attention networks achieve competitive performance compared to the state- of-the-art approaches, without using any prior knowledge, user interaction or external resource in training or testing.

I. INTRODUCTION

Fine-grained classification is difficult because visually similar categories differ subtly while instances within a category vary substantially. DVAN addresses this challenge by diversifying recurrent visual attention across multiple image locations and scales to build an incremental representation.

  • Challenge: Fine-grained classification distinguishes visually and semantically similar subordinate categories, including bird species, dog breeds, and car classes.Its applications include multimedia information retrieval and content analysis.
  • Challenge: Subtle inter-class differences and large intra-class variation arise from scale, viewpoint, background, and occlusion changes.These differences can be difficult even for humans to recognize.
  • Motivation: Existing approaches exploit discriminative local parts, but visual attention models struggle to find multiple discriminative regions simultaneously.The paper motivates diversity because fine-grained differences often occur in small object regions.
  • Approach: DVAN uses LSTM to find multiple attentive regions over time and generates attention canvases at different locations and scales.Some canvases cover the whole object, while others contain local parts.
  • Approach: DVAN combines a coarse-grained global view with fine-grained diversified local parts to dynamically build an incremental object representation.This representation is designed to capture both the general picture and local details.
  • Results: Experiments on three benchmark datasets show competitive performance with state-of-the-art methods without prior knowledge or user interaction.The paper identifies CUB-2011, Stanford Dogs, and Stanford Cars as evaluation datasets.

II. RELATED WORK

Related work approaches fine-grained classification through deep representations, part alignment or localization, and diversified visual-attention frameworks. These methods differ in how they identify and combine discriminative object regions.

  • Overview: Fine-grained classification methods are organized into four groups according to their use of additional information or human interaction.The supplied related-work passages include representation, alignment, part-localization, and human-in-the-loop approaches.
  • Discriminative Representation: Deep representation methods use convolutional features, while bilinear architectures model local pairwise feature interactions with two independent subnetworks.These approaches build discriminative representations rather than relying on handcrafted features.
  • Alignment: Alignment approaches discover or segment common object-part patterns, align images, and extract features from corresponding regions.The cited examples include unsupervised template learning and alignment categorization.
  • Part Localization: Part-based methods use predefined or automatically detected parts, combining local appearance features with global features or learning separate object and part detectors.Examples include dog-part features, POOF, and part-based R-CNNs.
  • Visual Attention: DVAN is presented as a framework that prepares multiple attention canvases, learns CNN features, predicts diversified attention maps, and classifies from pooled features.Its framework is described as combining attention canvas generation, CNN feature learning, diversified visual attention, and classification.

4) Human-in-the-Loop Approaches:

Human-in-the-loop methods require interactive localization of discriminative regions and therefore face scalability concerns. DVAN instead uses automatically generated canvases and diversified attention to localize object parts without such interaction.

  • Human-in-the-Loop Approaches: Human-in-the-loop approaches require people to identify the most discriminative regions interactively.The cited limitation is reduced scalability for large-scale image classification.
  • Visual Attention: Visual-attention models provide an alternative by automatically localizing discriminative regions, with soft attention trained differentiably and hard attention using stochastic points.Soft attention is generally more efficient because it can be trained end-to-end without sampling.
  • DVAN Architecture: The diversity-promoting attention model and canvas-generation approach are designed to guarantee diversity and information gain during visual attention.This design is identified as a core contribution of the model.
  • DVAN Architecture: DVAN generates several image regions at different scales as attention canvases, then applies CNN feature learning and diversified attention to highlight important locations.The architecture uses these components before attentive feature integration and classification.

B. Visual Attention Model

DVAN’s visual attention model integrates attentive features with an attention-map prediction process. LSTM recurrent processing combines sequential attentive features while CNN feature maps and prior hidden states determine successive attention maps.

  • B. Visual Attention Model: The visual attention model has two modules: attentive feature integration and attention map prediction.These correspond to the top and bottom panels of Fig. 3.
  • Attention Feature Integration: LSTM performs attentive feature integration because its long short-term memory models sequential attentive regions.The LSTM uses input, forget, output, and candidate-state gates to update cell and hidden states over time.
  • Attention Feature Integration: The attentive feature x_t is generated by attention-map prediction and fed into the LSTM for integration with previous features.The pooled feature is computed from CNN feature maps using the predicted attention map.
  • Attention Map Prediction: At each time step, DVAN can focus on K^2 locations, with each location represented by a feature vector whose dimension equals the number of feature maps.The feature maps are partitioned into spatial slices X_t,i.
  • Attention Map Prediction: CNN feature maps and the previous LSTM hidden state jointly determine the new attention map.Weights connect the previous hidden state and feature maps to each spatial attention-map location.

3) Attention Model Initialization:

DVAN initializes its LSTM states from averaged convolutional feature maps and uses the resulting hidden state for classification and subsequent attention prediction.

  • 3) Attention Model Initialization:: The LSTM cell and hidden states are initialized by two MLPs using the average over all feature maps.These initial states calculate the first attention map and initial attentive feature.
  • 3) Attention Model Initialization:: The LSTM hidden state, after tanh activation, serves as the classification feature and guides the next attention-map prediction.Classification uses a fully convolutional layer followed by softmax to produce category probabilities.

4) Classification:

Vanilla attention can repeatedly select the same discriminative region, limiting information gain when fine-grained differences occur elsewhere. DVAN promotes temporally diverse attention through a diversity loss, spatial-overlap constraint, and multiple attention canvases.

  • Attention diversity: Vanilla attention repeatedly focuses on the same bird head and neck across time steps, missing potentially informative wings or tails.These regions may distinguish visually similar species when head and neck features are insufficient.
  • Attention diversity: Fine-grained classification requires collecting information from multiple small regions because subtle differences may be spatially localized.The diversity of the attention process is introduced to capture sufficient discriminative information.
  • Diversity objective: DVAN penalizes correlation between temporally adjacent attention maps, although minimizing correlation alone may not ensure sufficient diversity.The diversity metric assigns larger values to more similar neighboring maps.
  • Diversity objective: A hard spatial constraint limits overlap between neighboring attention regions, shifting attention toward different image locations over time.The threshold is expressed relative to the number of pixels in the original image.
  • Diversity objective: The final loss combines classification loss, a diversity penalty, and a hard attention-canvas constraint.The penalty coefficient λ controls the extent of the diversity penalty when neighboring attention locations change little.

D. Multi-scale Attention Canvas Generation

DVAN generates attention canvases from different image locations and scales, then sequences them from coarse views to enlarged local regions. This supplies diverse candidates whose attention maps can cover bird parts from head to tail.

  • Canvas construction: DVAN crops multiple attention canvases at different locations and scales to provide diverse candidates for visual attention.Some canvases contain the main object, while others contain enlarged local regions.
  • Canvas construction: Smaller window sizes enlarge local regions after resizing, while smaller strides generate more attention canvases.All canvases are normalized to a uniform size, such as 224 × 224 for VGG Net.
  • Canvas sequence: The canvases cover most input-image regions and are ordered so coarse views precede fine local views.This sequence lets the model attend to the object’s main body before detecting local parts.
  • Canvas sequence: Diversified attention maps attend to different bird locations and scales, including the head, body, legs, and tail.The passage characterizes this pattern as reasonable for fine-grained classification.

IV. EXPERIMENTS

The experiments evaluate DVAN on three fine-grained classification benchmarks with complex image contents and available annotations. The model is assessed through ablations, comparisons with state-of-the-art methods, and attention-map visualization without using bounding-box or part annotations.

  • Datasets: The experiments evaluate DVAN on CUB-200-2011, Stanford Dogs, and Stanford Cars datasets.These benchmarks cover birds, dog breeds, and car classes.
  • Datasets: The datasets contain complex image contents that make fine-grained classification challenging.Representative examples include cluttered backgrounds and substantial variation across the benchmark images.
  • Datasets: CUB-200-2011 contains 11,778 images from 200 bird categories, with 5,994 training and 5,794 testing images.It provides image-level labels, bounding boxes, attribute annotations, and part landmarks.
  • Datasets: Stanford Dogs contains 20,580 images across 120 breeds, while Stanford Cars contains 16,185 images across 196 car classes.Stanford Dogs provides tight bounding boxes, and Stanford Cars commonly contains cluttered backgrounds.
  • Experimental setting: DVAN does not use the datasets’ bounding-box or part-level annotations during the experiments.The authors identify this annotation independence as an advantage of the proposed method.

B. Implementation Details

DVAN is evaluated with multiple attention canvases, alternative pooling strategies, and varying diversity parameter λ. Dynamic attention pooling performs best among the compared pooling designs, reaching 79.0%.

  • Implementation: Three window sizes—224 × 224, 168 × 168, and 112 × 112—generate attention canvases from resized images.Their strides are 32, 44, and 48, respectively.
  • Pooling comparisons: DVAN is compared with single-image and multiple-canvas VGG-16 baselines, replacing diversified attention with fully connected layers in the latter.The multiple-canvas baseline averages softmax predictions across canvases, as does DVAN.
  • Pooling comparisons: DVAN-Avg and DVAN-Max outperform VGG-16-multi-canvas, supporting LSTM-based sequential modeling of attention canvases.DVAN-Avg reaches 76.8%, while average pooling retains more information than max pooling in this comparison.
  • Pooling comparisons: Dynamic attention pooling raises performance to 79.0%, outperforming both DVAN-Avg and DVAN-Max.The model learns discriminative regions and pools feature maps according to attention probabilities.
  • Parameter setting: λ controls diversity among neighboring attention maps: λ = 0 imposes no successive-map diversification, whereas larger values force attention toward different regions.Performance improves initially but declines when excessive diversity scatters fixations and loses important information; the best setting is λ = 1.

3) Effect of Parameter

DVAN benefits from diversified attention across scales, with performance generally improving as more canvas scales are added. However, the smallest scale is not uniformly helpful, as shown by Stanford Cars.

  • Effect of Parameter: With λ = 0, attention covers major image areas but includes substantial background, whereas λ = 1 focuses more accurately on the bird torso or local parts.Larger λ values scatter attention unevenly and can shift it away from discriminative regions, degrading performance.
  • Performance of Different Scales: Multi-scale canvases at different locations contribute to attention diversity and let DVAN observe objects from coarse to fine granularity.Large windows tend to capture whole objects, while small windows attend detailed regions.
  • Performance of Different Scales: 79.0% accuracy on CUB-200-2011 and 81.5% on Stanford Dogs are achieved with three-scale attention.The reported trend is that adding canvas scales significantly improves performance.
  • Performance of Different Scales: Two-scale attention outperforms three-scale attention on Stanford Cars, indicating that a small window does not always add useful information.Very small canvases may contain only localized regions such as tires or windshields.

5) Effect of CNN Feature Learning:

DVAN learns convolutional features that support stronger SVM classification than VGG-16 and VGG-16-multi-canvas features. Comparisons with prior methods are also reported across fine-grained datasets, with more complex part-based models sometimes performing better.

  • Effect of CNN Feature Learning: DVAN is end-to-end trainable because gradients from diversified visual attention back-propagate into CNN feature learning.The reported feature comparison associates this training design with superior learned convolutional features relative to the two VGG baselines.
  • Effect of CNN Feature Learning: 68.2% accuracy is obtained by an SVM using DVAN pool5 features, exceeding SVMs trained on VGG-16 or VGG-16-multi-canvas pool5 features.The comparison is intended to assess the quality of learned convolutional features.
  • Comparison with state-of-the-art methods: On CUB-200-2011, POOF achieves 73.3% using bounding-box and part annotations for part feature extraction and comparison.The paper compares DVAN with state-of-the-art methods using additional localization information.
  • Comparison with state-of-the-art methods: More complicated NAC and PD models outperform DVAN in the cited comparison because they use detected parts and richer part-based representations.The passage describes NAC and PD as relying on detected parts, with PD using Spatial Weighted Fisher Vector pooling.
  • Comparison with state-of-the-art methods: Alignment localization on Stanford Dogs reaches only 50.1% despite using pregiven bounding boxes during training and testing.The paper characterizes this unsupervised alignment approach as poorly adapted in that comparison.

3) Performance on Stanford Cars:

On Stanford Cars, DVAN achieved competitive performance while diversifying attention across locations and scales, though extremely small local parts could reduce classification accuracy.

  • Performance comparison: 82.7% accuracy was achieved by Fisher vector, while ELLF and symbiotic segmentation obtained 73.9% and 78.0%, respectively.The passage identifies these methods and values in the Stanford Cars comparison.
  • Diversified attention visualization: DVAN visualizations show attention progressing from broad object regions to higher-resolution local parts across time steps and canvas scales.Small canvases cover more of the object, whereas large canvases focus on smaller regions at higher resolution.
  • Diversified attention visualization: For cars, attention moves from the frontal part to hoods, bumpers, side doors, front bumps, and wheels.These regions are attended sequentially after enlarging local parts.
  • Limitation: Tiny local car parts can reduce classification accuracy because they may lose the ability to distinguish car models.This is reported as an experimental observation.
  • Overall conclusion: Across three datasets, DVAN achieves competitive performance without bounding-box or part-location information.The conclusion states that the method uses no such strongly supervised localization information.
Loading 1606.08572v2…