PaperlayerView exact revision on arXiv

Source-linked AI summary

Object-Part Attention Model for Fine-grained Image Classification

Yuxin Peng, Xiangteng He, Junjie Zhao

arXiv:1704.01740v2cs.CV

TL;DR

Fine-grained classification must separate visually similar subcategories despite large within-category variation, while existing methods often require costly annotations or ignore object-part spatial relationships. OPAM jointly learns object- and part-level attention and uses spatial constraints for weakly supervised part selection. On four datasets, it reports the best performance among more than 10 state-of-the-art methods.

  • Problem

    Existing fine-grained methods often rely on labor-consuming object or part annotations and ignore spatial relationships useful for selecting discriminative parts.

  • Method

    OPAM jointly localizes objects and selects discriminative parts through two-level attention, while object-part spatial constraints enforce representativeness, reduce overlap, and enhance discrimination.

  • Results

    OPAM achieves the best results among state-of-the-art methods on four widely used datasets.

  • Takeaways & Limitations

    The approach performs weakly supervised fine-grained classification without object or part annotations while exploiting multi-view, multi-scale, subtle, and local features.

Abstract

from arXiv · show

Fine-grained image classification is to recognize hundreds of subcategories belonging to the same basic-level category, such as 200 subcategories belonging to the bird, which is highly challenging due to large variance in the same subcategory and small variance among different subcategories. Existing methods generally first locate the objects or parts and then discriminate which subcategory the image belongs to. However, they mainly have two limitations: (1) Relying on object or part annotations which are heavily labor consuming. (2) Ignoring the spatial relationships between the object and its parts as well as among these parts, both of which are significantly helpful for finding discriminative parts. Therefore, this paper proposes the object-part attention model (OPAM) for weakly supervised fine-grained image classification, and the main novelties are: (1) Object-part attention model integrates two level attentions: object-level attention localizes objects of images, and part-level attention selects discriminative parts of object. Both are jointly employed to learn multi-view and multi-scale features to enhance their mutual promotions. (2) Object-part spatial constraint model combines two spatial constraints: object spatial constraint ensures selected parts highly representative, and part spatial constraint eliminates redundancy and enhances discrimination of selected parts. Both are jointly employed to exploit the subtle and local differences for distinguishing the subcategories. Importantly, neither object nor part annotations are used in our proposed approach, which avoids the heavy labor consumption of labeling. Comparing with more than 10 state-of-the-art methods on 4 widely-used datasets, our OPAM approach achieves the best performance.

I. INTRODUCTION

Fine-grained classification distinguishes closely related subcategories within a basic-level category, where within-class variation is large and between-class variation is small. OPAM addresses annotation costs and neglected spatial relationships through joint object-part attention and spatial constraints.

  • Motivation: Fine-grained classification must distinguish hundreds of subcategories within basic-level categories such as birds, cars, pets, and flowers.Unlike basic-level classification, it determines the exact subcategory, such as American Crow versus Fish Crow.
  • Limitations: Object and part annotations are labor consuming, motivating weakly supervised methods that use neither annotation during training nor testing.Existing approaches commonly rely on bounding boxes or part locations to find discriminative regions.
  • Motivation: Large variation within subcategories and small variation among subcategories make subtle, local differences crucial for classification.Discriminative cues can include a bird’s back color, bill shape, or feather texture.
  • Limitations: Ignoring spatial relationships can select parts with excessive background or overlap, producing noisy and redundant features.The neglected relationships include those between objects and parts and those among parts.
  • Contributions: OPAM jointly uses object-level attention for representative object features and part-level attention for discriminative local features, supporting multi-view and multi-scale learning.The approach avoids object and part annotations and promotes mutual interaction between the two attention levels.
  • Contributions: Object-part spatial constraints keep selected parts representative of the object while reducing overlap and enhancing part discrimination.The combined constraints exploit subtle local distinctions for fine-grained classification.

II. RELATED WORK

Related work spans handcrafted descriptors, deep feature-learning systems, attention mechanisms, detection-based part localization, and weakly supervised approaches. OPAM is presented within this progression as a model combining object localization, part selection, and spatial relationships.

  • Traditional and deep methods: Traditional methods use low-level descriptors such as SIFT and Bag-of-Words, but handcrafted features limit performance.
  • Ensemble of Networks Based Methods: Ensemble methods combine representations from multiple neural networks, while bilinear CNNs multiply local features from two CNNs before pooling.Some ensemble approaches require external ontology labels, whereas bilinear CNNs use end-to-end feature extraction and classification.
  • Attention-Based Methods: Attention-based methods dynamically localize informative image regions or sequential glimpses to improve fine-grained recognition.Fully convolutional attention and related methods use saliency or neural activation patterns to identify discriminative regions.
  • Part Detection Based Methods: Detection-based methods generate many candidate patches and select those with high classification scores as object or part detections.R-CNN-style pipelines depend on bottom-up candidate generation before selecting informative patches.
  • Weakly Supervised Methods: Weakly supervised methods avoid object and part annotations, but prior approaches may ignore spatial relationships among objects and parts.The paper identifies spatial structure as useful for selecting discriminative parts and reducing noisy or redundant selections.
  • Part Detection Based Methods: OPAM’s overview separates object-level attention for object features from part-level attention for subtle local features before predicting subcategories.

III. OUR OPAM APPROACH

OPAM automatically filters candidate patches, learns multi-view and multi-scale features, and localizes objects from saliency maps using only image-level subcategory labels. These stages provide object-focused inputs for subsequent discriminative part selection.

  • Approach overview: OPAM uses object-level attention followed by part-level attention without object or part annotations during training or testing.The design reflects the paper’s weakly supervised setting and its object-then-parts localization strategy.
  • Patch Filtering: Selective search generates many candidate patches that expand the training data with regions potentially containing objects.
  • Patch Filtering: FilterNet removes noisy patches using subcategory-specific softmax activation as a confidence score and a threshold for selection.The retained patches are relevant to the object and support multiple views and scales.
  • Object Localization: CAM produces a class-specific saliency map, which is binarized and processed by connectivity extraction to obtain automatically localized object regions.
  • Patch Filtering: ClassNet learns multi-view and multi-scale features from filtered patches and supplies internal features for later part clustering and semantic alignment.Patch filtering occurs only during training and uses image-level subcategory labels.

2) Saliency Extraction:

The model extracts class-specific saliency maps to localize objects, then selects discriminative parts using object-part spatial constraints that enforce representativeness and reduce redundancy.

  • Saliency Extraction: CAM saliency maps localize representative regions for each predicted subcategory, followed by binarization and connectivity extraction to obtain object regions.The predicted subcategory, rather than the image-level label, is used during saliency extraction.
  • Object-Part Spatial Constraint: The object-part spatial constraint model selects n discriminative parts from bottom-up candidate patches by optimizing a joint scoring function.The selected parts are represented as P = {p1, p2, ..., pn}.
  • Object-Part Spatial Constraint: The combined constraints improve part representativeness and discrimination by exploiting spatial relationships between objects and parts and among parts.Ignoring these relationships can produce background-heavy or redundant parts.
  • Object Spatial Constraint: Object spatial constraint keeps selected parts inside automatically localized object regions, excluding parts with zero IoU from selection.Its product formulation requires all selected parts to satisfy the object constraint.
  • Part Spatial Constraint: Part spatial constraint reduces overlap and outside-object area while maximizing average saliency across the union of selected parts.The scoring terms use log(AU − AI − AO) and log(Mean(MAU)).

2) Part Alignment:

Part alignment groups selected image patches by semantic similarity, using spectral clusters of penultimate-layer neurons to organize differently aligned parts.

  • Part Alignment: Selected parts are initially unordered, so the method aligns parts with the same semantic meaning into common clusters.The motivation is that differently meaningful parts contribute differently to the final prediction.
  • Part Clusters: Spectral clustering partitions penultimate convolutional-layer neurons into two groups using cosine similarities between neuron weights.The two largest eigenvectors provide the coordinates illustrated in Fig. 6.
  • Alignment Procedure: Each selected part is warped to a neuron receptive-field size, forwarded through the network, and assigned to the cluster with the highest summed activation.Cluster scores sum activations from neurons belonging to each cluster.
  • Layer Selection: A validation-set grid search selects the penultimate convolutional layer because it performs better for part alignment and classification than other tested layers.The validation set withholds 10% of training data.

C. Final Prediction

OPAM fine-tunes separate classifiers for original images, localized objects, and selected parts, then merges their prediction scores for the final subcategory decision.

  • Classifiers: ClassNet, ObjectNet, and PartNet classify original images, localized objects, and selected discriminative parts, respectively.The networks are fine-tuned with the corresponding image focus.
  • Attention Levels: Object-level attention supplies multi-view, multi-scale object patches, while part-level attention supplies patches containing subtle and local features.The three image focuses have different but complementary representations.
  • Final Prediction: The final prediction merges the softmax scores from the three classifiers using cross-validation-selected weights, choosing the subcategory with the highest combined score.The scores are combined from ClassNet, ObjectNet, and PartNet.

IV. EXPERIMENTS

The experiments evaluate OPAM on four widely used fine-grained image-classification datasets against more than 10 state-of-the-art methods.

  • Datasets: OPAM is tested on CUB-200-2011, Cars-196, Oxford-IIIT Pet, and Oxford-Flower-102.These are four widely used datasets for fine-grained image classification.
  • Comparisons: The evaluation compares OPAM with more than 10 state-of-the-art methods to verify its effectiveness.

A. Datasets and evaluation metric

Experiments use four widely used fine-grained image-classification datasets spanning birds, cars, pets, and flowers, with accuracy as the evaluation metric.

  • Four datasets are used: CUB-200-2011, Cars-196, Oxford-IIIT Pet, and Oxford-Flower-102.They contain 200 bird, 196 car, 37 pet, and 102 flower subcategories, respectively.
  • CUB-200-2011 contains 11,788 images from 200 bird subcategories, split into 5,994 training and 5,794 testing images.
  • Cars-196 contains 16,185 images from 196 car subcategories, split into 8,144 training and 8,041 testing images.
  • Oxford-IIIT Pet contains 7,349 images from 37 pet subcategories, split into 3,680 training and 3,669 testing images.
  • Oxford-Flower-102 contains 8,189 images, divided into 1,020 training, 1,020 validation, and 6,149 testing images.
  • Accuracy is used to evaluate classification performance, computed from the number of correctly classified testing images divided by all testing images.

B. Details of the networks

The experiments modify VGGNet for object localization and classification, then compare OPAM with state-of-the-art methods across four datasets. OPAM achieves the best reported results, including 85.83% on CUB-200-2011 and 92.19%, 93.81%, and 97.10% on the other three datasets.

  • Network design: VGGNet is used experimentally, with CNN architectures modified separately for localization and classification functions.The CNN can be replaced with other CNN architectures.
  • Object localization: Object-level attention extracts an image saliency map for object localization using a higher-resolution 14 × 14 mapping.Layers after conv5 3 are removed, and a 3 × 3 convolutional layer with 1024 neurons precedes global average pooling and softmax.
  • Classification: Classification uses ClassNet for original-image predictions, ObjectNet for object predictions, and PartNet for part predictions.These networks share the VGGNet with batch normalization architecture but are fine-tuned on different training data.
  • Results: 85.83% accuracy makes OPAM the best compared method under the setting without object or part annotations on CUB-200-2011.This is 1.20% higher than FOAF’s 84.63% and 1.29% higher than PD’s 84.54%.
  • Results: 84.10% accuracy for both Bilinear-CNN variants is 1.73% below OPAM on CUB-200-2011.OPAM also outperforms STN and methods using object or object-and-part annotations.
  • Results: 92.19%, 93.81%, and 97.10% are OPAM’s results on Cars-196, Oxford-IIIT Pet, and Oxford-Flower-102, respectively.These improve over the best compared methods by 0.89%, 0.36%, and 0.70%, respectively.

D. Performances of components in our OPAM approach

Component evaluations show that object-level attention improves classification, while part-level attention contributes complementary information whose combination with object and original images produces the strongest results.

  • Object-level and part-level attention: Object-level attention improves accuracy by 2.92%, 2.00%, 2.84% and 0.62% across the four datasets versus “Original”.Combining “Object-level” with “Original” yields larger gains of 3.97%, 4.36%, 4.06% and 1.85%, respectively.
  • Object-level and part-level attention: Part-level attention can fail when objects are hard to distinguish from background or heavily occluded, causing inaccurate object localization and part selection.Its lower standalone accuracy also reflects that subtle local features contain less information than the original image.
  • Object-level and part-level attention: 84.73% versus 83.74% and 80.65% on CUB-200-2011 shows that combining both attention levels outperforms either one alone.Adding both attention levels to “Original” improves accuracy by 5.01%, 5.40%, 5.67% and 2.4% across the four datasets.
  • Object-level and part-level attention: OPAM localizes objects and selects discriminative parts across Birds, Cars, Cats and Flowers, with explicit head and body semantics for several datasets.For flowers, parts instead denote discriminative regions, accommodating images containing one or multiple flowers.
  • Object-level and part-level attention: “Original+Part-level” outperforms “Object-level+Part-level” because original images retain outside-object and useful background information.The full “Original+Object-level+Part-level” combination benefits from complementary information among image, object and part views.

2) Effectivenesses of object-part spatial constraint model and part alignment:

The object-part spatial constraint model augments part alignment by enforcing representative, nonredundant part selection, targeting subtle local differences for subcategory discrimination.

  • Spatial constraints and part alignment: OPSC adds object and part spatial constraints to the previous part-alignment approach for discriminative part selection.Object spatial constraint promotes representativeness, while part spatial constraint reduces redundancy and enhances discrimination.
  • Spatial constraints and part alignment: Without object-part spatial constraints, selected parts contain more background noise, less object area and greater overlap, producing redundant information.The comparison is illustrated between part alignment alone and the combined OPSC+PA approach in Fig. 10.

3) Effectiveness of patch filtering:

Patch filtering selects object-relevant patches that provide multiple views and scales of the original images for ClassNet training, while the paper identifies more precise localization as future work.

  • Patch filtering: Patch filtering selects object-relevant candidate patches that provide multiple views and scales of original images for training ClassNet.The experiment compares fine-tuning on filtered patches (“ft-patches”) with fine-tuning only on the original image (“ft-original”).
  • Patch filtering: More effective and precise part localization is identified as future work for improving fine-grained representation and classification performance.
Loading 1704.01740v2…