PaperlayerView exact revision on arXiv

Source-linked AI summary

Selective Convolutional Descriptor Aggregation for Fine-Grained Image Retrieval

Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, Zhi-Hua Zhou

arXiv:1604.04994v2cs.CV

TL;DR

Fine-grained image retrieval seeks same-variety images without supervision, despite subtle category differences. SCDA unsupervisedly localizes main objects, aggregates selected descriptors, and produces compact features; it outperforms baselines for fine-grained retrieval and is comparable to state-of-the-art general retrieval methods.

  • Problem

    Fine-grained image retrieval must identify same-variety images without supervision, even though categories differ only subtly.

  • Method

    SCDA unsupervisedly localizes the main object, removes background or noise, and aggregates selected deep descriptors into a short feature vector without additional learning.

  • Results

    SCDA achieves the best or outperforms baseline retrieval results for fine-grained images and obtains comparable results on general-purpose image retrieval datasets.

  • Takeaways & Limitations

    SCDA features exhibit well-defined visual attributes, including subtle object attributes, which may explain their high fine-grained retrieval accuracy.

Abstract

from arXiv · show

Deep convolutional neural network models pre-trained for the ImageNet classification task have been successfully adopted to tasks in other domains, such as texture description and object proposal generation, but these tasks require annotations for images in the new domain. In this paper, we focus on a novel and challenging task in the pure unsupervised setting: fine-grained image retrieval. Even with image labels, fine-grained images are difficult to classify, let alone the unsupervised retrieval task. We propose the Selective Convolutional Descriptor Aggregation (SCDA) method. SCDA firstly localizes the main object in fine-grained images, a step that discards the noisy background and keeps useful deep descriptors. The selected descriptors are then aggregated and dimensionality reduced into a short feature vector using the best practices we found. SCDA is unsupervised, using no image label or bounding box annotation. Experiments on six fine-grained datasets confirm the effectiveness of SCDA for fine-grained image retrieval. Besides, visualization of the SCDA features shows that they correspond to visual attributes (even subtle ones), which might explain SCDA's high mean average precision in fine-grained retrieval. Moreover, on general image retrieval datasets, SCDA achieves comparable retrieval results with state-of-the-art general image retrieval approaches.

I. INTRODUCTION

Fine-grained image retrieval seeks same-variety matches without supervision despite subtle category differences and changing appearance. SCDA addresses this by unsupervised object localization, descriptor aggregation, and dimensionality reduction, achieving strong retrieval results across fine-grained and general datasets.

  • Motivation: Strong fine-grained recognition systems commonly require expensive annotations such as object or part bounding boxes.Prior unsupervised retrieval attempts also required image-level labels to construct training triplets.
  • Motivation: Fine-grained retrieval must distinguish varieties with subtle differences while accommodating changes in pose, scale, rotation, color, and angle.Unlike general retrieval, it searches for images of the same type rather than merely similar visual content.
  • SCDA: SCDA uses only an ImageNet-pre-trained CNN to localize the main object without image labels, bounding boxes, object proposals, or additional learning.It selects useful convolutional descriptors, removes background or noise, and aggregates and reduces them into a vector representation.
  • SCDA: SCDA combines multiple CNN layers and dimensionality reduction to produce a shorter, more accurate representation than existing deep learning-based methods.The method is evaluated on six fine-grained datasets and standard general-purpose retrieval datasets.
  • Results: SCDA achieves the best reported fine-grained retrieval results and comparable results on general-purpose retrieval datasets.Its compressed features also correspond to visual attributes, including subtle ones, which might explain its fine-grained retrieval success.

II. RELATED WORK

Prior retrieval methods aggregate deep activations without evaluating descriptor usefulness, while fine-grained recognition typically relies on labels or stronger annotations. SCDA instead targets unsupervised fine-grained retrieval by selecting descriptors that correspond to the main object.

  • Deep Learning for Image Retrieval: Deep image retrieval methods explored descriptors and aggregation schemes, including multi-scale pooling and VLAD-style encoding.These approaches generally encode extracted CNN activations into a single representation.
  • Deep Learning for Image Retrieval: Existing deep retrieval approaches do not evaluate descriptor usefulness, whereas SCDA selects descriptors and removes background or noise through unsupervised object localization.SCDA also introduces retrieval-oriented practices for its representation.
  • Fine-Grained Image Tasks: Fine-grained recognition methods learn discriminative features, align objects, or use parts, but often depend on image-level, object, or part annotations.Such annotations are described as expensive and unrealistic in many applications.
  • Fine-Grained Image Tasks: Few prior works addressed unsupervised fine-grained retrieval, and Deep Ranking required image-level labels to build triplets.The cited approach therefore was not unsupervised and did not scale well to large-scale retrieval.
  • Fine-Grained Image Tasks: SCDA differs from prior fine-grained image search in both its use of pre-trained CNNs and its single-species database setting.The paper presents this as the first deep-learning attempt at fine-grained image retrieval.

III. SELECTIVE CONVOLUTIONAL DESCRIPTOR AGGREGATION

SCDA represents an image through convolutional descriptors arranged in an activation tensor. The method’s descriptor-selection stage precedes aggregation into a compact retrieval feature.

  • Method Overview: SCDA is organized around descriptor selection followed by feature aggregation.The method section introduces notation, the selection process, and aggregation details in that order.
  • Preliminary: A feature map is one channel’s convolution result, activations are all channel maps, and a descriptor is the d-dimensional vector at one tensor cell.These definitions distinguish spatial feature maps from per-location channel vectors.
  • Preliminary: For an input image of size H × W, a convolution layer produces an h × w × d tensor containing h × w spatial cells with d-dimensional descriptors.The same tensor can also be viewed as d feature maps of size h × w.
  • Preliminary: With VGG-16 and a 224×224 input, pool5 yields a 7×7×512 tensor, equivalent to 49 descriptors of dimension 512.It also comprises 512 feature maps, each sized 7×7.

B. Selecting Convolutional Descriptors

SCDA identifies object-relevant convolutional descriptors by aggregating activations across channels, thresholding positions, and retaining the largest connected component before forming the representation.

  • Descriptor selection: SCDA uses a pre-trained CNN to find useful convolutional features without fine-tuning or supervision from the target fine-grained dataset.The method discards irrelevant and noisy image regions while localizing the main object.
  • Aggregation map: Activations from the 512 pool5 feature maps are summed across depth into an h × w aggregation map.Each position’s summed response indicates how likely its corresponding region is part of the object.
  • Mask construction: Positions whose aggregation response exceeds the mean are marked in a binary mask map M.The threshold is the mean activation across all h × w positions.
  • Noise removal: The largest connected component f M removes smaller activated regions caused by background noise, retaining the main object region.Examples include removing plant, cloud, and grass regions from otherwise activated masks.
  • Descriptor selection: Descriptors are selected where f M is one and then aggregated into the final retrieval representation.The selected regions are visualized as red areas with black boundaries, and irregular regions can be enclosed by minimum rectangles for localization.

2) Qualitative Evaluation:

Qualitative evaluation shows that SCDA’s unsupervised localization often approximates annotated object regions, while the connected-component filter substantially improves localization accuracy.

  • Bounding-box evaluation: Predicted bounding boxes generally approximate ground-truth boxes, with especially close agreement reported for many Aircrafts and Cars images.Some predictions contain less background or cover both dogs, but fine details such as birds’ tails may be missed.
  • Localization results: 78.86% object localization accuracy is reported for Stanford Dogs, compared with 94.91% for Aircrafts and 90.96% for Cars.These results are reported using the percentage of correctly localized whole-object boxes.
  • Localization results: Unsupervised whole-object localization is reported as slightly lower or comparable to methods using ground-truth boxes and parts annotations.The comparison uses PCP, defined here as whole-object boxes with more than 50% intersection-over-union.
  • Ablation: Removing largest-component filtering reduces localization accuracy to 45.18%, 68.67%, 59.83% and 79.36% on CUB200-2011, Stanford Dogs, Aircrafts and Cars, respectively.The reported comparison supports retaining f M rather than the unfiltered mask M.

C. Aggregating Convolutional Descriptors

The paper compares encoding and pooling strategies for selected convolutional descriptors, finding that concatenated average- and max-pooling is the most effective aggregation scheme for fine-grained retrieval.

  • C. Aggregating Convolutional Descriptors: The study compares VLAD, Fisher Vector, global average-pooling, and global max-pooling for aggregating selected convolutional descriptors.VLAD and Fisher Vector use higher-dimensional encodings, while pooling produces 1 × d representations.
  • C. Aggregating Convolutional Descriptors: For VLAD and Fisher Vector, square-root and ℓ2 normalization are applied; pooling features use ℓ2 normalization before cosine-similarity retrieval.The experiments compare these normalized representations on two datasets.
  • C. Aggregating Convolutional Descriptors: Larger VLAD and Fisher Vector representations lead to lower retrieval accuracy, while simpler pooling methods perform better.The authors relate the weaker encoding results to the small number of selected deep descriptors per image.
  • C. Aggregating Convolutional Descriptors: The proposed avg&maxPool representation concatenates global average- and max-pooled features and consistently outperforms the other aggregation methods.This concatenated representation is used as the SCDA feature for the whole fine-grained image.

D. Multiple Layer Ensemble

SCDA extends its representation by combining features from pool5 and relu5_2, using jointly supported object masks and optional horizontal-flip augmentation.

  • D. Multiple Layer Ensemble: A second SCDA feature is extracted from relu5_2 and combined with the pool5 feature through a multiple-layer ensemble.The relu5_2 layer is three layers before pool5 in VGG-16.
  • D. Multiple Layer Ensemble: The relu5_2 mask is intersected with the upsampled largest pool5 connected component to retain descriptors supported by both masks.The relu5_2 activations detect the bird more accurately but also contain more noisy parts than pool5.
  • D. Multiple Layer Ensemble: Adding more layers such as pool4 improves top-1 mAP by only about 0.01%∼0.04% while substantially increasing feature dimensionality.The authors therefore retain the proposed SCDA representations rather than adding more layers.

IV. EXPERIMENTS AND RESULTS

Experiments evaluate SCDA on six fine-grained and two general-purpose retrieval datasets against baseline and state-of-the-art methods, with SCDA variants outperforming the compared fine-grained approaches.

  • IV. EXPERIMENTS AND RESULTS: The evaluation covers six fine-grained datasets and two standard general-purpose retrieval datasets.The fine-grained datasets include CUB200-2011, Stanford Dogs, Oxford Flowers 102, Oxford-IIIT Pets, Aircrafts, and Cars.
  • IV. EXPERIMENTS AND RESULTS: The experiments report top-1 and top-5 mAP and compare SCDA with handcrafted, deep-learning, and state-of-the-art general retrieval baselines.The general retrieval approaches include SPoC, CroW, and R-MAC.
  • IV. EXPERIMENTS AND RESULTS: SCDA substantially outperforms the unselected pool5 representation, while selectVLAD and selectFV use larger features but obtain lower retrieval mAP.The pool5 baseline concatenates average- and max-pooled descriptors into a 1,024-d representation.
  • IV. EXPERIMENTS AND RESULTS: General-purpose retrieval methods SPoC, CroW, and R-MAC do not achieve satisfactory results on fine-grained images.The authors conclude that these general retrieval methods cannot be directly applied to FGIR.
  • IV. EXPERIMENTS AND RESULTS: SCDA flip+ is the best among the compared methods, while horizontal flipping and multiple-layer ensembling improve retrieval performance over SCDA.SCDA remains the preferred option when low-dimensional features are required.

1) Post-Processing:

Post-processing compresses SCDA flip+ with dimensionality-reduction methods, with 512-d SVD whitening selected as the optimal FGIR representation despite unsupervised failures on tiny visual differences.

  • 1) Post-Processing:: The study compares SVD, PCA, PCA whitening, and SVD whitening for compressing SCDA flip+ to 256-d and 512-d.PCA whitening is omitted because its results were much worse than those of the other methods.
  • 1) Post-Processing:: 512-d SVD+whitening improves retrieval by 2%∼4% over the original SCDA flip+ on CUB200-2011 and Oxford Flowers.The same feature improves retrieval by 7%∼13% on Aircrafts and Cars.
  • 1) Post-Processing:: The authors select 512-d SVD+whitening as the optimal FGIR feature because it generally outperforms the other compressed SCDA representations with fewer dimensions.Retrieval examples are presented using this representation.
  • 1) Post-Processing:: Successful retrievals handle differing backgrounds, whereas failures occur when query and returned images differ only by tiny visual details that unsupervised localization cannot accurately detect.The examples span birds, animals, flowers, aircrafts, and cars.
  • 1) Post-Processing:: After SVD and whitening, leading SCDA dimensions correspond directly to discriminative semantic visual properties useful for retrieval.Examples include color, shape, pose, viewpoint, and other subtle attributes across fine-grained categories.

D. General Image Retrieval Results

SCDA remains effective beyond fine-grained retrieval, achieving comparable general-purpose retrieval performance while supporting classification experiments with compact representations.

  • General image retrieval: SCDA flip (512-d) achieved the highest mAP among the proposed features on the general image retrieval datasets.The features were compressed using SVD whitening.
  • General image retrieval: Compressed SCDA flip (512-d) outperformed SPoC and CroW and was comparable with R-MAC on general-purpose retrieval.
  • Fine-grained classification: SCDA classification accuracy was comparable to or better than methods trained with strong supervised annotations.This result concerns the directly fine-tuned SCDA strategy.
  • Fine-grained classification: The end-to-end SCDA model achieved slightly lower classification accuracy than SCDA fine-tuning but used only 15.53M parameters.Its architecture contains no fully connected layers.
  • Fine-grained classification: Without fine-tuning, the 512-d SCDA feature achieved 73.7% classification accuracy on CUB200-2011, versus 56.5% for CroW.The comparison used only the max-pooling part of SCDA for fairness.

F. Additional Experiments on Completely Disjoint Classes

Additional experiments support SCDA’s generalization across disjoint fine-grained classes and highlight trade-offs among accuracy, compression, representation design, and inference speed.

  • Disjoint-class generalization: SCDA flip+ outperformed other baseline methods on the Moth dataset, whose 2,120 images span 675 highly similar classes disjoint from ImageNet.Because some classes contain fewer than five images, only top-1 mAP was reported.
  • Inference speed: SCDA has computational speed comparable with CroW and is significantly faster than R-MAC, while running about 1 frame/sec slower than SPoC.SCDA flip+ is recommended when retrieval accuracy is preferred; SCDA remains scalable for large datasets.
  • Representation design: Concatenating max- and average-pooled convolutional descriptors achieved the best fine-grained retrieval performance.
  • Representation design: Convolutional descriptor representations outperformed fully connected features across aggregation methods for fine-grained retrieval.
  • Compression and feature quality: SVD whitening reduced SCDA dimensionality and improved retrieval performance, with large gains reported for Aircrafts and Cars.The compressed feature also described subtle attributes of the main objects.

V. CONCLUSIONS

The paper concludes that unsupervised SCDA can reuse an ImageNet-pretrained CNN for fine-grained retrieval by selecting, aggregating, and compressing localized descriptors.

  • SCDA uses an ImageNet-pretrained CNN without additional learning or supervision for fine-grained image retrieval.
  • SCDA localizes the main object and aggregates selected deep descriptors into a short feature vector.
  • SCDA outperformed baseline and general image retrieval methods on fine-grained retrieval while achieving comparable performance on standard general-purpose datasets.
  • SCDA features exhibited well-defined semantic visual attributes, which may explain their high fine-grained retrieval accuracy.
  • Future work includes using selected descriptor weights to identify object parts and exploring unsupervised object segmentation.
Loading 1604.04994v2…