PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Activation Constellations: Unsupervised Part Model Discovery with Convolutional Networks

Marcel Simon, Erik Rodner

arXiv:1504.08289v3cs.CV

TL;DR

Fine-grained recognition often depends on part or bounding-box annotations, motivating annotation-free part-model learning. The paper discovers spatial constellations among activation patterns from pre-trained CNNs and applies them to recognition and fine-tuning. It reports strong results across fine-grained and generic classification, including 81.0% on CUB200-2011 and 84.1% on Caltech-256.

  • Problem

    Fine-grained recognition needs localized part models, but existing approaches often rely on expensive part or bounding-box annotations.

  • Method

    The method uses activation maps from a pre-trained CNN as part proposals and estimates generative spatial constellation models from their predicted locations.

  • Results

    The approach improves state-of-the-art fine-grained recognition without ground-truth part or bounding-box annotations and reaches 81.0% on CUB200-2011 and 84.1% on Caltech 256.

  • Takeaways & Limitations

    The same unsupervised part-model approach applies to fine-grained and generic recognition and can guide data augmentation during fine-tuning.

  • Takeaways & Limitations

    The current method assumes that a single CNN channel corresponds to an object part, and the authors identify combining channels as a possible improvement.

Abstract

from arXiv · show

Part models of object categories are essential for challenging recognition tasks, where differences in categories are subtle and only reflected in appearances of small parts of the object. We present an approach that is able to learn part models in a completely unsupervised manner, without part annotations and even without given bounding boxes during learning. The key idea is to find constellations of neural activation patterns computed using convolutional neural networks. In our experiments, we outperform existing approaches for fine-grained recognition on the CUB200-2011, NA birds, Oxford PETS, and Oxford Flowers dataset in case no part or bounding box annotations are available and achieve state-of-the-art performance for the Stanford Dog dataset. We also show the benefits of neural constellation models as a data augmentation technique for fine-tuning. Furthermore, our paper unites the areas of generic and fine-grained classification, since our approach is suitable for both scenarios. The source code of our method is available online at http://www.inf-cv.uni-jena.de/part_discovery

1. Introduction

The paper learns part models without part or bounding-box annotations by finding spatial constellations of activation patterns from a pre-trained CNN. These models support weakly supervised part-based classification, fine-grained and generic recognition, and data augmentation.

  • Part models capture localized discriminative features important for fine-grained recognition, but conventional learning often requires part annotations or labeled bounding boxes.
  • The method uses intermediate CNN activation maps as implicit part detectors and learns constellations whose detectors fire consistently at relative locations.
  • Using pre-learned CNN detectors avoids hand-engineered proposals and explicit modeling of individual-part appearance, simplifying and accelerating part-model creation.
  • The approach improves state-of-the-art fine-grained recognition on three datasets, including CUB200-2011, without ground-truth part or bounding-box annotations.
  • The same approach supports generic object recognition and guided data augmentation during fine-tuning, including stronger discrimination than fine-tuning with ground-truth object bounding boxes.

2. Related work

Related work includes supervised and unsupervised part-constellation methods for fine-grained recognition and detection. The paper distinguishes its CNN-based, annotation-free proposal selection from approaches relying on manual parts, hand-engineered descriptors, bounding boxes, or object-specific pre-training.

  • Part constellation models: Supervised part-constellation methods rely on ground-truth part or bounding-box annotations, whereas unsupervised methods select useful parts from proposals without annotation.
  • Part constellation models: Earlier constellation models used manually annotated parts or generic SIFT interest points to represent relative part positions and scales.
  • Part constellation models: Compared with earlier constellation approaches, the paper uses CNN-based proposals with implicit correspondence and reduces inference complexity from exponential to linear in the number of modeled parts.
  • Part constellation models: Other unsupervised approaches select randomly initialized templates using co-occurrence, diversity, or fitness, but the paper argues spatial relations better select CNN-based proposals.
  • Fine-grained recognition with part models: Detection methods commonly require bounding boxes, while this approach learns a generative constellation model from proposals without bounding boxes or negative examples.
  • Fine-grained recognition with part models: Xiao et al. cluster CNN channels but require a classifier pre-trained for the target objects; this paper uses a CNN pre-trained on a weakly related object dataset.

3. Deep neural activation maps

Deep CNN channels provide implicit detectors for increasingly abstract patterns, including object parts. The method converts their activation information into higher-resolution localization maps and part proposals for feature extraction.

  • Later CNN layers respond to increasingly abstract patterns that can correspond to whole objects or object parts.
  • Each channel produces a two-dimensional activation map whose outputs can be interpreted as detection scores for object-part detectors.
  • Activation maps improve localization resolution over channel outputs, making them better suited for identifying image areas to which a channel is sensitive.
  • Part proposal locations are obtained by selecting the point of maximum activation in each channel’s map.
  • Each channel yields one two-dimensional part proposal, while classification extracts image patches at predicted part locations.
  • Implicit part detectors are learned automatically during CNN training, and proposal computation takes 110ms per image for 10 parts on a GPU-equipped standard PC.

4. Unsupervised part model discovery

The method discovers unsupervised part constellations by selecting CNN-generated proposals and fitting multi-view spatial models to their locations. Learning uses latent object roots, views, and visibility under a maximum-a-posteriori objective optimized iteratively.

  • The part model selects a small subset of robust CNN proposals for localized feature extraction and subsequent fine-grained classification.
  • Proposal locations are normalized two-dimensional channel maxima, with zero activation marking a hidden part.
  • A model view contains M selected proposals and corresponding shift vectors describing ideal offsets from a shared latent object root.
  • Latent view variables assign one active view per training image, while proposal-based visibility is supplied rather than estimated during learning.
  • The model is learned by maximum-a-posteriori estimation of proposal-selection, shift, view, and root parameters.
  • The optimization alternates over model and latent variables in an EM-like procedure, selecting proposals that fit each view and updating roots and shifts.

5. Experiments

The experiments evaluate unsupervised part discovery for data augmentation, fine-grained recognition without annotations, and generic classification. Across these settings, filtered part proposals improve fine-tuning and recognition while avoiding expensive part or bounding-box annotations.

  • Experimental scope: The experiments cover data augmentation for fine-tuning, fine-grained recognition without annotations, and generic image classification.The evaluation uses five fine-grained datasets and Caltech 256.
  • Experimental setup: The method uses class-specific part models with five views and ten parts per view, selecting frequently used parts for classification.Part models are learned repeatedly, and the model with the best objective value is retained.
  • Data augmentation: Filtered object proposals are built by retaining patches containing at least three predicted parts, yielding about 48 regions per CUB200-2011 image.The selected regions are used for fine-tuning without ground-truth bounding-box annotations.
  • Fine-grained recognition: 76.3% accuracy is achieved on NA birds with the constellation approach, compared with 63.9% for the no-parts GoogLeNet baseline.The passage states that the approach outperforms the provided baseline without part annotation.
  • Fine-grained recognition: 68.61% accuracy is reached on Stanford dogs with a weaker Krizhevsky architecture, improving by more than 16% over other non-deep architectures.A separate very deep GoogLeNet result of 75% is reported for comparison.
  • Generic classification: On Caltech 256, the method improves the global-feature baseline by 1% with AlexNet and 1.6% with VGG19.The paper contrasts this with more expensive dense evaluation and uses the same approach for generic classification.
  • Generic classification: Replacing semantic part detectors with generic detectors is sufficient to apply the fine-grained classification approach to generic image classification.The paper frames this as a way to extend part-based recognition beyond categories with shared semantic parts.

6. Conclusions

The paper presents unsupervised part selection for fine-grained and generic image classification, modeling useful proposals and their spatial relationships. It reports strong recognition results and identifies current modeling limitations.

  • The method selects generic parts without supervision for both fine-grained and generic image classification.A CNN pre-trained for classification supplies learned part detectors, and a constellation model analyzes predicted part locations across training images.
  • The resulting part model captures useful part proposals together with their spatial relationships across different object views.
  • 81.0% recognition is achieved on CUB200-2011 without additional annotation, alongside 84.1% accuracy on Caltech 256.
  • The approach also benefits data augmentation and generic object classification, extending its use beyond fine-grained recognition.
  • A current limitation is assuming that a single CNN channel corresponds to one object part.The authors propose combining channels to improve localization accuracy and jointly learning constellation models with subsequent classification.

7. Changelog

The changelog records results added for NA birds and an update to the camera-ready version.

  • Results for NA birds were added in version V3.
  • Version V2 updated the paper to the camera-ready version.
Loading 1504.08289v3…