Neural Codes for Image Retrieval

Artem Babenko; Anton Slesarev; Alexandr Chigorin; Victor Lempitsky

Source-linked AI summary

Neural Codes for Image Retrieval

Artem Babenko, Anton Slesarev, Alexandr Chigorin, Victor Lempitsky

arXiv:1404.1777v2Apr 7, 2014cs.CV

TL;DR

The paper asks whether CNN activations can serve as effective image-retrieval descriptors despite being learned for classification. It evaluates pretrained and retrained neural codes, then studies PCA and discriminative compression, finding strong retrieval performance and robust short codes, with low-resolution inputs limiting accuracy.

Problem
The paper investigates the quantitative retrieval performance of CNN features learned for classification and how task adaptation and compression affect those descriptors.
Method
The authors evaluate holistic neural codes from CNN layers, retrain networks on related image datasets, and compare PCA with discriminative dimensionality reduction for compact codes.
Results
Neural codes perform competitively across retrieval benchmarks, PCA preserves accuracy at 128 dimensions, and discriminative reduction reaches 0.368 mAP on Oxford Buildings with 16-dimensional features.
Takeaways & Limitations
Neural codes provide compact holistic descriptors whose retrieval accuracy is comparatively resilient to compression, while related-data retraining can improve matching retrieval datasets.
Takeaways & Limitations
Resizing all images to 224×224 loses discriminative texture information and leaves room for retrieval-accuracy improvement.

Abstract

from arXiv · show

It has been shown that the activations invoked by an image within the top layers of a large convolutional neural network provide a high-level descriptor of the visual content of the image. In this paper, we investigate the use of such descriptors (neural codes) within the image retrieval application. In the experiments with several standard retrieval benchmarks, we establish that neural codes perform competitively even when the convolutional neural network has been trained for an unrelated classification task (e.g.\ Image-Net). We also evaluate the improvement in the retrieval performance of neural codes, when the network is retrained on a dataset of images that are similar to images encountered at test time. We further evaluate the performance of the compressed neural codes and show that a simple PCA compression provides very good short codes that give state-of-the-art accuracy on a number of datasets. In general, neural codes turn out to be much more resilient to such compression in comparison other state-of-the-art descriptors. Finally, we show that discriminative dimensionality reduction trained on a dataset of pairs of matched photographs improves the performance of PCA-compressed neural codes even further. Overall, our quantitative experiments demonstrate the promise of neural codes as visual descriptors for image retrieval.

1 Introduction

The paper quantitatively evaluates CNN activations as neural codes for image retrieval, including pretrained, retrained, and compressed representations. Neural codes perform well across retrieval benchmarks, with task-specific retraining and discriminative compression providing further gains.

1 Introduction: The study quantitatively evaluates CNN features, termed neural codes, for image retrieval across several retrieval setups.The evaluation covers features from a CNN trained for Image-Net classification and their variations.
1 Introduction: Neural codes from an Image-Net-trained CNN perform well on INRIA Holidays, Oxford Buildings, Oxford Building 105K, and UKB, though not better than other state-of-the-art holistic features.The best standard-dataset performance comes from the middle of the fully connected layer hierarchy, and the best layer varies by retrieval setup.
1 Introduction: Retraining on related images improves retrieval on matching datasets but can degrade performance on unrelated datasets.Landmark retraining improves Holidays and Oxford Buildings while degrading UKB; Multi-view RGB-D retraining improves UKB while other datasets degrade or remain unchanged.
1 Introduction: PCA compresses neural codes to 128 dimensions with virtually no retrieval-accuracy loss and less degradation than other holistic descriptors.This compression behavior makes neural codes attractive for large-scale retrieval, where descriptor memory is a major bottleneck.
1 Introduction: At 16 dimensions, PCA causes considerable degradation, while discriminative dimensionality reduction achieves 0.368 mAP on Oxford Buildings.The discriminative reduction is trained on around 900K matched photograph pairs and substantially outperforms PCA for very short codes.

2 Related work

Related work spans classifier-derived descriptors, aggregated local-feature representations, dimensionality reduction, and CNN-based retrieval. This paper differs from concurrent neural-code work by emphasizing holistic, compact, and compressed descriptors.

2 Related work: Earlier work used responses from discriminatively trained multiclass or binary classifiers as image descriptors, including classemes and face descriptors.These shallow approaches repurpose classifier outputs for retrieval or specialized description tasks.
2 Related work: State-of-the-art holistic descriptors aggregate local gradient-based features, including Fisher Vectors and triangulation embedding.Some alternatives achieve high performance but have substantially larger memory footprints than uncompressed Fisher vectors.
2 Related work: Prior dimensionality-reduction work learned low-dimensional retrieval representations from Image-Net on top of hand-crafted SIFT and local color histogram features.Its architecture resembles the investigated deep approach but derives representations from hand-crafted descriptors rather than CNN neural codes.
2 Related work: The paper places its discriminative dimensionality reduction within existing dimensionality-reduction and metric-learning research.The method is described as a variant similar to prior discriminative dimensionality-reduction work.
2 Related work: Concurrent work extracted many neural codes per image with a jumping-window CNN, whereas this paper maps each whole image to one compact vector.The paper also studies compressed holistic descriptors, emphasizing computational and memory efficiency.
2 Related work: Related work also examined retraining Image-Net-derived networks for smaller classification datasets.This paper instead investigates how CNN retraining on different datasets affects retrieval performance.

3 Using Pretrained Neural Codes

The paper evaluates ImageNet-trained CNN activations as neural codes for image retrieval across four benchmarks. Layer 6 performs best overall, although performance varies by query and remains competitive rather than superior to state-of-the-art holistic descriptors.

Layer comparison: Layer 6 performs best overall, but its advantage is not uniform across queries.Simple code combinations perform worse than L6(I)-codes alone, while nonlinear combinations provide only marginal improvement.
Comparison with holistic descriptors: L6(I)-codes perform in the same ballpark as state-of-the-art holistic descriptors but are not superior to them.Their strong performance is notable despite the disparity between ImageNet classification and the retrieval tasks.
Layer comparison: Layer 5 gives the best result in one Holidays retrieval example, presumably because it relies more on low-level texture than high-level concepts.The query is the left-most image in each row, and correct answers are outlined in green.
Layer comparison: Layer 7 gives the best result in another Holidays retrieval example, presumably because it relies more on high-level concepts.The query is the left-most image in each row, and correct answers are outlined in green.

4 Retrained neural codes

The paper adapts neural codes by retraining the CNN on datasets whose classes and image statistics resemble target retrieval data. Related retraining substantially benefits matching domains, while mismatched domains can lose performance.

Adaptation datasets: Retraining on related datasets is designed to improve neural-code retrieval performance by matching training image statistics and classes to test-time data.The paper collects a landmark dataset and also evaluates retraining on turntable images of household objects.
Landmarks dataset: The collected Landmarks dataset uses landmark Wikipedia pages, image-search results, and manual filtering into relevance categories.The CNN is trained with 672 landmark output classes and initialized from the ImageNet model.
Retrieval results: Landmark retraining produces a very large improvement on Oxford and Oxford 105K, with a smaller but still considerable improvement on Holidays.Adapted L6(I) features on Holidays outperform previously published holistic-feature systems unless much higher dimensionality is used.
Comparison with holistic descriptors: Table 1 compares full-size holistic descriptors up to 32K dimensions and reports that neural codes benefit from retraining on related datasets.The comparison covers Landmarks for Oxford and Holidays and turntable sequences for UKB.
Retrieval results: Retraining on Landmarks does not improve SUN retrieval, apparently because SUN classes represent scene usage types rather than the same object.Landmark, Holidays, and Oxford classes correspond more closely to shared objects such as buildings.
Retrieval results: After Landmarks retraining, UKB performance drops because its small-object indoor-view classes are more similar to some ImageNet classes.The paper therefore evaluates turntable-sequence retraining for UKB.

5 Compressed neural codes

The paper evaluates PCA and discriminative dimensionality reduction for compact neural codes, finding that PCA preserves retrieval quality well while learned projections improve extreme compression.

PCA compression: PCA compression leaves neural-code performance almost unaffected through 256 dimensions, with graceful degradation under more extreme compression.PCA is trained on 100,000 random images from the Landmark dataset.
Discriminative dimensionality reduction: Discriminative dimensionality reduction learns a low-rank projection that brings matched-image codes closer and separates unmatched codes, adding tolerance to viewpoint changes.The training uses diverse same-object image pairs collected from the Landmark dataset.
PCA compression: 128-dimensional landmark-retrained neural codes establish new state-of-the-art among low-dimensional global descriptors on Holidays, Oxford, and Oxford 105K.The advantage of retrained codes persists across PCA compression rates.
Discriminative dimensionality reduction: Discriminative reduction improves over PCA particularly for extreme compression of original neural codes on Oxford.Table 4 compares the two strategies, with the largest gain at 16 dimensions.
Discriminative dimensionality reduction: Discriminative reduction provides no additional improvement for neural codes retrained on Landmarks, presumably because retraining and reduction use overlapping training data.For D = 64 and 128, the projection is learned after PCA compression to 1024 dimensions to reduce overfitting.

6 Discussion

The discussion finds that neural codes transfer well from classification to retrieval and improve with task-related retraining. Compression remains effective, although image downsampling limits retrieval accuracy and leaves room for improvement.

Transfer and retraining: Neural codes perform well despite classification and retrieval datasets being quite different, and retraining on related photographs further improves performance.The discussion presents this as a central conclusion from the retrieval experiments.
Limitations: 224×224 image downsampling discards potentially discriminative texture information and leaves room to improve retrieval accuracy.Experiments with Fisher Vectors suggest a 0.03 mAP drop on Holidays under similar conditions.
Layer selection: The best retrieval performance occurs two layers below the network outputs, an effect that persists after retraining on related images.The authors speculate that top layers are overly tuned to classification while bottom layers lack sufficient nuisance invariance.
Compression: PCA affects neural-code performance less than VLAD, Fisher Vectors, or triangulation embedding, possibly because deeper CNN codes retain less irrelevant information.This explanation is presented as a possible account of the more effective compression.
Compression: Compressed neural codes using PCA or discriminative reduction achieve very short codes with very good, state-of-the-art performance.The discussion summarizes the paper’s compression findings without restricting the conclusion to a single dataset.
Future directions: Training the full architecture directly on matched image pairs is identified as a possible direction for obtaining neural codes without classification-based training.The authors also note that automatically collecting a sufficiently diverse training collection would itself be challenging.