PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Instance Retrieval: A Survey

Wei Chen, Yu Liu, Weiping Wang, Erwin Bakker, Theodoros Georgiou, Paul Fieguth, Li Liu, Michael S. Lew

arXiv:2101.11282v4cs.CV

TL;DR

Instance retrieval must find the same visual instance across changing conditions while searching very large image collections efficiently. This survey synthesizes deep-learning approaches through their extraction, embedding, aggregation, matching, and fine-tuning strategies. It reports broad progress, including supervised fine-tuning gains over off-the-shelf models, while noting persistent domain-shift and robustness challenges.

  • Problem

    Instance retrieval requires accurate and efficient search for the same instance under viewpoint, scale, illumination, weather, and background changes across large image collections.

  • Method

    The survey organizes deep-learning instance retrieval methods by feature extraction, feature embedding and aggregation, feature matching, and network fine-tuning.

  • Results

    Fine-tuning usually improves accuracy over off-the-shelf models; on Oxford-5k, one result increased from 66.9% to 81.5% with a single-margin Siamese loss.

  • Takeaways & Limitations

    The survey connects representative methods, benchmarks, evaluation protocols, challenges, and future directions for deep-learning-based instance retrieval.

  • Takeaways & Limitations

    Instance retrieval challenges remain incompletely addressed, including domain shift and limited invariance to viewpoint, scale, illumination, weather, and clutter.

Abstract

from arXiv · show

In recent years a vast amount of visual content has been generated and shared from many fields, such as social media platforms, medical imaging, and robotics. This abundance of content creation and sharing has introduced new challenges, particularly that of searching databases for similar content-Content Based Image Retrieval (CBIR)-a long-established research area in which improved efficiency and accuracy are needed for real-time retrieval. Artificial intelligence has made progress in CBIR and has significantly facilitated the process of instance search. In this survey we review recent instance retrieval works that are developed based on deep learning algorithms and techniques, with the survey organized by deep network architecture types, deep features, feature embedding and aggregation methods, and network fine-tuning strategies. Our survey considers a wide variety of recent methods, whereby we identify milestone work, reveal connections among various methods and present the commonly used benchmarks, evaluation results, common challenges, and propose promising future directions.

1 INTRODUCTION

Instance-level image retrieval searches for the same visual instance under changing conditions, where large-scale collections make both accuracy and efficiency essential. This survey organizes deep-learning progress around feature extraction, embedding and aggregation, and network fine-tuning.

  • IIR seeks images containing the same instance despite changes in distance, viewpoint, background, illumination, and weather.
  • Searching efficiently is as critical as searching accurately across collections containing thousands to billions of images.
  • The survey focuses on deep-learning methods for IIR, particularly retrieval accuracy and efficiency.
  • Its three themes are deep feature extraction, feature embedding and aggregation, and network fine-tuning for learning representations.
  • Accuracy challenges include invariance, distraction, discriminativity, and fine-tuning, while high-dimensional features create efficiency trade-offs.
  • Off-the-shelf approaches keep parameters fixed, whereas fine-tuned models update them toward retrieval and address domain shifts.

2 GENERAL FRAMEWORK OF IIR

Deep-learning IIR uses a pipeline of feature extraction, embedding and aggregation, and feature matching. Global features support efficient initial retrieval, while local features commonly rerank the resulting candidates.

  • Feature extraction uses whole images in single-pass methods or extracted regions in multiple-pass methods.
  • Activations may come from fully connected or convolutional layers and can be combined across layers or models.
  • Feature embedding maps local features into a higher-dimensional space, while aggregation summarizes them into global vectors.
  • Feature matching measures image similarity and returns a ranked list using global or local features.
  • Local-feature matching usually reranks globally matched images because separately storing and comparing local features adds memory and search costs.

3 RETRIEVAL WITH OFF-THE-SHELF DCNN MODELS

Off-the-shelf DCNNs provide fixed retrieval features with little engineering effort, but classification training can leave them mismatched to retrieval domains and objectives. The survey therefore contrasts fixed extraction with representation-improving strategies and identifies domain shift as a central limitation.

  • Off-the-shelf DCNNs use classification-trained networks with frozen parameters as retrieval feature detectors.
  • Off-the-shelf retrieval is organized around deep feature extraction, network feedforward schemes, and feature selection.
  • The approach faces a model-transfer or domain-shift challenge when classification-trained models are applied to retrieval tasks.
  • Classification features may remain adequate for class decisions while lacking capacity for retrieval, where feature matching itself is central.

3.1 Deep Feature Extraction

Deep feature extraction varies in feedforward strategy, receptive field, and feature fusion. Single-pass processing is efficient, while multiple-pass regional processing addresses transformation and clutter challenges at higher computational cost.

  • Network Feedforward Scheme: Single-pass methods feed the whole image once into an off-the-shelf model, making them relatively efficient.
  • Network Feedforward Scheme: Patch generation includes sliding windows, spatial pyramids, dense sampling, and region proposals.
  • Network Feedforward Scheme: Multiple-pass methods generate and process several patches, improving robustness to translation, scaling, rotation, and background distraction.
  • Deep Feature Selection: Fully connected features provide global descriptors but can include irrelevant information and lack local geometric invariance.
  • Deep Feature Selection: Convolutional features preserve local structural information and are more robust to image transformations because of their smaller receptive fields.
  • Deep Feature Selection: Embedding and aggregation methods map regional convolutional vectors into higher-dimensional spaces and summarize them into global descriptors.

MAC R-MAC

The section presents symbolic fragments involving c_ij, A, αβ, H, and W, but no interpretable method description is available.

  • The passage includes c_ij and A alongside αβ and other symbols.
  • The available fragments do not state the MAC R-MAC procedure or its role in instance retrieval.
  • A second fragment contains H, W, α, and a multiplication symbol.

GeM

GeM is presented among single-pass convolutional-feature aggregation methods, alongside approaches that combine or weight feature representations. The survey also discusses layer and model fusion as ways to exploit complementary feature capacities.

  • GeM: GeM applies a channel-wise powers operation to convolutional features as an aggregation method.
  • GeM: Feature fusion can combine fully-connected and convolutional features because they provide complementary global semantic and local cues.
  • GeM: Fusing two convolutional layers with one fully-connected layer achieves the best performance on the Holiday and UKBench datasets.
  • GeM: Intra-model fusion combines compatible deep models, whereas inter-model fusion combines models with differing structures and receptive fields.
  • GeM: Early fusion concatenates candidate-model features before metric learning, while alternative strategies learn separate metrics and combine rankings afterward.
  • GeM: Layer- and model-level fusion require choosing features with different description capacities and complementary properties.

3.2 Feature Embedding and Aggregation

Feature embedding and aggregation transform local or convolutional features into discriminative global or local descriptors for instance retrieval. The survey covers pooling, codebook-based embeddings, attention-like weighting, matching strategies, and their efficiency trade-offs.

  • 3.2 Feature Embedding and Aggregation: Feature embedding and aggregation target discriminativity while producing final global and/or local features for retrieving specific instances.
  • 3.2 Feature Embedding and Aggregation: Global fully-connected features require dimensionality reduction and normalization, whereas patch-level activations can be embedded with VLAD to improve rotation invariance.
  • 3.2 Feature Embedding and Aggregation: Sum or average pooling is less discriminative because it includes all activated outputs, while max pooling suits sparse features with low activation probability.
  • 3.2 Feature Embedding and Aggregation: Attention and weighted pooling emphasize relevant regions or reduce bursty descriptors, while global features enable efficient matching but not spatial verification.
  • 3.2 Feature Embedding and Aggregation: BoW maps local descriptors to nearest codebook centroids and concatenates normalized occurrence counts into a sparse representation.
  • 3.2 Feature Embedding and Aggregation: BoW and VLAD depend on centroid counts and produce high-dimensional or sparse vectors, creating efficiency and indexing challenges at scale.
  • 3.2 Feature Embedding and Aggregation: VLAD stores residual sums for K visual words, producing a d × K representation that captures first-order feature statistics.
  • 3.2 Feature Embedding and Aggregation: FV uses a Gaussian Mixture Model to encode first- and second-order statistics, capturing more statistics than BoW and VLAD.

4 RETRIEVAL VIA LEARNING DCNN REPRESENTATIONS

Deep learning retrieval systems fine-tune DCNN representations using classification or pairwise supervisory signals, with global and local constraints shaping similarity. The surveyed methods address domain adaptation, metric learning, sample mining, direct retrieval optimization, and emerging transformer alternatives.

  • Overview: Off-the-shelf deep features may be insufficient for accurate retrieval, motivating supervised or unsupervised fine-tuning before organizing features globally or locally.Supervised methods use labels or pairwise information; unsupervised methods are needed when suitable ground-truth labels are unavailable.
  • Fine-tuning via Classification Loss: Classification fine-tuning updates a pretrained DCNN with cross entropy, using class labels to enforce intra-class similarity and inter-class diversity.Image-level features compute logits, while local-region or patch descriptors may require aggregation afterward.
  • Fine-tuning via Pairwise Ranking Loss: Pairwise ranking methods learn metrics from similar and dissimilar samples through pairwise or triplet constraints, either globally on image features or locally on region proposals.Triplet training uses an anchor with one similar and one dissimilar sample; local supervision satisfies constraints over selected regions.
  • Fine-tuning via Pairwise Ranking Loss: Transformation-matrix learning freezes pretrained network parameters while optimizing a matrix that increases similarity for related pairs and decreases it for unrelated pairs.The approach is illustrated as the transformation-matrix scheme in Figure 7(b), with cosine similarity and a margin guiding training.
  • Fine-tuning via Pairwise Ranking Loss: Siamese loss is effective for learning similarity, while double-margin variants relax penalties on similar pairs and improve retrieval mAP.The double-margin formulation applies contrastive force to similar pairs only when their distance exceeds m1.
  • Discussion: Pairwise ranking requires careful sample mining and weighting, and cross entropy or pairwise losses may not optimize retrieval mAP directly; diffusion methods also remain computationally demanding.Transformers provide an alternative backbone with competitive instance-retrieval performance and learned feature aggregation, while diffusion-based unsupervised fine-tuning seeks more efficient computation.

5 STATE OF THE ART PERFORMANCE

Across six datasets, the survey finds steadily improving instance-retrieval performance, with gains associated with feature extraction, aggregation, embedding, fine-tuning, and preprocessing choices. Results vary across methods and benchmarks, so the authors emphasize general trends rather than direct method-to-method comparisons.

  • Datasets: The survey compares retrieval methods across commonly used benchmarks including UKBench, Holidays, Oxford-5k, INSTRE, and Google Landmarks Dataset.UKBench contains 10,200 object images; Holidays contains 1,491 images; Oxford-5k contains 5,062 images; INSTRE contains 28,543 images; GLD-v2 includes 4.1M training and 762k index images.
  • Overall trends: Performance improved from 2014 to 2020 as DCNN feature extraction, regional features, feature fusion, aggregation, embedding, and network fine-tuning advanced.The overview summarizes results over six datasets and identifies these developments as major contributors to progress.
  • Off-the-shelf models: 91.4% mAP on Holidays was achieved by combining three layers, compared with 80.2% for a nonfusion method.The comparison is reported for layer-level feature fusion versus the nonfusion method in.
  • Fine-tuned models: Fine-tuning improved Oxford-5k performance from 66.9% to 81.5% with a single-margin Siamese loss, while triplet-loss training reached 86.1% on Oxford-5k and 94.5% on Paris-6k.The survey also reports classification-based fine-tuning increasing Oxford-5k mAP from 55.7% to 83.8%.
  • Fine-tuned models: Unsupervised fine-tuning can approach or exceed supervised results when suitable feature embedding is used, despite difficulty mining relevance without labels.Manifold learning achieved 85.4% on Oxford-5k and 96.3% on Paris-6k, while an IME layer raised Oxford-5k precision from 83.8% to 92.0%.
  • Off-the-shelf models: Different aggregation methods applied to the same off-the-shelf DCNN produce different retrieval performance.The survey presents these results as a reference for selecting feature aggregation methods with convolutional features.

6 CONCLUSIONS AND OUTLOOKS

The survey synthesizes deep-learning advances in instance retrieval while organizing representative methods, benchmarks, evaluation protocols, and challenges. It concludes that important gaps remain in robustness, efficiency, labeling, adaptation, privacy, and video retrieval.

  • The survey presents a taxonomy of recent deep-learning instance-retrieval methods and discusses datasets, evaluation protocols, and state-of-the-art performance.
  • Accurate and robust feature representations: Future progress requires more accurate and robust representations for large intra-class variation and difficult objects beyond rigid, planar, textured benchmarks.
  • Accurate and robust feature representations: Instance retrieval systems must balance accuracy with adversarial robustness because deep networks can be fooled by nearly imperceptible perturbations.
  • Compact and efficient deep representations: Large, energy-hungry DCNNs remain difficult to deploy on mobile and wearable devices, motivating compact, efficient, reusable representations.
  • Learning with fewer labels and continual learning: Practical instance retrieval also needs learning with fewer labels and continual learning under noisy, long-tailed, non-stationary data.
  • Privacy-aware and video instance retrieval: Privacy-aware systems and video instance retrieval are additional directions motivated by personalized search, surveillance, and the growth of video data.
Loading 2101.11282v4…