PaperlayerView exact revision on arXiv

Source-linked AI summary

A Decade Survey of Content Based Image Retrieval using Deep Learning

Shiv Ram Dubey

arXiv:2012.00641v2cs.CVcs.AIcs.MM

TL;DR

Content-based image retrieval must rank database images against a query using representative features, while earlier hand-designed descriptors provide limited image characterization. This paper surveys deep-learning developments from 2011 to 2020, organizes them through a multi-axis taxonomy, and synthesizes major approaches, trends, and benchmark-oriented analyses. The survey identifies deep learning, including newer generative, autoencoder, and reinforcement-learning models, as driving progress in image retrieval.

  • Problem

    Content-based image retrieval requires representative features that support discriminative, robust, and efficient similarity ranking, while hand-designed feature engineering has limited image characterization.

  • Method

    The paper comprehensively surveys deep-learning image retrieval from 2011 to 2020 using taxonomies covering supervision, networks, descriptor types, retrieval types, evolution, and other aspects.

  • Results

    The survey reports that deep-learning models are driving image-retrieval progress, with generative adversarial, autoencoder, and reinforcement-learning networks showing superior performance.

  • Takeaways & Limitations

    The survey provides chronological overviews, cross-perspective analyses, benchmark comparisons, and insights intended to help researchers understand progress and make choices.

Abstract

from arXiv · show

The content based image retrieval aims to find the similar images from a large scale dataset against a query image. Generally, the similarity between the representative features of the query image and dataset images is used to rank the images for retrieval. In early days, various hand designed feature descriptors have been investigated based on the visual cues such as color, texture, shape, etc. that represent the images. However, the deep learning has emerged as a dominating alternative of hand-designed feature engineering from a decade. It learns the features automatically from the data. This paper presents a comprehensive survey of deep learning based developments in the past decade for content based image retrieval. The categorization of existing state-of-the-art methods from different perspectives is also performed for greater understanding of the progress. The taxonomy used in this survey covers different supervision, different networks, different descriptor type and different retrieval type. A performance analysis is also performed using the state-of-the-art methods. The insights are also presented for the benefit of the researchers to observe the progress and to make the best choices. The survey presented in this paper will help in further research progress in image retrieval using deep learning.

I. INTRODUCTION

Content-based image retrieval ranks database images by similarity to a query, making discriminative, robust, and efficient feature representation central to performance. This survey reviews the decade-long transition from hand-engineered descriptors toward deep-learning approaches and organizes the literature across multiple methodological perspectives.

  • Image retrieval ranks database images by similarity to a query, so similarity computation should be discriminative, robust, and efficient.
  • Hand-engineered descriptors manually represent visual cues such as color, texture, shape, and gradients, but their image-characterization performance was limited.
  • The survey also discusses distance metric learning, which has been extensively explored for representing feature vectors and supporting image retrieval.
  • Deep learning shifted feature representation toward hierarchical, data-learned features, with CNNs replacing traditional hand-engineered representation pipelines for image data.
  • The survey categorizes deep-learning retrieval methods by supervision, network type, descriptor type, retrieval type, and other aspects, providing a taxonomy and broad coverage.

II. BACKGROUND

The background introduces retrieval evaluation, benchmark datasets, and the decade-long shift from hand-engineered descriptors toward deep learning features, hash codes, and retrieval architectures.

  • Evaluation Measures: Precision, recall, f-score, and especially mean average precision are common metrics for judging image retrieval performance.Precision measures correctly retrieved images among retrieved images, recall measures correctly retrieved images among relevant images, and f-score is their harmonic mean.
  • Datasets: Benchmark datasets span object categories, scenes, digits, apparel, landmarks, and large-scale social-photography collections.Examples include Sun397, MIRFlickr-1M, MNIST, SVHN, UT-ZAP50K, Yahoo-1M, and Google Landmarks.
  • Datasets: Dataset choice depends on the application scenario, while most datasets are class-imbalanced because they are collected in unconstrained environments.CIFAR-10 and MNIST are identified as exceptions with equal samples per category.
  • Evolution: CNN activations became neural-code descriptors, often compressed with PCA, while later methods learned hash codes using ranking, quantization, similarity, and other objectives.The chronological development includes autoencoders, CNNs, region-based descriptors, deep hashing, and progressive or incremental hashing models.
  • Evolution: Deep learning retrieval methods evolved from early autoencoder and CNN features toward objectives and architectures designed for discriminative, robust, and compact representations.The survey describes weaker discriminative ability and quantization information loss in many 2011–2015 methods, followed by substantial growth from 2016 to 2020.

IV. DIFFERENT SUPERVISION CATEGORIZATION

The survey categorizes retrieval methods by supervision, covering supervised learning and related hashing frameworks that exploit labels, pairwise information, or other training signals.

  • Supervision Taxonomy: The supervision taxonomy includes supervised, unsupervised, semi-supervised, weakly-supervised, pseudo-supervised, and self-supervised approaches.These categories organize methods according to how supervision is obtained for feature learning and retrieval.
  • Supervised Approaches: Supervised models learn class-specific and discriminative features, with CNNs used to generate image representations, hash codes, and class labels.Reported applications include MNIST, CIFAR-10, and NUS-WIDE.
  • Supervised Approaches: Deep supervised hashing frameworks combine classification information, pairwise labels, retrieval objectives, and quantization constraints.Examples include DSDH, SSDH, and scalable hashing designs that reduce real-to-binary code loss, balance bits, and decrease bit redundancy.
  • Supervised Approaches: Later supervised methods extend retrieval learning through asymmetric hashing, deep product quantization, multi-level hashing, and angular hashing losses.Supervised deep feature embedding is also combined with hand-crafted features in one cited approach.

B. Unsupervised Approaches

Unsupervised retrieval methods avoid class labels by imposing constraints on hash codes or reconstructed outputs, but the survey reports generally lower performance than supervised approaches.

  • Unsupervised Approaches: Unsupervised models learn retrieval features without class labels by constraining hash codes and/or generated outputs.These constraints support representation learning when large-scale labeled data are difficult to obtain.
  • Unsupervised Approaches: DeepBit and related methods use quantization, balanced-bit, and bit-independence constraints for unsupervised image retrieval.Other approaches add discriminative clustering, hard positive and negative examples, or triplet-based objectives.
  • Unsupervised Approaches: Unsupervised retrieval has also been applied to patch representations, triplet hashing, scalable retrieval, and remote-sensing images through deep transfer learning.The cited methods use patch convolutional kernels, pseudo-structured triplets, and transfer learning for specialized settings.
  • Comparison: Unsupervised methods generally perform worse than supervised approaches despite avoiding the need for labeled data.This performance trade-off motivates semi-supervised, weakly-supervised, pseudo-supervised, and self-supervised alternatives.
  • Alternative Supervision: Semi-supervised methods combine labeled and unlabeled data, whereas weakly-supervised, pseudo-supervised, and self-supervised methods use weaker labels, generated labels, or temporal and generative signals.Examples include adversarial learning, teacher-student training, multimodal hashing, word2vec tag embeddings, clustering-based pseudo labels, and temporal hashing.

D. Summary

The survey summarizes supervision objectives and presents network architectures used for deep learning retrieval, emphasizing CNNs, autoencoders, and related networks across the 2011–2020 evolution.

  • Summary: Supervised approaches use class-specific semantic information and generally achieve better performance by learning fine-grained class information.Classification error is combined with objectives related to hash-code generation.
  • Summary: Unsupervised approaches constrain hash codes or reconstruct data, while semi-supervised methods learn from labeled and unlabeled data across different modalities.Pseudo-supervised methods generate labels with other methods, and self-supervised methods use temporal or generative information.
  • Network Types: The survey’s architecture taxonomy includes CNNs, autoencoders, siamese and triplet networks, recurrent networks, generative adversarial networks, attention networks, and reinforcement learning networks.The chronological architecture view covers methods developed from 2011 to 2020.
  • Summary: Minimal quantization error, independent bits, low-dimensional features, and discriminative codes are common objectives across retrieval methods.These objectives target compact and useful representations for efficient retrieval.
  • Convolutional Neural Networks: CNNs have been used extensively for feature learning, neural codes, descriptors, and hash codes across supervised, unsupervised, semi-supervised, and cross-modal retrieval.CNN activations support efficient retrieval through varied losses and representation-learning modes.
  • Autoencoders: Autoencoders reconstruct input images from latent representations, with an encoder mapping the image to latent space and a decoder reconstructing the image.The latent space is used to generate hash codes; the encoder is generally a CNN and the decoder an up-CNN.

B. Autoencoder Networks based Image Retrieval

Autoencoder-based retrieval learns image representations by encoding images into latent features and reconstructing them, while related architectures use pairwise, triplet, adversarial, and attention-based objectives to improve retrieval descriptors.

  • Autoencoder Networks: Autoencoders transform an input image into latent feature space and reconstruct it through encoder and decoder networks.The encoder maps I to z, while the decoder maps z to reconstructed image I′.
  • Autoencoder Networks: Reconstruction error between the original and reconstructed images trains the autoencoder using L1 or L2 loss.
  • Autoencoder Networks: Autoencoder variants have been used for deep feature learning, denoising, cross-modal retrieval, and binary-code generation.
  • Siamese and Triplet Networks: Siamese networks learn from image-pair distances, whereas triplet networks minimize anchor-positive distance and maximize anchor-negative distance.
  • GANs and Attention: GANs and attention modules extend retrieval representations by improving feature discrimination, robustness, or focus on salient image regions.

F. Recurrent Neural Networks for Image Retrieval

The survey summarizes recurrent and reinforcement-learning approaches as network-driven methods for producing effective, less redundant retrieval representations and hash codes.

  • Recurrent Neural Networks: Recurrent neural networks have been used for hierarchical hashing, textual-visual cross retrieval, and triplet hashing.
  • Reinforcement Learning: Reinforcement learning generates hash codes through policy gradients, using similarity preservation as the reward function.
  • Reinforcement Learning: Deep reinforcement learning can remove redundant or harmful hash bits, reducing ambiguity in similarity computation.
  • Network-Driven Approaches: CNNs learn hash codes and descriptors under constraints including classification error, quantization error, and bit independence.
  • Network-Driven Approaches: Autoencoders, siamese and triplet networks, GANs, attention, recurrent networks, and reinforcement learning cover complementary retrieval objectives.
  • Descriptor Types: Descriptor categories include binary hash codes, real-valued descriptors, and feature aggregation for efficient or discriminative retrieval.

A. Binary Descriptors

Deep retrieval methods use binary codes for efficient large-scale search, real-valued descriptors for finer discrimination, and feature aggregation to combine complementary representations.

  • Binary Descriptors: Binary descriptions are learned with deep, convolutional, autoencoder, siamese, triplet, adversarial, and variational networks.
  • Binary Descriptors: Binary hashing struggles to represent fine-grained similarity and can assign similar codes to different images.
  • Real-Valued Descriptors: Real-valued descriptors improve feature discrimination but increase computational complexity during feature matching.
  • Feature Aggregation: Feature aggregation combines information from different networks, subnetworks, or layers to improve image retrieval performance.
  • Binary Descriptors: Binary-code methods target compactness, lower bit redundancy, generalization, and discriminative ability under different supervision modes.

VII. RETRIEVAL TYPE

Deep-learning image retrieval spans cross-modal, sketch-based, multi-label, instance, object, and semantic retrieval settings, adapting representations to different data types and retrieval targets.

  • Cross-modal Retrieval: Cross-modal retrieval measures similarity between heterogeneous data objects from more than one modality.
  • Sketch Based Image Retrieval: Sketch-based retrieval maps a sketch-domain query to images and includes fine-grained, zero-shot, re-ranking, and adversarial approaches.
  • Multi-Label Retrieval: Multi-label retrieval represents images using multiple categorical labels and strategies such as multilevel similarity and category-aware hashing.
  • Instance Retrieval: Instance retrieval methods learn representations for individual image instances using global, local, grouped, and multiple-instance approaches.
  • Object Retrieval: Object retrieval derives representations from specific image objects, including detected regions and multi-view representations for 3D objects.
  • Semantic Retrieval: Semantic retrieval methods jointly exploit visual-semantic embeddings, hash and classification losses, quantizers, and aggregated CNN features.

G. Fine-Grained Image Retrieval

Fine-grained and specialized retrieval methods use tailored feature-selection, aggregation, semantic, and asymmetric architectures. Across retrieval types, deep networks support heterogeneous modalities, multiple visual scenarios, object regions, and semantic preservation.

  • Cross-modal and sketch-based retrieval learn joint features across modalities, with recent methods using adversarial networks.
  • Multi-label and instance retrieval address images containing more than one type of visual scenario through different network architectures.
  • Object retrieval methods use region proposal networks to select image features.
  • Semantic retrieval uses abstract semantic features, while reconstruction-based networks are suited to semantic-preserving hashing.
  • G. Fine-Grained Image Retrieval: Fine-grained image retrieval uses different feature-selection and aggregation networks.
  • Asymmetric hashing processes query and gallery images with different networks.

A. Progress in Retrieval Loss

Deep retrieval research develops specialized losses, representations, and architectures to improve discriminative feature learning and hashing. The survey compares methods across datasets and retrieval depths, identifying strong performers for particular benchmarks and metrics.

  • A. Progress in Retrieval Loss: Siamese, triplet quantization, listwise, and other losses guide discriminative feature learning and can directly optimize retrieval objectives.Listwise loss was used to optimize global mean average precision end-to-end.
  • The surveyed methods target applications including cloth, biomedical, face, remote sensing, landmark, social image, and video retrieval.
  • IX. PERFORMANCE COMPARISON: DTH showed outstanding mAP@5000 performance on CIFAR-10 and NUS-WIDE, alongside DSAH, SPDAQ, DPAH, and DRDH as promising methods.
  • TBH was appealing on CIFAR-10 at mAP@1000, DPAH outperformed other approaches on ImageNet, and SDH performed appealingly on MNIST.
  • DRDH achieved superlative mAP@54000 on CIFAR-10, while Deep Triplet Quantization was identified as favorable for feature learning.
  • SPDAQ and UADH were identified as encouraging using mAP over all retrieved CIFAR-10 images.

X. CONCLUSION AND FUTURE DIRECTIVES

The survey synthesizes a decade of deep-learning progress in content-based image retrieval through a broad taxonomy, chronological coverage, dataset summary, and mAP-based performance analysis. It identifies deep models, new objectives, semantic feature learning, attention, and transfer learning as major trends, while proposing compact representations and self-supervised learning as future directions.

  • X. CONCLUSION AND FUTURE DIRECTIVES: The survey covers deep-learning retrieval methods from 2011 to 2020 using taxonomies for supervision, networks, descriptor data types, retrieval types, and related aspects.
  • Its analysis includes chronological evolution, common large-scale datasets, and mAP comparisons for different numbers of retrieved images.
  • Deep-learning models are driving retrieval progress, with GANs, autoencoders, reinforcement-learning networks, improved objectives, semantic feature learning, attention, and transfer learning highlighted as trends.
  • Future work includes improved models and objectives, lower-loss quantization, semantic-preserving and attention-focused feature learning, while balancing discriminativeness, robustness, and search speed.
  • Low-dimensional compact features or hash codes and self-supervised learning are proposed as directions for faster search and potentially stronger retrieval performance.
Loading 2012.00641v2…