Source-linked AI summary
Recent Advance in Content-based Image Retrieval: A Literature Survey
Wengang Zhou, Houqiang Li, Qi Tian
TL;DR
Content-based image retrieval addresses the mismatch between textual search and visual content, while remaining challenged by intention and semantic gaps. This survey categorizes CBIR methods from 2003 to 2016 across five framework modules and identifies eight future research directions.
Problem
CBIR seeks to retrieve relevant images using visual-content representations, addressing mismatches between text queries and visual content as well as intention and semantic gaps.
Method
The survey organizes research from 2003 to 2016 around query formation, image representation, indexing, retrieval scoring, and search re-ranking.
Results
The paper categorizes representative strategies for each module and summarizes eight potential directions for advancing CBIR.
Takeaways & Limitations
Future CBIR research can explore complementary textual, behavioral, and speech clues through cross-modal retrieval and multimodal feature fusion.
Takeaways & Limitations
Deep-learning representations remain sensitive to rotation and scaling, while computational efficiency and memory overhead remain important design constraints.
Abstract
from arXiv · showhide
The explosive increase and ubiquitous accessibility of visual data on the Web have led to the prosperity of research activity in image search or retrieval. With the ignorance of visual content as a ranking clue, methods with text search techniques for visual retrieval may suffer inconsistency between the text words and visual content. Content-based image retrieval (CBIR), which makes use of the representation of visual content to identify relevant images, has attracted sustained attention in recent two decades. Such a problem is challenging due to the intention gap and the semantic gap problems. Numerous techniques have been developed for content-based image retrieval in the last decade. The purpose of this paper is to categorize and evaluate those algorithms proposed during the period of 2003 to 2016. We conclude with several promising directions for future research.
1 INTRODUCTION
CBIR addresses visual retrieval by representing image content rather than relying only on text, but it must bridge intention and semantic gaps. This survey reviews post-2003 methodological advances across representation, organization, and similarity measurement.
- Research challenges: CBIR faces intention and semantic gaps: users struggle to express desired visual content, while low-level features poorly describe high-level concepts.These gaps motivate extensive academic and industrial efforts.
- Foundational advances: SIFT introduced invariant local features that capture visual content while resisting rotation, scaling, and illumination changes.Its descriptive and discriminative properties helped advance large-scale visual retrieval.
- Foundational advances: The Bag-of-Visual-Words model quantizes local features into compact image representations compatible with inverted-file indexing for scalable retrieval.BoW connects local-feature representation with efficient database search.
- Core technical issues: CBIR algorithms center on image representation, image organization, and image similarity measurement, with large databases motivating inverted files and hashing.Similarity is commonly derived from visual-feature matching with weighting schemes or match kernels.
- Core technical issues: Effective image representations should be descriptive, discriminative, and invariant to transformations such as translation, rotation, resizing, and illumination changes.Representation transforms images into feature spaces to support comparison while reducing background and transformation effects.
- Survey scope: The survey focuses on research after 2003 and emphasizes methodological progress in a generic CBIR framework, complementing earlier and related surveys.It reviews representative strategies across the framework’s components and discusses evaluation and future directions.
2 GENERAL FLOWCHART OVERVIEW
The general CBIR framework separates database preparation from query-time retrieval. The survey reviews strategies for each pipeline module and evaluates their associated problems and methods.
- Off-line stage: The off-line stage crawls database images, converts them into vectors, and builds indexes for later retrieval.This stage prepares the visual database before users submit queries.
- On-line stage: The on-line stage analyzes user intention, forms and represents the query, scores images, reranks results, and supports retrieval browsing.These modules operate when responding to a query.
- Survey organization: The survey reviews related work in each module and evaluates strategies addressing the corresponding retrieval issues.The framework provides the organizing structure for the subsequent discussion.
3 QUERY FORMATION
CBIR supports several visual query formations, including example images, sketches, color maps, and structured object queries. These schemes differ in expressiveness, interaction, and practical limitations.
- Query schemes: Query formation strongly affects retrieval, and the survey considers example images, sketches, color maps, context maps, and structured object queries.Different query schemes can produce significantly different retrieval results.
- Query schemes: Query by example uses an image, optionally constrained by a bounding box, to retrieve images with the same or similar semantics.Its objective input supports quantitative analysis, making it the most widely explored query style.
- Query schemes: Sketch queries use contour images and can align more closely with users’ semantic representations, though early work focused on artworks and simple patterns.Natural-image sketch retrieval later used the representative edgel approach.
- Query schemes: Color-map queries specify spatial color distributions on a grid and retrieve images with similar regional colors.They support user interaction but are limited in the concepts they can represent and challenged by color or illumination changes.
- Query schemes: Structured object queries arrange text concepts spatially to express generalized objects or scenes with context.This approach was also explored with a latent ranking SVM model.
- Open query issues: Single-image queries may inadequately capture user intent, motivating collaborative representation or result fusion for multiple probe images.The issue is especially relevant to video retrieval involving temporal sequences.
4 IMAGE REPRESENTATION
Image representation converts visual content into comparable fixed-length features, addressing feature extraction, aggregation, and similarity computation. The survey covers handcrafted and learning-based representations, including spatial context and quantization.
- Representation framework: CBIR represents images as fixed-length vectors so their visual similarity can be computed efficiently rather than compared directly at pixel level.The representation pipeline includes feature extraction, encoding and aggregation, and indexing.
- Spatial context: Spatial context is treated as an important property because it improves the distinctiveness of visual representations.The similarity formulation and representation questions motivate incorporating spatial information during encoding and aggregation.
- Handcrafted features: Handcrafted representations include global color, shape, texture, and structure descriptors, alongside local features such as SIFT and binary descriptors.Local descriptors are designed for robustness to transformations and distinctive matching across large image corpora.
- Learning-based features: Learning-based features include attributes, topic models, and deep neural network activations that provide increasingly semantic-aware image representations.Deep models can learn high-level abstractions close to human cognition from their different layers.
- Learning-based features: Classification-trained deep features may not reflect retrieval content well, motivating retrieval-specific fine-tuning and supervised or unsupervised retrieval-oriented learning.The survey describes landmark fine-tuning, convolutional kernel networks, and binary-code supervision as partial responses.
5 DATABASE INDEXING
Database indexing organizes visual representations for efficient retrieval at large scale. The survey discusses inverted-file and hashing-based approaches, including quantization, reranking, semantic indexing, and recall-speed trade-offs.
- Indexing overview: CBIR indexing addresses efficient retrieval from large databases, with inverted-file and hashing-based techniques as the two popular categories.Index structures reduce the response time needed to identify relevant target images.
- Inverted file indexing: Inverted files represent a sparse image–visual-word matrix and compare only database images sharing visual words with the query.This substantially reduces the number of candidate images requiring comparison.
- Inverted file indexing: Inverted-file entries can store image IDs, Hamming codes, and geometric clues such as feature position, scale, and orientation for verification.These stored signals support descriptor matching and geometric consistency checks.
- Inverted file indexing: Variants extend inverted indexing through complementary feature fusion, semantic decomposition, product quantization, multiple quantizers, and query-adaptive pruning.The survey includes c-MI, semantic components, inverted multi-index, joint quantizers, and Q-Index strategies.
- Hashing-based indexing: Hashing is suited to dense vectors such as GIST and VLAD, retrieving collision-based candidates before reranking them by exact distance.A stated scalability concern is that raw database vectors must remain in memory for reranking.
- Memory reduction: Sparse projections and min-Hash-style descriptors further reduce memory overhead by compressing visual-word occurrence representations before indexing.These methods approximate or map occurrence vectors to lower-dimensional representations for more compact indexing.
6 IMAGE SCORING
Image scoring assigns relevance through distances between aggregated feature vectors or votes from matched local features. The survey covers distance-based refinements, weighting schemes, contextual cues, and burstiness corrections.
- Relevance scores derive either from distances between aggregated image features or from votes cast by matched visual features.
- Distance Based Scoring: Feature aggregation represents images as fixed-size vectors, whose content relevance can be measured with Lp-normalized distances.
- Distance Based Scoring: L1-norm can outperform L2-norm with the BoW model, while optimization schemes extend feature distances to partial image similarity.
- Distance Based Scoring: Contextual dissimilarity addresses neighborhood reversibility, where images are not reliably among the nearest neighbors of their neighbors.
- Voting Based Scoring: Voting-based scores include local-match counts, TF-IDF-weighted matches, bundled-feature weighting, and Lp-norm IDF extensions.
- Voting Based Scoring: Descriptor and spatial context refine voting, while burstiness penalties reduce the influence of visual elements appearing unusually often.
7 SEARCH RERANKING
Search reranking improves an initial result list by exploiting geometric context, query expansion, relevance feedback, or fusion across retrieval methods. These approaches address false correspondences and complementary rankings or scores.
- Geometric Verification: Geometric verification suppresses false correspondences caused by descriptor ambiguity and quantization loss by using local-feature geometric context.
- Geometric Verification: Transformation-based verification models object or scene changes with affine or multiple homographies and separates inliers from outliers.
- Geometric Verification: RANSAC-like methods sample correspondence sets to find models with many inliers, whereas Hough methods vote in transformation spaces.
- Geometric Verification: Hough pyramid matching addresses transformation-space binning flexibility and has complexity linear in the number of correspondences.
- Query Expansion: Query expansion enriches the original query with features from highly ranked or spatially verified images to improve recall.
- Retrieval Fusion: Retrieval fusion combines complementary rankings or scores from different image representations, using rank aggregation or effectiveness-weighted score products.
8 DATASET AND PERFORMANCE EVALUATION
CBIR evaluation combines benchmark datasets with measures of accuracy, computational efficiency, and memory cost. The surveyed datasets vary in task, scale, object or scene content, and use of distractor images.
- Datasets: Evaluation uses ground-truth datasets, often combined with million-scale distractors because existing ground-truth collections are relatively small.
- Datasets: Table 1 identifies whether datasets are ground-truth, distractor, or mixed databases, and Figure 3 displays sample images from seven datasets.
- Datasets: INSTRE provides single-instance and multiple-instance subsets, while ZuBuD contains 201 Zurich buildings with five views per building.
- Datasets: The benchmarks cover scene, landmark, object, logo, artwork, and mobile visual-search tasks with varied viewpoints, clutter, lighting, and distortion.
- Performance Evaluation: Accuracy is summarized with rank-based measures including AP for binary relevance, mAP across queries, NDCG for multiple relevance levels, and UKBench N-S.
- Performance Evaluation: Efficiency covers codebook construction, feature indexing, and online querying, while memory cost concerns the quantizer and database index loaded for retrieval.
9 FUTURE DIRECTIONS
The survey identifies future work in reducing dataset overfitting, better supporting user intention, improving query quality, and moving learned representations toward retrieval-specific training.
- Evaluation: Benchmark progress can encourage overfitting to specific ground-truth datasets, motivating broader evaluation beyond established records.
- Intention-Oriented Query Formation: Because example, color-map, or sketch queries often underspecify user intention, retrieval interfaces should reduce effort while actively involving users.
- Query Selection: Query quality depends on resolution, noise, affine distortion, and background clutter, so mobile systems can guide users to retake better photos.
9.3 Deep Learning in CBIR
Deep learning is proposed to address semantic limitations in CBIR, but learned representations must remain robust to common image transformations. Scalable supervised or semi-supervised learning is identified as an important direction.
- Hand-crafted image representations remain insufficient for capturing semantics in visual retrieval.Most existing methods are unsupervised because multimedia visual data are highly diverse and large-scale.
- Scalable supervised or semi-supervised learning is proposed to advance semantic-aware retrieval.
- Deep-learning representations must be flexible and robust to changes such as rotation and scaling.Convolutional filters can make feature maps sensitive to large translation, rotation, and scaling changes.
9.4 Unsupervised Database Mining
Unsupervised database mining can exploit relevance context among database images when labels are unavailable and category numbers are unlimited. Large databases may contain mutually relevant subsets that can be uncovered offline.
- Traditional CBIR processes database images independently without considering their potential relevance context.
- The absence of labels and unlimited potential category numbers constrain sophisticated supervised learning in CBIR.
- Large databases are likely to contain subsets of mutually relevant images that unsupervised techniques can uncover during offline processing.
9.5 Cross-modal Retrieval
Cross-modal retrieval can combine visual content with textual, behavioral, and speech clues because these modalities are complementary. The survey also points toward end-to-end retrieval frameworks that jointly optimize multiple modules.
- Textual information, user click logs, and video speech provide complementary clues for identifying visual content.The survey suggests fusing multimodal features with different models for collaborative retrieval.
- Multimodal representation leaves open problems in collaborative quantization, indexing, and search reranking.
- CBIR pipelines contain modules such as feature extraction, codebook learning, feature quantization, and image indexing that are often independently optimized.
- An end-to-end retrieval framework could analogously optimize these modules collaboratively, as CNN components are jointly optimized for image classification.The analogy maps convolutional filters to codewords, convolution responses to soft quantization, and max pooling to local aggregation.
9.7 Social Media Mining with CBIR
CBIR can use social-media visual data to study community structure, user behavior, recommendations, and crowd sentiment. Bridging academic and industrial deployment gaps requires research on open challenges grounded in real scenarios.
- Social media’s community-based content creation, sharing, and interaction make its visual data useful for uncovering community structure.
- CBIR applied to social-media visuals can support understanding individual behavior and recommending products and services.
- Social-media visual analysis can help analyze crowd sentiment for supervision and forewarning.
- Academic CBIR research differs from industrial applications because deployment structures and data availability differ.The survey proposes open grand challenges involving industry and academia to investigate difficulties in real scenarios.
10 CONCLUSIONS
The paper surveys recent advances in content-based image retrieval through five framework modules and identifies eight potential directions for future progress.
- The survey organizes content-based image retrieval around query formation, image representation, image indexing, retrieval scoring, and search re-ranking.It discusses key problems and categorizes representative strategies for each component.
- The paper categorizes a variety of representative strategies and methods for each component of the general retrieval framework.
- The survey summarizes eight potential directions that may boost the advance of content-based image retrieval in the near future.