Source-linked AI summary
Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval
Li Liu, Fumin Shen, Yuming Shen, Xianglong Liu, Ling Shao
TL;DR
SBIR must match ambiguous free-hand sketches to natural images despite geometric distortion and the cost of continuous-valued search. DSH jointly learns binary codes and deep hash functions using a semi-heterogeneous three-CNN architecture with auxiliary sketch-tokens. Across TU-Berlin Extension and Sketchy, it reports superior retrieval accuracy with reduced retrieval time and memory footprint.
Problem
SBIR methods struggle with geometric distortion between free-hand sketches and natural images, while continuous-valued nearest-neighbor search limits large-scale efficiency.
Method
DSH jointly learns binary codes and deep hash functions using three CNNs for sketches, natural images, and auxiliary sketch-tokens that bridge the modalities.
Results
DSH consistently outperforms other methods in SBIR accuracy while significantly reducing retrieval time and memory load on two large-scale datasets.
Takeaways & Limitations
DSH provides a hashing framework for category-level SBIR that combines cross-view retrieval with lower retrieval and storage costs.
Takeaways & Limitations
Hand-crafted SBIR methods remain limited because matching edge maps to non-aligned sketches with large variation and ambiguity is difficult.
Abstract
from arXiv · showhide
Free-hand sketch-based image retrieval (SBIR) is a specific cross-view retrieval task, in which queries are abstract and ambiguous sketches while the retrieval database is formed with natural images. Work in this area mainly focuses on extracting representative and shared features for sketches and natural images. However, these can neither cope well with the geometric distortion between sketches and images nor be feasible for large-scale SBIR due to the heavy continuous-valued distance computation. In this paper, we speed up SBIR by introducing a novel binary coding method, named \textbf{Deep Sketch Hashing} (DSH), where a semi-heterogeneous deep architecture is proposed and incorporated into an end-to-end binary coding framework. Specifically, three convolutional neural networks are utilized to encode free-hand sketches, natural images and, especially, the auxiliary sketch-tokens which are adopted as bridges to mitigate the sketch-image geometric distortion. The learned DSH codes can effectively capture the cross-view similarities as well as the intrinsic semantic correlations between different categories. To the best of our knowledge, DSH is the first hashing work specifically designed for category-level SBIR with an end-to-end deep architecture. The proposed DSH is comprehensively evaluated on two large-scale datasets of TU-Berlin Extension and Sketchy, and the experiments consistently show DSH's superior SBIR accuracies over several state-of-the-art methods, while achieving significantly reduced retrieval time and memory footprint.
1. Introduction
SBIR uses free-hand sketches to retrieve same-category natural images, but sketch–image distortion and continuous-valued search limit robustness and scalability. DSH addresses these challenges with binary codes and a semi-heterogeneous deep architecture using auxiliary sketch-tokens.
- Motivation: SBIR retrieves natural images in the query sketch’s category, offering an intuitive way to express shape, pose, and fine-grained details.Sketch queries can be more convenient than describing targets with many words.
- Challenges: Free-hand sketches focus on salient object structures, causing distorted shapes and scales relative to natural images.This geometric mismatch creates a substantial domain gap between the two modalities.
- Challenges: Current deep SBIR methods remain unreliable for natural images with varied viewpoints and complex backgrounds because geometric distortions become significant.They tend to perform better on single-object images with simple contours and clean backgrounds.
- Challenges: Most SBIR methods search continuous-valued features with nearest-neighbor complexity O(Nd), limiting large-scale use under constrained computation and memory.Fast retrieval on substantial image collections is especially important for wearable and mobile devices.
- DSH approach: DSH jointly learns binary codes and deep hash functions in an end-to-end framework to preserve cross-view similarities and category-level semantic correlations.An alternating optimization scheme is used to produce the hash codes.
- DSH approach: DSH uses three CNNs for natural images, free-hand sketches, and auxiliary sketch-tokens within a semi-heterogeneous architecture.The design uses late fusion for images and sketch-tokens and shared weights for sketches and sketch-tokens to address the domain gap.
- Results: Experiments consistently report higher SBIR performance than state-of-the-art methods while reducing retrieval time and memory load.The paper evaluates DSH on TU-Berlin Extension and Sketchy.
2. Deep Sketch Hashing
Deep Sketch Hashing (DSH) combines auxiliary sketch-tokens, a semi-heterogeneous three-CNN architecture, and end-to-end discrete binary-code learning for category-level SBIR. Its objective preserves cross-view similarity and latent semantic relationships while using alternating optimization to learn compact hash codes and deep hash functions.
- Inputs and motivation: Sketch-tokens bridge sketch-image distortion by representing essential natural-image edges without detailed texture information.Each token is obtained by thresholding contour-likelihood scores at 60% of the maximum score.
- Inputs and motivation: DSH uses natural images, free-hand sketches, and corresponding sketch-tokens as inputs to three CNN-based hash functions.Sketch-tokens retain essential image edges and resemble free-hand sketches in stroke patterns and appearance.
- Semi-heterogeneous architecture: The cross-weight late-fusion net combines natural images with sketch-tokens, while the shared-weight sketch net links sketch-token and free-hand-sketch representations.The natural-image and sketch-token branches interact through cross-weights; the sketch-token and sketch branches share convolutional and fully connected weights.
- Objective formulation: DSH learns binary codes that preserve cross-view similarities and intrinsic semantic correlations across categories.The pairwise loss aligns image and sketch codes using a semantic similarity matrix, while semantic factorization uses word embeddings and a shared basis.
- Objective formulation: The final non-convex binary optimization jointly learns codes, semantic factors, and deep hash functions through alternating updates.The optimization sequentially updates D, BI, BS, and the deep hash functions while retaining binary constraints.
3. Optimization
DSH solves its non-convex binary optimization through alternating updates of codes, regression variables, and deep-network parameters, retaining discrete constraints throughout. The procedure iterates these updates until convergence and then produces binary retrieval codes.
- Optimization: DSH alternates updates of D, BI, BS, and deep hash functions F1/F2 because the binary-constrained objective is non-convex and nonsmooth.The method first pre-trains classification networks, then optimizes the semi-heterogeneous model iteratively.
- Optimization: The D update reduces to a quadratic regression problem that can be solved analytically.
- Optimization: BI is optimized under BI ∈{−1, +1}m×n1 by updating one binary-code row at a time while fixing the remaining rows.This discrete cyclic coordinate procedure optimizes BI bit by bit toward a stationary point.
- Optimization: The algorithm initializes binary codes and the cross-view similarity matrix, then updates D, BI, BS, and deep parameters once per epoch.The deep parameters Θ1 and Θ2 are updated using mini-batch back-propagation after code updates.
- Optimization: DSH alternates D →BI → BS →{Θ1, Θ2} for T = 10 ∼ 15 epochs, with convergence guaranteed because the objective is lower-bounded.After training, sketches and image–sketch-token pairs are mapped to binary codes for retrieval.
4. Experiments
DSH is evaluated on TU-Berlin Extension and Sketchy against SBIR, cross-modality hashing, and feature-learning baselines using accuracy, retrieval efficiency, visualization, and component analyses. It consistently outperforms comparison methods, while sketch-tokens and joint losses improve performance.
- Datasets and Protocols: The evaluation uses TU-Berlin Extension and Sketchy, with large natural-image galleries and category-level sketch query sets.TU-Berlin Extension has 250 categories and 204,489 gallery images; Sketchy uses 73,002 gallery images across 125 categories.
- Datasets and Protocols: DSH is compared with hand-crafted, deep SBIR, cross-modality hashing, and cross-view feature-learning methods.The comparison includes HOG-family methods, Siamese CNN, Sketch-a-Net, GN Triplet, 3D shape, CMFH, CMSSH, CVH, SCM, SePH, DCMH, CCA, PLSR, XQDA, and CVFL.
- Results and Discussions: 0.138/0.142 and 0.210/0.105 improvements in MAP/precision@200 over the best comparison methods are reported on the two datasets, respectively.The gains are attributed to DSH's semi-heterogeneous architecture and auxiliary sketch-tokens, which mitigate geometric distortion.
- Results and Discussions: DSH consistently outperforms cross-modality hashing and feature-learning methods, with superiority also shown by precision-recall and HD2 curves across code lengths.The evaluation additionally reports t-SNE distributions, query examples, retrieval time per query, and memory loads.
- Results and Discussions: Removing sketch-tokens decreases MAP by around 0.073 and 0.101 on the respective datasets, while using either loss alone performs worse than the full model.The component analysis supports the roles of sketch-tokens, cross-view pairwise loss, and semantic factorization loss.
5. Conclusion
The conclusion presents DSH as a deep hashing framework for fast SBIR, jointly optimizing binary codes and deep hash functions while using sketch-tokens to mitigate geometric distortion. Experiments validate its superiority in retrieval accuracy and time/storage complexity.
- Conclusion: DSH uses a semi-heterogeneous architecture to encode sketches, natural images, and auxiliary sketch-tokens that mitigate geometric distortion.Binary codes and deep hash functions are jointly optimized in an alternating manner.
- Conclusion: Extensive experiments validate DSH's superiority over state-of-the-art methods in retrieval accuracy and time/storage complexity.