Source-linked AI summary
Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval
Sounak Dey, Pau Riba, Anjan Dutta, Josep Llados, Yi-Zhe Song
TL;DR
ZS-SBIR seeks to retrieve photos from unseen categories using human sketches despite scarce data, large domain gaps, abstraction, and large-scale retrieval requirements. The paper introduces QuickDraw-Extended and a joint visual–semantic embedding framework with domain disentanglement, reporting improvements over prior methods on existing and new benchmarks.
Problem
ZS-SBIR must retrieve photos from unseen categories despite scarce sketches, large sketch–photo domain gaps, sketch abstraction, and the need for large-scale retrieval.
Method
The paper builds QuickDraw-Extended and a cross-domain zero-shot embedding model combining visual triplet ranking, domain disentanglement, mutual-information learning, and semantic information.
Results
The reduced model significantly outperforms state-of-the-art results on established ZS-SBIR datasets, while the full framework shows superior performance on the new benchmark.
Takeaways & Limitations
QuickDraw-Extended and the proposed framework provide a more practical large-scale ZS-SBIR setting centered on highly abstract amateur sketches.
Takeaways & Limitations
Performance comparisons on TUBerlin-Extended are affected by train/test split choices and fluctuate substantially across random splits.
Abstract
from arXiv · showhide
In this paper, we investigate the problem of zero-shot sketch-based image retrieval (ZS-SBIR), where human sketches are used as queries to conduct retrieval of photos from unseen categories. We importantly advance prior arts by proposing a novel ZS-SBIR scenario that represents a firm step forward in its practical application. The new setting uniquely recognizes two important yet often neglected challenges of practical ZS-SBIR, (i) the large domain gap between amateur sketch and photo, and (ii) the necessity for moving towards large-scale retrieval. We first contribute to the community a novel ZS-SBIR dataset, QuickDraw-Extended, that consists of 330,000 sketches and 204,000 photos spanning across 110 categories. Highly abstract amateur human sketches are purposefully sourced to maximize the domain gap, instead of ones included in existing datasets that can often be semi-photorealistic. We then formulate a ZS-SBIR framework to jointly model sketches and photos into a common embedding space. A novel strategy to mine the mutual information among domains is specifically engineered to alleviate the domain gap. External semantic knowledge is further embedded to aid semantic transfer. We show that, rather surprisingly, retrieval performance significantly outperforms that of state-of-the-art on existing datasets that can already be achieved using a reduced version of our model. We further demonstrate the superior performance of our full model by comparing with a number of alternatives on the newly proposed dataset. The new dataset, plus all training and testing code of our model, will be publicly released to facilitate future research
1. Introduction
The paper frames practical ZS-SBIR around scarce sketch data, large sketch–photo domain gaps, highly abstract drawings, unseen categories, and the need for large-scale retrieval. It introduces QuickDraw-Extended and a cross-domain embedding model combining domain and semantic strategies, with strong results on established benchmarks.
- Motivation: Existing SBIR datasets provide limited sketch volume and variety, motivating retrieval across categories without training data.They contain fewer than a thousand sketches per category and at most a few hundred classes.
- Motivation: ZS-SBIR must address sketch–photo domain gaps, sketch abstraction, and semantic transfer from seen to unseen categories.The paper argues that practical progress also requires a dataset capturing these challenges.
- Contributions: QuickDraw-Extended contains 330,000 sketches and 204,000 photos across 110 categories, including 3,000 amateur sketches per category.Its test set spans 166 million comparisons, versus 10 million for Sketchy-Extended and 1.9 million for TUBerlin-Extended.
- Contributions: The proposed model uses a visually attended triplet-ranking base, domain disentanglement, mutual-information learning, and semantic information for cross-domain transfer.A Gradient Reversal Layer encourages domain-agnostic embeddings by extracting shared information from sketches and photos.
- Results: A reduced model significantly outperforms prior state-of-the-art results on TUBerlin-Extended and Sketchy-Extended.The full framework is additionally evaluated on the newly proposed dataset with component ablations.
2. Related Work
Prior SBIR research spans handcrafted descriptors, cross-domain deep learning, and ranking losses, while ZS-SBIR combines sketch–image retrieval with zero-shot semantic transfer. Existing ZS-SBIR studies include multimodal and generative approaches, but the combination remains relatively underexplored.
- SBIR Datasets: Existing SBIR datasets include Sketchy, TU-Berlin, and fine-grained shoe and chair benchmarks, but their scale, abstraction, or pairing limits practical generalization.Sketchy has 75,471 sketches across 125 categories, while TU-Berlin originally has roughly 80 sketches per class.
- SBIR Methods: Cross-domain SBIR methods use handcrafted features or deep networks with contrastive, triplet, and HOLEF ranking losses.These methods target the domain shift between sketches and images.
- ZS-SBIR: Zero-shot learning commonly learns joint embeddings for visual features and class semantic descriptors.Existing models differ by their embedding spaces and visual-to-semantic projection functions.
- ZS-SBIR: Few studies combine SBIR and zero-shot learning; prior approaches include multimodal networks and conditional generative models.These methods respectively mitigate sketch–image heterogeneity or generate photo features from sketches.
3. QuickDraw-Extended Dataset
QuickDraw-Extended is a large-scale ZS-SBIR benchmark designed to address the domain gap between amateur sketches and photos and support retrieval across unseen categories. It uses highly abstract, variable sketches and a large photo gallery to make practical ZS-SBIR more realistic and difficult.
- Dataset motivation: QuickDraw-Extended addresses the large domain gap between amateur sketches and photos and the need for large-scale ZS-SBIR retrieval.These limitations are identified as shortcomings of existing datasets and motivations for the new benchmark.
- Dataset design: The dataset uses rough conceptual abstractions without one-to-one sketch-image correspondence, avoids ambiguous overlapping classes, and preserves large intra-class variability.The variability comes from the high abstraction level of sketches produced by different drawers.
- Dataset construction: QuickDraw-Extended contains 110 categories, with 80 used for training and 30 for testing, and uses Flickr images as its retrieval gallery.Classes unsuitable for SBIR are discarded, gallery outliers are manually filtered, and the test split excludes classes present in ImageNet when using pre-trained models.
- Scale: The benchmark comparison reports six times more sketches and more than double the photos per class than previous benchmarks.Table 1 compares benchmark size, class partitions, and the number of sketch-image comparisons performed during testing.
- Qualitative comparison: Figure 2 qualitatively shows increasing sketch abstraction and decreasing sketch-image alignment from the top dataset examples to the bottom.The comparison highlights the intended increase in domain difficulty for QuickDraw-Extended.
4. A ZS-SBIR framework
The framework maps sketches and photos into a common embedding space using attention-based encoders and a combined objective that enforces ranking, domain alignment, and semantic information.
- Problem Formulation: The framework uses separate embedding functions to map photos and sketches into a common space for retrieval across seen and unseen categories.Training and testing categories are disjoint in the zero-shot setting.
- Embedding Functions: Attention-based CNN encoders localize important features in both sketch and photo modalities before producing embeddings of dimension D.The final fully connected layers are replaced to match the desired embedding size.
- Triplet Loss: Triplet loss pulls same-class sketch-photo embeddings together and pushes different-class embeddings apart with a margin.The ranking condition requires the negative distance to exceed the positive distance by at least µ.
- Domain Loss: Domain loss uses a gradient reversal layer and a domain classifier to encourage embeddings that are indistinguishable between sketches and photos.The reversal layer leaves the forward pass unchanged while reversing encoder gradients during backpropagation.
- Semantic Loss: Semantic loss reconstructs category semantics from embeddings and applies gradient reversal to negative samples to separate semantically similar classes.The semantic decoder reconstructs the positive category’s representation, with λs fixed at 0.5.
- Training Objective: The final objective is a weighted sum of triplet, domain, and semantic losses, with equal weighting factors in the reported model.Training updates the encoder parameters iteratively using this combined loss.
5. Experimental Validation
The framework is evaluated on three ZS-SBIR benchmarks through state-of-the-art comparisons, ablations, and qualitative retrieval examples. Results show strong performance on existing datasets, while QuickDraw-Extended remains challenging and split choice affects TUBerlin-Extended comparisons.
- Experiments validate the approach on Sketchy-Extended, TUBerlin-Extended, and QuickDraw-Extended, with comparisons against state-of-the-art methods.
- The model performs best overall across metrics and nearly all datasets, with an almost doubled gap over state of the art on Sketchy-Extended.
- TUBerlin-Extended results fluctuate across random splits, and unspecified comparison splits prevent a fully fair comparison with ZSIH.The authors report that class selection may affect the experiment and that more experiments are needed to confirm this explanation.
- Ablations find that attention strongly improves the baseline, while semantic and domain losses become more important as dataset difficulty and sketch abstraction increase.Semantics provide better extrapolation to unseen data than domain loss, and the best model uses all three losses.
- Qualitative examples show that combining semantic, domain, and triplet losses performs well despite substantial variation in visual abstraction.Some retrievals reflect semantic or shape similarities, such as skyscraper with door or saw and helicopter with windmill.
6. Conclusions
The paper presents a practical ZS-SBIR setting centered on amateur sketches and large-scale retrieval. It introduces QuickDraw-Extended and a model combining visual and semantic information, reporting improved performance over recent state-of-the-art methods.
- The paper targets practical ZS-SBIR challenges involving unconstrained retrieval and the large domain gap between amateur sketches and photos.
- QuickDraw-Extended provides highly abstract amateur sketches collected with the Google Quick, Draw! game for the practical ZS-SBIR scenario.
- The proposed system combines visual and semantic information to generate an image embedding.
- Experiments show that the framework overcomes recent state-of-the-art methods in the ZS-SBIR setting.