Source-linked AI summary
A Zero-Shot Framework for Sketch-based Image Retrieval
Sasi Kiran Yelamarthi, Shiva Krishna Reddy, Ashish Mishra, Anurag Mittal
TL;DR
Conventional SBIR evaluation rewards class-level retrieval without ensuring shape correspondence, allowing class-specific mappings that fail on unseen classes. The paper introduces a zero-shot benchmark and conditional generative models that stochastically complete sketch information, reporting significant improvement over existing approaches.
Problem
Conventional SBIR evaluation emphasizes class-based retrieval rather than shape or attribute correspondence, while existing test splits can permit class-specific mappings and training-image overlap.
Method
The paper creates a zero-shot SBIR benchmark from a careful Sketchy split and proposes conditional generative models that generate possible image vectors from sketch features.
Results
The proposed generative models achieve significant improvement over existing SBIR methods and adapted zero-shot image-classification approaches in the zero-shot setting.
Takeaways & Limitations
Zero-shot SBIR provides an automatic coarse-grained evaluation setting intended to discourage class-specific learning while avoiding the labor and biases of fine-grained annotation.
Abstract
from arXiv · showhide
Sketch-based image retrieval (SBIR) is the task of retrieving images from a natural image database that correspond to a given hand-drawn sketch. Ideally, an SBIR model should learn to associate components in the sketch (say, feet, tail, etc.) with the corresponding components in the image having similar shape characteristics. However, current evaluation methods simply focus only on coarse-grained evaluation where the focus is on retrieving images which belong to the same class as the sketch but not necessarily having the same shape characteristics as in the sketch. As a result, existing methods simply learn to associate sketches with classes seen during training and hence fail to generalize to unseen classes. In this paper, we propose a new benchmark for zero-shot SBIR where the model is evaluated in novel classes that are not seen during training. We show through extensive experiments that existing models for SBIR that are trained in a discriminative setting learn only class specific mappings and fail to generalize to the proposed zero-shot setting. To circumvent this, we propose a generative approach for the SBIR task by proposing deep conditional generative models that take the sketch as an input and fill the missing information stochastically. Experiments on this new benchmark created from the "Sketchy" dataset, which is a large-scale database of sketch-photo pairs demonstrate that the performance of these generative models is significantly better than several state-of-the-art approaches in the proposed zero-shot framework of the coarse-grained SBIR task.
1 Introduction
The paper identifies weaknesses in conventional SBIR evaluation and introduces zero-shot SBIR with a new Sketchy split and generative models for unseen classes.
- Motivation: SBIR models should align sketch components with corresponding image components, but conventional class-based evaluation does not test these shape-level associations.Existing evaluation can reward retrieving images from the sketch’s class even when their shapes or attributes do not match.
- Motivation: Class-overlapping test splits can let models learn class-specific mappings, and corresponding images for withheld sketches may still appear during training.These settings make retrieval easier without requiring generalization to genuinely novel classes.
- Proposed framework: ZS-SBIR uses unseen test classes as an automatic surrogate for fine-grained evaluation, discouraging class-specific learning without human ranking annotations or bias.The proposed setting is also motivated by applications where new product classes emerge after training.
- Proposed approach: The authors propose conditional generative models that take sketch features and stochastically generate possible image vectors by filling missing information.The approach uses models based on Adversarial Autoencoders and Variational Autoencoders.
- Proposed framework: The benchmark carefully splits Sketchy so test classes do not overlap with ImageNet’s 1000 classes, while evaluating retrieval on novel classes.This preserves the possibility of ImageNet pretraining without violating the zero-shot assumption.
- Evaluation: Existing SBIR models and adapted zero-shot classifiers are evaluated against the proposed approach in the new zero-shot setting.The paper reports better performance for the proposed generative approach than these comparison methods.
2 Related Work
The related work reviews conventional SBIR methods and zero-shot learning, emphasizing the domain gap between sketches and images and the absence of prior zero-shot SBIR benchmarks.
- Sketch-based image retrieval: Conventional SBIR projects sketches and images into a common feature space for retrieval using handcrafted descriptors, learned features, or binary codes.The reviewed handcrafted methods include HOG, HELO, and LKS within retrieval pipelines, while CNNs were later introduced for sketch representation.
- Zero-shot learning: Zero-shot learning recognizes images from novel classes without training examples from those classes and has expanded beyond image classification to other vision tasks.The paper notes applications including image tagging, visual question answering, and action recognition.
- Research gap: The authors state that zero-shot learning had not previously been explored for sketch-based image retrieval.This motivates introducing a dedicated zero-shot SBIR benchmark.
3 Zero shot setting for SBIR
The paper defines zero-shot SBIR by separating sketch-image pairs and retrieval images according to disjoint training and test classes. The proposed Sketchy benchmark uses 104 training classes and 21 test classes, with test classes excluded from ImageNet-1K.
- Zero-shot formulation: Zero-shot SBIR partitions class labels into Ytrain and Ytest, then separates paired data so no test-class sketches occur during training.The model trains on Str and evaluates test sketches against images in Dte.
- Zero-shot formulation: The retrieval objective is to return Dte images sharing the query sketch’s class, preventing retrieval through learned class-label mappings.The retrieval model may use Dtr for validation and hyperparameter tuning.
- Benchmark construction: The benchmark uses Sketchy, containing 75,471 sketches and 12,500 images across 125 classes, with an augmented 73,002-image database.The augmented database adds 60,502 ImageNet images to the original 12,500-image database.
- Benchmark construction: The proposed split assigns 104 classes to training and 21 to testing, ensuring test classes are absent from ImageNet’s 1,000 classes.This preserves the zero-shot assumption while allowing ImageNet pretraining.
4 Limitations of existing SBIR methods
The paper evaluates discriminative SBIR approaches in the proposed zero-shot setting and finds substantial generalization failures. Their performance decline is associated with class-specific learning, while DSH also lacks semantic representations for test classes.
- Evaluation protocol: The evaluated baselines include Siamese, Triplet, and DSH models, with retrieval assessed using the top 200 images and precision and mAP.The evaluation also compares performance with the traditional SBIR setting.
- Evaluation findings: More than 50% performance drops in zero-shot SBIR indicate that existing discriminative approaches do not generalize well beyond training classes.The paper attributes this decline to models associating sketches and images with class labels.
- Evaluation findings: Siamese networks perform best among the existing SBIR methods, whereas Triplet loss performs more poorly in the zero-shot setting.The paper attributes Triplet’s weaker performance partly to insufficient training images.
- Evaluation findings: Coarse-grained Triplet training outperforms fine-grained Triplet training on the proposed zero-shot task.The compared strategies differ in whether negatives are sampled only from different classes or from all non-corresponding images.
- Evaluation findings: DSH underperforms Siamese and Triplet networks in ZS-SBIR despite being the state-of-the-art SBIR model in its original setting.The paper suggests its semantic factorization loss uses only training-class embeddings and does not reduce the semantic gap for test classes.
5 Generative Models for ZS-SBIR
The paper proposes conditional generative models for zero-shot SBIR that generate image features from sketch features without class labels, then retrieve database images using sampled and clustered representations.
- Generative formulation: The generative approach models image-feature distributions conditioned on sketch features, encouraging associations based on outlines and local shapes rather than class labels.Training uses paired sketch-image features from seen classes, while class labels are omitted.
- Model architectures: CVAE approximates the conditional image distribution with a latent-variable encoder-decoder, while CAAE replaces the VAE’s KL-divergence term with adversarial training.Both architectures condition image generation on the sketch vector.
- Model architectures: A reconstruction regularizer forces generated image features to reconstruct the input sketch features through a neural network trained end-to-end with the generative model.The regularization coefficient λ is tuned as a hyper-parameter.
- Retrieval methodology: At test time, the decoder samples image features from p(z) = N(0, I) for each unseen-class sketch, clusters them with K-Means, and retrieves 200 database images.The method uses K cluster centers and compares them with database image features using a VGG-16-based distance; K = 5 performs best empirically.
- Evaluation: The evaluation compares generative models with state-of-the-art SBIR methods and adapted zero-shot image-classification algorithms using Precision and mAP on the proposed zero-shot split.Image features are 4096-dimensional VGG-16 features, while sketch features come from a network tuned on training sketches.
6 Results
On unseen classes, the proposed generative models outperform existing methods and produce retrieved or generated features that preserve sketch-related visual structure. CVAE performs better than CAAE, while reconstruction regularization improves precision.
- Quantitative results: Both proposed generative models outperform existing SBIR models and adapted zero-shot classification methods in the ZS-SBIR evaluation.The authors interpret this as evidence of better latent-alignment learning in the generative approach.
- Qualitative analysis: CVAE retrievals closely match the outlines of sketches from classes absent during training, including false positives with similar shapes but different class labels.The examples are presented as qualitative evidence in the proposed zero-shot setting.
- Performance comparisons: SAE achieves the best performance among prior methods, while the proposed models generate multiple nonlinear samples instead of one image through a linear projection.The authors attribute the proposed models’ superior performance to these generalizations over SAE.
- Feature visualization: Generated image features are largely close to true test-image features and capture multiple distributional modalities in the t-SNE visualization.Test features and predicted features are shown separately, with colors representing classes.
- Ablation and model comparison: CVAE performs significantly better than CAAE, and adding reconstruction loss improves precision by 3%.The paper associates CVAE’s advantage with instability during adversarial training and smoother CVAE training error.
7 Conclusion
The paper introduces ZS-SBIR as a benchmark using a careful Sketchy split, then shows existing SBIR models degrade substantially while conditional generative models improve performance.
- ZS-SBIR uses a careful split of the Sketchy dataset to benchmark retrieval on unseen classes.
- Existing state-of-the-art SBIR models drop significantly under this zero-shot evaluation, exposing class-specific learning in discriminative models.
- The paper extends existing zero-shot image classification methods to the ZS-SBIR task.
- The authors formulate SBIR generatively and propose two conditional generative models that significantly improve over existing methods.