CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not

• Sketch scarcity drove SBIR toward zero-shot retrieval, where semantic transfer remained rudimentary; CLIP offers rich cross-modal semantics for category transfer and sketch-photo matching.Source: §1 · p. 1
• The CLIP-adapted model surpasses prior ZS-SBIR methods by a large margin.The introduction reports this result without specifying a quantitative value.Source: Figure 1 · p. 1, §1 · p. 2
• The paper adapts CLIP for conventional category-level ZS-SBIR and the more challenging fine-grained, instance-level setting.It presents this as the first adaptation of CLIP to ZS-SBIR.Source: §1 · pp. 1–2
• The method uses modality-specific visual prompts and category-template text prompts to adapt CLIP while retaining most of its generality.The visual prompts are trained with triplet loss while CLIP is otherwise frozen; a small subset of Layer Normalisation parameters is also fine-tuned.Source: §1 · pp. 1–2
• For fine-grained retrieval, the method regularizes cross-category relative distances and uses patch shuffling to encourage structural correspondence.The regularizer minimizes KL divergence between category distance distributions so a global margin can work across categories; shuffled sketch-photo patches are matched by permutation order.Source: §1 · p. 2

Category-level SBIR

• Category-level SBIR learns joint sketch-photo manifolds, while ZS-SBIR targets unseen classes and hashing methods address computational ease.Deep frameworks use triplet-ranking objectives; ZS-SBIR was explored for cross-category generalisation [16], while binary hash-codes [35] were studied for computational ease.Source: §2 · p. 2

Fine-grained SBIR

• FG-SBIR methods expanded from triplet-ranking siamese networks to attention, textual tags, cross-domain generation, co-attention, and reinforcement learning.Further work addressed sketch style-diversity [52], data-scarcity [5], and stroke redundancy [6]; approaches to novel classes included prototypical sketch-trait manifolds and adaptation using supporting sketch-photo pairs [8].Source: §2 · p. 2

Zero-Shot SBIR

• Prior ZS-SBIR methods reduced sketch-photo domain gaps using image-to-image translation, adversarial training, gradient reversal, graph convolution networks, distilled contrastive relationships, shared convolutional or ViT encoders, and reconstruction-based test-time training on test-set sketches.Semantic transfer for ZS-SBIR was mostly limited to using word embeddings directly [18] or indirectly [37].Source: §2 · p. 2

CLIP in Vision Tasks

• CLIP places images and text in a shared embedding space, supporting zero-shot or few-shot downstream tasks through text-derived classification weights.Its cross-modal training uses semantic text and large-scale image-text data, and prompt engineering has been applied across retrieval, image generation, continual learning, detection, recognition, and segmentation.Source: §2 · p. 3

Prompt Learning for Vision Tasks

• Prompt tuning learns task-specific continuous context vectors, and this work learns one prompt for CLIP’s sketch and photo branches to generalize to unseen classes.The approach targets cross-category FG-ZS-SBIR, adapting CLIP’s zero-shot potential to the task.Source: §2 · p. 3

Overview of CLIP

• CLIP [46] pairs image and text encoders; its image encoder uses ResNet-50 [26] or ViT [17] and projects image features into a joint vision-language space.For ViT, the image is divided into patches and processed with a learnable class token.Source: §3 · p. 3
• CLIP [46] trains with a contrastive loss that brings matched text-photo pairs together and separates unmatched pairs; category prompts support downstream classification.The text encoder transforms sentence embeddings into text features, while the image feature is used to calculate category prediction probabilities.Source: §3 · p. 3

Prompt Learning

• Prompt learning [10] keeps foundation-model weights frozen and adapts downstream behavior with task-specific prompts, preserving pretrained knowledge without separate model copies.The paper uses shallow visual prompts [29], inserted with patch embeddings at the first layer, which it reports as empirically optimal and easy to reproduce.Source: §3 · p. 3

Baseline Categorical SBIR

• Categorical SBIR retrieves a photo from the query sketch’s category by bringing same-category sketch-photo features together and separating different-category photos with triplet loss.The baseline uses separate sketch and photo embedding functions and a gallery containing photos from multiple categories.Source: §4 · pp. 3–4

Naively Adapting CLIP for ZS-SBIR

• Naively fine-tuning CLIP’s image encoder with triplet loss collapses performance through catastrophic forgetting, while an added MLP does not ensure CLIP’s generalisation is preserved.The section therefore selects prompt learning for stable optimisation and retention of open-vocabulary generalisation.Source: §4 · p. 4

Prompt Learning for ZS-SBIR

• Sketch- and photo-specific visual prompts adapt frozen CLIP image encoders to their respective image distributions, while training a small subset of layer-normalisation parameters for stability.The remaining encoder parameters stay frozen; the trainable parameters are optimized with the standard triplet loss.Source: §4 · p. 4

Classification Loss using CLIP’s Text Encoder

• A CLIP text-encoder classification loss complements triplet loss in the joint sketch-photo embedding space, using handcrafted templates such as “a photo of a [category]”.The templates produce class weight vectors from seen-category names, leveraging CLIP’s semantic-visual associations.Source: §4 · p. 4
• Because sketches model fine-grained details, the section raises the question of whether sketch queries benefit over text-keyword queries and motivates cross-category fine-grained ZS-SBIR.Off-the-shelf CLIP can perform zero-shot image retrieval using a category-level text query such as “a photo of a [query]”.Source: §4 · p. 4

Background on FG-SBIR

• Cross-category zero-shot FG-SBIR retrieves instance-level photos from unseen categories, unlike category-level ZS-SBIR, which retrieves at category level.Fine-grained SBIR targets instance-level sketch-photo matching within a category, whereas many earlier methods were limited to training and evaluating on a single category.Source: §5 · pp. 4–5

Extending CLIP-based ZS-SBIR to FG-ZS-SBIR

• The CLIP-based model replaces category-level triplets with hard triplets among different instances of the same category and uses a shared sketch-photo prompt and backbone.Its other components include separate sketch/photo branches, CLIP text-encoder classification loss, and fine-tuned layer norms; the common prompt and shared backbone were found to work better for fine-grained matching.Source: §5 · p. 5

Stabilising Margin (µ) across Categories

• A KL-divergence regularizer aligns relative-distance distributions across categories so one global triplet-loss margin can work across categories.This addresses category-varying optimal margins; the regularizer uses sketch/photo samples from every category in a batch.Source: §5 · p. 5

Patch-shuffling for Zero-Shot Fine-grained Transfer

• Patch-shuffling triplets train matched sketch-photo permutations to be closer than mismatched permutations, supporting fine-grained shape transfer to novel classes.The authors report that this objective transfers better than auxiliary patch-order prediction and is cheaper to train than the Sinkhorn operation used by Pang et al..Source: §5 · p. 6

6. Experiments

• Evaluation uses Sketchy, TUBerlin, and QuickDraw Extended for category-level ZS-SBIR, and Sketchy fine-grained associations for cross-category FG-ZS-SBIR.For ZS-SBIR, the reported metrics are mAP@all and precision, with P@100 for TUBerlin and mAP@200 for Sketchy-ext; FG-ZS-SBIR uses Acc.@1 and Acc.@5. Prompts are trained for 60 epochs with CLIP ViT-B/32, while CLIP is fixed except for LayerNorm.Source: §6 · p. 6

6.1. Competitors

• Comparisons include prior translation, semantic-transfer, distillation, graph, transformer, and sketch-retrieval methods, alongside CLIP fine-tuning, linear-probe, and prompt-adaptation baselines.The CLIP prompt baselines include sketch-conditioned, independent sketch/photo, multimodal, and deep prompts; unlike the proposed method, the last three types keep LayerNorm frozen.Source: §6.1 · pp. 6–7

6.2. Performance Analysis:

• ≈24.8% (avg) improvement over existing ZS-SBIR state-of-the-arts is reported, and adapted CLIP surpasses the two existing cross-category FG-ZS-SBIR state-of-the-arts.The ZS-SBIR comparison spans all three datasets.Source: Table 1 · p. 7, §6.2 · p. 7
• CLIP’s ZS-SBIR and FG-ZS-SBIR performance remains relatively stable as training data size and the number of seen categories vary.The study varies per-class training data from 10% to 100% and seen Sketchy classes from 20 to 104.Source: §6.2 · p. 7, Figure 4 · p. 8

6.3. Ablation Study

• Removing patch shuffling and f-Divergence lowers FG-ZS-SBIR accuracy by 3.15% and 3.75%, respectively, while removing the class-discrimination loss severely affects performance.Using 2 × 2 patches instead of 3 × 3 gains 1.2% in Acc.@1 on Sketchy; not fine-tuning LayerNorm slightly lowers performance in both paradigms.Source: §6.3 · p. 8, Table 2 · p. 8
• 4.57%/0.172 (Acc@1/mAP@200) performance drops on Sketchy when the proposed method uses word2vec instead of CLIP’s text encoder.The paper attributes the difference to CLIP’s training on image-text pairs rather than text-only data.Source: §6.3 · p. 8
• Prompt accuracy rises from 26.15%/0.675 at K = 1 to 28.68%/0.723 at K = 3, then falls to 28.26%/0.718 at K = 4 (Acc@1/mAP@200).Learning text prompts instead of using handcrafted prompts drops performance by 2.36%/0.078 (Acc.@1/mAP@200) on Sketchy.Source: §6.3 · p. 8
• Keyword retrieval scores 0.523/0.612 mAP/P@200 versus 0.723/0.725 for ZS-SBIR, and 4.6% Acc@1 versus 18.68% for FG-ZS-SBIR.The fine-grained comparison uses Song et al.’s dataset of fine-grained sketch-photo-text triplets.Source: §6.3 · p. 8

7. Conclusion

• The prompt-based CLIP method significantly surpasses prior state-of-the-art methods in category-level and fine-grained zero-shot SBIR.The conclusion reports no numerical margin for these improvements.Source: §7 · p. 8
• For fine-grained ZS-SBIR, the method makes relative sketch-photo feature distances uniform across categories and learns structural correspondences through patch shuffling.Source: §7 · p. 8
• The authors highlight the potential of combining foundation models such as CLIP with sketch-related tasks.Source: §7 · p. 8

A. Prompt Design for ZS-SBIR and ZS-FG-SBIR

• ZS-SBIR uses separate CLIP image encoders with sketch-specific and photo-specific prompts, training only the prompts and transformer-layer LayerNorms while keeping the rest of CLIP frozen.The design is shown with its training objectives in Figure 5.Source: A. Prompt Design for ZS-SBIR and… · p. 12, Figure 5 · p. 12
• FG-ZS-SBIR uses one shared CLIP image encoder and common prompt for sketches and photos, with only the prompt and transformer-layer LayerNorms trainable.Unlike ZS-SBIR's category-level triplets, it considers hard triplets within each category.Source: A. Prompt Design for ZS-SBIR and… · p. 12
• FG-ZS-SBIR adds an f-divergence loss to make relative sketch-photo distances uniform across categories and a Patch-shuffling loss to learn structural correspondences within sketch-photo pairs.These two losses supplement the losses used for ZS-SBIR.Source: A. Prompt Design for ZS-SBIR and… · p. 12

B. Datasets

• ZS-SBIR uses Sketchy-extended [35], with 75,471 sketches across 125 categories and 60,502 added ImageNet images [48], split into 104 training and 21 test classes.Source: B. Datasets · p. 12
• TUBerlin has 250 categories with 80 free-hand sketches each and 204,489 added images, split into 220 training and 30 test classes [16].Source: B. Datasets · p. 12
• QuickDraw Extended uses a 110-category subset with 330,000 sketches and 204,000 photos [16], split into 80 training and 30 test classes with no test-photo overlap with ImageNet [48].The full QuickDraw collection contains over 50 million sketches across 345 categories, drawn in under 20 seconds per sketch.Source: B. Datasets · p. 13
• Fine-grained evaluation uses Sketchy for one-to-one sketch-photo matching across categories, with at least five associated sketches per photo and the same 104/21 class split.Source: B. Datasets · p. 13

C. More on f-Divergence

• The regulariser aims to make sketch-photo relative distances uniform across categories so one global margin parameter µ can work for all categories.The section defines a category-specific distribution of relative distances over sketch-photo triplets.Source: C. More on f-Divergence · p. 13
• KL-divergence measures information distance between two distributions, while comparing multiple distributions is less studied.The section frames f-divergence as a multi-distribution generalisation of information distance and notes it is under-explored in computer vision.Source: C. More on f-Divergence · p. 13
• The method adopts Sgarro’s average-divergence definition as a simple f-divergence for multiple distributions.f-divergence is defined using a convex function f : [0, ∞) → R.Source: C. More on f-Divergence · p. 13

D. Some Qualitative Results on Sketchy

• Figure 8 compares the baseline and proposed method qualitatively for ZS-SBIR on Sketchy (ext) [35].The baseline follows [16].Source: D. Some Qualitative Results on Sketchy · p. 14, Figure 8 · p. 14
• Figure 9 compares the baseline and proposed method qualitatively for FG-ZS-SBIR on Sketchy.The baseline follows.Source: D. Some Qualitative Results on Sketchy · p. 14, Figure 9 · p. 14
• Figure 9 orders retrieved images by rank, marks top-5 true matches with green frames, and labels each true match’s rank.For each sketch query, the leftmost image is rank 1.Source: Figure 9 · p. 14

E. Limitations

• The assumption that CLIP covers almost all classes during training may fail in niche cases, and thorough zero-shot evaluation on unseen classes is challenging given its training on 400M image-text pairs.The authors reserve both limitations for future work and describe them as universal to CLIP-based applications.Source: E. Limitations · p. 14