PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not

Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, Yi-Zhe Song

arXiv:2303.13440v3cs.CV

TL;DR

ZS-SBIR must transfer semantics across categories and sketch-photo modalities despite scarce sketch data, and fine-grained retrieval additionally requires instance-level matching. The paper adapts CLIP with prompt learning, a distance-distribution regularizer, and patch shuffling for category-level and fine-grained settings. It reports substantial gains over prior methods and identifies scope boundaries for CLIP-based evaluation.

  • Problem

    Data scarcity motivates ZS-SBIR, but semantic transfer for sketch-photo retrieval remained limited, while fine-grained retrieval adds non-uniform distances and structural-correspondence challenges.

  • Method

    The paper adapts CLIP with modality-specific prompt learning, a regularizer for uniform relative distances across categories, and patch shuffling for structural sketch-photo correspondence.

  • Results

    The method surpasses prior state-of-the-art methods significantly for both category-level and fine-grained ZS-SBIR.

  • Takeaways & Limitations

    The results support the potential of combining CLIP and prompt learning for sketch-related tasks under data scarcity beyond ZS-SBIR.

  • Takeaways & Limitations

    The method assumes CLIP covers almost all classes during training, and thorough zero-shot evaluation on unseen classes is challenging because CLIP was trained on internet-scale data.

Abstract

from arXiv · show

In this paper, we leverage CLIP for zero-shot sketch based image retrieval (ZS-SBIR). We are largely inspired by recent advances on foundation models and the unparalleled generalisation ability they seem to offer, but for the first time tailor it to benefit the sketch community. We put forward novel designs on how best to achieve this synergy, for both the category setting and the fine-grained setting ("all"). At the very core of our solution is a prompt learning setup. First we show just via factoring in sketch-specific prompts, we already have a category-level ZS-SBIR system that overshoots all prior arts, by a large margin (24.8%) - a great testimony on studying the CLIP and ZS-SBIR synergy. Moving onto the fine-grained setup is however trickier, and requires a deeper dive into this synergy. For that, we come up with two specific designs to tackle the fine-grained matching nature of the problem: (i) an additional regularisation loss to ensure the relative separation between sketches and photos is uniform across categories, which is not the case for the gold standard standalone triplet loss, and (ii) a clever patch shuffling technique to help establishing instance-level structural correspondences between sketch-photo pairs. With these designs, we again observe significant performance gains in the region of 26.9% over previous state-of-the-art. The take-home message, if any, is the proposed CLIP and prompt learning paradigm carries great promise in tackling other sketch-related tasks (not limited to ZS-SBIR) where data scarcity remains a great challenge. Project page: https://aneeshan95.github.io/Sketch_LVM/

1. Introduction

Data scarcity motivates zero-shot sketch-based image retrieval, while semantic transfer across categories and modalities remains limited. The paper adapts CLIP through prompt learning for both category-level and fine-grained retrieval.

  • Data scarcity in the sketch community motivates zero-shot SBIR, where successful models must transfer semantics across object categories and sketch-photo modalities.
  • The paper is the first to propose combining foundation models such as CLIP with cross-modal ZS-SBIR for category-level and fine-grained instance-level retrieval.
  • CLIP contributes a rich semantic space for category transfer and broad cross-modal knowledge suited to SBIR.
  • Prompt learning adapts CLIP to ZS-SBIR using modality-specific visual prompts injected into the transformer while keeping the remaining CLIP parameters frozen.The prompts are trained with the gold-standard triplet-loss paradigm on extracted sketch-photo features.
  • Fine-grained ZS-SBIR requires uniform relative sketch-photo distances across categories and instance-level structural correspondences beyond semantic consistency.
  • The proposed fine-grained solution combines a KL-divergence regularizer over category-wise relative-distance distributions with patch shuffling of paired sketches and photos.
  • 24.8% improvement over prior arts is reported for category-level ZS-SBIR, while the approach also surpasses prior state-of-the-art methods for fine-grained retrieval.

2. Related Work

Related work centers on category-level ZS-SBIR, while fine-grained cross-category retrieval remains comparatively underexplored. CLIP offers large-scale cross-modal pretraining and open-vocabulary generalisation for this setting.

  • Category-level ZS-SBIR extends sketch-based retrieval to unseen test-time categories, alongside work on test-time training and binary hash codes.
  • Fine-grained cross-category ZS-SBIR remains an area to be explored, motivating adaptation of CLIP for semantic transfer and FG-ZS-SBIR.
  • CLIP trains on approximately 400M image-text pairs and represents images using rich semantic textual information rather than only discretized labels.

3. Preliminaries

CLIP uses separate image and text encoders to produce a joint vision-language embedding space. Prompt learning adapts this frozen foundation model by training task-specific prompt parameters.

  • CLIP's image encoder divides an input image into fixed-size patches, processes them with a class token through transformer layers, and projects the result into a visual feature.
  • CLIP's text encoder transforms sentence word embeddings through a transformer to obtain a textual feature in the joint vision-language space.
  • Contrastive training maximizes cosine similarity for matched text-photo pairs while minimizing it for unmatched pairs.
  • Prompt learning keeps foundation-model weights frozen and trains task-specific prompt parameters to preserve pre-learned generalisable knowledge.
  • The paper uses a shallow visual prompt inserted only at the first transformer layer alongside patch embeddings.

4. CLIP for Zero-Shot Category-level SBIR

Category-level SBIR retrieves same-category photos for a query sketch, while ZS-SBIR evaluates generalisation to unseen categories. The paper adapts CLIP with visual prompts and text-based classification to improve this transfer.

  • Category-level SBIR: Category-level SBIR learns a joint sketch-photo manifold with triplet loss, bringing same-category pairs closer and separating photos from different categories.
  • Zero-Shot SBIR: ZS-SBIR evaluates retrieval on novel categories rather than categories seen during training.
  • Naively Adapting CLIP for ZS-SBIR: Naively replacing ImageNet-pretrained VGG-16 with CLIP's visual encoder does not adequately address zero-shot transfer when trained directly with triplet loss.
  • Prompt Learning for ZS-SBIR: The proposed category-level adaptation learns separate sketch and photo visual prompts while using CLIP image encoders for both modalities.
  • Prompt Learning for ZS-SBIR: Prompt and branch-specific layer-normalisation parameters are updated while the rest of CLIP remains frozen, using the standard triplet loss.
  • Classification Loss using CLIP’s Text Encoder: A CLIP text-encoder classification loss complements triplet learning by supplying class-specific text features from handcrafted prompts.
  • Fine-Grained Motivation: The category-level setting motivates fine-grained ZS-SBIR because sketches model fine-grained details beyond category-level matching.

5. CLIP for Zero-Shot Fine-grained SBIR

FG-ZS-SBIR extends zero-shot sketch-based retrieval from category-level matching to instance-level matching across unseen categories. The method adapts CLIP with shared prompting, hard-triplet training, distance regularisation, and patch shuffling to address category variation and structural correspondence.

  • FG-SBIR matches sketches and photos at the instance level within categories, whereas category-level SBIR retrieves category-specific photos.
  • Cross-category FG-SBIR trains across multiple categories with hard negatives from the same category but different instances and an auxiliary class-discrimination objective.
  • CLIP-based FG-ZS-SBIR uses an image encoder with sketch/photo prompting, category-level components, text-encoder classification, and fine-tuned layer norms before replacing category triplets with hard triplets.
  • A KL-divergence regulariser aligns relative sketch-photo distance distributions across categories so one global triplet margin can work across them.The method computes category-specific relative-distance distributions and minimises divergence between category pairs.
  • Patch shuffling creates augmented triplets that pull sketches toward paired photos with the same patch permutation and push them from differently permuted photos.The technique targets fine-grained shape correspondence for transfer to novel classes.
  • The patch-shuffling triplet objective provides better fine-grained shape transfer than auxiliary patch-order prediction while costing less than a complex Sinkhorn operation.

6. Experiments

Experiments evaluate CLIP-based zero-shot retrieval across category-level and fine-grained settings, comparing against prior frameworks, baselines, and design ablations. Results show strong gains from prompt-based adaptation and the proposed fine-grained components.

  • Experimental setup: The experiments use Sketchy, TUBerlin, and QuickDraw Extended for ZS-SBIR, while fine-grained evaluation uses Sketchy with one-to-one sketch-photo associations.The zero-shot splits use disjoint training and testing categories; FG-ZS-SBIR uses Sketchy’s fine-grained associations.
  • ZS-SBIR performance: 24.8% (average) is the improvement of successful CLIP adaptation over existing state-of-the-art methods for ZS-SBIR.The method surpasses prior methods across all three ZS-SBIR datasets, while naive fine-tuning collapses.
  • ZS-SBIR performance: Prompt-based learning outperforms linear probing and more elaborate prompt variants, supporting a simple shallow prompt design for ZS-SBIR.The marginal difference between simple and more complicated prompt adaptations motivates avoiding additional architectural complexity.
  • Ablation study: Removing patch shuffling and f-Divergence lowers FG-ZS-SBIR accuracy by 3.15% and 3.75%, respectively.The ablation attributes these drops to losing structural sketch-photo correspondence and uniform relative distances across categories.
  • Representation analysis: Replacing CLIP’s text encoder with word2vec drops Sketchy performance by 4.57% in Acc@1 and 0.172 in mAP@200.The comparison supports CLIP’s image-text-trained visual-semantic associations over text-only word2vec information.
  • Representation analysis: Prompt accuracy rises from 26.15%/0.675 at K = 1 to 28.68%/0.723 at K = 3, then saturates at 28.26%/0.718 for K = 4.The reported Acc@1/mAP@200 values on Sketchy identify K = 3 as optimal.

7. Conclusion

The paper concludes that prompt-based CLIP adaptation improves zero-shot sketch-based image retrieval at both category and fine-grained levels. Fine-grained gains come from uniforming cross-category feature distances and learning structural sketch-photo correspondences.

  • Conclusion: CLIP’s open-vocabulary generalisation potential, accessed through prompt-based design, enhances both category-level and fine-grained zero-shot SBIR.The method surpasses prior state-of-the-art methods significantly in both settings.
  • Conclusion: Fine-grained ZS-SBIR is improved by uniform relative sketch-photo distances across categories and patch-shuffling-based structural correspondence learning.These are presented as the two strategies addressing fine-grained matching challenges.

Supplementary material for CLIP for All Things Zero-Shot Sketch-Based Image Retrieval,

The supplementary material details how visual prompts are incorporated into CLIP encoders for category-level and fine-grained ZS-SBIR. The two settings differ in whether prompts and encoders are modality-specific or shared.

  • Prompt design for ZS-SBIR: ZS-SBIR uses separate CLIP image encoders with distinct sketch and photo prompts for the two modalities.The prompts are incorporated into the respective encoders, as illustrated in Figure 5.
  • Prompt design for FG-ZS-SBIR: FG-ZS-SBIR uses one shared CLIP image encoder with a common prompt for both sketch and photo branches.Its training also keeps the CLIP model frozen except for LayerNorm parameters and the prompt.
  • Prompt design for FG-ZS-SBIR: The FG-ZS-SBIR design adds hard-triplet and additional loss objectives beyond the category-level setup.These additions target fine-grained matching requirements alongside the shared image encoder.

B. Datasets

The experiments use three datasets for category-level ZS-SBIR and Sketchy for fine-grained cross-category evaluation. Their splits reserve unseen categories or require fine-grained sketch-photo associations.

  • ZS-SBIR datasets: Sketchy Extended contains 75,471 sketches across 125 categories plus 60,502 ImageNet images, split into 104 training and 21 testing classes.Each category has 100 images, with at least five associated hand-drawn sketches per photo.
  • ZS-SBIR datasets: TUBerlin contains 250 categories with 80 free-hand sketches each and is split into 220 training and 30 testing classes.The dataset was extended with 204,489 images.
  • ZS-SBIR datasets: QuickDraw Extended uses 110 categories with 330,000 sketches and 204,000 photos, split into 80 training and 30 testing classes.The subset augments QuickDraw sketches with Flickr images.
  • FG-ZS-SBIR dataset: Fine-grained ZS-SBIR evaluation uses Sketchy’s one-to-one sketch-photo associations with the same 104-training and 21-testing class split.Multiple sketches associated with each photo support the fine-grained evaluation illustrated in Figure 7.

C. More on f-Divergence

The paper extends KL-divergence from pairwise comparison toward multiple distributions using an f-divergence framework. It adopts Sgarro’s average-divergence definition as a simple multi-distribution formulation.

  • C. More on f-Divergence: f-divergence generalizes information distance to comparisons involving multiple distributions through a convex function f.The paper notes that multi-distribution divergence is under-explored in computer vision.
  • C. More on f-Divergence: The method adopts Sgarro’s average divergence as a simple definition of f-divergence for multiple distributions.

D. Some Qualitative Results on Sketchy

The paper presents qualitative comparisons between baseline methods and its method on category-level and fine-grained ZS-SBIR using Sketchy datasets. The fine-grained figure orders retrieved images by rank and marks top-five true matches.

  • D. Some Qualitative Results on Sketchy: Qualitative results compare a baseline with the proposed method for ZS-SBIR on Sketchy (ext).
  • D. Some Qualitative Results on Sketchy: Qualitative results also compare a baseline with the proposed method for FG-ZS-SBIR on Sketchy.
  • D. Some Qualitative Results on Sketchy: In the fine-grained figure, retrieved images are arranged in increasing rank beside each sketch query.The left-most image is rank 1 for each category.
  • D. Some Qualitative Results on Sketchy: True matches appearing in the top five are marked with green frames, and their retrieval ranks are displayed beside the corresponding queries.

E. Limitations

The authors identify two limitations: CLIP may not cover nearly all classes in niche cases, and its internet-scale training complicates thorough zero-shot evaluation on unseen classes. They state that both limitations apply broadly to CLIP-based applications.

  • E. Limitations: CLIP’s assumed coverage of almost all training classes may fail for certain niche cases.
  • E. Limitations: CLIP’s training on 400M image-text pairs makes thorough zero-shot evaluation on an unseen class challenging.
  • E. Limitations: The authors characterize both limitations as universal to CLIP-based applications.
Loading 2303.13440v3…